bot: update RSS feed
This commit is contained in:
github-actions[bot]
394
feed.xml
394
feed.xml
@@ -7,335 +7,153 @@
|
|||||||
<docs>http://www.rssboard.org/rss-specification</docs>
|
<docs>http://www.rssboard.org/rss-specification</docs>
|
||||||
<generator>python-feedgen</generator>
|
<generator>python-feedgen</generator>
|
||||||
<language>en</language>
|
<language>en</language>
|
||||||
<lastBuildDate>Wed, 11 Jun 2025 00:02:30 +0000</lastBuildDate>
|
<lastBuildDate>Thu, 12 Jun 2025 00:02:34 +0000</lastBuildDate>
|
||||||
<item>
|
<item>
|
||||||
<title>EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions</title>
|
<title>QQSUM: A Novel Task and Model of Quantitative Query-Focused Summarization for Review-based Product Question Answering</title>
|
||||||
<link>https://arxiv.org/abs/2505.23473</link>
|
<link>https://arxiv.org/abs/2506.04020</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23473.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiaorui Wu, Xiaofeng Mao, Xin Zhang, Fei Li, Chong Teng, Yuxiang Peng, Li Zheng, Donghong Ji, Zhuang Li</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large language models (LLMs) frequently refuse to respond to pseudo-malicious instructions: semantically harmless input queries triggering unnecessary LLM refusals due to conservative safety alignment, significantly impairing user experience. Collecting such instructions is crucial for evaluating and mitigating over-refusals, but existing instruction curation methods, like manual creation or instruction rewriting, either lack scalability or fail to produce sufficiently diverse and effective refusal-inducing prompts. To address these limitations, we introduce EVOREFUSE, a prompt optimization approach that generates diverse pseudo-malicious instructions consistently eliciting confident refusals across LLMs. EVOREFUSE employs an evolutionary algorithm exploring the instruction space in more diverse directions than existing methods via mutation strategies and recombination, and iteratively evolves seed instructions to maximize evidence lower bound on LLM refusal probability. Using EVOREFUSE, we create two novel datasets: EVOREFUSE-TEST, a benchmark of 582 pseudo-malicious instructions that outperforms the next-best benchmark with 140.41% higher average refusal triggering rate across 9 LLMs, 34.86% greater lexical diversity, and 40.03% improved LLM response confidence scores; and EVOREFUSE-ALIGN, which provides 3,000 pseudo-malicious instructions with responses for supervised and preference-based alignment training. LLAMA3.1-8B-INSTRUCT supervisedly fine-tuned on EVOREFUSE-ALIGN achieves up to 14.31% fewer over-refusals than models trained on the second-best alignment dataset, without compromising safety. Our analysis with EVOREFUSE-TEST reveals models trigger over-refusals by overly focusing on sensitive keywords while ignoring broader context.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04020.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> An Quang Tang, Xiuzhen Zhang, Minh Ngoc Dinh, Zhuang Li</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Review-based Product Question Answering (PQA) allows e-commerce platforms to automatically address customer queries by leveraging insights from user reviews. However, existing PQA systems generate answers with only a single perspective, failing to capture the diversity of customer opinions. In this paper we introduce a novel task Quantitative Query-Focused Summarization (QQSUM), which aims to summarize diverse customer opinions into representative Key Points (KPs) and quantify their prevalence to effectively answer user queries. While Retrieval-Augmented Generation (RAG) shows promise for PQA, its generated answers still fall short of capturing the full diversity of viewpoints. To tackle this challenge, our model QQSUM-RAG, which extends RAG, employs few-shot learning to jointly train a KP-oriented retriever and a KP summary generator, enabling KP-based summaries that capture diverse and representative opinions. Experimental results demonstrate that QQSUM-RAG achieves superior performance compared to state-of-the-art RAG baselines in both textual quality and quantification accuracy of opinions. Our source code is available at: https://github.com/antangrocket1312/QQSUMM</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23473</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.04020</guid>
|
||||||
<pubDate>Thu, 29 May 2025 14:26:46 +0000</pubDate>
|
<pubDate>Wed, 04 Jun 2025 14:50:32 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Model Immunization from a Condition Number Perspective</title>
|
<title>Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation</title>
|
||||||
<link>https://arxiv.org/abs/2505.23760</link>
|
<link>https://arxiv.org/abs/2506.04614</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23760.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Amber Yijia Zheng, Cedar Site Bai, Brian Bullins, Raymond A. Yeh</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Model immunization aims to pre-train models that are difficult to fine-tune on harmful tasks while retaining their utility on other non-harmful tasks. Though prior work has shown empirical evidence for immunizing text-to-image models, the key understanding of when immunization is possible and a precise definition of an immunized model remain unclear. In this work, we propose a framework, based on the condition number of a Hessian matrix, to analyze model immunization for linear models. Building on this framework, we design an algorithm with regularization terms to control the resulting condition numbers after pre-training. Empirical results on linear models and non-linear deep-nets demonstrate the effectiveness of the proposed algorithm on model immunization. The code is available at https://github.com/amberyzheng/model-immunization-cond-num.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04614.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuyang Wanyan, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Jiabo Ye, Yutong Kou, Ming Yan, Fei Huang, Xiaoshan Yang, Weiming Dong, Changsheng Xu</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> In recent years, Multimodal Large Language Models (MLLMs) have been extensively utilized for multimodal reasoning tasks, including Graphical User Interface (GUI) automation. Unlike general offline multimodal tasks, GUI automation is executed in online interactive environments, necessitating step-by-step decision-making based on real-time status of the environment. This task has a lower tolerance for decision-making errors at each step, as any mistakes may cumulatively disrupt the process and potentially lead to irreversible outcomes like deletions or payments. To address these issues, we introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution, by reasoning about the potential outcome and correctness of actions. Specifically, we propose a Suggestion-aware Gradient Relative Policy Optimization (S-GRPO) strategy to construct our pre-operative critic model GUI-Critic-R1, incorporating a novel suggestion reward to enhance the reliability of the model's feedback. Furthermore, we develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test, filling existing gaps in GUI critic data. Static experiments on the GUI-Critic-Test across both mobile and web domains reveal that our GUI-Critic-R1 offers significant advantages in critic accuracy compared to current MLLMs. Dynamic evaluation on GUI automation benchmark further highlights the effectiveness and superiority of our model, as evidenced by improved success rates and operational efficiency.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23760</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.04614</guid>
|
||||||
<pubDate>Thu, 29 May 2025 17:59:48 +0000</pubDate>
|
<pubDate>Thu, 05 Jun 2025 04:12:36 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Hidden in Plain Sight: Probing Implicit Reasoning in Multimodal Language Models</title>
|
<title>MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models</title>
|
||||||
<link>https://arxiv.org/abs/2506.00258</link>
|
<link>https://arxiv.org/abs/2506.04688</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.00258.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qianqi Yan, Hongquan Li, Shan Jiang, Yang Zhao, Xinze Guan, Ching-Chen Kuo, Xin Eric Wang</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Multimodal large language models (MLLMs) are increasingly deployed in open-ended, real-world environments where inputs are messy, underspecified, and not always trustworthy. Unlike curated benchmarks, these settings frequently involve instructions that refer to missing objects or contradictory facts, rely on ambiguous references, or request infeasible actions. In such cases, success hinges not on task execution alone, but on a model's ability to detect when something is silently wrong. This paper presents a systematic analysis of how current MLLMs handle such implicit reasoning scenarios: cases where the flaw is not explicitly stated but must be inferred from context. Using a curated diagnostic suite spanning four categories of real-world failure modes, we evaluate six MLLMs, including o3 and GPT-4o, and find that models frequently fail to surface hidden issues, even when they possess the necessary perceptual and reasoning skills. Explicit prompting reveals that the underlying capabilities exist but are often suppressed in favor of user compliance. We further show that simple inference-time interventions, such as cautious persona prompting and, in particular, requiring a clarifying question, can dramatically recover performance. Our findings highlight a persistent gap between reasoning competence and behavioral compliance in current MLLMs and suggest practical strategies for making these models more trustworthy in underconstrained environments.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04688.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Gio Paik, Geewook Kim, Jinbae Im</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> This paper introduces MMRefine, a MultiModal Refinement benchmark designed to evaluate the error refinement capabilities of Multimodal Large Language Models (MLLMs). As the emphasis shifts toward enhancing reasoning during inference, MMRefine provides a framework that evaluates MLLMs' abilities to detect and correct errors across six distinct scenarios beyond just comparing final accuracy before and after refinement. Furthermore, the benchmark analyzes the refinement performance by categorizing errors into six error types. Experiments with various open and closed MLLMs reveal bottlenecks and factors impeding refinement performance, highlighting areas for improvement in effective reasoning enhancement. Our code and dataset are publicly available at https://github.com/naver-ai/MMRefine.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.00258</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.04688</guid>
|
||||||
<pubDate>Fri, 30 May 2025 21:47:28 +0000</pubDate>
|
<pubDate>Thu, 05 Jun 2025 07:11:36 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists</title>
|
<title>ECoRAG: Evidentiality-guided Compression for Long Context RAG</title>
|
||||||
<link>https://arxiv.org/abs/2506.01241</link>
|
<link>https://arxiv.org/abs/2506.05167</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.01241.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang, Jasmine Gump, Tessa Bialek, Vivek Sankaran, Margo Schlanger, Lu Wang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> This paper introduces ExpertLongBench, an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications. Beyond question answering, the application-driven tasks in ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and strict adherence to domain-specific requirements. Notably, each task in ExpertLongBench includes a rubric, designed or validated by domain experts, to specify task requirements and guide output evaluation. Furthermore, we propose CLEAR, an evaluation framework that supports accurate evaluation of long-form model outputs in our benchmark. To achieve fine-grained, expert-aligned evaluation, CLEAR derives checklists from both model outputs and references by extracting information corresponding to items in the task-specific rubric. Checklist items for model outputs are then compared with corresponding items for reference outputs to assess their correctness, enabling grounded evaluation. We benchmark 11 large language models (LLMs) and analyze components in CLEAR, showing that (1) existing LLMs, with the top performer achieving only a 26.8% F1 score, require significant improvement for expert-level tasks; (2) models can generate content corresponding to the required aspects, though often not accurately; and (3) accurate checklist extraction and comparison in CLEAR can be achieved by open-weight models for more scalable and low-cost usage.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05167.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yeonseok Jeong, Jinsu Kim, Dohyeon Lee, Seung-won Hwang</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Large Language Models (LLMs) have shown remarkable performance in Open-Domain Question Answering (ODQA) by leveraging external documents through Retrieval-Augmented Generation (RAG). To reduce RAG overhead, from longer context, context compression is necessary. However, prior compression methods do not focus on filtering out non-evidential information, which limit the performance in LLM-based RAG. We thus propose Evidentiality-guided RAG, or ECoRAG framework. ECoRAG improves LLM performance by compressing retrieved documents based on evidentiality, ensuring whether answer generation is supported by the correct evidence. As an additional step, ECoRAG reflects whether the compressed content provides sufficient evidence, and if not, retrieves more until sufficient. Experiments show that ECoRAG improves LLM performance on ODQA tasks, outperforming existing compression methods. Furthermore, ECoRAG is highly cost-efficient, as it not only reduces latency but also minimizes token usage by retaining only the necessary information to generate the correct answer. Code is available at https://github.com/ldilab/ECoRAG.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.01241</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.05167</guid>
|
||||||
<pubDate>Mon, 02 Jun 2025 01:39:02 +0000</pubDate>
|
<pubDate>Thu, 05 Jun 2025 15:43:49 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>NetPress: Dynamically Generated LLM Benchmarks for Network Applications</title>
|
<title>RKEFino1: A Regulation Knowledge-Enhanced Large Language Model</title>
|
||||||
<link>https://arxiv.org/abs/2506.03231</link>
|
<link>https://arxiv.org/abs/2506.05700</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.03231.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yajie Zhou, Jiajun Ruan, Eric S. Wang, Sadjad Fouladi, Francis Y. Yan, Kevin Hsieh, Zaoxing Liu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Despite growing interest in domain-specific benchmarking of large language models (LLMs) and agents, current evaluations remain limited to static, small-scale datasets, especially in high-stakes tasks like network operations that demand reliability for deployments. We present NetPress, an automated benchmark generation framework for evaluating LLM agents in network applications. NetPress introduces a unified abstraction with state and action, enabling dynamic generation of diverse query sets along with corresponding ground truths. At runtime, users can specify benchmark configurations to generate millions of queries on the fly. In addition to dynamic benchmark construction, NetPress integrates with network emulators to provide realistic environment feedback, supporting comprehensive evaluation across correctness, safety, and latency. We instantiate NetPress on three representative applications, revealing interesting fine-grained differences in agent behavior that static, correctness-only benchmarks often miss. NetPress moves LLM evaluation toward realistic, scalable testing in infrastructure-centric domains, helping close the gap between benchmark performance and real-world deployment readiness. Code is available at https://github.com/Froot-NetSys/NetPress.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05700.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yan Wang, Yueru He, Ruoyu Xiang, Jeff Zhao</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent advances in large language models (LLMs) hold great promise for financial applications but introduce critical accuracy and compliance challenges in Digital Regulatory Reporting (DRR). To address these issues, we propose RKEFino1, a regulation knowledge-enhanced financial reasoning model built upon Fino1, fine-tuned with domain knowledge from XBRL, CDM, and MOF. We formulate two QA tasks-knowledge-based and mathematical reasoning-and introduce a novel Numerical NER task covering financial entities in both sentences and tables. Experimental results demonstrate the effectiveness and generalization capacity of RKEFino1 in compliance-critical financial tasks. We have released our model on Hugging Face.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.03231</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.05700</guid>
|
||||||
<pubDate>Tue, 03 Jun 2025 14:04:22 +0000</pubDate>
|
<pubDate>Fri, 06 Jun 2025 03:02:52 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Robust Preference Optimization via Dynamic Target Margins</title>
|
<title>MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models</title>
|
||||||
<link>https://arxiv.org/abs/2506.03690</link>
|
<link>https://arxiv.org/abs/2506.05928</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.03690.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jie Sun, Junkang Wu, Jiancan Wu, Zhibo Zhu, Xingyu Lu, Jun Zhou, Lintao Ma, Xiang Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> The alignment of Large Language Models (LLMs) is crucial for ensuring their safety and reliability in practical applications. Direct Preference Optimization (DPO) has emerged as an efficient method that directly optimizes models using preference pairs, significantly reducing resource demands. However, the effectiveness of DPO heavily depends on the data quality, which is frequently compromised by noise. In this work, we propose gamma-PO, a dynamic target margin preference optimization algorithm that adjust reward margins at the pairwise level. By introducing instance-specific margin calibration, gamma-PO strategically prioritizes high-confidence pairs (those demonstrating higher reward margins) while suppressing potential noise from ambiguous pairs. Moreover, gamma-PO is a plug-and-play method, compatible with variants of DPO that rely on reward margin between preference pairs. Across benchmarks such as AlpacaEval2 and Arena-Hard, gamma-PO achieves an average 4.4\% improvement over other baselines, setting new benchmarks for state-of-the-art performance. Additionally, gamma-PO requires minimal code changes and has a negligible impact on training efficiency, making it a robust solution for enhancing LLMs alignment. Our codes are available at https://github.com/sunjie279/gammaPO{https://github.com/sunjie279/gammaPO}.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05928.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jie Cao, Tianwei Lin, Hongyang He, Rolan Yan, Wenqiao Zhang, Juncheng Li, Dongping Zhang, Siliang Tang, Yueting Zhuang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Recent studies integrate Low-Rank Adaptation (LoRA) and Mixture-of-Experts (MoE) to further enhance the performance of parameter-efficient fine-tuning (PEFT) methods in Large Language Model (LLM) applications. Existing methods employ homogeneous MoE-LoRA architectures composed of LoRA experts with either similar or identical structures and capacities. However, these approaches often suffer from representation collapse and expert load imbalance, which negatively impact the potential of LLMs. To address these challenges, we propose a heterogeneous Mixture-of-Adapters (MoA) approach. This method dynamically integrates PEFT adapter experts with diverse structures, leveraging their complementary representational capabilities to foster expert specialization, thereby enhancing the effective transfer of pre-trained knowledge to downstream tasks. MoA supports two variants: (i) Soft MoA achieves fine-grained integration by performing a weighted fusion of all expert outputs; (ii) Sparse MoA activates adapter experts sparsely based on their contribution, achieving this with negligible performance degradation. Experimental results demonstrate that heterogeneous MoA outperforms homogeneous MoE-LoRA methods in both performance and parameter efficiency. Our project is available at https://github.com/DCDmllm/MoA.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.03690</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.05928</guid>
|
||||||
<pubDate>Wed, 04 Jun 2025 08:19:37 +0000</pubDate>
|
<pubDate>Fri, 06 Jun 2025 09:54:19 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Agents of Change: Self-Evolving LLM Agents for Strategic Planning</title>
|
<title>Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models</title>
|
||||||
<link>https://arxiv.org/abs/2506.04651</link>
|
<link>https://arxiv.org/abs/2506.06751</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04651.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nikolas Belle, Dakota Barnes, Alfonso Amayuelas, Ivan Bercovich, Xin Eric Wang, William Wang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Recent advances in LLMs have enabled their use as autonomous agents across a range of tasks, yet they continue to struggle with formulating and adhering to coherent long-term strategies. In this paper, we investigate whether LLM agents can self-improve when placed in environments that explicitly challenge their strategic planning abilities. Using the board game Settlers of Catan, accessed through the open-source Catanatron framework, we benchmark a progression of LLM-based agents, from a simple game-playing agent to systems capable of autonomously rewriting their own prompts and their player agent's code. We introduce a multi-agent architecture in which specialized roles (Analyzer, Researcher, Coder, and Player) collaborate to iteratively analyze gameplay, research new strategies, and modify the agent's logic or prompt. By comparing manually crafted agents to those evolved entirely by LLMs, we evaluate how effectively these systems can diagnose failure and adapt over time. Our results show that self-evolving agents, particularly when powered by models like Claude 3.7 and GPT-4o, outperform static baselines by autonomously adopting their strategies, passing along sample behavior to game-playing agents, and demonstrating adaptive reasoning over multiple iterations.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06751.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mikhail Salnikov, Dmitrii Korzh, Ivan Lazichny, Elvir Karimov, Artyom Iudin, Ivan Oseledets, Oleg Y. Rogov, Alexander Panchenko, Natalia Loukachevitch, Elena Tutubalina</p><p><b>Upvotes:</b> 52</p><p><b>Summary:</b> This paper evaluates geopolitical biases in LLMs with respect to various countries though an analysis of their interpretation of historical events with conflicting national perspectives (USA, UK, USSR, and China). We introduce a novel dataset with neutral event descriptions and contrasting viewpoints from different countries. Our findings show significant geopolitical biases, with models favoring specific national narratives. Additionally, simple debiasing prompts had a limited effect in reducing these biases. Experiments with manipulated participant labels reveal models' sensitivity to attribution, sometimes amplifying biases or recognizing inconsistencies, especially with swapped labels. This work highlights national narrative biases in LLMs, challenges the effectiveness of simple debiasing methods, and offers a framework and dataset for future geopolitical bias research.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.04651</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.06751</guid>
|
||||||
<pubDate>Thu, 05 Jun 2025 05:45:24 +0000</pubDate>
|
<pubDate>Sat, 07 Jun 2025 10:45:17 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>MegaHan97K: A Large-Scale Dataset for Mega-Category Chinese Character Recognition with over 97K Categories</title>
|
<title>Interpretable and Reliable Detection of AI-Generated Images via Grounded Reasoning in MLLMs</title>
|
||||||
<link>https://arxiv.org/abs/2506.04807</link>
|
<link>https://arxiv.org/abs/2506.07045</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04807.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuyi Zhang, Yongxin Shi, Peirong Zhang, Yixin Zhao, Zhenhua Yang, Lianwen Jin</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Foundational to the Chinese language and culture, Chinese characters encompass extraordinarily extensive and ever-expanding categories, with the latest Chinese GB18030-2022 standard containing 87,887 categories. The accurate recognition of this vast number of characters, termed mega-category recognition, presents a formidable yet crucial challenge for cultural heritage preservation and digital applications. Despite significant advances in Optical Character Recognition (OCR), mega-category recognition remains unexplored due to the absence of comprehensive datasets, with the largest existing dataset containing merely 16,151 categories. To bridge this critical gap, we introduce MegaHan97K, a mega-category, large-scale dataset covering an unprecedented 97,455 categories of Chinese characters. Our work offers three major contributions: (1) MegaHan97K is the first dataset to fully support the latest GB18030-2022 standard, providing at least six times more categories than existing datasets; (2) It effectively addresses the long-tail distribution problem by providing balanced samples across all categories through its three distinct subsets: handwritten, historical and synthetic subsets; (3) Comprehensive benchmarking experiments reveal new challenges in mega-category scenarios, including increased storage demands, morphologically similar character recognition, and zero-shot learning difficulties, while also unlocking substantial opportunities for future research. To the best of our knowledge, the MetaHan97K is likely the dataset with the largest classes not only in the field of OCR but may also in the broader domain of pattern recognition. The dataset is available at https://github.com/SCUT-DLVCLab/MegaHan97K.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07045.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yikun Ji, Hong Yan, Jun Lan, Huijia Zhu, Weiqiang Wang, Qi Fan, Liqing Zhang, Jianfu Zhang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> The rapid advancement of image generation technologies intensifies the demand for interpretable and robust detection methods. Although existing approaches often attain high accuracy, they typically operate as black boxes without providing human-understandable justifications. Multi-modal Large Language Models (MLLMs), while not originally intended for forgery detection, exhibit strong analytical and reasoning capabilities. When properly fine-tuned, they can effectively identify AI-generated images and offer meaningful explanations. However, existing MLLMs still struggle with hallucination and often fail to align their visual interpretations with actual image content and human reasoning. To bridge this gap, we construct a dataset of AI-generated images annotated with bounding boxes and descriptive captions that highlight synthesis artifacts, establishing a foundation for human-aligned visual-textual grounded reasoning. We then finetune MLLMs through a multi-stage optimization strategy that progressively balances the objectives of accurate detection, visual localization, and coherent textual explanation. The resulting model achieves superior performance in both detecting AI-generated images and localizing visual flaws, significantly outperforming baseline methods.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.04807</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.07045</guid>
|
||||||
<pubDate>Thu, 05 Jun 2025 09:33:06 +0000</pubDate>
|
<pubDate>Sun, 08 Jun 2025 08:47:44 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation</title>
|
<title>Mathesis: Towards Formal Theorem Proving from Natural Languages</title>
|
||||||
<link>https://arxiv.org/abs/2506.05062</link>
|
<link>https://arxiv.org/abs/2506.07047</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05062.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Noy Sternlicht, Ariel Gera, Roy Bar-Haim, Tom Hope, Noam Slonim</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> We introduce Debate Speech Evaluation as a novel and challenging benchmark for assessing LLM judges. Evaluating debate speeches requires a deep understanding of the speech at multiple levels, including argument strength and relevance, the coherence and organization of the speech, the appropriateness of its style and tone, and so on. This task involves a unique set of cognitive abilities that have previously received limited attention in systematic LLM benchmarking. To explore such skills, we leverage a dataset of over 600 meticulously annotated debate speeches and present the first in-depth analysis of how state-of-the-art LLMs compare to human judges on this task. Our findings reveal a nuanced picture: while larger models can approximate individual human judgments in some respects, they differ substantially in their overall judgment behavior. We also investigate the ability of frontier LLMs to generate persuasive, opinionated speeches, showing that models may perform at a human level on this task.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07047.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yu Xuejun, Jianyuan Zhong, Zijin Feng, Pengyi Zhai, Roozbeh Yousefzadeh, Wei Chong Ng, Haoxiong Liu, Ziyi Shou, Jing Xiong, Yudong Zhou, Claudia Beth Ong, Austen Jeremy Sugiarto, Yaoxi Zhang, Wai Ming Tai, Huan Cao, Dongcai Lu, Jiacheng Sun, Qiang Xu, Shen Xin, Zhenguo Li</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Recent advances in large language models show strong promise for formal reasoning. However, most LLM-based theorem provers have long been constrained by the need for expert-written formal statements as inputs, limiting their applicability to real-world problems expressed in natural language. We tackle this gap with Mathesis, the first end-to-end theorem proving pipeline processing informal problem statements. It contributes Mathesis-Autoformalizer, the first autoformalizer using reinforcement learning to enhance the formalization ability of natural language problems, aided by our novel LeanScorer framework for nuanced formalization quality assessment. It also proposes a Mathesis-Prover, which generates formal proofs from the formalized statements. To evaluate the real-world applicability of end-to-end formal theorem proving, we introduce Gaokao-Formal, a benchmark of 488 complex problems from China's national college entrance exam. Our approach is carefully designed, with a thorough study of each component. Experiments demonstrate Mathesis's effectiveness, with the autoformalizer outperforming the best baseline by 22% in pass-rate on Gaokao-Formal. The full system surpasses other model combinations, achieving 64% accuracy on MiniF2F with pass@32 and a state-of-the-art 18% on Gaokao-Formal.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05062</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.07047</guid>
|
||||||
<pubDate>Thu, 05 Jun 2025 14:06:51 +0000</pubDate>
|
<pubDate>Sun, 08 Jun 2025 09:04:14 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs</title>
|
<title>Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models</title>
|
||||||
<link>https://arxiv.org/abs/2506.05598</link>
|
<link>https://arxiv.org/abs/2506.07177</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05598.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Michael J Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, Diyi Yang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Recent calls for pluralistic alignment of Large Language Models (LLMs) encourage adapting models to diverse user preferences. However, most prior work on personalized reward models heavily rely on additional identity information, such as demographic details or a predefined set of preference categories. To this end, we introduce SynthesizeMe, an approach to inducing synthetic user personas from user interactions for personalized reward modeling. SynthesizeMe first generates and verifies reasoning to explain user preferences, then induces synthetic user personas from that reasoning, and finally filters to informative prior user interactions in order to build personalized prompts for a particular user. We show that using SynthesizeMe induced prompts improves personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining SynthesizeMe derived prompts with a reward model achieves top performance on PersonalRewardBench: a new curation of user-stratified interactions with chatbots collected from 854 users of Chatbot Arena and PRISM.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07177.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sangwon Jang, Taekyung Ki, Jaehyeong Jo, Jaehong Yoon, Soo Ye Kim, Zhe Lin, Sung Ju Hwang</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. For practical training-free guidance, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, compatible with any video models. Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05598</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.07177</guid>
|
||||||
<pubDate>Thu, 05 Jun 2025 21:23:16 +0000</pubDate>
|
<pubDate>Sun, 08 Jun 2025 14:54:41 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Proactive Assistant Dialogue Generation from Streaming Egocentric Videos</title>
|
<title>Solving Inequality Proofs with Large Language Models</title>
|
||||||
<link>https://arxiv.org/abs/2506.05904</link>
|
<link>https://arxiv.org/abs/2506.07927</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05904.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, Seungwhan Moon</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactive, proactive assistance based on streaming visual inputs, yet their development is constrained by the costly and labor-intensive process of data collection and system evaluation. To address these limitations, we present a comprehensive framework with three key contributions. First, we introduce a novel data curation pipeline that synthesizes dialogues from annotated egocentric videos, resulting in \dataset, a large-scale synthetic dialogue dataset spanning multiple domains. Second, we develop a suite of automatic evaluation metrics, validated through extensive human studies. Third, we propose an end-to-end model that processes streaming video inputs to generate contextually appropriate responses, incorporating novel techniques for handling data imbalance and long-duration videos. This work lays the foundation for developing real-time, proactive AI assistants capable of guiding users through diverse tasks. Project page: https://pro-assist.github.io/</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07927.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiayi Sheng, Luna Lyu, Jikai Jin, Tony Xia, Alex Gu, James Zou, Pan Lu</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Inequality proving, crucial across diverse scientific and mathematical fields, tests advanced reasoning skills such as discovering tight bounds and strategic theorem application. This makes it a distinct, demanding frontier for large language models (LLMs), offering insights beyond general mathematical problem-solving. Progress in this area is hampered by existing datasets that are often scarce, synthetic, or rigidly formal. We address this by proposing an informal yet verifiable task formulation, recasting inequality proving into two automatically checkable subtasks: bound estimation and relation prediction. Building on this, we release IneqMath, an expert-curated dataset of Olympiad-level inequalities, including a test set and training corpus enriched with step-wise solutions and theorem annotations. We also develop a novel LLM-as-judge evaluation framework, combining a final-answer judge with four step-wise judges designed to detect common reasoning flaws. A systematic evaluation of 29 leading LLMs on IneqMath reveals a surprising reality: even top models like o1 achieve less than 10% overall accuracy under step-wise scrutiny; this is a drop of up to 65.5% from their accuracy considering only final answer equivalence. This discrepancy exposes fragile deductive chains and a critical gap for current LLMs between merely finding an answer and constructing a rigorous proof. Scaling model size and increasing test-time computation yield limited gains in overall proof correctness. Instead, our findings highlight promising research directions such as theorem-guided reasoning and self-refinement. Code and data are available at https://ineqmath.github.io/.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05904</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.07927</guid>
|
||||||
<pubDate>Fri, 06 Jun 2025 09:23:29 +0000</pubDate>
|
<pubDate>Mon, 09 Jun 2025 16:43:38 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Bootstrapping World Models from Dynamics Models in Multimodal Foundation Models</title>
|
<title>Squeeze3D: Your 3D Generation Model is Secretly an Extreme Neural Compressor</title>
|
||||||
<link>https://arxiv.org/abs/2506.06006</link>
|
<link>https://arxiv.org/abs/2506.07932</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06006.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. Ponti</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> To what extent do vision-and-language foundation models possess a realistic world model (observation times action rightarrow observation) and a dynamics model (observation times observation rightarrow action), when actions are expressed through language? While open-source foundation models struggle with both, we find that fine-tuning them to acquire a dynamics model through supervision is significantly easier than acquiring a world model. In turn, dynamics models can be used to bootstrap world models through two main strategies: 1) weakly supervised learning from synthetic data and 2) inference time verification. Firstly, the dynamics model can annotate actions for unlabelled pairs of video frame observations to expand the training data. We further propose a new objective, where image tokens in observation pairs are weighted by their importance, as predicted by a recognition model. Secondly, the dynamics models can assign rewards to multiple samples of the world model to score them, effectively guiding search at inference time. We evaluate the world models resulting from both strategies through the task of action-centric image editing on Aurora-Bench. Our best model achieves a performance competitive with state-of-the-art image editing models, improving on them by a margin of 15% on real-world subsets according to GPT4o-as-judge, and achieving the best average human evaluation across all subsets of Aurora-Bench.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07932.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Rishit Dagli, Yushi Guan, Sankeerth Durvasula, Mohammadreza Mofayezi, Nandita Vijaykumar</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> We propose Squeeze3D, a novel framework that leverages implicit prior knowledge learnt by existing pre-trained 3D generative models to compress 3D data at extremely high compression ratios. Our approach bridges the latent spaces between a pre-trained encoder and a pre-trained generation model through trainable mapping networks. Any 3D model represented as a mesh, point cloud, or a radiance field is first encoded by the pre-trained encoder and then transformed (i.e. compressed) into a highly compact latent code. This latent code can effectively be used as an extremely compressed representation of the mesh or point cloud. A mapping network transforms the compressed latent code into the latent space of a powerful generative model, which is then conditioned to recreate the original 3D model (i.e. decompression). Squeeze3D is trained entirely on generated synthetic data and does not require any 3D datasets. The Squeeze3D architecture can be flexibly used with existing pre-trained 3D encoders and existing generative models. It can flexibly support different formats, including meshes, point clouds, and radiance fields. Our experiments demonstrate that Squeeze3D achieves compression ratios of up to 2187x for textured meshes, 55x for point clouds, and 619x for radiance fields while maintaining visual quality comparable to many existing methods. Squeeze3D only incurs a small compression and decompression latency since it does not involve training object-specific networks to compress an object.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06006</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.07932</guid>
|
||||||
<pubDate>Fri, 06 Jun 2025 11:50:18 +0000</pubDate>
|
<pubDate>Mon, 09 Jun 2025 16:52:10 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Astra: Toward General-Purpose Mobile Robots via Hierarchical Multimodal Learning</title>
|
<title>Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction</title>
|
||||||
<link>https://arxiv.org/abs/2506.06205</link>
|
<link>https://arxiv.org/abs/2506.07976</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06205.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sheng Chen, Peiyu He, Jiaxin Hu, Ziyang Liu, Yansheng Wang, Tao Xu, Chi Zhang, Chongchong Zhang, Chao An, Shiyu Cai, Duo Cao, Kangping Chen, Shuai Chu, Tianwei Chu, Mingdi Dan, Min Du, Weiwei Fang, Pengyou Fu, Junkai Hu, Xiaowei Jiang, Zhaodi Jiang, Fuxuan Li, Jun Li, Minghui Li, Mingyao Li, Yanchang Li, Zhibin Li, Guangming Liu, Kairui Liu, Lihao Liu, Weizhi Liu, Xiaoshun Liu, Yufei Liu, Yunfei Liu, Qiang Lu, Yuanfei Luo, Xiang Lv, Hongying Ma, Sai Ma, Lingxian Mi, Sha Sa, Hongxiang Shu, Lei Tian, Chengzhi Wang, Jiayu Wang, Kaijie Wang, Qingyi Wang, Renwen Wang, Tao Wang, Wei Wang, Xirui Wang, Chao Wei, Xuguang Wei, Zijun Xia, Zhaohao Xiao, Tingshuai Yan, Liyan Yang, Yifan Yang, Zhikai Yang, Zhong Yin, Li Yuan, Liuchun Yuan, Chi Zhang, Jinyang Zhang, Junhui Zhang, Linge Zhang, Zhenyi Zhang, Zheyu Zhang, Dongjie Zhu, Hang Li, Yangang Zhang</p><p><b>Upvotes:</b> 20</p><p><b>Summary:</b> Modern robot navigation systems encounter difficulties in diverse and complex indoor environments. Traditional approaches rely on multiple modules with small models or rule-based systems and thus lack adaptability to new environments. To address this, we developed Astra, a comprehensive dual-model architecture, Astra-Global and Astra-Local, for mobile robot navigation. Astra-Global, a multimodal LLM, processes vision and language inputs to perform self and goal localization using a hybrid topological-semantic graph as the global map, and outperforms traditional visual place recognition methods. Astra-Local, a multitask network, handles local path planning and odometry estimation. Its 4D spatial-temporal encoder, trained through self-supervised learning, generates robust 4D features for downstream tasks. The planning head utilizes flow matching and a novel masked ESDF loss to minimize collision risks for generating local trajectories, and the odometry head integrates multi-sensor inputs via a transformer encoder to predict the relative pose of the robot. Deployed on real in-house mobile robots, Astra achieves high end-to-end mission success rate across diverse indoor environments.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07976.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, Aviral Kumar</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting in the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose to scale test-time interaction, an untapped dimension of test-time scaling that increases the agent's interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting-based interaction scaling without any training can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI (Test-Time Interaction), a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using a Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WebVoyager and WebArena benchmarks. We further show that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06205</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.07976</guid>
|
||||||
<pubDate>Fri, 06 Jun 2025 16:08:47 +0000</pubDate>
|
<pubDate>Mon, 09 Jun 2025 17:50:02 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Cartridges: Lightweight and general-purpose long context representations via self-study</title>
|
<title>Aligning Text, Images, and 3D Structure Token-by-Token</title>
|
||||||
<link>https://arxiv.org/abs/2506.06266</link>
|
<link>https://arxiv.org/abs/2506.08002</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06266.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Re</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Large language models are often used to answer queries grounded in large text corpora (e.g. codebases, legal documents, or chat histories) by placing the entire corpus in the context window and leveraging in-context learning (ICL). Although current models support contexts of 100K-1M tokens, this setup is costly to serve because the memory consumption of the KV cache scales with input length. We explore an alternative: training a smaller KV cache offline on each corpus. At inference time, we load this trained KV cache, which we call a Cartridge, and decode a response. Critically, the cost of training a Cartridge can be amortized across all the queries referencing the same corpus. However, we find that the naive approach of training the Cartridge with next-token prediction on the corpus is not competitive with ICL. Instead, we propose self-study, a training recipe in which we generate synthetic conversations about the corpus and train the Cartridge with a context-distillation objective. We find that Cartridges trained with self-study replicate the functionality of ICL, while being significantly cheaper to serve. On challenging long-context benchmarks, Cartridges trained with self-study match ICL performance while using 38.6x less memory and enabling 26.4x higher throughput. Self-study also extends the model's effective context length (e.g. from 128k to 484k tokens on MTOB) and surprisingly, leads to Cartridges that can be composed at inference time without retraining.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08002.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aadarsh Sahoo, Vansh Tibrewal, Georgia Gkioxari</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Creating machines capable of understanding the world in 3D is essential in assisting designers that build and edit 3D environments and robots navigating and interacting within a three-dimensional space. Inspired by advances in language and image modeling, we investigate the potential of autoregressive models for a new modality: structured 3D scenes. To this end, we propose a unified LLM framework that aligns language, images, and 3D scenes and provide a detailed ''cookbook'' outlining critical design choices for achieving optimal training and performance addressing key questions related to data representation, modality-specific objectives, and more. We evaluate performance across four core 3D tasks -- rendering, recognition, instruction-following, and question-answering -- and four 3D datasets, synthetic and real-world. We extend our approach to reconstruct complex 3D object shapes by enriching our 3D modality with quantized shape encodings, and show our model's effectiveness on real-world 3D object recognition tasks. Project webpage: https://glab-caltech.github.io/kyvo/</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06266</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.08002</guid>
|
||||||
<pubDate>Fri, 06 Jun 2025 17:48:23 +0000</pubDate>
|
<pubDate>Mon, 09 Jun 2025 17:59:37 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Saffron-1: Towards an Inference Scaling Paradigm for LLM Safety Assurance</title>
|
<title>Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion</title>
|
||||||
<link>https://arxiv.org/abs/2506.06444</link>
|
<link>https://arxiv.org/abs/2506.08009</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06444.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ruizhong Qiu, Gaotang Li, Tianxin Wei, Jingrui He, Hanghang Tong</p><p><b>Upvotes:</b> 60</p><p><b>Summary:</b> Existing safety assurance research has primarily focused on training-phase alignment to instill safe behaviors into LLMs. However, recent studies have exposed these methods' susceptibility to diverse jailbreak attacks. Concurrently, inference scaling has significantly advanced LLM reasoning capabilities but remains unexplored in the context of safety assurance. Addressing this gap, our work pioneers inference scaling for robust and effective LLM safety against emerging threats. We reveal that conventional inference scaling techniques, despite their success in reasoning tasks, perform poorly in safety contexts, even falling short of basic approaches like Best-of-N Sampling. We attribute this inefficiency to a newly identified challenge, the exploration--efficiency dilemma, arising from the high computational overhead associated with frequent process reward model (PRM) evaluations. To overcome this dilemma, we propose SAFFRON, a novel inference scaling paradigm tailored explicitly for safety assurance. Central to our approach is the introduction of a multifurcation reward model (MRM) that significantly reduces the required number of reward model evaluations. To operationalize this paradigm, we further propose: (i) a partial supervision training objective for MRM, (ii) a conservative exploration constraint to prevent out-of-distribution explorations, and (iii) a Trie-based key--value caching strategy that facilitates cache sharing across sequences during tree search. Extensive experiments validate the effectiveness of our method. Additionally, we publicly release our trained multifurcation reward model (Saffron-1) and the accompanying token-level safety reward dataset (Safety4M) to accelerate future research in LLM safety. Our code, model, and data are publicly available at https://github.com/q-rz/saffron , and our project homepage is at https://q-rz.github.io/p/saffron .</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08009.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, Eli Shechtman</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models. It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs during inference. Unlike prior methods that denoise future frames based on ground-truth context frames, Self Forcing conditions each frame's generation on previously self-generated outputs by performing autoregressive rollout with key-value (KV) caching during training. This strategy enables supervision through a holistic loss at the video level that directly evaluates the quality of the entire generated sequence, rather than relying solely on traditional frame-wise objectives. To ensure training efficiency, we employ a few-step diffusion model along with a stochastic gradient truncation strategy, effectively balancing computational cost and performance. We further introduce a rolling KV cache mechanism that enables efficient autoregressive video extrapolation. Extensive experiments demonstrate that our approach achieves real-time streaming video generation with sub-second latency on a single GPU, while matching or even surpassing the generation quality of significantly slower and non-causal diffusion models. Project website: http://self-forcing.github.io/</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06444</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.08009</guid>
|
||||||
<pubDate>Fri, 06 Jun 2025 18:05:45 +0000</pubDate>
|
<pubDate>Mon, 09 Jun 2025 17:59:55 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>What Is Seen Cannot Be Unseen: The Disruptive Effect of Knowledge Conflict on Large Language Models</title>
|
<title>Seeing Voices: Generating A-Roll Video from Audio with Mirage</title>
|
||||||
<link>https://arxiv.org/abs/2506.06485</link>
|
<link>https://arxiv.org/abs/2506.08279</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06485.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kaiser Sun, Fan Bai, Mark Dredze</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Large language models frequently rely on both contextual input and parametric knowledge to perform tasks. However, these sources can come into conflict, especially when retrieved documents contradict the model's parametric knowledge. We propose a diagnostic framework to systematically evaluate LLM behavior under context-memory conflict, where the contextual information diverges from their parametric beliefs. We construct diagnostic data that elicit these conflicts and analyze model performance across multiple task types. Our findings reveal that (1) knowledge conflict has minimal impact on tasks that do not require knowledge utilization, (2) model performance is consistently higher when contextual and parametric knowledge are aligned, (3) models are unable to fully suppress their internal knowledge even when instructed, and (4) providing rationales that explain the conflict increases reliance on contexts. These insights raise concerns about the validity of model-based evaluation and underscore the need to account for knowledge conflict in the deployment of LLMs.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08279.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aditi Sundararaman, Amogh Adishesha, Andrew Jaegle, Dan Bigioi, Hyoung-Kyu Song, Jon Kyl, Justin Mao, Kevin Lan, Mojtaba Komeili, ShahRukh Athar, Sheila Babayan, Stanislau Beliasau, William Buchwalter</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> From professional filmmaking to user-generated content, creators and consumers have long recognized that the power of video depends on the harmonious integration of what we hear (the video's audio track) with what we see (the video's image sequence). Current approaches to video generation either ignore sound to focus on general-purpose but silent image sequence generation or address both visual and audio elements but focus on restricted application domains such as re-dubbing. We introduce Mirage, an audio-to-video foundation model that excels at generating realistic, expressive output imagery from scratch given an audio input. When integrated with existing methods for speech synthesis (text-to-speech, or TTS), Mirage results in compelling multimodal video. When trained on audio-video footage of people talking (A-roll) and conditioned on audio containing speech, Mirage generates video of people delivering a believable interpretation of the performance implicit in input audio. Our central technical contribution is a unified method for training self-attention-based audio-to-video generation models, either from scratch or given existing weights. This methodology allows Mirage to retain generality as an approach to audio-to-video generation while producing outputs of superior subjective quality to methods that incorporate audio-specific architectures or loss components specific to people, speech, or details of how images or audio are captured. We encourage readers to watch and listen to the results of Mirage for themselves (see paper and comments for links).</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06485</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.08279</guid>
|
||||||
<pubDate>Fri, 06 Jun 2025 19:20:23 +0000</pubDate>
|
<pubDate>Mon, 09 Jun 2025 22:56:02 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Self-Adapting Improvement Loops for Robotic Learning</title>
|
<title>Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability</title>
|
||||||
<link>https://arxiv.org/abs/2506.06658</link>
|
<link>https://arxiv.org/abs/2506.08300</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06658.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Calvin Luo, Zilai Zeng, Mingxi Jia, Yilun Du, Chen Sun</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Video generative models trained on expert demonstrations have been utilized as performant text-conditioned visual planners for solving robotic tasks. However, generalization to unseen tasks remains a challenge. Whereas improved generalization may be facilitated by leveraging learned prior knowledge from additional pre-collected offline data sources, such as web-scale video datasets, in the era of experience we aim to design agents that can continuously improve in an online manner from self-collected behaviors. In this work we thus propose the Self-Adapting Improvement Loop (SAIL), where an in-domain video model iteratively updates itself on self-produced trajectories, collected through adaptation with an internet-scale pretrained video model, and steadily improves its performance for a specified task of interest. We apply SAIL to a diverse suite of MetaWorld tasks, as well as two manipulation tasks on a real robot arm, and find that performance improvements continuously emerge over multiple iterations for novel tasks initially unseen during original in-domain video model training. Furthermore, we discover that SAIL is surprisingly robust regarding if and how the self-collected experience is filtered, and the quality of the initial in-domain demonstrations. Through adaptation with summarized internet-scale data, and learning through online experience, we thus demonstrate a way to iteratively bootstrap a high-performance video model for solving novel robotic tasks through self-improvement.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08300.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Matteo Cargnelutti, Catherine Brobston, John Hess, Jack Cushman, Kristi Mukk, Aristana Scourtas, Kyle Courtney, Greg Leppert, Amanda Watson, Martha Whitehead, Jonathan Zittrain</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Large language models (LLMs) use data to learn about the world in order to produce meaningful correlations and predictions. As such, the nature, scale, quality, and diversity of the datasets used to train these models, or to support their work at inference time, have a direct impact on their quality. The rapid development and adoption of LLMs of varying quality has brought into focus the scarcity of publicly available, high-quality training data and revealed an urgent need to ground the stewardship of these datasets in sustainable practices with clear provenance chains. To that end, this technical report introduces Institutional Books 1.0, a large collection of public domain books originally digitized through Harvard Library's participation in the Google Books project, beginning in 2006. Working with Harvard Library, we extracted, analyzed, and processed these volumes into an extensively-documented dataset of historic texts. This analysis covers the entirety of Harvard Library's collection scanned as part of that project, originally spanning 1,075,899 volumes written in over 250 different languages for a total of approximately 250 billion tokens. As part of this initial release, the OCR-extracted text (original and post-processed) as well as the metadata (bibliographic, source, and generated) of the 983,004 volumes, or 242B tokens, identified as being in the public domain have been made available. This report describes this project's goals and methods as well as the results of the analyses we performed, all in service of making this historical collection more accessible and easier for humans and machines alike to filter, read and use.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06658</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.08300</guid>
|
||||||
<pubDate>Sat, 07 Jun 2025 04:34:37 +0000</pubDate>
|
<pubDate>Tue, 10 Jun 2025 00:11:30 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering</title>
|
<title>DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs</title>
|
||||||
<link>https://arxiv.org/abs/2506.06905</link>
|
<link>https://arxiv.org/abs/2506.08500</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06905.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Akash Gupta, Amos Storkey, Mirella Lapata</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, is inconsistent and does not always improve monotonically with increasing examples. We hypothesize that this occurs due to the LMM being overwhelmed by additional information present in the image embeddings, which is not required for the downstream task. To address this, we propose a meta-learning approach that provides an alternative for inducing few-shot capabilities in LMMs, using a fixed set of soft prompts that are distilled from task-relevant image features and can be adapted at test time using a few examples. To facilitate this distillation, we introduce an attention-mapper module that can be easily integrated with the popular LLaVA v1.5 architecture and is jointly learned with soft prompts, enabling task adaptation in LMMs under low-data regimes with just a few gradient steps. Evaluation on the VL-ICL Bench shows that our method consistently outperforms ICL and related prompt-tuning approaches, even under image perturbations, improving task induction and reasoning across visual question answering tasks.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08500.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Arie Cattan, Alon Jacovi, Ori Ram, Jonathan Herzig, Roee Aharoni, Sasha Goldshtein, Eran Ofek, Idan Szpektor, Avi Caciularu</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Retrieval Augmented Generation (RAG) is a commonly used approach for enhancing large language models (LLMs) with relevant and up-to-date information. However, the retrieved sources can often contain conflicting information and it remains unclear how models should address such discrepancies. In this work, we first propose a novel taxonomy of knowledge conflict types in RAG, along with the desired model behavior for each type. We then introduce CONFLICTS, a high-quality benchmark with expert annotations of conflict types in a realistic RAG setting. CONFLICTS is the first benchmark that enables tracking progress on how models address a wide range of knowledge conflicts. We conduct extensive experiments on this benchmark, showing that LLMs often struggle to appropriately resolve conflicts between sources. While prompting LLMs to explicitly reason about the potential conflict in the retrieved documents significantly improves the quality and appropriateness of their responses, substantial room for improvement in future research remains.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06905</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.08500</guid>
|
||||||
<pubDate>Sat, 07 Jun 2025 19:37:22 +0000</pubDate>
|
<pubDate>Tue, 10 Jun 2025 06:52:57 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity</title>
|
<title>RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling</title>
|
||||||
<link>https://arxiv.org/abs/2506.06941</link>
|
<link>https://arxiv.org/abs/2506.08672</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06941.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Recent generations of language models have introduced Large Reasoning Models (LRMs) that generate detailed thinking processes before providing answers. While these models demonstrate improved performance on reasoning benchmarks, their fundamental capabilities, scaling properties, and limitations remain insufficiently understood. Current evaluations primarily focus on established math and coding benchmarks, emphasizing final answer accuracy. However, this evaluation paradigm often suffers from contamination and does not provide insights into the reasoning traces. In this work, we systematically investigate these gaps with the help of controllable puzzle environments that allow precise manipulation of complexity while maintaining consistent logical structures. This setup enables the analysis of not only final answers but also the internal reasoning traces, offering insights into how LRMs think. Through extensive experiments, we show that LRMs face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counterintuitive scaling limit: their reasoning effort increases with problem complexity up to a point, then declines despite having remaining token budget. By comparing LRMs with their standard LLM counterparts under same inference compute, we identify three performance regimes: (1) low-complexity tasks where standard models outperform LRMs, (2) medium-complexity tasks where LRMs demonstrates advantage, and (3) high-complexity tasks where both models face complete collapse. We found that LRMs have limitations in exact computation: they fail to use explicit algorithms and reason inconsistently across scales. We also investigate the reasoning traces in more depth, studying the patterns of explored solutions and analyzing the models' computational behavior, shedding light on their strengths, limitations, and raising questions about their reasoning capabilities.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08672.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Liu, Jiaqi Li, Zilong Zheng</p><p><b>Upvotes:</b> 23</p><p><b>Summary:</b> Rule-based reasoning has been acknowledged as one of the fundamental problems in reasoning, while deviations in rule formats, types, and complexity in real-world applications pose severe challenges. Recent studies have shown that large reasoning models (LRMs) have remarkable reasoning capabilities, and their performance is substantially enhanced by reinforcement learning (RL). However, it remains an open question whether small reasoning models (SRMs) can learn rule-based reasoning effectively with robust generalization across diverse tasks and domains. To address this, we introduce Reinforced Rule-based Reasoning, a.k.a. RuleReasoner, a simple yet effective method to conduct rule-based reasoning via a wide collection of curated tasks and a novel domain-aware dynamic sampling approach. Specifically, RuleReasoner resamples each training batch by updating the sampling weights of different domains based on historical rewards. This facilitates domain augmentation and flexible online learning schedules for RL, obviating the need for pre-hoc human-engineered mix-training recipes used in existing methods. Empirical evaluations on in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that RuleReasoner outperforms frontier LRMs by a significant margin (Delta4.1% average points on eight ID tasks and Delta10.4% average points on three OOD tasks over OpenAI-o1). Notably, our approach also exhibits higher computational efficiency compared to prior dynamic sampling methods for RL.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06941</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.08672</guid>
|
||||||
<pubDate>Sat, 07 Jun 2025 22:42:29 +0000</pubDate>
|
<pubDate>Tue, 10 Jun 2025 10:31:21 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning</title>
|
<title>DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval</title>
|
||||||
<link>https://arxiv.org/abs/2506.07044</link>
|
<link>https://arxiv.org/abs/2506.08887</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07044.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, Yu Rong</p><p><b>Upvotes:</b> 79</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08887.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Leqi Shen, Guoqiang Gong, Tianxiang Hao, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, Jungong Han, Guiguang Ding</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> The parameter-efficient adaptation of the image-text pretraining model CLIP for video-text retrieval is a prominent area of research. While CLIP is focused on image-level vision-language matching, video-text retrieval demands comprehensive understanding at the video level. Three key discrepancies emerge in the transfer from image-level to video-level: vision, language, and alignment. However, existing methods mainly focus on vision while neglecting language and alignment. In this paper, we propose Discrepancy Reduction in Vision, Language, and Alignment (DiscoVLA), which simultaneously mitigates all three discrepancies. Specifically, we introduce Image-Video Features Fusion to integrate image-level and video-level features, effectively tackling both vision and language discrepancies. Additionally, we generate pseudo image captions to learn fine-grained image-level alignment. To mitigate alignment discrepancies, we propose Image-to-Video Alignment Distillation, which leverages image-level alignment knowledge to enhance video-level alignment. Extensive experiments demonstrate the superiority of our DiscoVLA. In particular, on MSRVTT with CLIP (ViT-B/16), DiscoVLA outperforms previous methods by 1.5% in R@1, reaching a final score of 50.5% R@1. The code is available at https://github.com/LunarShen/DsicoVLA.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07044</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.08887</guid>
|
||||||
<pubDate>Sun, 08 Jun 2025 08:47:30 +0000</pubDate>
|
<pubDate>Tue, 10 Jun 2025 15:16:40 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>GeometryZero: Improving Geometry Solving for LLM with Group Contrastive Policy Optimization</title>
|
<title>Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better</title>
|
||||||
<link>https://arxiv.org/abs/2506.07160</link>
|
<link>https://arxiv.org/abs/2506.09040</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07160.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yikun Wang, Yibin Wang, Dianyi Wang, Zimian Peng, Qipeng Guo, Dacheng Tao, Jiaqi Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent advances in large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, particularly in mathematical reasoning, amid which geometry problem solving remains a challenging area where auxiliary construction plays a enssential role. Existing approaches either achieve suboptimal performance or rely on massive LLMs (e.g., GPT-4o), incurring massive computational costs. We posit that reinforcement learning with verifiable reward (e.g., GRPO) offers a promising direction for training smaller models that effectively combine auxiliary construction with robust geometric reasoning. However, directly applying GRPO to geometric reasoning presents fundamental limitations due to its dependence on unconditional rewards, which leads to indiscriminate and counterproductive auxiliary constructions. To address these challenges, we propose Group Contrastive Policy Optimization (GCPO), a novel reinforcement learning framework featuring two key innovations: (1) Group Contrastive Masking, which adaptively provides positive or negative reward signals for auxiliary construction based on contextual utility, and a (2) length reward that promotes longer reasoning chains. Building on GCPO, we develop GeometryZero, a family of affordable-size geometric reasoning models that judiciously determine when to employ auxiliary construction. Our extensive empirical evaluation across popular geometric benchmarks (Geometry3K, MathVista) demonstrates that GeometryZero models consistently outperform baselines (e.g. GRPO), achieving an average improvement of 4.29% across all benchmarks.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.09040.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Dianyi Wang, Wei Song, Yikun Wang, Siyuan Wang, Kaicheng Yu, Zhongyu Wei, Jiaqi Wang</p><p><b>Upvotes:</b> 26</p><p><b>Summary:</b> Typical large vision-language models (LVLMs) apply autoregressive supervision solely to textual sequences, without fully incorporating the visual modality into the learning process. This results in three key limitations: (1) an inability to utilize images without accompanying captions, (2) the risk that captions omit critical visual details, and (3) the challenge that certain vision-centric content cannot be adequately conveyed through text. As a result, current LVLMs often prioritize vision-to-language alignment while potentially overlooking fine-grained visual information. While some prior works have explored autoregressive image generation, effectively leveraging autoregressive visual supervision to enhance image understanding remains an open challenge. In this paper, we introduce Autoregressive Semantic Visual Reconstruction (ASVR), which enables joint learning of visual and textual modalities within a unified autoregressive framework. We show that autoregressively reconstructing the raw visual appearance of images does not enhance and may even impair multimodal understanding. In contrast, autoregressively reconstructing the semantic representation of images consistently improves comprehension. Notably, we find that even when models are given continuous image features as input, they can effectively reconstruct discrete semantic tokens, resulting in stable and consistent improvements across a wide range of multimodal understanding benchmarks. Our approach delivers significant performance gains across varying data scales (556k-2M) and types of LLM bacbones. Specifically, ASVR improves LLaVA-1.5 by 5% in average scores across 14 multimodal benchmarks. The code is available at https://github.com/AlenjandroWang/ASVR.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07160</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.09040</guid>
|
||||||
<pubDate>Sun, 08 Jun 2025 14:18:15 +0000</pubDate>
|
<pubDate>Tue, 10 Jun 2025 17:57:50 +0000</pubDate>
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07240</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07240.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Roy Eisenstadt, Itamar Zimerman, Lior Wolf</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recently, techniques such as explicit structured reasoning have demonstrated strong test-time scaling behavior by enforcing a separation between the model's internal "thinking" process and the final response. A key factor influencing answer quality in this setting is the length of the thinking stage. When the reasoning is too short, the model may fail to capture the complexity of the task. Conversely, when it is too long, the model may overthink, leading to unnecessary computation and degraded performance. This paper explores and exploits the underlying mechanisms by which LLMs understand and regulate the length of their reasoning during explicit thought processes. First, we show that LLMs encode their progress through the reasoning process and introduce an interactive progress bar visualization, which is then used to reveal insights on the model's planning dynamics. Second, we manipulate the internal progress encoding during inference to reduce unnecessary steps and generate a more concise and decisive chain of thoughts. Our empirical results demonstrate that this "overclocking" method mitigates overthinking, improves answer accuracy, and reduces inference latency. Our code is publicly available.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07240</guid>
|
|
||||||
<pubDate>Sun, 08 Jun 2025 17:54:33 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Pre-trained Large Language Models Learn Hidden Markov Models In-context</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07298</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07298.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yijia Dai, Zhaolin Gao, Yahya Satter, Sarah Dean, Jennifer J. Sun</p><p><b>Upvotes:</b> 16</p><p><b>Summary:</b> Hidden Markov Models (HMMs) are foundational tools for modeling sequential data with latent Markovian structure, yet fitting them to real-world data remains computationally challenging. In this work, we show that pre-trained large language models (LLMs) can effectively model data generated by HMMs via in-context learning (ICL)x2013their ability to infer patterns from examples within a prompt. On a diverse set of synthetic HMMs, LLMs achieve predictive accuracy approaching the theoretical optimum. We uncover novel scaling trends influenced by HMM properties, and offer theoretical conjectures for these empirical observations. We also provide practical guidelines for scientists on using ICL as a diagnostic tool for complex data. On real-world animal decision-making tasks, ICL achieves competitive performance with models designed by human experts. To our knowledge, this is the first demonstration that ICL can learn and predict HMM-generated sequencesx2013an advance that deepens our understanding of in-context learning in LLMs and establishes its potential as a powerful tool for uncovering hidden structure in complex scientific data.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07298</guid>
|
|
||||||
<pubDate>Sun, 08 Jun 2025 21:49:38 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>ConfQA: Answer Only If You Are Confident</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07309</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07309.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yin Huang, Yifan Ethan Xu, Kai Sun, Vera Yan, Alicia Sun, Haidar Khan, Jimmy Nguyen, Mohammad Kachuee, Zhaojiang Lin, Yue Liu, Aaron Colak, Anuj Kumar, Wen-tau Yih, Xin Luna Dong</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Can we teach Large Language Models (LLMs) to refrain from hallucinating factual statements? In this paper we present a fine-tuning strategy that we call ConfQA, which can reduce hallucination rate from 20-40% to under 5% across multiple factuality benchmarks. The core idea is simple: when the LLM answers a question correctly, it is trained to continue with the answer; otherwise, it is trained to admit "I am unsure". But there are two key factors that make the training highly effective. First, we introduce a dampening prompt "answer only if you are confident" to explicitly guide the behavior, without which hallucination remains high as 15%-25%. Second, we leverage simple factual statements, specifically attribute values from knowledge graphs, to help LLMs calibrate the confidence, resulting in robust generalization across domains and question types. Building on this insight, we propose the Dual Neural Knowledge framework, which seamlessly select between internally parameterized neural knowledge and externally recorded symbolic knowledge based on ConfQA's confidence. The framework enables potential accuracy gains to beyond 95%, while reducing unnecessary external retrievals by over 30%.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07309</guid>
|
|
||||||
<pubDate>Sun, 08 Jun 2025 22:51:46 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07434</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07434.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Feifan Song, Shaohang Wei, Wen Luo, Yuxuan Fan, Tianyu Liu, Guoyin Wang, Houfeng Wang</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Large Language Models (LLMs) require alignment with human preferences to avoid generating offensive, false, or meaningless content. Recently, low-resource methods for LLM alignment have been popular, while still facing challenges in obtaining both high-quality and aligned content. Motivated by the observation that the difficulty of generating aligned responses is concentrated at the beginning of decoding, we propose a novel framework, Weak-to-Strong Decoding (WSD), to enhance the alignment ability of base models by the guidance of a small aligned model. The small model first drafts well-aligned beginnings, followed by the large base model to continue the rest, controlled by a well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign, to fine-tune a small-sized Pilot-3B as the draft model, which effectively enhances different base models under the WSD framework to outperform all baseline methods, while avoiding degradation on downstream tasks, termed as the alignment tax. Extensive experiments are further conducted to examine the impact of different settings and time efficiency, as well as analyses on the intrinsic mechanisms of WSD in depth.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07434</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 05:21:22 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07463</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07463.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guang Liu, Liangdong Wang, Jijie Li, Yang Yu, Yao Xu, Jiabei Chen, Yu Bai, Feng Liao, Yonghua Lin</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> We introduce CCI4.0, a large-scale bilingual pre-training dataset engineered for superior data quality and diverse human-like reasoning trajectory. CCI4.0 occupies roughly 35 TB of disk space and comprises two sub-datasets: CCI4.0-M2-Base and CCI4.0-M2-CoT. CCI4.0-M2-Base combines a 5.2 TB carefully curated Chinese web corpus, a 22.5 TB English subset from Nemotron-CC, and diverse sources from math, wiki, arxiv, and code. Although these data are mostly sourced from well-processed datasets, the quality standards of various domains are dynamic and require extensive expert experience and labor to process. So, we propose a novel pipeline justifying data quality mainly based on models through two-stage deduplication, multiclassifier quality scoring, and domain-aware fluency filtering. We extract 4.5 billion pieces of CoT(Chain-of-Thought) templates, named CCI4.0-M2-CoT. Differing from the distillation of CoT from larger models, our proposed staged CoT extraction exemplifies diverse reasoning patterns and significantly decreases the possibility of hallucination. Empirical evaluations demonstrate that LLMs pre-trained in CCI4.0 benefit from cleaner, more reliable training signals, yielding consistent improvements in downstream tasks, especially in math and code reflection tasks. Our results underscore the critical role of rigorous data curation and human thinking templates in advancing LLM performance, shedding some light on automatically processing pretraining corpora.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07463</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 06:14:19 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>SpatialLM: Training Large Language Models for Structured Indoor Modeling</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07491</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07491.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, Zihan Zhou</p><p><b>Upvotes:</b> 29</p><p><b>Summary:</b> SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07491</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 07:10:58 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07527</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07527.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, Wentao Zhang</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Recent advances in large language model (LLM) reasoning have shown that sophisticated behaviors such as planning and self-reflection can emerge through reinforcement learning (RL). However, despite these successes, RL in its current form remains insufficient to induce capabilities that exceed the limitations of the base model, as it is primarily optimized based on existing knowledge of the model rather than facilitating the acquisition of new information. To address this limitation, we employ supervised fine-tuning (SFT) to learn what RL cannot, which enables the incorporation of new knowledge and reasoning patterns by leveraging high-quality demonstration data. We analyze the training dynamics of RL and SFT for LLM reasoning and find that RL excels at maintaining and improving performance on questions within the model's original capabilities, while SFT is more effective at enabling progress on questions beyond the current scope of the model. Motivated by the complementary strengths of RL and SFT, we introduce a novel training approach, ReLIFT (Reinforcement Learning Interleaved with Online Fine-Tuning). In ReLIFT, the model is primarily trained using RL, but when it encounters challenging questions, high-quality solutions are collected for fine-tuning, and the training process alternates between RL and fine-tuning to enhance the model's reasoning abilities. ReLIFT achieves an average improvement of over +5.2 points across five competition-level benchmarks and one out-of-distribution benchmark compared to other zero-RL models. Furthermore, we demonstrate that ReLIFT outperforms both RL and SFT while using only 13\% of the detailed demonstration data, highlighting its scalability. These results provide compelling evidence that ReLIFT overcomes the fundamental limitations of RL and underscores the significant potential.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07527</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 08:11:20 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07530</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07530.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongyu Wang, Chuyan Xiong, Ruiping Wang, Xilin Chen</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Vision-Language-Action (VLA) models have shown impressive capabilities across a wide range of robotics manipulation tasks. However, their growing model size poses significant challenges for deployment on resource-constrained robotic systems. While 1-bit pretraining has proven effective for enhancing the inference efficiency of large language models with minimal performance loss, its application to VLA models remains underexplored. In this work, we present BitVLA, the first 1-bit VLA model for robotics manipulation, in which every parameter is ternary, i.e., {-1, 0, 1}. To further reduce the memory footprint of the vision encoder, we propose the distillation-aware training strategy that compresses the full-precision encoder to 1.58-bit weights. During this process, a full-precision encoder serves as a teacher model to better align latent representations. Despite the lack of large-scale robotics pretraining, BitVLA achieves performance comparable to the state-of-the-art model OpenVLA-OFT with 4-bit post-training quantization on the LIBERO benchmark, while consuming only 29.8% of the memory. These results highlight BitVLA's promise for deployment on memory-constrained edge devices. We release the code and model weights in https://github.com/ustcwhy/BitVLA.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07530</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 08:15:11 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07553</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07553.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingchao Wang, Haote Yang, Jiang Wu, Yifan He, Xingjian Wei, Yinfan Wang, Chengjin Liu, Lingli Ge, Lijun Wu, Bin Wang, Dahua Lin, Conghui He</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Optical Chemical Structure Recognition (OCSR) is crucial for digitizing chemical knowledge by converting molecular images into machine-readable formats. While recent vision-language models (VLMs) have shown potential in this task, their image-captioning approach often struggles with complex molecular structures and inconsistent annotations. To overcome these challenges, we introduce GTR-Mol-VLM, a novel framework featuring two key innovations: (1) the Graph Traversal as Visual Chain of Thought mechanism that emulates human reasoning by incrementally parsing molecular graphs through sequential atom-bond predictions, and (2) the data-centric principle of Faithfully Recognize What You've Seen, which addresses the mismatch between abbreviated structures in images and their expanded annotations. To support model development, we constructed GTR-CoT-1.3M, a large-scale instruction-tuning dataset with meticulously corrected annotations, and introduced MolRec-Bench, the first benchmark designed for a fine-grained evaluation of graph-parsing accuracy in OCSR. Comprehensive experiments demonstrate that GTR-Mol-VLM achieves superior results compared to specialist models, chemistry-domain VLMs, and commercial general-purpose VLMs. Notably, in scenarios involving molecular images with functional group abbreviations, GTR-Mol-VLM outperforms the second-best baseline by approximately 14 percentage points, both in SMILES-based and graph-based metrics. We hope that this work will drive OCSR technology to more effectively meet real-world needs, thereby advancing the fields of cheminformatics and AI for Science. We will release GTR-CoT at https://github.com/opendatalab/GTR-CoT.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07553</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 08:47:10 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07564</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07564.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Peiran Li, Xinkai Zou, Zhuohang Wu, Ruifeng Li, Shuo Xing, Hanwen Zheng, Zhikai Hu, Yuping Wang, Haoxi Li, Qin Yuan, Yingmo Zhang, Zhengzhong Tu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Recent advances in large language models (LLMs) and vision-language models (VLMs) have enabled powerful autonomous agents capable of complex reasoning and multi-modal tool use. Despite their growing capabilities, today's agent frameworks remain fragile, lacking principled mechanisms for secure information flow, reliability, and multi-agent coordination. In this work, we introduce SAFEFLOW, a new protocol-level framework for building trustworthy LLM/VLM-based agents. SAFEFLOW enforces fine-grained information flow control (IFC), precisely tracking provenance, integrity, and confidentiality of all the data exchanged between agents, tools, users, and environments. By constraining LLM reasoning to respect these security labels, SAFEFLOW prevents untrusted or adversarial inputs from contaminating high-integrity decisions. To ensure robustness in concurrent multi-agent settings, SAFEFLOW introduces transactional execution, conflict resolution, and secure scheduling over shared state, preserving global consistency across agents. We further introduce mechanisms, including write-ahead logging, rollback, and secure caches, that further enhance resilience against runtime errors and policy violations. To validate the performances, we built SAFEFLOWBENCH, a comprehensive benchmark suite designed to evaluate agent reliability under adversarial, noisy, and concurrent operational conditions. Extensive experiments demonstrate that agents built with SAFEFLOW maintain impressive task performance and security guarantees even in hostile environments, substantially outperforming state-of-the-art. Together, SAFEFLOW and SAFEFLOWBENCH lay the groundwork for principled, robust, and secure agent ecosystems, advancing the frontier of reliable autonomy.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07564</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 09:04:37 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07645</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07645.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Maciej Chrabąszcz, Katarzyna Lorenc, Karolina Seweryn</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large language models (LLMs) have demonstrated impressive capabilities across various natural language processing (NLP) tasks in recent years. However, their susceptibility to jailbreaks and perturbations necessitates additional evaluations. Many LLMs are multilingual, but safety-related training data contains mainly high-resource languages like English. This can leave them vulnerable to perturbations in low-resource languages such as Polish. We show how surprisingly strong attacks can be cheaply created by altering just a few characters and using a small proxy model for word importance calculation. We find that these character and word-level attacks drastically alter the predictions of different LLMs, suggesting a potential vulnerability that can be used to circumvent their internal safety mechanisms. We validate our attack construction methodology on Polish, a low-resource language, and find potential vulnerabilities of LLMs in this language. Additionally, we show how it can be extended to other languages. We release the created datasets and code for further research.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07645</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 11:09:39 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Through the Valley: Path to Effective Long CoT Training for Small Language Models</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07712</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07712.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Renjie Luo, Jiaxi Li, Chen Huang, Wei Lu</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Long chain-of-thought (CoT) supervision has become a common strategy to enhance reasoning in language models. While effective for large models, we identify a phenomenon we call Long CoT Degradation, in which small language models (SLMs; <=3B parameters) trained on limited long CoT data experience significant performance deterioration. Through extensive experiments on the Qwen2.5, LLaMA3 and Gemma3 families, we demonstrate that this degradation is widespread across SLMs. In some settings, models trained on only 8k long CoT examples lose up to 75% of their original performance before fine-tuning. Strikingly, we further observe that for some particularly small models, even training on 220k long CoT examples fails to recover or surpass their original performance prior to fine-tuning. Our analysis attributes this effect to error accumulation: while longer responses increase the capacity for multi-step reasoning, they also amplify the risk of compounding mistakes. Furthermore, we find that Long CoT Degradation may negatively impacts downstream reinforcement learning (RL), although this can be alleviated by sufficiently scaled supervised fine-tuning (SFT). Our findings challenge common assumptions about the benefits of long CoT training for SLMs and offer practical guidance for building more effective small-scale reasoning models.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07712</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 12:56:41 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Image Reconstruction as a Tool for Feature Analysis</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07803</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07803.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Eduard Allakhverdov, Dmitrii Tarasov, Elizaveta Goncharova, Andrey Kuznetsov</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> Vision encoders are increasingly used in modern applications, from vision-only models to multimodal systems such as vision-language models. Despite their remarkable success, it remains unclear how these architectures represent features internally. Here, we propose a novel approach for interpreting vision features via image reconstruction. We compare two related model families, SigLIP and SigLIP2, which differ only in their training objective, and show that encoders pre-trained on image-based tasks retain significantly more image information than those trained on non-image tasks such as contrastive learning. We further apply our method to a range of vision encoders, ranking them by the informativeness of their feature representations. Finally, we demonstrate that manipulating the feature space yields predictable changes in reconstructed images, revealing that orthogonal rotations (rather than spatial transformations) control color encoding. Our approach can be applied to any vision encoder, shedding light on the inner structure of its feature space. The code and model weights to reproduce the experiments are available in GitHub.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07803</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 14:32:18 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Improving large language models with concept-aware fine-tuning</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07833</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07833.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Michael K. Chen, Xikun Zhang, Jiaxing Huang, Dacheng Tao</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Large language models (LLMs) have become the cornerstone of modern AI. However, the existing paradigm of next-token prediction fundamentally limits their ability to form coherent, high-level concepts, making it a critical barrier to human-like understanding and reasoning. Take the phrase "ribonucleic acid" as an example: an LLM will first decompose it into tokens, i.e., artificial text fragments ("rib", "on", ...), then learn each token sequentially, rather than grasping the phrase as a unified, coherent semantic entity. This fragmented representation hinders deeper conceptual understanding and, ultimately, the development of truly intelligent systems. In response, we introduce Concept-Aware Fine-Tuning (CAFT), a novel multi-token training method that redefines how LLMs are fine-tuned. By enabling the learning of sequences that span multiple tokens, this method fosters stronger concept-aware learning. Our experiments demonstrate significant improvements compared to conventional next-token finetuning methods across diverse tasks, including traditional applications like text summarization and domain-specific ones like de novo protein design. Multi-token prediction was previously only possible in the prohibitively expensive pretraining phase; CAFT, to our knowledge, is the first to bring the multi-token setting to the post-training phase, thus effectively democratizing its benefits for the broader community of practitioners and researchers. Finally, the unexpected effectiveness of our proposed method suggests wider implications for the machine learning research community. All code and data are available at https://github.com/michaelchen-lab/caft-llm</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07833</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 14:55:00 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07848</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07848.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Teng Hu, Zhentao Yu, Zhengguang Zhou, Jiangning Zhang, Yuan Zhou, Qinglin Lu, Ran Yi</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Despite recent advances in video generation, existing models still lack fine-grained controllability, especially for multi-subject customization with consistent identity and interaction. In this paper, we propose PolyVivid, a multi-subject video customization framework that enables flexible and identity-consistent generation. To establish accurate correspondences between subject images and textual entities, we design a VLLM-based text-image fusion module that embeds visual identities into the textual space for precise grounding. To further enhance identity preservation and subject interaction, we propose a 3D-RoPE-based enhancement module that enables structured bidirectional fusion between text and image embeddings. Moreover, we develop an attention-inherited identity injection module to effectively inject fused identity features into the video generation process, mitigating identity drift. Finally, we construct an MLLM-based data pipeline that combines MLLM-based grounding, segmentation, and a clique-based subject consolidation strategy to produce high-quality multi-subject data, effectively enhancing subject distinction and reducing ambiguity in downstream video generation. Extensive experiments demonstrate that PolyVivid achieves superior performance in identity fidelity, video realism, and subject alignment, outperforming existing open-source and commercial baselines.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07848</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 15:11:09 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>MiniCPM4: Ultra-Efficient LLMs on End Devices</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07900</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07900.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Jie Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, Ning Ding, Shengdan Fan, Yewei Fang, Zixuan Fu, Wenyu Guan, Yitong Guan, Junshao Guo, Yufeng Han, Bingxiang He, Yuxiang Huang, Cunliang Kong, Qiuzuo Li, Siyuan Li, Wenhao Li, Yanghao Li, Yishan Li, Zhen Li, Dan Liu, Biyuan Lin, Yankai Lin, Xiang Long, Quanyu Lu, Yaxi Lu, Peiyan Luo, Hongya Lyu, Litu Ou, Yinxu Pan, Zekai Qu, Qundong Shi, Zijun Song, Jiayuan Su, Zhou Su, Ao Sun, Xianghui Sun, Peijun Tang, Fangzheng Wang, Feng Wang, Shuo Wang, Yudong Wang, Yesai Wu, Zhenyu Xiao, Jie Xie, Zihao Xie, Yukun Yan, Jiarui Yuan, Kaihuo Zhang, Lei Zhang, Linyue Zhang, Xueren Zhang, Yudi Zhang, Hengyu Zhao, Weilin Zhao, Weilun Zhao, Yuanqian Zhao, Zhi Zheng, Ge Zhou, Jie Zhou, Wei Zhou, Zihan Zhou, Zixuan Zhou, Zhiyuan Liu, Guoyang Zeng, Chao Jia, Dahai Li, Maosong Sun</p><p><b>Upvotes:</b> 55</p><p><b>Summary:</b> This paper introduces MiniCPM4, a highly efficient large language model (LLM) designed explicitly for end-side devices. We achieve this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems. Specifically, in terms of model architecture, we propose InfLLM v2, a trainable sparse attention mechanism that accelerates both prefilling and decoding phases for long-context processing. Regarding training data, we propose UltraClean, an efficient and accurate pre-training data filtering and generation strategy, and UltraChat v2, a comprehensive supervised fine-tuning dataset. These datasets enable satisfactory model performance to be achieved using just 8 trillion training tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient pre-training strategy search, and improve existing post-training methods by introducing chunk-wise rollout for load-balanced reinforcement learning and data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose CPM.cu that integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding. To meet diverse on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B parameters, respectively. Sufficient evaluation results show that MiniCPM4 outperforms open-source models of similar size across multiple benchmarks, highlighting both its efficiency and effectiveness. Notably, MiniCPM4-8B demonstrates significant speed improvements over Qwen3-8B when processing long sequences. Through further adaptation, MiniCPM4 successfully powers diverse applications, including trustworthy survey generation and tool use with model context protocol, clearly showcasing its broad usability.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07900</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 16:16:50 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>CyberV: Cybernetics for Test-time Scaling in Video Understanding</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07971</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07971.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiahao Meng, Shuyang Sun, Yue Tan, Lu Qi, Yunhai Tong, Xiangtai Li, Longyin Wen</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Current Multimodal Large Language Models (MLLMs) may struggle with understanding long or complex videos due to computational demands at test time, lack of robustness, and limited accuracy, primarily stemming from their feed-forward processing nature. These limitations could be more severe for models with fewer parameters. To address these limitations, we propose a novel framework inspired by cybernetic principles, redesigning video MLLMs as adaptive systems capable of self-monitoring, self-correction, and dynamic resource allocation during inference. Our approach, CyberV, introduces a cybernetic loop consisting of an MLLM Inference System, a Sensor, and a Controller. Specifically, the sensor monitors forward processes of the MLLM and collects intermediate interpretations, such as attention drift, then the controller determines when and how to trigger self-correction and generate feedback to guide the next round. This test-time adaptive scaling framework enhances frozen MLLMs without requiring retraining or additional components. Experiments demonstrate significant improvements: CyberV boosts Qwen2.5-VL-7B by 8.3% and InternVL3-8B by 5.5% on VideoMMMU, surpassing the competitive proprietary model GPT-4o. When applied to Qwen2.5-VL-72B, it yields a 10.0% improvement, achieving performance even comparable to human experts. Furthermore, our method demonstrates consistent gains on general-purpose benchmarks, such as VideoMME and WorldSense, highlighting its effectiveness and generalization capabilities in making MLLMs more robust and accurate for dynamic video understanding. The code is released at https://github.com/marinero4972/CyberV.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07971</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:45:18 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07977</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07977.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingjing Chang, Yixiao Fang, Peng Xing, Shuhan Wu, Wei Cheng, Rui Wang, Xianfang Zeng, Gang Yu, Hai-Bao Chen</p><p><b>Upvotes:</b> 37</p><p><b>Summary:</b> Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts. However, rapid T2I model advancements reveal limitations in early benchmarks, lacking comprehensive evaluations, for example, the evaluation on reasoning, text rendering and style. Notably, recent state-of-the-art models, with their rich knowledge modeling capabilities, show promising results on the image generation problems requiring strong reasoning ability, yet existing evaluation systems have not adequately addressed this frontier. To systematically address these gaps, we introduce OneIG-Bench, a meticulously designed comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions, including prompt-image alignment, text rendering precision, reasoning-generated content, stylization, and diversity. By structuring the evaluation, this benchmark enables in-depth analysis of model performance, helping researchers and practitioners pinpoint strengths and bottlenecks in the full pipeline of image generation. Specifically, OneIG-Bench enables flexible evaluation by allowing users to focus on a particular evaluation subset. Instead of generating images for the entire set of prompts, users can generate images only for the prompts associated with the selected dimension and complete the corresponding evaluation accordingly. Our codebase and dataset are now publicly available to facilitate reproducible evaluation studies and cross-model comparisons within the T2I research community.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07977</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:50:21 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>τ^2-Bench: Evaluating Conversational Agents in a Dual-Control Environment</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07982</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07982.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider. This differs from real-world scenarios like technical support, where users need to actively participate in modifying the state of the (shared) world. In order to address this gap, we introduce tau^2-bench, with four key contributions: 1) A novel Telecom dual-control domain modeled as a Dec-POMDP, where both agent and user make use of tools to act in a shared, dynamic environment that tests both agent coordination and communication, 2) A compositional task generator that programmatically creates diverse, verifiable tasks from atomic components, ensuring domain coverage and controlled complexity, 3) A reliable user simulator tightly coupled with the environment, whose behavior is constrained by tools and observable states, improving simulation fidelity, 4) Fine-grained analysis of agent performance through multiple ablations including separating errors arising from reasoning vs communication/coordination. In particular, our experiments show significant performance drops when agents shift from no-user to dual-control, highlighting the challenges of guiding users. Overall, tau^2-bench provides a controlled testbed for agents that must both reason effectively and guide user actions.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07982</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:52:18 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.07986</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07986.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhengyao Lv, Tianlin Pan, Chenyang Si, Zhaoxi Chen, Wangmeng Zuo, Ziwei Liu, Kwan-Yee K. Wong</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress in text-driven visual generation. However, even state-of-the-art MM-DiT models like FLUX struggle with achieving precise alignment between text prompts and generated content. We identify two key issues in the attention mechanism of MM-DiT, namely 1) the suppression of cross-modal attention due to token imbalance between visual and textual modalities and 2) the lack of timestep-aware attention weighting, which hinder the alignment. To address these issues, we propose Temperature-Adjusted Cross-modal Attention (TACA), a parameter-efficient method that dynamically rebalances multimodal interactions through temperature scaling and timestep-dependent adjustment. When combined with LoRA fine-tuning, TACA significantly enhances text-image alignment on the T2I-CompBench benchmark with minimal computational overhead. We tested TACA on state-of-the-art models like FLUX and SD3.5, demonstrating its ability to improve image-text alignment in terms of object appearance, attribute binding, and spatial relationships. Our findings highlight the importance of balancing cross-modal attention in improving semantic fidelity in text-to-image diffusion models. Our codes are publicly available at https://github.com/Vchitect/TACA</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07986</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:54:04 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Dynamic View Synthesis as an Inverse Problem</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.08004</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08004.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hidir Yesiltepe, Pinar Yanardag</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> In this work, we address dynamic view synthesis from monocular videos as an inverse problem in a training-free setting. By redesigning the noise initialization phase of a pre-trained video diffusion model, we enable high-fidelity dynamic view synthesis without any weight updates or auxiliary modules. We begin by identifying a fundamental obstacle to deterministic inversion arising from zero-terminal signal-to-noise ratio (SNR) schedules and resolve it by introducing a novel noise representation, termed K-order Recursive Noise Representation. We derive a closed form expression for this representation, enabling precise and efficient alignment between the VAE-encoded and the DDIM inverted latents. To synthesize newly visible regions resulting from camera motion, we introduce Stochastic Latent Modulation, which performs visibility aware sampling over the latent space to complete occluded regions. Comprehensive experiments demonstrate that dynamic view synthesis can be effectively performed through structured latent manipulation in the noise initialization phase.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08004</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:59:47 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Dreamland: Controllable World Creation with Simulator and Generative Models</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.08006</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08006.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sicheng Mo, Ziyang Leng, Leon Liu, Weizhen Wang, Honglin He, Bolei Zhou</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Large-scale video generative models can synthesize diverse and realistic visual content for dynamic world creation, but they often lack element-wise controllability, hindering their use in editing scenes and training embodied AI agents. We propose Dreamland, a hybrid world generation framework combining the granular control of a physics-based simulator and the photorealistic content output of large-scale pretrained generative models. In particular, we design a layered world abstraction that encodes both pixel-level and object-level semantics and geometry as an intermediate representation to bridge the simulator and the generative model. This approach enhances controllability, minimizes adaptation cost through early alignment with real-world distributions, and supports off-the-shelf use of existing and future pretrained generative models. We further construct a D3Sim dataset to facilitate the training and evaluation of hybrid generation pipelines. Experiments demonstrate that Dreamland outperforms existing baselines with 50.8% improved image quality, 17.9% stronger controllability, and has great potential to enhance embodied agent training. Code and data will be made available.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08006</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:59:52 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Reinforcement Pre-Training</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.08007</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08007.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, Furu Wei</p><p><b>Upvotes:</b> 143</p><p><b>Summary:</b> In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08007</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:59:53 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.08012</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08012.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08012</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:59:57 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Play to Generalize: Learning to Reason Through Game Play</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.08011</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08011.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yunfei Xie, Yinsong Ma, Shiyi Lan, Alan Yuille, Junfei Xiao, Chen Wei</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Developing generalizable reasoning capabilities in multimodal large language models (MLLMs) remains challenging. Motivated by cognitive science literature suggesting that gameplay promotes transferable cognitive skills, we propose a novel post-training paradigm, Visual Game Learning, or ViGaL, where MLLMs develop out-of-domain generalization of multimodal reasoning through playing arcade-like games. Specifically, we show that post-training a 7B-parameter MLLM via reinforcement learning (RL) on simple arcade-like games, e.g. Snake, significantly enhances its downstream performance on multimodal math benchmarks like MathVista, and on multi-discipline questions like MMMU, without seeing any worked solutions, equations, or diagrams during RL, suggesting the capture of transferable reasoning skills. Remarkably, our model outperforms specialist models tuned on multimodal reasoning data in multimodal reasoning benchmarks, while preserving the base model's performance on general visual benchmarks, a challenge where specialist models often fall short. Our findings suggest a new post-training paradigm: synthetic, rule-based games can serve as controllable and scalable pre-text tasks that unlock generalizable multimodal reasoning abilities in MLLMs.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08011</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:59:57 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Vision Transformers Don't Need Trained Registers</title>
|
|
||||||
<link>https://arxiv.org/abs/2506.08010</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08010.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nick Jiang, Amil Dravid, Alexei Efros, Yossi Gandelsman</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> We investigate the mechanism underlying a previously identified phenomenon in Vision Transformers -- the emergence of high-norm tokens that lead to noisy attention maps. We observe that in multiple models (e.g., CLIP, DINOv2), a sparse set of neurons is responsible for concentrating high-norm activations on outlier tokens, leading to irregular attention patterns and degrading downstream visual processing. While the existing solution for removing these outliers involves retraining models from scratch with additional learned register tokens, we use our findings to create a training-free approach to mitigate these artifacts. By shifting the high-norm activations from our discovered register neurons into an additional untrained token, we can mimic the effect of register tokens on a model already trained without registers. We demonstrate that our method produces cleaner attention and feature maps, enhances performance over base models across multiple downstream visual tasks, and achieves results comparable to models explicitly trained with register tokens. We then extend test-time registers to off-the-shelf vision-language models to improve their interpretability. Our results suggest that test-time registers effectively take on the role of register tokens at test-time, offering a training-free solution for any pre-trained model released without them.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08010</guid>
|
|
||||||
<pubDate>Mon, 09 Jun 2025 17:59:57 +0000</pubDate>
|
|
||||||
</item>
|
</item>
|
||||||
</channel>
|
</channel>
|
||||||
</rss>
|
</rss>
|
||||||
|
|||||||
Reference in New Issue
Block a user