bot: update RSS feed
This commit is contained in:
github-actions[bot]
608
feed.xml
608
feed.xml
@@ -7,391 +7,447 @@
|
||||
<docs>http://www.rssboard.org/rss-specification</docs>
|
||||
<generator>python-feedgen</generator>
|
||||
<language>en</language>
|
||||
<lastBuildDate>Fri, 30 May 2025 00:02:53 +0000</lastBuildDate>
|
||||
<lastBuildDate>Sat, 31 May 2025 00:02:34 +0000</lastBuildDate>
|
||||
<item>
|
||||
<title>Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment</title>
|
||||
<link>https://arxiv.org/abs/2505.11821</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.11821.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, Mingyi Hong</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> This paper investigates approaches to enhance the reasoning capabilities of Large Language Model (LLM) agents using Reinforcement Learning (RL). Specifically, we focus on multi-turn tool-use scenarios, which can be naturally modeled as Markov Decision Processes (MDPs). While existing approaches often train multi-turn LLM agents with trajectory-level advantage estimation in bandit settings, they struggle with turn-level credit assignment across multiple decision steps, limiting their performance on multi-turn reasoning tasks. To address this, we introduce a fine-grained turn-level advantage estimation strategy to enable more precise credit assignment in multi-turn agent interactions. The strategy is general and can be incorporated into various RL algorithms such as Group Relative Preference Optimization (GRPO). Our experimental evaluation on multi-turn reasoning and search-based tool-use tasks with GRPO implementations highlights the effectiveness of the MDP framework and the turn-level credit assignment in advancing the multi-turn reasoning capabilities of LLM agents in complex decision-making settings. Our method achieves 100% success in tool execution and 50% accuracy in exact answer matching, significantly outperforming baselines, which fail to invoke tools and achieve only 20-30% exact match accuracy.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.11821</guid>
|
||||
<pubDate>Sat, 17 May 2025 04:09:46 +0000</pubDate>
|
||||
<title>Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?</title>
|
||||
<link>https://arxiv.org/abs/2505.14321</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.14321.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bo Feng, Zhengfeng Lai, Shiyu Li, Zizhen Wang, Simon Wang, Ping Huang, Meng Cao</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Existing video understanding benchmarks often conflate knowledge-based and purely image-based questions, rather than clearly isolating a model's temporal reasoning ability, which is the key aspect that distinguishes video understanding from other modalities. We identify two major limitations that obscure whether higher scores truly indicate stronger understanding of the dynamic content in videos: (1) strong language priors, where models can answer questions without watching the video; and (2) shuffling invariance, where models maintain similar performance on certain questions even when video frames are temporally shuffled. To alleviate these issues, we propose VBenchComp, an automated pipeline that categorizes questions into different domains: LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions can be answered without viewing the video; Semantic questions remain answerable even when the video frames are shuffled; and Temporal questions require understanding the correct temporal order of frames. The rest of the questions are labeled as Others. This can enable fine-grained evaluation of different capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that are hidden by traditional overall scores, and we offer insights and recommendations for designing future benchmarks that more accurately assess video LLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.14321</guid>
|
||||
<pubDate>Tue, 20 May 2025 13:07:55 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Safe-Sora: Safe Text-to-Video Generation via Graphical Watermarking</title>
|
||||
<link>https://arxiv.org/abs/2505.12667</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.12667.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zihan Su, Xuerui Qiu, Hongbin Xu, Tangyu Jiang, Junhao Zhuang, Chun Yuan, Ming Li, Shengfeng He, Fei Richard Yu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> The explosive growth of generative video models has amplified the demand for reliable copyright preservation of AI-generated content. Despite its popularity in image synthesis, invisible generative watermarking remains largely underexplored in video generation. To address this gap, we propose Safe-Sora, the first framework to embed graphical watermarks directly into the video generation process. Motivated by the observation that watermarking performance is closely tied to the visual similarity between the watermark and cover content, we introduce a hierarchical coarse-to-fine adaptive matching mechanism. Specifically, the watermark image is divided into patches, each assigned to the most visually similar video frame, and further localized to the optimal spatial region for seamless embedding. To enable spatiotemporal fusion of watermark patches across video frames, we develop a 3D wavelet transform-enhanced Mamba architecture with a novel spatiotemporal local scanning strategy, effectively modeling long-range dependencies during watermark embedding and retrieval. To the best of our knowledge, this is the first attempt to apply state space models to watermarking, opening new avenues for efficient and robust watermark protection. Extensive experiments demonstrate that Safe-Sora achieves state-of-the-art performance in terms of video quality, watermark fidelity, and robustness, which is largely attributed to our proposals. We will release our code upon publication.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.12667</guid>
|
||||
<pubDate>Mon, 19 May 2025 03:31:31 +0000</pubDate>
|
||||
<title>PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions</title>
|
||||
<link>https://arxiv.org/abs/2505.17818</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17818.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daeun Kyung, Hyunseung Chung, Seongsu Bae, Jiho Kim, Jae Ho Sohn, Taerim Kim, Soo Kyung Kim, Edward Choi</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Doctor-patient consultations require multi-turn, context-aware communication tailored to diverse patient personas. Training or evaluating doctor LLMs in such settings requires realistic patient interaction systems. However, existing simulators often fail to reflect the full range of personas seen in clinical practice. To address this, we introduce PatientSim, a patient simulator that generates realistic and diverse patient personas for clinical scenarios, grounded in medical expertise. PatientSim operates using: 1) clinical profiles, including symptoms and medical history, derived from real-world data in the MIMIC-ED and MIMIC-IV datasets, and 2) personas defined by four axes: personality, language proficiency, medical history recall level, and cognitive confusion level, resulting in 37 unique combinations. We evaluated eight LLMs for factual accuracy and persona consistency. The top-performing open-source model, Llama 3.3, was validated by four clinicians to confirm the robustness of our framework. As an open-source, customizable platform, PatientSim provides a reproducible and scalable solution that can be customized for specific training needs. Offering a privacy-compliant environment, it serves as a robust testbed for evaluating medical dialogue systems across diverse patient presentations and shows promise as an educational tool for healthcare.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17818</guid>
|
||||
<pubDate>Fri, 23 May 2025 12:34:48 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex</title>
|
||||
<link>https://arxiv.org/abs/2505.15813</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.15813.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Muquan Yu, Mu Nan, Hossein Adeli, Jacob S. Prince, John A. Pyles, Leila Wehbe, Margaret M. Henderson, Michael J. Tarr, Andrew F. Luo</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Understanding functional representations within higher visual cortex is a fundamental question in computational neuroscience. While artificial neural networks pretrained on large-scale datasets exhibit striking representational alignment with human neural responses, learning image-computable models of visual cortex relies on individual-level, large-scale fMRI datasets. The necessity for expensive, time-intensive, and often impractical data acquisition limits the generalizability of encoders to new subjects and stimuli. BraInCoRL uses in-context learning to predict voxelwise neural responses from few-shot examples without any additional finetuning for novel subjects and stimuli. We leverage a transformer architecture that can flexibly condition on a variable number of in-context image stimuli, learning an inductive bias over multiple subjects. During training, we explicitly optimize the model for in-context learning. By jointly conditioning on image features and voxel activations, our model learns to directly generate better performing voxelwise models of higher visual cortex. We demonstrate that BraInCoRL consistently outperforms existing voxelwise encoder designs in a low-data regime when evaluated on entirely novel images, while also exhibiting strong test-time scaling behavior. The model also generalizes to an entirely new visual fMRI dataset, which uses different subjects and fMRI data acquisition parameters. Further, BraInCoRL facilitates better interpretability of neural signals in higher visual cortex by attending to semantically relevant stimuli. Finally, we show that our framework enables interpretable mappings from natural language queries to voxel selectivity.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.15813</guid>
|
||||
<pubDate>Wed, 21 May 2025 17:59:41 +0000</pubDate>
|
||||
<title>CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays</title>
|
||||
<link>https://arxiv.org/abs/2505.18087</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18087.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hyungyung Lee, Geon Choi, Jung-Oh Lee, Hangyul Yoon, Hyuk Gi Hong, Edward Choi</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Recent progress in Large Vision-Language Models (LVLMs) has enabled promising applications in medical tasks, such as report generation and visual question answering. However, existing benchmarks focus mainly on the final diagnostic answer, offering limited insight into whether models engage in clinically meaningful reasoning. To address this, we present CheXStruct and CXReasonBench, a structured pipeline and benchmark built on the publicly available MIMIC-CXR-JPG dataset. CheXStruct automatically derives a sequence of intermediate reasoning steps directly from chest X-rays, such as segmenting anatomical regions, deriving anatomical landmarks and diagnostic measurements, computing diagnostic indices, and applying clinical thresholds. CXReasonBench leverages this pipeline to evaluate whether models can perform clinically valid reasoning steps and to what extent they can learn from structured guidance, enabling fine-grained and transparent assessment of diagnostic reasoning. The benchmark comprises 18,988 QA pairs across 12 diagnostic tasks and 1,200 cases, each paired with up to 4 visual inputs, and supports multi-path, multi-stage evaluation including visual grounding via anatomical region selection and diagnostic measurements. Even the strongest of 10 evaluated LVLMs struggle with structured reasoning and generalization, often failing to link abstract knowledge with anatomically grounded visual interpretation. The code is available at https://github.com/ttumyche/CXReasonBench</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18087</guid>
|
||||
<pubDate>Fri, 23 May 2025 16:44:21 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding</title>
|
||||
<link>https://arxiv.org/abs/2505.17330</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17330.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Amit Agarwal, Srikant Panda, Kulbhushan Pachauri</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> In this work, we propose Few Shot Domain Adapting Graph (FS-DAG), a scalable and efficient model architecture for visually rich document understanding (VRDU) in few-shot settings. FS-DAG leverages domain-specific and language/vision specific backbones within a modular framework to adapt to diverse document types with minimal data. The model is robust to practical challenges such as handling OCR errors, misspellings, and domain shifts, which are critical in real-world deployments. FS-DAG is highly performant with less than 90M parameters, making it well-suited for complex real-world applications for Information Extraction (IE) tasks where computational resources are limited. We demonstrate FS-DAG's capability through extensive experiments for information extraction task, showing significant improvements in convergence speed and performance compared to state-of-the-art methods. Additionally, this work highlights the ongoing progress in developing smaller, more efficient models that do not compromise on performance. Code : https://github.com/oracle-samples/fs-dag</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17330</guid>
|
||||
<pubDate>Thu, 22 May 2025 22:53:58 +0000</pubDate>
|
||||
<title>TokBench: Evaluating Your Visual Tokenizer before Visual Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.18142</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18142.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junfeng Wu, Dongliang Luo, Weizhi Zhao, Zhihao Xie, Yuanhao Wang, Junyi Li, Xudong Xie, Yuliang Liu, Xiang Bai</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> In this work, we reveal the limitations of visual tokenizers and VAEs in preserving fine-grained features, and propose a benchmark to evaluate reconstruction performance for two challenging visual contents: text and face. Visual tokenizers and VAEs have significantly advanced visual generation and multimodal modeling by providing more efficient compressed or quantized image representations. However, while helping production models reduce computational burdens, the information loss from image compression fundamentally limits the upper bound of visual generation quality. To evaluate this upper bound, we focus on assessing reconstructed text and facial features since they typically: 1) exist at smaller scales, 2) contain dense and rich textures, 3) are prone to collapse, and 4) are highly sensitive to human vision. We first collect and curate a diverse set of clear text and face images from existing datasets. Unlike approaches using VLM models, we employ established OCR and face recognition models for evaluation, ensuring accuracy while maintaining an exceptionally lightweight assessment process <span style="font-weight: bold; color: rgb(214, 21, 21);">requiring just 2GB memory and 4 minutes</span> to complete. Using our benchmark, we analyze text and face reconstruction quality across various scales for different image tokenizers and VAEs. Our results show modern visual tokenizers still struggle to preserve fine-grained features, especially at smaller scales. We further extend this evaluation framework to video, conducting comprehensive analysis of video tokenizers. Additionally, we demonstrate that traditional metrics fail to accurately reflect reconstruction performance for faces and text, while our proposed metrics serve as an effective complement.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18142</guid>
|
||||
<pubDate>Fri, 23 May 2025 17:52:16 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Benchmarking Recommendation, Classification, and Tracing Based on Hugging Face Knowledge Graph</title>
|
||||
<link>https://arxiv.org/abs/2505.17507</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17507.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qiaosheng Chen, Kaijia Huang, Xiao Zhou, Weiqing Luo, Yuanning Cui, Gong Cheng</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> The rapid growth of open source machine learning (ML) resources, such as models and datasets, has accelerated IR research. However, existing platforms like Hugging Face do not explicitly utilize structured representations, limiting advanced queries and analyses such as tracing model evolution and recommending relevant datasets. To fill the gap, we construct HuggingKG, the first large-scale knowledge graph built from the Hugging Face community for ML resource management. With 2.6 million nodes and 6.2 million edges, HuggingKG captures domain-specific relations and rich textual attributes. It enables us to further present HuggingBench, a multi-task benchmark with three novel test collections for IR tasks including resource recommendation, classification, and tracing. Our experiments reveal unique characteristics of HuggingKG and the derived tasks. Both resources are publicly available, expected to advance research in open source resource sharing and management.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17507</guid>
|
||||
<pubDate>Fri, 23 May 2025 06:00:20 +0000</pubDate>
|
||||
<title>System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts</title>
|
||||
<link>https://arxiv.org/abs/2505.18962</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18962.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, Bang Liu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Chain-of-thought (CoT) reasoning enables large language models (LLMs) to move beyond fast System-1 responses and engage in deliberative System-2 reasoning. However, this comes at the cost of significant inefficiency due to verbose intermediate output. Recent latent-space reasoning methods improve efficiency by operating on hidden states without decoding into language, yet they treat all steps uniformly, failing to distinguish critical deductions from auxiliary steps and resulting in suboptimal use of computational resources. In this paper, we propose System-1.5 Reasoning, an adaptive reasoning framework that dynamically allocates computation across reasoning steps through shortcut paths in latent space. Specifically, System-1.5 Reasoning introduces two types of dynamic shortcuts. The model depth shortcut (DS) adaptively reasons along the vertical depth by early exiting non-critical tokens through lightweight adapter branches, while allowing critical tokens to continue through deeper Transformer layers. The step shortcut (SS) reuses hidden states across the decoding steps to skip trivial steps and reason horizontally in latent space. Training System-1.5 Reasoning involves a two-stage self-distillation process: first distilling natural language CoT into latent-space continuous thought, and then distilling full-path System-2 latent reasoning into adaptive shortcut paths (System-1.5 Reasoning). Experiments on reasoning tasks demonstrate the superior performance of our method. For example, on GSM8K, System-1.5 Reasoning achieves reasoning performance comparable to traditional CoT fine-tuning methods while accelerating inference by over 20x and reducing token generation by 92.31% on average.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18962</guid>
|
||||
<pubDate>Sun, 25 May 2025 03:35:49 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States</title>
|
||||
<link>https://arxiv.org/abs/2505.17663</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17663.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Xiao, Jiashuo Wang, Qiancheng Xu, Changhe Song, Chunpu Xu, Yi Cheng, Wenjie Li, Pengfei Liu</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> As Large Language Models (LLMs) increasingly participate in human-AI interactions, evaluating their Theory of Mind (ToM) capabilities - particularly their ability to track dynamic mental states - becomes crucial. While existing benchmarks assess basic ToM abilities, they predominantly focus on static snapshots of mental states, overlooking the temporal evolution that characterizes real-world social interactions. We present DynToM, a novel benchmark specifically designed to evaluate LLMs' ability to understand and track the temporal progression of mental states across interconnected scenarios. Through a systematic four-step framework, we generate 1,100 social contexts encompassing 5,500 scenarios and 78,100 questions, each validated for realism and quality. Our comprehensive evaluation of ten state-of-the-art LLMs reveals that their average performance underperforms humans by 44.7\%, with performance degrading significantly when tracking and reasoning about the shift of mental states. This performance gap highlights fundamental limitations in current LLMs' ability to model the dynamic nature of human mental states.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17663</guid>
|
||||
<pubDate>Fri, 23 May 2025 09:27:40 +0000</pubDate>
|
||||
<title>Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator</title>
|
||||
<link>https://arxiv.org/abs/2505.19236</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19236.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qian Cao, Xiting Wang, Yuzhuo Yuan, Yahui Liu, Fang Luo, Ruihua Song</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Creativity evaluation remains a challenging frontier for large language models (LLMs). Current evaluations heavily rely on inefficient and costly human judgments, hindering progress in enhancing machine creativity. While automated methods exist, ranging from psychological testing to heuristic- or prompting-based approaches, they often lack generalizability or alignment with human judgment. To address these issues, in this paper, we propose a novel pairwise-comparison framework for assessing textual creativity, leveraging shared contextual instructions to improve evaluation consistency. We introduce CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic creative instruction-response pairs spanning diverse open-domain tasks. Through training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval demonstrates remarkable superiority over existing methods in alignment with human judgments. Experimental results underscore the indispensable significance of integrating both human-generated and synthetic data in training highly robust evaluators, and showcase the practical utility of CrEval in boosting the creativity of LLMs. We will release all data, code, and models publicly soon to support further research.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19236</guid>
|
||||
<pubDate>Sun, 25 May 2025 17:25:23 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality</title>
|
||||
<link>https://arxiv.org/abs/2505.18227</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18227.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, Marinka Zitnik</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks. Each token is then mapped to an embedding, enabling parallel attention computations while preserving the input's essential information. Due to the quadratic computational complexity of transformer self-attention mechanisms, token reduction has primarily been used as an efficiency strategy. This is especially true in single vision and language domains, where it helps balance computational costs, memory usage, and inference latency. Despite these advances, this paper argues that token reduction should transcend its traditional efficiency-oriented role in the era of large generative models. Instead, we position it as a fundamental principle in generative modeling, critically influencing both model architecture and broader applications. Specifically, we contend that across vision, language, and multimodal systems, token reduction can: (i) facilitate deeper multimodal integration and alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain coherence over long inputs, and (iv) enhance training stability, etc. We reframe token reduction as more than an efficiency measure. By doing so, we outline promising future directions, including algorithm design, reinforcement learning-guided token reduction, token optimization for in-context learning, and broader ML and scientific domains. We highlight its potential to drive new model architectures and learning strategies that improve robustness, increase interpretability, and better align with the objectives of generative modeling.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18227</guid>
|
||||
<pubDate>Fri, 23 May 2025 11:30:30 +0000</pubDate>
|
||||
<title>A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.19286</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19286.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Utkarsh Sahu, Zhisheng Qi, Yongjia Lei, Ryan A. Rossi, Franck Dernoncourt, Nesreen K. Ahmed, Mahantesh M Halappanavar, Yao Ma, Yu Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Large language models have been extensively studied as neural knowledge bases for their knowledge access, editability, reasoning, and explainability. However, few works focus on the structural patterns of their knowledge. Motivated by this gap, we investigate these structural patterns from a graph perspective. We quantify the knowledge of LLMs at both the triplet and entity levels, and analyze how it relates to graph structural properties such as node degree. Furthermore, we uncover the knowledge homophily, where topologically close entities exhibit similar levels of knowledgeability, which further motivates us to develop graph machine learning models to estimate entity knowledge based on its local neighbors. This model further enables valuable knowledge checking by selecting triplets less known to LLMs. Empirical results show that using selected triplets for fine-tuning leads to superior performance.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19286</guid>
|
||||
<pubDate>Sun, 25 May 2025 19:34:15 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Just as Humans Need Vaccines, So Do Models: Model Immunization to Combat Falsehoods</title>
|
||||
<link>https://arxiv.org/abs/2505.17870</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17870.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shaina Raza, Rizwan Qureshi, Marcelo Lotif, Aman Chadha, Deval Pandya, Christos Emmanouilidis</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Generative AI models often learn and reproduce false information present in their training corpora. This position paper argues that, analogous to biological immunization, where controlled exposure to a weakened pathogen builds immunity, AI models should be fine tuned on small, quarantined sets of explicitly labeled falsehoods as a "vaccine" against misinformation. These curated false examples are periodically injected during finetuning, strengthening the model ability to recognize and reject misleading claims while preserving accuracy on truthful inputs. An illustrative case study shows that immunized models generate substantially less misinformation than baselines. To our knowledge, this is the first training framework that treats fact checked falsehoods themselves as a supervised vaccine, rather than relying on input perturbations or generic human feedback signals, to harden models against future misinformation. We also outline ethical safeguards and governance controls to ensure the safe use of false data. Model immunization offers a proactive paradigm for aligning AI systems with factuality.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17870</guid>
|
||||
<pubDate>Fri, 23 May 2025 13:20:23 +0000</pubDate>
|
||||
<title>ChartLens: Fine-grained Visual Attribution in Charts</title>
|
||||
<link>https://arxiv.org/abs/2505.19360</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19360.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Dinesh Manocha</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> The growing capabilities of multimodal large language models (MLLMs) have advanced tasks like chart understanding. However, these models often suffer from hallucinations, where generated text sequences conflict with the provided visual data. To address this, we introduce Post-Hoc Visual Attribution for Charts, which identifies fine-grained chart elements that validate a given chart-associated response. We propose ChartLens, a novel chart attribution algorithm that uses segmentation-based techniques to identify chart objects and employs set-of-marks prompting with MLLMs for fine-grained visual attribution. Additionally, we present ChartVA-Eval, a benchmark with synthetic and real-world charts from diverse domains like finance, policy, and economics, featuring fine-grained attribution annotations. Our evaluations show that ChartLens improves fine-grained attributions by 26-66%.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19360</guid>
|
||||
<pubDate>Sun, 25 May 2025 23:17:32 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>First Finish Search: Efficient Test-Time Scaling in Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.18149</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18149.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Test-time scaling (TTS), which involves dynamic allocation of compute during inference, offers a promising way to improve reasoning in large language models. While existing TTS methods work well, they often rely on long decoding paths or require a large number of samples to be generated, increasing the token usage and inference latency. We observe the surprising fact that for reasoning tasks, shorter traces are much more likely to be correct than longer ones. Motivated by this, we introduce First Finish Search (FFS), a training-free parallel decoding strategy that launches n independent samples and returns as soon as any one completes. We evaluate FFS alongside simple decoding, beam search, majority voting, and budget forcing on four reasoning models (DeepSeek-R1, R1-Distill-Qwen-32B, QwQ-32B and Phi-4-Reasoning-Plus) and across four datasets (AIME24, AIME25-I, AIME25-II and GPQA Diamond). With DeepSeek-R1, FFS achieves 82.23% accuracy on the AIME datasets, a 15% improvement over DeepSeek-R1's standalone accuracy, nearly matching OpenAI's o4-mini performance. Our theoretical analysis explains why stopping at the shortest trace is likely to yield a correct answer and identifies the conditions under which early stopping may be suboptimal. The elegance and simplicity of FFS demonstrate that straightforward TTS strategies can perform remarkably well, revealing the untapped potential of simple approaches at inference time.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18149</guid>
|
||||
<pubDate>Fri, 23 May 2025 17:57:43 +0000</pubDate>
|
||||
<title>Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting</title>
|
||||
<link>https://arxiv.org/abs/2505.19716</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19716.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifan Wu, Jingze Shi, Bingheng Wu, Jiayi Zhang, Xiaotian Lin, Nan Tang, Yuyu Luo</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Existing chain-of-thought (CoT) distillation methods can effectively transfer reasoning abilities to base models but suffer from two major limitations: excessive verbosity of reasoning traces and inadequate adaptability to problem difficulty. Long reasoning traces significantly increase inference costs, and uniform-length solutions prevent base models from learning adaptive reasoning strategies. To address these issues, we propose a difficulty-aware prompting (DAP) method to dynamically shorten reasoning traces without performance loss. In our approach, a large teacher model first judges each problem's difficulty and then rewrites its reasoning traces to an appropriate shorter length, yielding concise yet complete reasoning traces. Leveraging the DAP pipeline, we curate a distilled dataset called LiteCoT consisting of 100K concise reasoning examples, with solutions averaging only 720 tokens (an order of magnitude shorter than typical CoTs). Using LiteCoT, we distilled a new family of reasoning models called Liter (1.5B, 7B, and 32B) based on the Qwen2.5 architecture. Experiments show that a student model fine-tuned on just 100K of these difficulty-pruned CoT samples outperforms a model distilled on 800K original Long CoT samples, while significantly reducing training and inference costs. Our method also generalizes well: across 11 diverse benchmarks, the shorter difficulty-aware CoTs achieve equal or better accuracy than Long chains, using far fewer tokens. For example, on the challenging AIME24 exam, our approach reaches 74.2% Pass@1 using only about 5K inference tokens, surpassing other methods that consume many more tokens. Our code and data are available at https://github.com/Evanwu1125/LiteCoT.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19716</guid>
|
||||
<pubDate>Mon, 26 May 2025 09:04:44 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems</title>
|
||||
<link>https://arxiv.org/abs/2505.18366</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18366.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hansa Meghwani, Amit Agarwal, Priyaranjan Pattnayak, Hitesh Laxmichand Patel, Srikant Panda</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Enterprise search systems often struggle to retrieve accurate, domain-specific information due to semantic mismatches and overlapping terminologies. These issues can degrade the performance of downstream applications such as knowledge management, customer support, and retrieval-augmented generation agents. To address this challenge, we propose a scalable hard-negative mining framework tailored specifically for domain-specific enterprise data. Our approach dynamically selects semantically challenging but contextually irrelevant documents to enhance deployed re-ranking models. Our method integrates diverse embedding models, performs dimensionality reduction, and uniquely selects hard negatives, ensuring computational efficiency and semantic precision. Evaluation on our proprietary enterprise corpus (cloud services domain) demonstrates substantial improvements of 15\% in MRR@3 and 19\% in MRR@10 compared to state-of-the-art baselines and other negative sampling techniques. Further validation on public domain-specific datasets (FiQA, Climate Fever, TechQA) confirms our method's generalizability and readiness for real-world applications.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18366</guid>
|
||||
<pubDate>Fri, 23 May 2025 20:51:20 +0000</pubDate>
|
||||
<title>Multi-Domain Explainability of Preferences</title>
|
||||
<link>https://arxiv.org/abs/2505.20088</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20088.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nitay Calderon, Liat Ein-Dor, Roi Reichart</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Preference mechanisms, such as human preference, LLM-as-a-Judge (LaaJ), and reward models, are central to aligning and evaluating large language models (LLMs). Yet, the underlying concepts that drive these preferences remain poorly understood. In this work, we propose a fully automated method for generating local and global concept-based explanations of preferences across multiple domains. Our method utilizes an LLM to identify concepts that distinguish between chosen and rejected responses, and to represent them with concept-based vectors. To model the relationships between concepts and preferences, we propose a white-box Hierarchical Multi-Domain Regression model that captures both domain-general and domain-specific effects. To evaluate our method, we curate a dataset spanning eight challenging and diverse domains and explain twelve mechanisms. Our method achieves strong preference prediction performance, outperforming baselines while also being explainable. Additionally, we assess explanations in two application-driven settings. First, guiding LLM outputs with concepts from LaaJ explanations yields responses that those judges consistently prefer. Second, prompting LaaJs with concepts explaining humans improves their preference predictions. Together, our work establishes a new paradigm for explainability in the era of LLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20088</guid>
|
||||
<pubDate>Mon, 26 May 2025 15:01:56 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment</title>
|
||||
<link>https://arxiv.org/abs/2505.18600</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18600.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bryan Sangwoo Kim, Jeongsol Kim, Jong Chul Ye</p><p><b>Upvotes:</b> 21</p><p><b>Summary:</b> Modern single-image super-resolution (SISR) models deliver photo-realistic results at the scale factors on which they are trained, but collapse when asked to magnify far beyond that regime. We address this scalability bottleneck with Chain-of-Zoom (CoZ), a model-agnostic framework that factorizes SISR into an autoregressive chain of intermediate scale-states with multi-scale-aware prompts. CoZ repeatedly re-uses a backbone SR model, decomposing the conditional probability into tractable sub-problems to achieve extreme resolutions without additional training. Because visual cues diminish at high magnifications, we augment each zoom step with multi-scale-aware text prompts generated by a vision-language model (VLM). The prompt extractor itself is fine-tuned using Generalized Reward Policy Optimization (GRPO) with a critic VLM, aligning text guidance towards human preference. Experiments show that a standard 4x diffusion SR model wrapped in CoZ attains beyond 256x enlargement with high perceptual quality and fidelity. Project Page: https://bryanswkim.github.io/chain-of-zoom/ .</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18600</guid>
|
||||
<pubDate>Sat, 24 May 2025 08:50:08 +0000</pubDate>
|
||||
<title>Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities</title>
|
||||
<link>https://arxiv.org/abs/2505.20099</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20099.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chuangtao Ma, Yongrui Chen, Tianxing Wu, Arijit Khan, Haofen Wang</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Large language models (LLMs) have demonstrated remarkable performance on question-answering (QA) tasks because of their superior capabilities in natural language understanding and generation. However, LLM-based QA struggles with complex QA tasks due to poor reasoning capacity, outdated knowledge, and hallucinations. Several recent works synthesize LLMs and knowledge graphs (KGs) for QA to address the above challenges. In this survey, we propose a new structured taxonomy that categorizes the methodology of synthesizing LLMs and KGs for QA according to the categories of QA and the KG's role when integrating with LLMs. We systematically survey state-of-the-art advances in synthesizing LLMs and KGs for QA and compare and analyze these approaches in terms of strength, limitations, and KG requirements. We then align the approaches with QA and discuss how these approaches address the main challenges of different complex QA. Finally, we summarize the advancements, evaluation metrics, and benchmark datasets and highlight open challenges and opportunities.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20099</guid>
|
||||
<pubDate>Mon, 26 May 2025 15:08:23 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chains</title>
|
||||
<link>https://arxiv.org/abs/2505.18700</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18700.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chun Wang, Xiaoran Pan, Zihao Pan, Haofan Wang, Yiren Song</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Recent advances in Visual Language Models (VLMs) have demonstrated exceptional performance in visual reasoning tasks. However, geo-localization presents unique challenges, requiring the extraction of multigranular visual cues from images and their integration with external world knowledge for systematic reasoning. Current approaches to geo-localization tasks often lack robust reasoning mechanisms and explainability, limiting their effectiveness. To address these limitations, we propose the Geo Reason Enhancement (GRE) Suite, a novel framework that augments VLMs with structured reasoning chains for accurate and interpretable location inference. The GRE Suite is systematically developed across three key dimensions: dataset, model, and benchmark. First, we introduce GRE30K, a high-quality geo-localization reasoning dataset designed to facilitate fine-grained visual and contextual analysis. Next, we present the GRE model, which employs a multi-stage reasoning strategy to progressively infer scene attributes, local details, and semantic features, thereby narrowing down potential geographic regions with enhanced precision. Finally, we construct the Geo Reason Evaluation Benchmark (GREval-Bench), a comprehensive evaluation framework that assesses VLMs across diverse urban, natural, and landmark scenes to measure both coarse-grained (e.g., country, continent) and fine-grained (e.g., city, street) localization performance. Experimental results demonstrate that GRE significantly outperforms existing methods across all granularities of geo-localization tasks, underscoring the efficacy of reasoning-augmented VLMs in complex geographic inference. Code and data will be released at https://github.com/Thorin215/GRE.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18700</guid>
|
||||
<pubDate>Sat, 24 May 2025 13:48:57 +0000</pubDate>
|
||||
<title>Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking</title>
|
||||
<link>https://arxiv.org/abs/2505.20199</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20199.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Pengxiang Li, Shilin Yan, Joey Tsai, Renrui Zhang, Ruichuan An, Ziyu Guo, Xiaowei Gao</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Classifier-Free Guidance (CFG) significantly enhances controllability in generative models by interpolating conditional and unconditional predictions. However, standard CFG often employs a static unconditional input, which can be suboptimal for iterative generation processes where model uncertainty varies dynamically. We introduce Adaptive Classifier-Free Guidance (A-CFG), a novel method that tailors the unconditional input by leveraging the model's instantaneous predictive confidence. At each step of an iterative (masked) diffusion language model, A-CFG identifies tokens in the currently generated sequence for which the model exhibits low confidence. These tokens are temporarily re-masked to create a dynamic, localized unconditional input. This focuses CFG's corrective influence precisely on areas of ambiguity, leading to more effective guidance. We integrate A-CFG into a state-of-the-art masked diffusion language model and demonstrate its efficacy. Experiments on diverse language generation benchmarks show that A-CFG yields substantial improvements over standard CFG, achieving, for instance, a 3.9 point gain on GPQA. Our work highlights the benefit of dynamically adapting guidance mechanisms to model uncertainty in iterative generation.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20199</guid>
|
||||
<pubDate>Mon, 26 May 2025 16:40:22 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach</title>
|
||||
<link>https://arxiv.org/abs/2505.18882</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18882.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuchen Wu, Edward Sun, Kaijie Zhu, Jianxun Lian, Jose Hernandez-Orallo, Aylin Caliskan, Jindong Wang</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Large language models (LLMs) typically generate identical or similar responses for all users given the same prompt, posing serious safety risks in high-stakes applications where user vulnerabilities differ widely. Existing safety evaluations primarily rely on context-independent metrics - such as factuality, bias, or toxicity - overlooking the fact that the same response may carry divergent risks depending on the user's background or condition. We introduce personalized safety to fill this gap and present PENGUIN - a benchmark comprising 14,000 scenarios across seven sensitive domains with both context-rich and context-free variants. Evaluating six leading LLMs, we demonstrate that personalized user information significantly improves safety scores by 43.2%, confirming the effectiveness of personalization in safety alignment. However, not all context attributes contribute equally to safety enhancement. To address this, we develop RAISE - a training-free, two-stage agent framework that strategically acquires user-specific background. RAISE improves safety scores by up to 31.6% over six vanilla LLMs, while maintaining a low interaction cost of just 2.7 user queries on average. Our findings highlight the importance of selective information gathering in safety-critical domains and offer a practical solution for personalizing LLM responses without model retraining. This work establishes a foundation for safety research that adapts to individual user contexts rather than assuming a universal harm standard.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18882</guid>
|
||||
<pubDate>Sat, 24 May 2025 21:37:10 +0000</pubDate>
|
||||
<title>One-shot Entropy Minimization</title>
|
||||
<link>https://arxiv.org/abs/2505.20282</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20282.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zitian Gao, Lynx Chen, Joey Zhou, Bryan Dai</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> We trained 13,440 large language models and found that entropy minimization requires only a single unlabeled data and 10 steps optimization to achieve performance improvements comparable to or even greater than those obtained using thousands of data and carefully designed rewards in rule-based reinforcement learning. This striking result may prompt a rethinking of post-training paradigms for large language models. Our code is avaliable at https://github.com/zitian-gao/one-shot-em.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20282</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:58:30 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Can Large Language Models Infer Causal Relationships from Real-World Text?</title>
|
||||
<link>https://arxiv.org/abs/2505.18931</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18931.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ryan Saklad, Aman Chadha, Oleg Pavlov, Raha Moraffah</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Understanding and inferring causal relationships from texts is a core aspect of human cognition and is essential for advancing large language models (LLMs) towards artificial general intelligence. Existing work primarily focuses on synthetically generated texts which involve simple causal relationships explicitly mentioned in the text. This fails to reflect the complexities of real-world tasks. In this paper, we investigate whether LLMs are capable of inferring causal relationships from real-world texts. We develop a benchmark drawn from real-world academic literature which includes diverse texts with respect to length, complexity of relationships (different levels of explicitness, number of events, and causal relationships), and domains and sub-domains. To the best of our knowledge, our benchmark is the first-ever real-world dataset for this task. Our experiments on state-of-the-art LLMs evaluated on our proposed benchmark demonstrate significant challenges, with the best-performing model achieving an average F1 score of only 0.477. Analysis reveals common pitfalls: difficulty with implicitly stated information, in distinguishing relevant causal factors from surrounding contextual details, and with connecting causally relevant information spread across lengthy textual passages. By systematically characterizing these deficiencies, our benchmark offers targeted insights for further research into advancing LLM causal reasoning.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18931</guid>
|
||||
<pubDate>Sun, 25 May 2025 01:50:05 +0000</pubDate>
|
||||
<title>Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction</title>
|
||||
<link>https://arxiv.org/abs/2505.20755</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20755.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifei Wang, Weimin Bai, Colin Zhang, Debing Zhang, Weijian Luo, He Sun</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> In this paper, we unify more than 10 existing one-step diffusion distillation approaches, such as Diff-Instruct, DMD, SIM, SiD, f-distill, etc, inside a theory-driven framework which we name the \emph{Uni-Instruct}. Uni-Instruct is motivated by our proposed diffusion expansion theory of the f-divergence family. Then we introduce key theories that overcome the intractability issue of the original expanded f-divergence, resulting in an equivalent yet tractable loss that effectively trains one-step diffusion models by minimizing the expanded f-divergence family. The novel unification introduced by Uni-Instruct not only offers new theoretical contributions that help understand existing approaches from a high-level perspective but also leads to state-of-the-art one-step diffusion generation performances. On the CIFAR10 generation benchmark, Uni-Instruct achieves record-breaking Frechet Inception Distance (FID) values of \emph{1.46} for unconditional generation and \emph{1.38} for conditional generation. On the ImageNet-64times 64 generation benchmark, Uni-Instruct achieves a new SoTA one-step generation FID of \emph{1.02}, which outperforms its 79-step teacher diffusion with a significant improvement margin of 1.33 (1.02 vs 2.35). We also apply Uni-Instruct on broader tasks like text-to-3D generation. For text-to-3D generation, Uni-Instruct gives decent results, which slightly outperforms previous methods, such as SDS and VSD, in terms of both generation quality and diversity. Both the solid theoretical and empirical contributions of Uni-Instruct will potentially help future studies on one-step diffusion distillation and knowledge transferring of diffusion models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20755</guid>
|
||||
<pubDate>Tue, 27 May 2025 05:55:45 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Efficient Data Selection at Scale via Influence Distillation</title>
|
||||
<link>https://arxiv.org/abs/2505.19051</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19051.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mahdi Nikdan, Vincent Cohen-Addad, Dan Alistarh, Vahab Mirrokni</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Effective data selection is critical for efficient training of modern Large Language Models (LLMs). This paper introduces Influence Distillation, a novel, mathematically-justified framework for data selection that employs second-order information to optimally weight training samples. By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data for LLM fine-tuning, guiding it toward strong performance on the target domain. We derive these optimal weights for both Gradient Descent and Adam optimizers. To ensure scalability and reduce computational cost, we propose a landmark-based approximation: influence is precisely computed for a small subset of "landmark" samples and then efficiently propagated to all other samples to determine their weights. We validate Influence Distillation by applying it to instruction tuning on the Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU, across several models from the Llama and Qwen families. Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to 3.5times faster selection.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19051</guid>
|
||||
<pubDate>Sun, 25 May 2025 09:08:00 +0000</pubDate>
|
||||
<title>Differentiable Solver Search for Fast Diffusion Sampling</title>
|
||||
<link>https://arxiv.org/abs/2505.21114</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21114.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shuai Wang, Zexian Li, Qipeng zhang, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Diffusion models have demonstrated remarkable generation quality but at the cost of numerous function evaluations. Recently, advanced ODE-based solvers have been developed to mitigate the substantial computational demands of reverse-diffusion solving under limited sampling steps. However, these solvers, heavily inspired by Adams-like multistep methods, rely solely on t-related Lagrange interpolation. We show that t-related Lagrange interpolation is suboptimal for diffusion model and reveal a compact search space comprised of time steps and solver coefficients. Building on our analysis, we propose a novel differentiable solver search algorithm to identify more optimal solver. Equipped with the searched solver, rectified-flow models, e.g., SiT-XL/2 and FlowDCN-XL/2, achieve FID scores of 2.40 and 2.35, respectively, on ImageNet256 with only 10 steps. Meanwhile, DDPM model, DiT-XL/2, reaches a FID score of 2.33 with only 10 steps. Notably, our searched solver outperforms traditional solvers by a significant margin. Moreover, our searched solver demonstrates generality across various model architectures, resolutions, and model sizes.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21114</guid>
|
||||
<pubDate>Tue, 27 May 2025 12:33:43 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs</title>
|
||||
<link>https://arxiv.org/abs/2505.19075</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19075.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jaemin Kim, Hangeol Chang, Hyunmin Hwang, Choonghan Kim, Jong Chul Ye</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Large Language Models (LLMs) have demonstrated remarkable general capabilities, but enhancing skills such as reasoning often demands substantial computational resources and may compromise their generalization. While Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious alternative, they typically requires retraining for each LLM backbone due to architectural dependencies. To address these challenges, here we propose Universal Reasoner (UniR) - a single, lightweight, composable, and plug-and-play reasoning module that can be used with any frozen LLM to endow it with specialized reasoning capabilities. Specifically, UniR decomposes the reward into a standalone reasoning module that is trained independently using predefined rewards, effectively translating trajectory-level signals into token-level guidance. Once trained, UniR can be combined with any frozen LLM at inference time by simply adding its output logits to those of the LLM backbone. This additive structure naturally enables modular composition: multiple UniR modules trained for different tasks can be jointly applied by summing their logits, enabling complex reasoning via composition. Experimental results on mathematical reasoning and machine translation tasks show that UniR significantly outperforms existing baseline fine-tuning methods using the Llama3.2 model. Furthermore, UniR demonstrates strong weak-to-strong generalization: reasoning modules trained on smaller models effectively guide much larger LLMs. This makes UniR a cost-efficient, adaptable, and robust solution for enhancing reasoning in LLMs without compromising their core capabilities. Code is open-sourced at https://github.com/hangeol/UniR</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19075</guid>
|
||||
<pubDate>Sun, 25 May 2025 10:19:10 +0000</pubDate>
|
||||
<title>Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation</title>
|
||||
<link>https://arxiv.org/abs/2505.21190</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21190.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jong Hak Moon, Geon Choi, Paloma Rabaey, Min Gwan Kim, Hyuk Gi Hong, Jung-Oh Lee, Hangyul Yoon, Eun Woo Doe, Jiyoun Kim, Harshita Sharma, Daniel C. Castro, Javier Alvarez-Valle, Edward Choi</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Radiology reports convey detailed clinical observations and capture diagnostic reasoning that evolves over time. However, existing evaluation methods are limited to single-report settings and rely on coarse metrics that fail to capture fine-grained clinical semantics and temporal dependencies. We introduce LUNGUAGE,a benchmark dataset for structured radiology report generation that supports both single-report evaluation and longitudinal patient-level assessment across multiple studies. It contains 1,473 annotated chest X-ray reports, each reviewed by experts, and 80 of them contain longitudinal annotations to capture disease progression and inter-study intervals, also reviewed by experts. Using this benchmark, we develop a two-stage framework that transforms generated reports into fine-grained, schema-aligned structured representations, enabling longitudinal interpretation. We also propose LUNGUAGESCORE, an interpretable metric that compares structured outputs at the entity, relation, and attribute level while modeling temporal consistency across patient timelines. These contributions establish the first benchmark dataset, structuring framework, and evaluation metric for sequential radiology reporting, with empirical results demonstrating that LUNGUAGESCORE effectively supports structured report evaluation. The code is available at: https://github.com/SuperSupermoon/Lunguage</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21190</guid>
|
||||
<pubDate>Tue, 27 May 2025 13:40:00 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>LIMOPro: Reasoning Refinement for Efficient and Effective Test-time Scaling</title>
|
||||
<link>https://arxiv.org/abs/2505.19187</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19187.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Xiao, Jiashuo Wang, Ruifeng Yuan, Chunpu Xu, Kaishuai Xu, Wenjie Li, Pengfei Liu</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Large language models (LLMs) have demonstrated remarkable reasoning capabilities through test-time scaling approaches, particularly when fine-tuned with chain-of-thought (CoT) data distilled from more powerful large reasoning models (LRMs). However, these reasoning chains often contain verbose elements that mirror human problem-solving, categorized as progressive reasoning (the essential solution development path) and functional elements (verification processes, alternative solution approaches, and error corrections). While progressive reasoning is crucial, the functional elements significantly increase computational demands during test-time inference. We introduce PIR (Perplexity-based Importance Refinement), a principled framework that quantitatively evaluates the importance of each reasoning step based on its impact on answer prediction confidence. PIR systematically identifies and selectively prunes only low-importance functional steps while preserving progressive reasoning components, creating optimized training data that maintains the integrity of the core solution path while reducing verbosity. Models fine-tuned on PIR-optimized data exhibit superior test-time scaling properties, generating more concise reasoning chains while achieving improved accuracy (+0.9\% to +6.6\%) with significantly reduced token usage (-3\% to -41\%) across challenging reasoning benchmarks (AIME, AMC, and GPQA Diamond). Our approach demonstrates strong generalizability across different model sizes, data sources, and token budgets, offering a practical solution for deploying reasoning-capable LLMs in scenarios where efficient test-time scaling, response time, and computational efficiency are valuable constraints.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19187</guid>
|
||||
<pubDate>Sun, 25 May 2025 15:17:57 +0000</pubDate>
|
||||
<title>Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation</title>
|
||||
<link>https://arxiv.org/abs/2505.21784</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21784.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Safety reasoning is a recent paradigm where LLMs reason over safety policies before generating responses, thereby mitigating limitations in existing safety measures such as over-refusal and jailbreak vulnerabilities. However, implementing this paradigm is challenging due to the resource-intensive process of creating high-quality policy-embedded chain-of-thought (CoT) datasets while ensuring reasoning remains accurate and free from hallucinations or policy conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation for Safety Reasoning, a novel data generation recipe that leverages multi-agent deliberation to iteratively expand reasoning on safety policies. A data refiner stage in AIDSAFE ensures high-quality outputs by eliminating repetitive, redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong foundation for supervised fine-tuning (SFT)-based safety training. Additionally, to address the need of preference data in alignment stages, such as DPO training, we introduce a supplemental recipe that uses belief augmentation to create distinct selected and rejected CoT samples. Our evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy adherence and reasoning quality. Consequently, we show that fine-tuning open-source LLMs on these CoTs can significantly improve safety generalization and jailbreak robustness while maintaining acceptable utility and over-refusal accuracy. AIDSAFE-generated CoT datasets can be found here: https://huggingface.co/datasets/AmazonScience/AIDSAFE</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21784</guid>
|
||||
<pubDate>Tue, 27 May 2025 21:34:40 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research</title>
|
||||
<link>https://arxiv.org/abs/2505.19253</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19253.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, Chenyan Xiong</p><p><b>Upvotes:</b> 21</p><p><b>Summary:</b> Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries. However, most existing frameworks rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges in addition to their cost. To address these limitations, we introduce DeepResearchGym, an open-source sandbox that combines a reproducible search API with a rigorous evaluation protocol for benchmarking deep research systems. The API indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is freely available for research use. To evaluate deep research systems' outputs, we extend the Researchy Questions benchmark with automatic metrics through LLM-as-a-judge assessments to measure alignment with users' information needs, retrieval faithfulness, and report quality. Experimental results show that systems integrated with DeepResearchGym achieve performance comparable to those using commercial APIs, with performance rankings remaining consistent across evaluation metrics. A human evaluation study further confirms that our automatic protocol aligns with human preferences, validating the framework's ability to help support controlled assessment of deep research systems. Our code and API documentation are available at https://www.deepresearchgym.ai.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19253</guid>
|
||||
<pubDate>Sun, 25 May 2025 18:16:13 +0000</pubDate>
|
||||
<title>SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model</title>
|
||||
<link>https://arxiv.org/abs/2505.22126</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22126.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifan Chang, Yukang Feng, Jianwen Sun, Jiaxin Ai, Chuanhao Li, S. Kevin Zhou, Kaipeng Zhang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent years have seen rapid advances in AI-driven image generation. Early diffusion models emphasized perceptual quality, while newer multimodal models like GPT-4o-image integrate high-level reasoning, improving semantic understanding and structural composition. Scientific illustration generation exemplifies this evolution: unlike general image synthesis, it demands accurate interpretation of technical content and transformation of abstract ideas into clear, standardized visuals. This task is significantly more knowledge-intensive and laborious, often requiring hours of manual work and specialized tools. Automating it in a controllable, intelligent manner would provide substantial practical value. Yet, no benchmark currently exists to evaluate AI on this front. To fill this gap, we introduce SridBench, the first benchmark for scientific figure generation. It comprises 1,120 instances curated from leading scientific papers across 13 natural and computer science disciplines, collected via human experts and MLLMs. Each sample is evaluated along six dimensions, including semantic fidelity and structural accuracy. Experimental results reveal that even top-tier models like GPT-4o-image lag behind human performance, with common issues in text/visual clarity and scientific correctness. These findings highlight the need for more advanced reasoning-driven visual generation capabilities.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22126</guid>
|
||||
<pubDate>Wed, 28 May 2025 08:51:01 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understanding</title>
|
||||
<link>https://arxiv.org/abs/2505.20298</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20298.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jeonghun Baek, Kazuki Egashira, Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Hikaru Ikuta, Kiyoharu Aizawa</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Manga, or Japanese comics, is a richly multimodal narrative form that blends images and text in complex ways. Teaching large multimodal models (LMMs) to understand such narratives at a human-like level could help manga creators reflect on and refine their stories. To this end, we introduce two benchmarks for multimodal manga understanding: MangaOCR, which targets in-page text recognition, and MangaVQA, a novel benchmark designed to evaluate contextual understanding through visual question answering. MangaVQA consists of 526 high-quality, manually constructed question-answer pairs, enabling reliable evaluation across diverse narrative and visual scenarios. Building on these benchmarks, we develop MangaLMM, a manga-specialized model finetuned from the open-source LMM Qwen2.5-VL to jointly handle both tasks. Through extensive experiments, including comparisons with proprietary models such as GPT-4o and Gemini 2.5, we assess how well LMMs understand manga. Our benchmark and model provide a comprehensive foundation for evaluating and advancing LMMs in the richly narrative domain of manga.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20298</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:59:59 +0000</pubDate>
|
||||
<title>Train Sparse Autoencoders Efficiently by Utilizing Features Correlation</title>
|
||||
<link>https://arxiv.org/abs/2505.22255</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22255.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Vadim Kurochkin, Yaroslav Aksenov, Daniil Laptev, Daniil Gavrilov, Nikita Balagansky</p><p><b>Upvotes:</b> 16</p><p><b>Summary:</b> Sparse Autoencoders (SAEs) have demonstrated significant promise in interpreting the hidden states of language models by decomposing them into interpretable latent directions. However, training SAEs at scale remains challenging, especially when large dictionary sizes are used. While decoders can leverage sparse-aware kernels for efficiency, encoders still require computationally intensive linear operations with large output dimensions. To address this, we propose KronSAE, a novel architecture that factorizes the latent representation via Kronecker product decomposition, drastically reducing memory and computational overhead. Furthermore, we introduce mAND, a differentiable activation function approximating the binary AND operation, which improves interpretability and performance in our factorized framework.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22255</guid>
|
||||
<pubDate>Wed, 28 May 2025 11:41:11 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents</title>
|
||||
<link>https://arxiv.org/abs/2505.20411</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20411.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, Boris Yangel</p><p><b>Upvotes:</b> 62</p><p><b>Summary:</b> LLM-based agents have shown promising capabilities in a growing range of software engineering (SWE) tasks. However, advancing this field faces two critical challenges. First, high-quality training data is scarce, especially data that reflects real-world SWE scenarios, where agents must interact with development environments, execute code and adapt behavior based on the outcomes of their actions. Existing datasets are either limited to one-shot code generation or comprise small, manually curated collections of interactive tasks, lacking both scale and diversity. Second, the lack of fresh interactive SWE tasks affects evaluation of rapidly improving models, as static benchmarks quickly become outdated due to contamination issues. To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale. Additionally, we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering. We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20411</guid>
|
||||
<pubDate>Mon, 26 May 2025 18:01:00 +0000</pubDate>
|
||||
<title>GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control</title>
|
||||
<link>https://arxiv.org/abs/2505.22421</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22421.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Anthony Chen, Wenzhao Zheng, Yida Wang, Xueyang Zhang, Kun Zhan, Peng Jia, Kurt Keutzer, Shanghang Zhang</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> Recent advancements in world models have revolutionized dynamic environment simulation, allowing systems to foresee future states and assess potential actions. In autonomous driving, these capabilities help vehicles anticipate the behavior of other road users, perform risk-aware planning, accelerate training in simulation, and adapt to novel scenarios, thereby enhancing safety and reliability. Current approaches exhibit deficiencies in maintaining robust 3D geometric consistency or accumulating artifacts during occlusion handling, both critical for reliable safety assessment in autonomous navigation tasks. To address this, we introduce GeoDrive, which explicitly integrates robust 3D geometry conditions into driving world models to enhance spatial understanding and action controllability. Specifically, we first extract a 3D representation from the input frame and then obtain its 2D rendering based on the user-specified ego-car trajectory. To enable dynamic modeling, we propose a dynamic editing module during training to enhance the renderings by editing the positions of the vehicles. Extensive experiments demonstrate that our method significantly outperforms existing models in both action accuracy and 3D spatial awareness, leading to more realistic, adaptable, and reliable scene modeling for safer autonomous driving. Additionally, our model can generalize to novel trajectories and offers interactive scene editing capabilities, such as object editing and object trajectory control.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22421</guid>
|
||||
<pubDate>Wed, 28 May 2025 14:46:51 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>HoPE: Hybrid of Position Embedding for Length Generalization in Vision-Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.20444</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20444.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haoran Li, Yingjie Qin, Baoyuan Ou, Lai Xu, Ruiwen Xu</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Vision-Language Models (VLMs) have made significant progress in multimodal tasks. However, their performance often deteriorates in long-context scenarios, particularly long videos. While Rotary Position Embedding (RoPE) has been widely adopted for length generalization in Large Language Models (LLMs), extending vanilla RoPE to capture the intricate spatial-temporal dependencies in videos remains an unsolved challenge. Existing methods typically allocate different frequencies within RoPE to encode 3D positional information. However, these allocation strategies mainly rely on heuristics, lacking in-depth theoretical analysis. In this paper, we first study how different allocation strategies impact the long-context capabilities of VLMs. Our analysis reveals that current multimodal RoPEs fail to reliably capture semantic similarities over extended contexts. To address this issue, we propose HoPE, a Hybrid of Position Embedding designed to improve the long-context capabilities of VLMs. HoPE introduces a hybrid frequency allocation strategy for reliable semantic modeling over arbitrarily long context, and a dynamic temporal scaling mechanism to facilitate robust learning and flexible inference across diverse context lengths. Extensive experiments across four video benchmarks on long video understanding and retrieval tasks demonstrate that HoPE consistently outperforms existing methods, confirming its effectiveness. Code is available at https://github.com/hrlics/HoPE.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20444</guid>
|
||||
<pubDate>Mon, 26 May 2025 18:37:40 +0000</pubDate>
|
||||
<title>Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding</title>
|
||||
<link>https://arxiv.org/abs/2505.22618</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22618.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, Enze Xie</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities. However, the practical inference speed of open-sourced Diffusion LLMs often lags behind autoregressive models due to the lack of Key-Value (KV) Cache and quality degradation when decoding multiple tokens simultaneously. To bridge this gap, we introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop. Additionally, we identify the root cause of generation quality degradation in parallel decoding as the disruption of token dependencies under the conditional independence assumption. To address this, we propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality. Experimental results on LLaDA and Dream models across multiple LLM benchmarks demonstrate up to 27.6times throughput improvement with minimal accuracy loss, closing the performance gap with autoregressive models and paving the way for practical deployment of Diffusion LLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22618</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:39:15 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Prot2Token: A Unified Framework for Protein Modeling via Next-Token Prediction</title>
|
||||
<link>https://arxiv.org/abs/2505.20589</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20589.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mahdi Pourmirzaei, Farzaneh Esmaili, Salhuldin Alqarghuli, Mohammadreza Pourmirzaei, Ye Han, Kai Chen, Mohsen Rezaei, Duolin Wang, Dong Xu</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> The diverse nature of protein prediction tasks has traditionally necessitated specialized models, hindering the development of broadly applicable and computationally efficient Protein Language Models (PLMs). In this work, we introduce Prot2Token, a unified framework that overcomes these challenges by converting a wide spectrum of protein-related predictions, from sequence-level properties and residue-specific attributes to complex inter-protein interactions, into a standardized next-token prediction format. At its core, Prot2Token employs an autoregressive decoder, conditioned on embeddings from pre-trained protein encoders and guided by learnable task tokens, to perform diverse predictions. This architecture uniquely facilitates multi-task learning, enabling a single model to master numerous tasks with improved efficiency. We present extensive experimental validation across a variety of benchmarks, demonstrating Prot2Tokens strong predictive power in different types of protein-prediction tasks. Key results include significant speedups (e.g., near 1000x over AlphaFold2 with MSA) and performance often matching or exceeding specialized approaches. Beyond that, we introduce an auxiliary self-supervised decoder pre-training approach to improve spatially sensitive task performance. Prot2Token thus offers a significant step towards a versatile, high-throughput paradigm for protein modeling, promising to accelerate biological discovery and the development of novel therapeutics. The code is available at https://github.com/mahdip72/prot2token .</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20589</guid>
|
||||
<pubDate>Mon, 26 May 2025 23:50:36 +0000</pubDate>
|
||||
<title>The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason</title>
|
||||
<link>https://arxiv.org/abs/2505.22653</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22653.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ang Lv, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Rui Yan</p><p><b>Upvotes:</b> 45</p><p><b>Summary:</b> Recent studies on post-training large language models (LLMs) for reasoning through reinforcement learning (RL) typically focus on tasks that can be accurately verified and rewarded, such as solving math problems. In contrast, our research investigates the impact of reward noise, a more practical consideration for real-world scenarios involving the post-training of LLMs using reward models. We found that LLMs demonstrate strong robustness to substantial reward noise. For example, manually flipping 40% of the reward function's outputs in math tasks still allows a Qwen-2.5-7B model to achieve rapid convergence, improving its performance on math tasks from 5% to 72%, compared to the 75% accuracy achieved by a model trained with noiseless rewards. Surprisingly, by only rewarding the appearance of key reasoning phrases (namely reasoning pattern reward, RPR), such as ``first, I need to''-without verifying the correctness of answers, the model achieved peak downstream performance (over 70% accuracy for Qwen-2.5-7B) comparable to models trained with strict correctness verification and accurate rewards. Recognizing the importance of the reasoning process over the final results, we combined RPR with noisy reward models. RPR helped calibrate the noisy reward models, mitigating potential false negatives and enhancing the LLM's performance on open-ended tasks. These findings suggest the importance of improving models' foundational abilities during the pre-training phase while providing insights for advancing post-training techniques. Our code and scripts are available at https://github.com/trestad/Noisy-Rewards-in-Learning-to-Reason.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22653</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:59:03 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding</title>
|
||||
<link>https://arxiv.org/abs/2505.20715</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20715.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fuwen Luo, Shengfeng Lou, Chi Chen, Ziyue Wang, Chenliang Li, Weizhou Shen, Jiyue Guo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Video temporal understanding is crucial for multimodal large language models (MLLMs) to reason over events in videos. Despite recent advances in general video understanding, current MLLMs still struggle with fine-grained temporal reasoning. While reinforcement learning (RL) has been explored to address this issue recently, existing RL approaches remain limited in effectiveness. In this work, we propose MUSEG, a novel RL-based method that enhances temporal understanding by introducing timestamp-aware multi-segment grounding. MUSEG enables MLLMs to align queries with multiple relevant video segments, promoting more comprehensive temporal reasoning. To facilitate effective learning, we design a customized RL training recipe with phased rewards that progressively guides the model toward temporally grounded reasoning. Extensive experiments on temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG significantly outperforms existing methods and generalizes well across diverse temporal understanding scenarios. View our project at https://github.com/THUNLP-MT/MUSEG.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20715</guid>
|
||||
<pubDate>Tue, 27 May 2025 04:50:07 +0000</pubDate>
|
||||
<title>FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian</title>
|
||||
<link>https://arxiv.org/abs/2505.22759</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22759.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sara Papi, Marco Gaido, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> The development of speech foundation models (SFMs) like Whisper and SeamlessM4T has significantly advanced the field of speech processing. However, their closed nature--with inaccessible training data and code--poses major reproducibility and fair evaluation challenges. While other domains have made substantial progress toward open science by developing fully transparent models trained on open-source (OS) code and data, similar efforts in speech remain limited. To fill this gap, we introduce FAMA, the first family of open science SFMs for English and Italian, trained on 150k+ hours of OS speech data. Moreover, we present a new dataset containing 16k hours of cleaned and pseudo-labeled speech for both languages. Results show that FAMA achieves competitive performance compared to existing SFMs while being up to 8 times faster. All artifacts, including code, datasets, and models, are released under OS-compliant licenses, promoting openness in speech technology research.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22759</guid>
|
||||
<pubDate>Wed, 28 May 2025 18:19:34 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>CHIMERA: A Knowledge Base of Idea Recombination in Scientific Literature</title>
|
||||
<link>https://arxiv.org/abs/2505.20779</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20779.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Noy Sternlicht, Tom Hope</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> A hallmark of human innovation is the process of recombination -- creating original ideas by integrating elements of existing mechanisms and concepts. In this work, we automatically mine the scientific literature and build CHIMERA: a large-scale knowledge base (KB) of recombination examples. CHIMERA can be used to empirically explore at scale how scientists recombine concepts and take inspiration from different areas, or to train supervised machine learning models that learn to predict new creative cross-domain directions. To build this KB, we present a novel information extraction task of extracting recombination from scientific paper abstracts, collect a high-quality corpus of hundreds of manually annotated abstracts, and use it to train an LLM-based extraction model. The model is applied to a large corpus of papers in the AI domain, yielding a KB of over 28K recombination examples. We analyze CHIMERA to explore the properties of recombination in different subareas of AI. Finally, we train a scientific hypothesis generation model using the KB, which predicts new recombination directions that real-world researchers find inspiring. Our data and code are available at https://github.cs.huji.ac.il/tomhope-lab/CHIMERA</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20779</guid>
|
||||
<pubDate>Tue, 27 May 2025 06:36:04 +0000</pubDate>
|
||||
<title>StressTest: Can YOUR Speech LM Handle the Stress?</title>
|
||||
<link>https://arxiv.org/abs/2505.22765</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22765.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Iddo Yosha, Gallil Maimon, Yossi Adi</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Sentence stress refers to emphasis, placed on specific words within a spoken utterance to highlight or contrast an idea, or to introduce new information. It is often used to imply an underlying intention that is not explicitly stated. Recent advances in speech-aware language models (SLMs) have enabled direct processing of audio, allowing models to bypass transcription and access the full richness of the speech signal and perform audio reasoning tasks such as spoken question answering. Despite the crucial role of sentence stress in shaping meaning and speaker intent, it remains largely overlooked in evaluation and development of such models. In this work, we address this gap by introducing StressTest, a benchmark specifically designed to evaluate a model's ability to distinguish between interpretations of spoken sentences based on the stress pattern. We assess the performance of several leading SLMs and find that, despite their overall capabilities, they perform poorly on such tasks. To overcome this limitation, we propose a novel synthetic data generation pipeline, and create Stress17k, a training set that simulates change of meaning implied by stress variation. Then, we empirically show that optimizing models with this synthetic dataset aligns well with real-world recordings and enables effective finetuning of SLMs. Results suggest, that our finetuned model, StresSLM, significantly outperforms existing models on both sentence stress reasoning and detection tasks. Code, models, data, and audio samples - pages.cs.huji.ac.il/adiyoss-lab/stresstest.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22765</guid>
|
||||
<pubDate>Wed, 28 May 2025 18:32:56 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>AITEE -- Agentic Tutor for Electrical Engineering</title>
|
||||
<link>https://arxiv.org/abs/2505.21582</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21582.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Christopher Knievel, Alexander Bernhardt, Christian Bernhardt</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Intelligent tutoring systems combined with large language models offer a promising approach to address students' diverse needs and promote self-efficacious learning. While large language models possess good foundational knowledge of electrical engineering basics, they remain insufficiently capable of addressing specific questions about electrical circuits. In this paper, we present AITEE, an agent-based tutoring system for electrical engineering designed to accompany students throughout their learning process, offer individualized support, and promote self-directed learning. AITEE supports both hand-drawn and digital circuits through an adapted circuit reconstruction process, enabling natural interaction with students. Our novel graph-based similarity measure identifies relevant context from lecture materials through a retrieval augmented generation approach, while parallel Spice simulation further enhances accuracy in applying solution methodologies. The system implements a Socratic dialogue to foster learner autonomy through guided questioning. Experimental evaluations demonstrate that AITEE significantly outperforms baseline approaches in domain-specific knowledge application, with even medium-sized LLM models showing acceptable performance. Our results highlight the potential of agentic tutors to deliver scalable, personalized, and effective learning environments for electrical engineering education.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21582</guid>
|
||||
<pubDate>Tue, 27 May 2025 10:07:05 +0000</pubDate>
|
||||
<title>VidText: Towards Comprehensive Evaluation for Video Text Understanding</title>
|
||||
<link>https://arxiv.org/abs/2505.22810</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22810.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhoufaran Yang, Yan Shu, Zhifei Yang, Yan Zhang, Yu Li, Keyang Lu, Gangyan Zeng, Shaohui Liu, Yu Zhou, Nicu Sebe</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Visual texts embedded in videos carry rich semantic information, which is crucial for both holistic video understanding and fine-grained reasoning about local human actions. However, existing video understanding benchmarks largely overlook textual information, while OCR-specific benchmarks are constrained to static images, limiting their ability to capture the interaction between text and dynamic visual contexts. To address this gap, we propose VidText, a new benchmark designed for comprehensive and in-depth evaluation of video text understanding. VidText offers the following key features: 1) It covers a wide range of real-world scenarios and supports multilingual content, encompassing diverse settings where video text naturally appears. 2) It introduces a hierarchical evaluation framework with video-level, clip-level, and instance-level tasks, enabling assessment of both global summarization and local retrieval capabilities. 3) The benchmark also introduces a set of paired perception reasoning tasks, ranging from visual text perception to cross-modal reasoning between textual and visual information. Extensive experiments on 18 state-of-the-art Large Multimodal Models (LMMs) reveal that current models struggle across most tasks, with significant room for improvement. Further analysis highlights the impact of both model-intrinsic factors, such as input resolution and OCR capability, and external factors, including the use of auxiliary information and Chain-of-Thought reasoning strategies. We hope VidText will fill the current gap in video understanding benchmarks and serve as a foundation for future research on multimodal reasoning with video text in dynamic environments.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22810</guid>
|
||||
<pubDate>Wed, 28 May 2025 19:39:35 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Styl3R: Instant 3D Stylized Reconstruction for Arbitrary Scenes and Styles</title>
|
||||
<link>https://arxiv.org/abs/2505.21060</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21060.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Peng Wang, Xiang Liu, Peidong Liu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Stylizing 3D scenes instantly while maintaining multi-view consistency and faithfully resembling a style image remains a significant challenge. Current state-of-the-art 3D stylization methods typically involve computationally intensive test-time optimization to transfer artistic features into a pretrained 3D representation, often requiring dense posed input images. In contrast, leveraging recent advances in feed-forward reconstruction models, we demonstrate a novel approach to achieve direct 3D stylization in less than a second using unposed sparse-view scene images and an arbitrary style image. To address the inherent decoupling between reconstruction and stylization, we introduce a branched architecture that separates structure modeling and appearance shading, effectively preventing stylistic transfer from distorting the underlying 3D scene structure. Furthermore, we adapt an identity loss to facilitate pre-training our stylization model through the novel view synthesis task. This strategy also allows our model to retain its original reconstruction capabilities while being fine-tuned for stylization. Comprehensive evaluations, using both in-domain and out-of-domain datasets, demonstrate that our approach produces high-quality stylized 3D content that achieve a superior blend of style and scene appearance, while also outperforming existing methods in terms of multi-view consistency and efficiency.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21060</guid>
|
||||
<pubDate>Tue, 27 May 2025 11:47:15 +0000</pubDate>
|
||||
<title>CLIPGaussian: Universal and Multimodal Style Transfer Based on Gaussian Splatting</title>
|
||||
<link>https://arxiv.org/abs/2505.22854</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22854.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kornel Howil, Joanna Waczyńska, Piotr Borycki, Tadeusz Dziarmaga, Marcin Mazur, Przemysław Spurek</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Gaussian Splatting (GS) has recently emerged as an efficient representation for rendering 3D scenes from 2D images and has been extended to images, videos, and dynamic 4D content. However, applying style transfer to GS-based representations, especially beyond simple color changes, remains challenging. In this work, we introduce CLIPGaussians, the first unified style transfer framework that supports text- and image-guided stylization across multiple modalities: 2D images, videos, 3D objects, and 4D scenes. Our method operates directly on Gaussian primitives and integrates into existing GS pipelines as a plug-in module, without requiring large generative models or retraining from scratch. CLIPGaussians approach enables joint optimization of color and geometry in 3D and 4D settings, and achieves temporal coherence in videos, while preserving a model size. We demonstrate superior style fidelity and consistency across all tasks, validating CLIPGaussians as a universal and efficient solution for multimodal style transfer.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22854</guid>
|
||||
<pubDate>Wed, 28 May 2025 20:41:24 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>SageAttention2++: A More Efficient Implementation of SageAttention2</title>
|
||||
<link>https://arxiv.org/abs/2505.21136</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21136.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jintao Zhang, Xiaoming Xu, Jia Wei, Haofeng Huang, Pengle Zhang, Chendong Xiang, Jun Zhu, Jianfei Chen</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> The efficiency of attention is critical because its time complexity grows quadratically with sequence length. SageAttention2 addresses this by utilizing quantization to accelerate matrix multiplications (Matmul) in attention. To further accelerate SageAttention2, we propose to utilize the faster instruction of FP8 Matmul accumulated in FP16. The instruction is 2x faster than the FP8 Matmul used in SageAttention2. Our experiments show that SageAttention2++ achieves a 3.9x speedup over FlashAttention while maintaining the same attention accuracy as SageAttention2. This means SageAttention2++ effectively accelerates various models, including those for language, image, and video generation, with negligible end-to-end metrics loss. The code will be available at https://github.com/thu-ml/SageAttention.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21136</guid>
|
||||
<pubDate>Tue, 27 May 2025 12:50:36 +0000</pubDate>
|
||||
<title>When Models Reason in Your Language: Controlling Thinking Trace Language Comes at the Cost of Accuracy</title>
|
||||
<link>https://arxiv.org/abs/2505.22888</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22888.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jirui Qi, Shan Chen, Zidi Xiong, Raquel Fernández, Danielle S. Bitterman, Arianna Bisazza</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Recent Large Reasoning Models (LRMs) with thinking traces have shown strong performance on English reasoning tasks. However, their ability to think in other languages is less studied. This capability is as important as answer accuracy for real world applications because users may find the reasoning trace useful for oversight only when it is expressed in their own language. We comprehensively evaluate two leading families of LRMs on our XReasoning benchmark and find that even the most advanced models often revert to English or produce fragmented reasoning in other languages, revealing a substantial gap in multilingual reasoning. Prompt based interventions that force models to reason in the users language improve readability and oversight but reduce answer accuracy, exposing an important trade off. We further show that targeted post training on just 100 examples mitigates this mismatch, though some accuracy loss remains. Our results highlight the limited multilingual reasoning capabilities of current LRMs and outline directions for future work. Code and data are available at https://github.com/Betswish/mCoT-XReasoning.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22888</guid>
|
||||
<pubDate>Wed, 28 May 2025 21:44:12 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities</title>
|
||||
<link>https://arxiv.org/abs/2505.21191</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21191.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junyan Zhang, Yubo Gao, Yibo Yan, Jungang Li, Zhaorui Hou, Sicheng Tao, Shuliang Liu, Song Dai, Yonghua Hei, Junzhuo Li, Xuming Hu</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> The finetuning of Large Language Models (LLMs) has significantly advanced their instruction-following capabilities, yet the underlying computational mechanisms driving these improvements remain poorly understood. This study systematically examines how fine-tuning reconfigures LLM computations by isolating and analyzing instruction-specific sparse components, i.e., neurons in dense models and both neurons and experts in Mixture-of-Experts (MoE) architectures. In particular, we introduce HexaInst, a carefully curated and balanced instructional dataset spanning six distinct categories, and propose SPARCOM, a novel analytical framework comprising three key contributions: (1) a method for identifying these sparse components, (2) an evaluation of their functional generality and uniqueness, and (3) a systematic comparison of their alterations. Through experiments, we demonstrate functional generality, uniqueness, and the critical role of these components in instruction execution. By elucidating the relationship between fine-tuning-induced adaptations and sparse computational substrates, this work provides deeper insights into how LLMs internalize instruction-following behavior for the trustworthy LLM community.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21191</guid>
|
||||
<pubDate>Tue, 27 May 2025 13:40:28 +0000</pubDate>
|
||||
<title>cadrille: Multi-modal CAD Reconstruction with Online Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.22914</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22914.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Maksim Kolodiazhnyi, Denis Tarasov, Dmitrii Zhemchuzhnikov, Alexander Nikulin, Ilya Zisman, Anna Vorontsova, Anton Konushin, Vladislav Kurenkov, Danila Rukhovich</p><p><b>Upvotes:</b> 19</p><p><b>Summary:</b> Computer-Aided Design (CAD) plays a central role in engineering and manufacturing, making it possible to create precise and editable 3D models. Using a variety of sensor or user-provided data as inputs for CAD reconstruction can democratize access to design applications. However, existing methods typically focus on a single input modality, such as point clouds, images, or text, which limits their generalizability and robustness. Leveraging recent advances in vision-language models (VLM), we propose a multi-modal CAD reconstruction model that simultaneously processes all three input modalities. Inspired by large language model (LLM) training paradigms, we adopt a two-stage pipeline: supervised fine-tuning (SFT) on large-scale procedurally generated data, followed by reinforcement learning (RL) fine-tuning using online feedback, obtained programatically. Furthermore, we are the first to explore RL fine-tuning of LLMs for CAD tasks demonstrating that online RL algorithms such as Group Relative Preference Optimization (GRPO) outperform offline alternatives. In the DeepCAD benchmark, our SFT model outperforms existing single-modal approaches in all three input modalities simultaneously. More importantly, after RL fine-tuning, cadrille sets new state-of-the-art on three challenging datasets, including a real-world one.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22914</guid>
|
||||
<pubDate>Wed, 28 May 2025 22:32:31 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing</title>
|
||||
<link>https://arxiv.org/abs/2505.21600</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21600.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tianyu Fu, Yi Ge, Yichen You, Enshu Liu, Zhihang Yuan, Guohao Dai, Shengen Yan, Huazhong Yang, Yu Wang</p><p><b>Upvotes:</b> 58</p><p><b>Summary:</b> Large Language Models (LLMs) achieve impressive reasoning capabilities at the cost of substantial inference overhead, posing substantial deployment challenges. Although distilled Small Language Models (SLMs) significantly enhance efficiency, their performance suffers as they fail to follow LLMs' reasoning paths. Luckily, we reveal that only a small fraction of tokens genuinely diverge reasoning paths between LLMs and SLMs. Most generated tokens are either identical or exhibit neutral differences, such as minor variations in abbreviations or expressions. Leveraging this insight, we introduce **Roads to Rome (R2R)**, a neural token routing method that selectively utilizes LLMs only for these critical, path-divergent tokens, while leaving the majority of token generation to the SLM. We also develop an automatic data generation pipeline that identifies divergent tokens and generates token-level routing labels to train the lightweight router. We apply R2R to combine R1-1.5B and R1-32B models from the DeepSeek family, and evaluate on challenging math, coding, and QA benchmarks. With an average activated parameter size of 5.6B, R2R surpasses the average accuracy of R1-7B by 1.6x, outperforming even the R1-14B model. Compared to R1-32B, it delivers a 2.8x wall-clock speedup with comparable performance, advancing the Pareto frontier of test-time scaling efficiency. Our code is available at https://github.com/thu-nics/R2R.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21600</guid>
|
||||
<pubDate>Tue, 27 May 2025 16:57:20 +0000</pubDate>
|
||||
<title>Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape</title>
|
||||
<link>https://arxiv.org/abs/2505.22918</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22918.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ruichen Chen, Keith G. Mills, Liyao Jiang, Chao Gao, Di Niu</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Diffusion Transformers (DiT) have become the de-facto model for generating high-quality visual content like videos and images. A huge bottleneck is the attention mechanism where complexity scales quadratically with resolution and video length. One logical way to lessen this burden is sparse attention, where only a subset of tokens or patches are included in the calculation. However, existing techniques fail to preserve visual quality at extremely high sparsity levels and might even incur non-negligible compute overheads. % To address this concern, we propose Re-ttention, which implements very high sparse attention for visual generation models by leveraging the temporal redundancy of Diffusion Models to overcome the probabilistic normalization shift within the attention mechanism. Specifically, Re-ttention reshapes attention scores based on the prior softmax distribution history in order to preserve the visual quality of the full quadratic attention at very high sparsity levels. % Experimental results on T2V/T2I models such as CogVideoX and the PixArt DiTs demonstrate that Re-ttention requires as few as 3.1\% of the tokens during inference, outperforming contemporary methods like FastDiTAttn, Sparse VideoGen and MInference. Further, we measure latency to show that our method can attain over 45\% end-to-end % and over 92\% self-attention latency reduction on an H100 GPU at negligible overhead cost. Code available online here: https://github.com/cccrrrccc/Re-ttention{https://github.com/cccrrrccc/Re-ttention}</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22918</guid>
|
||||
<pubDate>Wed, 28 May 2025 22:39:12 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Right Side Up? Disentangling Orientation Understanding in MLLMs with Fine-grained Multi-axis Perception Tasks</title>
|
||||
<link>https://arxiv.org/abs/2505.21649</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21649.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Keanu Nichols, Nazia Tasnim, Yan Yuting, Nicholas Ikechukwu, Elva Zou, Deepti Ghadiyaram, Bryan Plummer</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Object orientation understanding represents a fundamental challenge in visual perception critical for applications like robotic manipulation and augmented reality. Current vision-language benchmarks fail to isolate this capability, often conflating it with positional relationships and general scene understanding. We introduce DORI (Discriminative Orientation Reasoning Intelligence), a comprehensive benchmark establishing object orientation perception as a primary evaluation target. DORI assesses four dimensions of orientation comprehension: frontal alignment, rotational transformations, relative directional relationships, and canonical orientation understanding. Through carefully curated tasks from 11 datasets spanning 67 object categories across synthetic and real-world scenarios, DORI provides insights on how multi-modal systems understand object orientations. Our evaluation of 15 state-of-the-art vision-language models reveals critical limitations: even the best models achieve only 54.2% accuracy on coarse tasks and 33.0% on granular orientation judgments, with performance deteriorating for tasks requiring reference frame shifts or compound rotations. These findings demonstrate the need for dedicated orientation representation mechanisms, as models show systematic inability to perform precise angular estimations, track orientation changes across viewpoints, and understand compound rotations - suggesting limitations in their internal 3D spatial representations. As the first diagnostic framework specifically designed for orientation awareness in multimodal systems, DORI offers implications for improving robotic control, 3D scene reconstruction, and human-AI interaction in physical environments. DORI data: https://huggingface.co/datasets/appledora/DORI-Benchmark</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21649</guid>
|
||||
<pubDate>Tue, 27 May 2025 18:22:44 +0000</pubDate>
|
||||
<title>Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates</title>
|
||||
<link>https://arxiv.org/abs/2505.22943</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22943.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jaewoo Ahn, Heeseung Yun, Dayoon Ko, Gunhee Kim</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> While pre-trained multimodal representations (e.g., CLIP) have shown impressive capabilities, they exhibit significant compositional vulnerabilities leading to counterintuitive judgments. We introduce Multimodal Adversarial Compositionality (MAC), a benchmark that leverages large language models (LLMs) to generate deceptive text samples to exploit these vulnerabilities across different modalities and evaluates them through both sample-wise attack success rate and group-wise entropy-based diversity. To improve zero-shot methods, we propose a self-training approach that leverages rejection-sampling fine-tuning with diversity-promoting filtering, which enhances both attack success rate and sample diversity. Using smaller language models like Llama-3.1-8B, our approach demonstrates superior performance in revealing compositional vulnerabilities across various multimodal representations, including images, videos, and audios.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22943</guid>
|
||||
<pubDate>Wed, 28 May 2025 23:45:55 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Towards Scalable Language-Image Pre-training for 3D Medical Imaging</title>
|
||||
<link>https://arxiv.org/abs/2505.21862</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21862.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chenhui Zhao, Yiwei Lyu, Asadur Chowdury, Edward Harake, Akhil Kondepudi, Akshay Rao, Xinhai Hou, Honglak Lee, Todd Hollon</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Language-image pre-training has demonstrated strong performance in 2D medical imaging, but its success in 3D modalities such as CT and MRI remains limited due to the high computational demands of volumetric data, which pose a significant barrier to training on large-scale, uncurated clinical studies. In this study, we introduce Hierarchical attention for Language-Image Pre-training (HLIP), a scalable pre-training framework for 3D medical imaging. HLIP adopts a lightweight hierarchical attention mechanism inspired by the natural hierarchy of radiology data: slice, scan, and study. This mechanism exhibits strong generalizability, e.g., +4.3% macro AUC on the Rad-ChestCT benchmark when pre-trained on CT-RATE. Moreover, the computational efficiency of HLIP enables direct training on uncurated datasets. Trained on 220K patients with 3.13 million scans for brain MRI and 240K patients with 1.44 million scans for head CT, HLIP achieves state-of-the-art performance, e.g., +32.4% balanced ACC on the proposed publicly available brain MRI benchmark Pub-Brain-5; +1.4% and +6.9% macro AUC on head CT benchmarks RSNA and CQ500, respectively. These results demonstrate that, with HLIP, directly pre-training on uncurated clinical datasets is a scalable and effective direction for language-image pre-training in 3D medical imaging. The code is available at https://github.com/Zch0414/hlip</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21862</guid>
|
||||
<pubDate>Wed, 28 May 2025 01:16:34 +0000</pubDate>
|
||||
<title>ATI: Any Trajectory Instruction for Controllable Video Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.22944</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22944.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Angtian Wang, Haibin Huang, Jacob Zhiyuan Fang, Yiding Yang, Chongyang Ma</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> We propose a unified framework for motion control in video generation that seamlessly integrates camera movement, object-level translation, and fine-grained local motion using trajectory-based inputs. In contrast to prior methods that address these motion types through separate modules or task-specific designs, our approach offers a cohesive solution by projecting user-defined trajectories into the latent space of pre-trained image-to-video generation models via a lightweight motion injector. Users can specify keypoints and their motion paths to control localized deformations, entire object motion, virtual camera dynamics, or combinations of these. The injected trajectory signals guide the generative process to produce temporally consistent and semantically aligned motion sequences. Our framework demonstrates superior performance across multiple video motion control tasks, including stylized motion effects (e.g., motion brushes), dynamic viewpoint changes, and precise local motion manipulation. Experiments show that our method provides significantly better controllability and visual quality compared to prior approaches and commercial solutions, while remaining broadly compatible with various state-of-the-art video generation backbones. Project page: https://anytraj.github.io/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22944</guid>
|
||||
<pubDate>Wed, 28 May 2025 23:49:18 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance</title>
|
||||
<link>https://arxiv.org/abs/2505.21876</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21876.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zun Wang, Jaemin Cho, Jialu Li, Han Lin, Jaehong Yoon, Yue Zhang, Mohit Bansal</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Recent approaches on 3D camera control in video diffusion models (VDMs) often create anchor videos to guide diffusion models as a structured prior by rendering from estimated point clouds following annotated camera trajectories. However, errors inherent in point cloud estimation often lead to inaccurate anchor videos. Moreover, the requirement for extensive camera trajectory annotations further increases resource demands. To address these limitations, we introduce EPiC, an efficient and precise camera control learning framework that automatically constructs high-quality anchor videos without expensive camera trajectory annotations. Concretely, we create highly precise anchor videos for training by masking source videos based on first-frame visibility. This approach ensures high alignment, eliminates the need for camera trajectory annotations, and thus can be readily applied to any in-the-wild video to generate image-to-video (I2V) training pairs. Furthermore, we introduce Anchor-ControlNet, a lightweight conditioning module that integrates anchor video guidance in visible regions to pretrained VDMs, with less than 1% of backbone model parameters. By combining the proposed anchor video data and ControlNet module, EPiC achieves efficient training with substantially fewer parameters, training steps, and less data, without requiring modifications to the diffusion model backbone typically needed to mitigate rendering misalignments. Although being trained on masking-based anchor videos, our method generalizes robustly to anchor videos made with point clouds during inference, enabling precise 3D-informed camera control. EPiC achieves SOTA performance on RealEstate10K and MiraData for I2V camera control task, demonstrating precise and robust camera control ability both quantitatively and qualitatively. Notably, EPiC also exhibits strong zero-shot generalization to video-to-video scenarios.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21876</guid>
|
||||
<pubDate>Wed, 28 May 2025 01:45:26 +0000</pubDate>
|
||||
<title>ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mind</title>
|
||||
<link>https://arxiv.org/abs/2505.22961</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22961.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Peixuan Han, Zijia Liu, Jiaxuan You</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Large language models (LLMs) have shown promising potential in persuasion, but existing works on training LLM persuaders are still preliminary. Notably, while humans are skilled in modeling their opponent's thoughts and opinions proactively and dynamically, current LLMs struggle with such Theory of Mind (ToM) reasoning, resulting in limited diversity and opponent awareness. To address this limitation, we introduce Theory of Mind Augmented Persuader (ToMAP), a novel approach for building more flexible persuader agents by incorporating two theory of mind modules that enhance the persuader's awareness and analysis of the opponent's mental state. Specifically, we begin by prompting the persuader to consider possible objections to the target central claim, and then use a text encoder paired with a trained MLP classifier to predict the opponent's current stance on these counterclaims. Our carefully designed reinforcement learning schema enables the persuader learns how to analyze opponent-related information and utilize it to generate more effective arguments. Experiments show that the ToMAP persuader, while containing only 3B parameters, outperforms much larger baselines, like GPT-4o, with a relative gain of 39.4% across multiple persuadee models and diverse corpora. Notably, ToMAP exhibits complex reasoning chains and reduced repetition during training, which leads to more diverse and effective arguments. The opponent-aware feature of ToMAP also makes it suitable for long conversations and enables it to employ more logical and opponent-aware strategies. These results underscore our method's effectiveness and highlight its potential for developing more persuasive language agents. Code is available at: https://github.com/ulab-uiuc/ToMAP.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22961</guid>
|
||||
<pubDate>Thu, 29 May 2025 01:03:41 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>SVRPBench: A Realistic Benchmark for Stochastic Vehicle Routing Problem</title>
|
||||
<link>https://arxiv.org/abs/2505.21887</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21887.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ahmed Heakl, Yahia Salaheldin Shaaban, Martin Takac, Salem Lahlou, Zangir Iklassov</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Robust routing under uncertainty is central to real-world logistics, yet most benchmarks assume static, idealized settings. We present SVRPBench, the first open benchmark to capture high-fidelity stochastic dynamics in vehicle routing at urban scale. Spanning more than 500 instances with up to 1000 customers, it simulates realistic delivery conditions: time-dependent congestion, log-normal delays, probabilistic accidents, and empirically grounded time windows for residential and commercial clients. Our pipeline generates diverse, constraint-rich scenarios, including multi-depot and multi-vehicle setups. Benchmarking reveals that state-of-the-art RL solvers like POMO and AM degrade by over 20% under distributional shift, while classical and metaheuristic methods remain robust. To enable reproducible research, we release the dataset and evaluation suite. SVRPBench challenges the community to design solvers that generalize beyond synthetic assumptions and adapt to real-world uncertainty.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21887</guid>
|
||||
<pubDate>Wed, 28 May 2025 02:03:31 +0000</pubDate>
|
||||
<title>Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement</title>
|
||||
<link>https://arxiv.org/abs/2505.23183</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23183.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Gabriele Sarti, Vilém Zouhar, Malvina Nissim, Arianna Bisazza</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Word-level quality estimation (WQE) aims to automatically identify fine-grained error spans in machine-translated outputs and has found many uses, including assisting translators during post-editing. Modern WQE techniques are often expensive, involving prompting of large language models or ad-hoc training on large amounts of human-labeled data. In this work, we investigate efficient alternatives exploiting recent advances in language model interpretability and uncertainty quantification to identify translation errors from the inner workings of translation models. In our evaluation spanning 14 metrics across 12 translation directions, we quantify the impact of human label variation on metric performance by using multiple sets of human labels. Our results highlight the untapped potential of unsupervised metrics, the shortcomings of supervised methods when faced with label uncertainty, and the brittleness of single-annotator evaluation practices.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23183</guid>
|
||||
<pubDate>Thu, 29 May 2025 07:20:36 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination</title>
|
||||
<link>https://arxiv.org/abs/2505.21925</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21925.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chong Zeng, Yue Dong, Pieter Peers, Hongzhi Wu, Xin Tong</p><p><b>Upvotes:</b> 23</p><p><b>Summary:</b> We present RenderFormer, a neural rendering pipeline that directly renders an image from a triangle-based representation of a scene with full global illumination effects and that does not require per-scene training or fine-tuning. Instead of taking a physics-centric approach to rendering, we formulate rendering as a sequence-to-sequence transformation where a sequence of tokens representing triangles with reflectance properties is converted to a sequence of output tokens representing small patches of pixels. RenderFormer follows a two stage pipeline: a view-independent stage that models triangle-to-triangle light transport, and a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values guided by the triangle-sequence from the view-independent stage. Both stages are based on the transformer architecture and are learned with minimal prior constraints. We demonstrate and evaluate RenderFormer on scenes with varying complexity in shape and light transport.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21925</guid>
|
||||
<pubDate>Wed, 28 May 2025 03:20:46 +0000</pubDate>
|
||||
<title>UniTEX: Universal High Fidelity Generative Texturing for 3D Shapes</title>
|
||||
<link>https://arxiv.org/abs/2505.23253</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23253.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yixun Liang, Kunming Luo, Xiao Chen, Rui Chen, Hongyu Yan, Weiyu Li, Jiarui Liu, Ping Tan</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> We present UniTEX, a novel two-stage 3D texture generation framework to create high-quality, consistent textures for 3D assets. Existing approaches predominantly rely on UV-based inpainting to refine textures after reprojecting the generated multi-view images onto the 3D shapes, which introduces challenges related to topological ambiguity. To address this, we propose to bypass the limitations of UV mapping by operating directly in a unified 3D functional space. Specifically, we first propose that lifts texture generation into 3D space via Texture Functions (TFs)--a continuous, volumetric representation that maps any 3D point to a texture value based solely on surface proximity, independent of mesh topology. Then, we propose to predict these TFs directly from images and geometry inputs using a transformer-based Large Texturing Model (LTM). To further enhance texture quality and leverage powerful 2D priors, we develop an advanced LoRA-based strategy for efficiently adapting large-scale Diffusion Transformers (DiTs) for high-quality multi-view texture synthesis as our first stage. Extensive experiments demonstrate that UniTEX achieves superior visual quality and texture integrity compared to existing approaches, offering a generalizable and scalable solution for automated 3D texture generation. Code will available in: https://github.com/YixunLiang/UniTEX.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23253</guid>
|
||||
<pubDate>Thu, 29 May 2025 08:58:41 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models</title>
|
||||
<link>https://arxiv.org/abs/2505.21960</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21960.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Senmao Li, Lei Wang, Kai Wang, Tao Liu, Jiehang Xie, Joost van de Weijer, Fahad Shahbaz Khan, Shiqi Yang, Yaxing Wang, Jian Yang</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Text-to-Image (T2I) diffusion models have made remarkable advancements in generative modeling; however, they face a trade-off between inference speed and image quality, posing challenges for efficient deployment. Existing distilled T2I models can generate high-fidelity images with fewer sampling steps, but often struggle with diversity and quality, especially in one-step models. From our analysis, we observe redundant computations in the UNet encoders. Our findings suggest that, for T2I diffusion models, decoders are more adept at capturing richer and more explicit semantic information, while encoders can be effectively shared across decoders from diverse time steps. Based on these observations, we introduce the first Time-independent Unified Encoder TiUE for the student model UNet architecture, which is a loop-free image generation approach for distilling T2I diffusion models. Using a one-pass scheme, TiUE shares encoder features across multiple decoder time steps, enabling parallel sampling and significantly reducing inference time complexity. In addition, we incorporate a KL divergence term to regularize noise prediction, which enhances the perceptual realism and diversity of the generated images. Experimental results demonstrate that TiUE outperforms state-of-the-art methods, including LCM, SD-Turbo, and SwiftBrushv2, producing more diverse and realistic results while maintaining the computational efficiency.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21960</guid>
|
||||
<pubDate>Wed, 28 May 2025 04:23:22 +0000</pubDate>
|
||||
<title>VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?</title>
|
||||
<link>https://arxiv.org/abs/2505.23359</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23359.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuanxin Liu, Kun Ouyang, Haoning Wu, Yi Liu, Lin Sui, Xinhao Li, Yan Zhong, Y. Charles, Xinyu Zhou, Xu Sun</p><p><b>Upvotes:</b> 27</p><p><b>Summary:</b> Recent studies have shown that long chain-of-thought (CoT) reasoning can significantly enhance the performance of large language models (LLMs) on complex tasks. However, this benefit is yet to be demonstrated in the domain of video understanding, since most existing benchmarks lack the reasoning depth required to demonstrate the advantages of extended CoT chains. While recent efforts have proposed benchmarks aimed at video reasoning, the tasks are often knowledge-driven and do not rely heavily on visual content. To bridge this gap, we introduce VideoReasonBench, a benchmark designed to evaluate vision-centric, complex video reasoning. To ensure visual richness and high reasoning complexity, each video in VideoReasonBench depicts a sequence of fine-grained operations on a latent state that is only visible in part of the video. The questions evaluate three escalating levels of video reasoning skills: recalling observed visual information, inferring the content of latent states, and predicting information beyond the video. Under such task setting, models have to precisely recall multiple operations in the video, and perform step-by-step reasoning to get correct final answers for these questions. Using VideoReasonBench, we comprehensively evaluate 18 state-of-the-art multimodal LLMs (MLLMs), finding that most perform poorly on complex video reasoning, e.g., GPT-4o achieves only 6.9% accuracy, while the thinking-enhanced Gemini-2.5-Pro significantly outperforms others with 56.0% accuracy. Our investigations on "test-time scaling" further reveal that extended thinking budget, while offering none or minimal benefits on existing video benchmarks, is essential for improving the performance on VideoReasonBench.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23359</guid>
|
||||
<pubDate>Thu, 29 May 2025 11:33:43 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.22019</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22019.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, Feng Zhao</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Effectively retrieving, reasoning and understanding visually rich information remains a challenge for RAG methods. Traditional text-based methods cannot handle visual-related information. On the other hand, current vision-based RAG approaches are often limited by fixed pipelines and frequently struggle to reason effectively due to the insufficient activation of the fundamental capabilities of models. As RL has been proven to be beneficial for model reasoning, we introduce VRAG-RL, a novel RL framework tailored for complex reasoning across visually rich information. With this framework, VLMs interact with search engines, autonomously sampling single-turn or multi-turn reasoning trajectories with the help of visual perception tokens and undergoing continual optimization based on these samples. Our approach highlights key limitations of RL in RAG domains: (i) Prior Multi-modal RAG approaches tend to merely incorporate images into the context, leading to insufficient reasoning token allocation and neglecting visual-specific perception; and (ii) When models interact with search engines, their queries often fail to retrieve relevant information due to the inability to articulate requirements, thereby leading to suboptimal performance. To address these challenges, we define an action space tailored for visually rich inputs, with actions including cropping and scaling, allowing the model to gather information from a coarse-to-fine perspective. Furthermore, to bridge the gap between users' original inquiries and the retriever, we employ a simple yet effective reward that integrates query rewriting and retrieval performance with a model-based reward. Our VRAG-RL optimizes VLMs for RAG tasks using specially designed RL strategies, aligning the model with real-world applications. The code is available at https://github.com/Alibaba-NLP/VRAG{https://github.com/Alibaba-NLP/VRAG}.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22019</guid>
|
||||
<pubDate>Wed, 28 May 2025 06:30:51 +0000</pubDate>
|
||||
<title>UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.23380</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23380.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Weijia Mao, Zhenheng Yang, Mike Zheng Shou</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Unified multimodal large language models such as Show-o and Janus have achieved strong performance across both generation and understanding tasks. However, these models typically rely on large-scale datasets and require substantial computation during the pretraining stage. In addition, several post-training methods have been proposed, but they often depend on external data or are limited to task-specific customization. In this work, we introduce UniRL, a self-improving post-training approach. Our approach enables the model to generate images from prompts and use them as training data in each iteration, without relying on any external image data. Moreover, it enables the two tasks to enhance each other: the generated images are used for understanding, and the understanding results are used to supervise generation. We explore supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to optimize the models. UniRL offers three key advantages: (1) it requires no external image data, as all training samples are generated by the model itself during training; (2) it not only improves individual task performance, but also reduces the imbalance between generation and understanding; and (3) it requires only several additional training steps during the post-training stage. We evaluate UniRL on top of Show-o and Janus, achieving a GenEval score of 0.77 for Show-o and 0.65 for Janus. Code and models will be released in https://github.com/showlab/UniRL.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23380</guid>
|
||||
<pubDate>Thu, 29 May 2025 12:00:15 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?</title>
|
||||
<link>https://arxiv.org/abs/2505.22129</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22129.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jinhong Ni, Chang-Bin Zhang, Qiang Zhang, Jing Zhang</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Recent prosperity of text-to-image diffusion models, e.g. Stable Diffusion, has stimulated research to adapt them to 360-degree panorama generation. Prior work has demonstrated the feasibility of using conventional low-rank adaptation techniques on pre-trained diffusion models to generate panoramic images. However, the substantial domain gap between perspective and panoramic images raises questions about the underlying mechanisms enabling this empirical success. We hypothesize and examine that the trainable counterparts exhibit distinct behaviors when fine-tuned on panoramic data, and such an adaptation conceals some intrinsic mechanism to leverage the prior knowledge within the pre-trained diffusion models. Our analysis reveals the following: 1) the query and key matrices in the attention modules are responsible for common information that can be shared between the panoramic and perspective domains, thus are less relevant to panorama generation; and 2) the value and output weight matrices specialize in adapting pre-trained knowledge to the panoramic domain, playing a more critical role during fine-tuning for panorama generation. We empirically verify these insights by introducing a simple framework called UniPano, with the objective of establishing an elegant baseline for future research. UniPano not only outperforms existing methods but also significantly reduces memory usage and training time compared to prior dual-branch approaches, making it scalable for end-to-end panorama generation with higher resolution. The code will be released.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22129</guid>
|
||||
<pubDate>Wed, 28 May 2025 08:54:04 +0000</pubDate>
|
||||
<title>Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization</title>
|
||||
<link>https://arxiv.org/abs/2505.23387</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23387.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mingzhe Du, Luu Tuan Tuan, Yue Liu, Yuhao Qing, Dong Huang, Xinyi He, Qian Liu, Zejun Ma, See-kiong Ng</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Large Language Models (LLMs) generate functionally correct solutions but often fall short in code efficiency, a critical bottleneck for real-world deployment. In this paper, we introduce a novel test-time iterative optimization framework to address this, employing a closed-loop system where LLMs iteratively refine code based on empirical performance feedback from an execution sandbox. We explore three training strategies: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization~(GRPO). Experiments on our Venus dataset and the APPS benchmark show that SFT and DPO rapidly saturate in efficiency gains. In contrast, GRPO, using reinforcement learning (RL) with execution feedback, continuously optimizes code performance, significantly boosting both pass@1 (from 47% to 62%) and the likelihood of outperforming human submissions in efficiency (from 31% to 45%). Our work demonstrates effective test-time code efficiency improvement and critically reveals the power of RL in teaching LLMs to truly self-improve code efficiency.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23387</guid>
|
||||
<pubDate>Thu, 29 May 2025 12:14:29 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Let's Predict Sentence by Sentence</title>
|
||||
<link>https://arxiv.org/abs/2505.22202</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22202.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hyeonbin Hwang, Byeongguk Jeon, Seungone Kim, Jiyeon Kim, Hoyeon Chang, Sohee Yang, Seungpil Won, Dohaeng Lee, Youbin Ahn, Minjoon Seo</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Autoregressive language models (LMs) generate one token at a time, yet human reasoning operates over higher-level abstractions - sentences, propositions, and concepts. This contrast raises a central question- Can LMs likewise learn to reason over structured semantic units rather than raw token sequences? In this work, we investigate whether pretrained LMs can be lifted into such abstract reasoning spaces by building on their learned representations. We present a framework that adapts a pretrained token-level LM to operate in sentence space by autoregressively predicting continuous embeddings of next sentences. We explore two embedding paradigms inspired by classical representation learning: 1) semantic embeddings, learned via autoencoding to preserve surface meaning; and 2) contextual embeddings, trained via next-sentence prediction to encode anticipatory structure. We evaluate both under two inference regimes: Discretized, which decodes each predicted embedding into text before re-encoding; and Continuous, which reasons entirely in embedding space for improved efficiency. Across four domains - mathematics, logic, commonsense, and planning - contextual embeddings under continuous inference show competitive performance with Chain-of-Thought (CoT) while reducing inference-time FLOPs on average by half. We also present early signs of scalability and modular adaptation. Finally, to visualize latent trajectories, we introduce SentenceLens, a diagnostic tool that decodes intermediate model states into interpretable sentences. Together, our results indicate that pretrained LMs can effectively transition to abstract, structured reasoning within latent embedding spaces.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22202</guid>
|
||||
<pubDate>Wed, 28 May 2025 10:28:35 +0000</pubDate>
|
||||
<title>KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction</title>
|
||||
<link>https://arxiv.org/abs/2505.23416</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23416.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, Hyun Oh Song</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Transformer-based large language models (LLMs) cache context as key-value (KV) pairs during inference. As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency. This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries. KVzip quantifies the importance of a KV pair using the underlying LLM to reconstruct original contexts from cached KV pairs, subsequently evicting pairs with lower importance. Extensive empirical evaluations demonstrate that KVzip reduces KV cache size by 3-4times and FlashAttention decoding latency by approximately 2times, with negligible performance loss in question-answering, retrieval, reasoning, and code comprehension tasks. Evaluations include various models such as LLaMA3.1-8B, Qwen2.5-14B, and Gemma3-12B, with context lengths reaching up to 170K tokens. KVzip significantly outperforms existing query-aware KV eviction methods, which suffer from performance degradation even at a 90% cache budget ratio under multi-query scenarios.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23416</guid>
|
||||
<pubDate>Thu, 29 May 2025 13:05:47 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Pitfalls of Rule- and Model-based Verifiers -- A Case Study on Mathematical Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.22203</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22203.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuzhen Huang, Weihao Zeng, Xingshan Zeng, Qi Zhu, Junxian He</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Trustworthy verifiers are essential for the success of reinforcement learning with verifiable reward (RLVR), which is the core methodology behind various large reasoning models such as DeepSeek-R1. In complex domains like mathematical reasoning, rule-based verifiers have been widely adopted in previous works to train strong reasoning models. However, the reliability of these verifiers and their impact on the RL training process remain poorly understood. In this work, we take mathematical reasoning as a case study and conduct a comprehensive analysis of various verifiers in both static evaluation and RL training scenarios. First, we find that current open-source rule-based verifiers often fail to recognize equivalent answers presented in different formats across multiple commonly used mathematical datasets, resulting in non-negligible false negative rates. This limitation adversely affects RL training performance and becomes more pronounced as the policy model gets stronger. Subsequently, we investigate model-based verifiers as a potential solution to address these limitations. While the static evaluation shows that model-based verifiers achieve significantly higher verification accuracy, further analysis and RL training results imply that they are highly susceptible to hacking, where they misclassify certain patterns in responses as correct (i.e., false positives). This vulnerability is exploited during policy model optimization, leading to artificially inflated rewards. Our findings underscore the unique risks inherent to both rule-based and model-based verifiers, aiming to offer valuable insights to develop more robust reward systems in reinforcement learning.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22203</guid>
|
||||
<pubDate>Wed, 28 May 2025 10:28:41 +0000</pubDate>
|
||||
<title>SWE-bench Goes Live!</title>
|
||||
<link>https://arxiv.org/abs/2505.23419</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23419.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs). While SWE-bench and its variants have become standard in this domain, they suffer from key limitations: they have not been updated since their initial releases, cover a narrow set of repositories, and depend heavily on manual effort for instance construction and environment setup. These factors hinder scalability and introduce risks of overfitting and data contamination. In this work, we present SWE-bench-Live, a live-updatable benchmark designed to overcome these challenges. Our initial release consists of 1,319 tasks derived from real GitHub issues created since 2024, spanning 93 repositories. Each task is accompanied by a dedicated Docker image to ensure reproducible execution. Central to our benchmark is \method, an automated curation pipeline that streamlines the entire process from instance creation to environment setup, removing manual bottlenecks and enabling scalability and continuous updates. We evaluate a range of state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a substantial performance gap compared to static benchmarks like SWE-bench, even under controlled evaluation conditions. To better understand this discrepancy, we perform detailed analyses across repository origin, issue recency, and task difficulty. By providing a fresh, diverse, and executable benchmark grounded in live repository activity, SWE-bench-Live facilitates rigorous, contamination-resistant evaluation of LLMs and agents in dynamic, real-world software development settings.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23419</guid>
|
||||
<pubDate>Thu, 29 May 2025 13:09:44 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.22232</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22232.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Köhler, Patrick Schramowski, Michael Fromm, Kristian Kersting</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets remains limited. Existing state-of-the-art datasets mostly rely on heuristic filtering methods, restricting both their cross-lingual transferability and scalability. Here, we introduce JQL, a systematic approach that efficiently curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. JQL distills LLMs' annotation capabilities into lightweight annotators based on pretrained multilingual embeddings. These models exhibit robust multilingual and cross-lingual performance, even for languages and scripts unseen during training. Evaluated empirically across 35 languages, the resulting annotation pipeline substantially outperforms current heuristic filtering methods like Fineweb2. JQL notably enhances downstream model training quality and increases data retention rates. Our research provides practical insights and valuable resources for multilingual data curation, raising the standards of multilingual dataset development.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22232</guid>
|
||||
<pubDate>Wed, 28 May 2025 11:06:54 +0000</pubDate>
|
||||
<title>SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents</title>
|
||||
<link>https://arxiv.org/abs/2505.23559</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23559.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu, Jiaxuan You</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> Recent advancements in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet concurrently raised critical ethical and safety concerns. To systematically address these challenges, we introduce SafeScientist, an innovative AI scientist framework explicitly designed to enhance safety and ethical responsibility in AI-driven scientific exploration. SafeScientist proactively refuses ethically inappropriate or high-risk tasks and rigorously emphasizes safety throughout the research process. To achieve comprehensive safety oversight, we integrate multiple defensive mechanisms, including prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and an ethical reviewer component. Complementing SafeScientist, we propose SciSafetyBench, a novel benchmark specifically designed to evaluate AI safety in scientific contexts, comprising 240 high-risk scientific tasks across 6 domains, alongside 30 specially designed scientific tools and 120 tool-related risk tasks. Extensive experiments demonstrate that SafeScientist significantly improves safety performance by 35\% compared to traditional AI scientist frameworks, without compromising scientific output quality. Additionally, we rigorously validate the robustness of our safety pipeline against diverse adversarial attack methods, further confirming the effectiveness of our integrated approach. The code and data will be available at https://github.com/ulab-uiuc/SafeScientist. red{Warning: this paper contains example data that may be offensive or harmful.}</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23559</guid>
|
||||
<pubDate>Thu, 29 May 2025 15:35:58 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Skywork Open Reasoner 1 Technical Report</title>
|
||||
<link>https://arxiv.org/abs/2505.22312</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22312.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, Yahui Zhou</p><p><b>Upvotes:</b> 44</p><p><b>Summary:</b> The success of DeepSeek-R1 underscores the significant role of reinforcement learning (RL) in enhancing the reasoning capabilities of large language models (LLMs). In this work, we present Skywork-OR1, an effective and scalable RL implementation for long Chain-of-Thought (CoT) models. Building on the DeepSeek-R1-Distill model series, our RL approach achieves notable performance gains, increasing average accuracy across AIME24, AIME25, and LiveCodeBench from 57.8% to 72.8% (+15.0%) for the 32B model and from 43.6% to 57.5% (+13.9%) for the 7B model. Our Skywork-OR1-32B model surpasses both DeepSeek-R1 and Qwen3-32B on the AIME24 and AIME25 benchmarks, while achieving comparable results on LiveCodeBench. The Skywork-OR1-7B and Skywork-OR1-Math-7B models demonstrate competitive reasoning capabilities among models of similar size. We perform comprehensive ablation studies on the core components of our training pipeline to validate their effectiveness. Additionally, we thoroughly investigate the phenomenon of entropy collapse, identify key factors affecting entropy dynamics, and demonstrate that mitigating premature entropy collapse is critical for improved test performance. To support community research, we fully open-source our model weights, training code, and training datasets.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22312</guid>
|
||||
<pubDate>Wed, 28 May 2025 12:56:04 +0000</pubDate>
|
||||
<title>On-Policy RL with Optimal Reward Baseline</title>
|
||||
<link>https://arxiv.org/abs/2505.23585</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23585.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, Furu Wei</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Reinforcement learning algorithms are fundamental to align large language models with human preferences and to enhance their reasoning capabilities. However, current reinforcement learning algorithms often suffer from training instability due to loose on-policy constraints and computational inefficiency due to auxiliary models. In this work, we propose On-Policy RL with Optimal reward baseline (OPO), a novel and simplified reinforcement learning algorithm designed to address these challenges. OPO emphasizes the importance of exact on-policy training, which empirically stabilizes the training process and enhances exploration. Moreover, OPO introduces the optimal reward baseline that theoretically minimizes gradient variance. We evaluate OPO on mathematical reasoning benchmarks. The results demonstrate its superior performance and training stability without additional models or regularization terms. Furthermore, OPO achieves lower policy shifts and higher output entropy, encouraging more diverse and less repetitive responses. These results highlight OPO as a promising direction for stable and effective reinforcement learning in large language model alignment and reasoning tasks. The implementation is provided at https://github.com/microsoft/LMOps/tree/main/opo.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23585</guid>
|
||||
<pubDate>Thu, 29 May 2025 15:58:04 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start</title>
|
||||
<link>https://arxiv.org/abs/2505.22334</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22334.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, Weiran Huang</p><p><b>Upvotes:</b> 29</p><p><b>Summary:</b> Recent advancements in large language models (LLMs) have demonstrated impressive chain-of-thought reasoning capabilities, with reinforcement learning (RL) playing a crucial role in this progress. While "aha moment" patterns--where models exhibit self-correction through reflection--are often attributed to emergent properties from RL, we first demonstrate that these patterns exist in multimodal LLMs (MLLMs) prior to RL training but may not necessarily correlate with improved reasoning performance. Building on these insights, we present a comprehensive study on enhancing multimodal reasoning through a two-stage approach: (1) supervised fine-tuning (SFT) as a cold start with structured chain-of-thought reasoning patterns, followed by (2) reinforcement learning via GRPO to further refine these capabilities. Our extensive experiments show that this combined approach consistently outperforms both SFT-only and RL-only methods across challenging multimodal reasoning benchmarks. The resulting models achieve state-of-the-art performance among open-source MLLMs at both 3B and 7B scales, with our 7B model showing substantial improvements over base models (e.g., 66.3 %rightarrow73.4 % on MathVista, 62.9 %rightarrow70.4 % on We-Math) and our 3B model achieving performance competitive with several 7B models. Overall, this work provides practical guidance for building advanced multimodal reasoning models. Our code is available at https://github.com/waltonfuture/RL-with-Cold-Start.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22334</guid>
|
||||
<pubDate>Wed, 28 May 2025 13:21:38 +0000</pubDate>
|
||||
<title>Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering</title>
|
||||
<link>https://arxiv.org/abs/2505.23604</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23604.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guangtao Zeng, Maohao Shen, Delin Chen, Zhenting Qi, Subhro Das, Dan Gutfreund, David Cox, Gregory Wornell, Wei Lu, Zhang-Wei Hong, Chuang Gan</p><p><b>Upvotes:</b> 21</p><p><b>Summary:</b> Language models (LMs) perform well on standardized coding benchmarks but struggle with real-world software engineering tasks such as resolving GitHub issues in SWE-Bench, especially when model parameters are less than 100B. While smaller models are preferable in practice due to their lower computational cost, improving their performance remains challenging. Existing approaches primarily rely on supervised fine-tuning (SFT) with high-quality data, which is expensive to curate at scale. An alternative is test-time scaling: generating multiple outputs, scoring them using a verifier, and selecting the best one. Although effective, this strategy often requires excessive sampling and costly scoring, limiting its practical application. We propose Evolutionary Test-Time Scaling (EvoScale), a sample-efficient method that treats generation as an evolutionary process. By iteratively refining outputs via selection and mutation, EvoScale shifts the output distribution toward higher-scoring regions, reducing the number of samples needed to find correct solutions. To reduce the overhead from repeatedly sampling and selection, we train the model to self-evolve using reinforcement learning (RL). Rather than relying on external verifiers at inference time, the model learns to self-improve the scores of its own generations across iterations. Evaluated on SWE-Bench-Verified, EvoScale enables our 32B model, Satori-SWE-32B, to match or exceed the performance of models with over 100B parameters while using a few samples. Code, data, and models will be fully open-sourced.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23604</guid>
|
||||
<pubDate>Thu, 29 May 2025 16:15:36 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Text2Grad: Reinforcement Learning from Natural Language Feedback</title>
|
||||
<link>https://arxiv.org/abs/2505.22338</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22338.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Traditional RLHF optimizes language models with coarse, scalar rewards that mask the fine-grained reasons behind success or failure, leading to slow and opaque learning. Recent work augments RL with textual critiques through prompting or reflection, improving interpretability but leaving model parameters untouched. We introduce Text2Grad, a reinforcement-learning paradigm that turns free-form textual feedback into span-level gradients. Given human (or programmatic) critiques, Text2Grad aligns each feedback phrase with the relevant token spans, converts these alignments into differentiable reward signals, and performs gradient updates that directly refine the offending portions of the model's policy. This yields precise, feedback-conditioned adjustments instead of global nudges. Text2Grad is realized through three components: (1) a high-quality feedback-annotation pipeline that pairs critiques with token spans; (2) a fine-grained reward model that predicts span-level reward on answer while generating explanatory critiques; and (3) a span-level policy optimizer that back-propagates natural-language gradients. Across summarization, code generation, and question answering, Text2Grad consistently surpasses scalar-reward RL and prompt-only baselines, providing both higher task metrics and richer interpretability. Our results demonstrate that natural-language feedback, when converted to gradients, is a powerful signal for fine-grained policy optimization. The code for our method is available at https://github.com/microsoft/Text2Grad</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22338</guid>
|
||||
<pubDate>Wed, 28 May 2025 13:23:49 +0000</pubDate>
|
||||
<title>Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model</title>
|
||||
<link>https://arxiv.org/abs/2505.23606</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23606.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qingyu Shi, Jinbin Bai, Zhuoran Zhao, Wenhao Chai, Kaidong Yu, Jianzong Wu, Shuangyong Song, Yunhai Tong, Xiangtai Li, Xuelong Li, Shuicheng Yan</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Unified generation models aim to handle diverse tasks across modalities -- such as text generation, image generation, and vision-language reasoning -- within a single architecture and decoding paradigm. Autoregressive unified models suffer from slow inference due to sequential decoding, and non-autoregressive unified models suffer from weak generalization due to limited pretrained backbones. We introduce Muddit, a unified discrete diffusion transformer that enables fast and parallel generation across both text and image modalities. Unlike prior unified diffusion models trained from scratch, Muddit integrates strong visual priors from a pretrained text-to-image backbone with a lightweight text decoder, enabling flexible and high-quality multimodal generation under a unified architecture. Empirical results show that Muddit achieves competitive or superior performance compared to significantly larger autoregressive models in both quality and efficiency. The work highlights the potential of purely discrete diffusion, when equipped with strong visual priors, as a scalable and effective backbone for unified generation.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23606</guid>
|
||||
<pubDate>Thu, 29 May 2025 16:15:48 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO</title>
|
||||
<link>https://arxiv.org/abs/2505.22453</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22453.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lai Wei, Yuting Li, Chen Wang, Yue Wang, Linghe Kong, Weiran Huang, Lichao Sun</p><p><b>Upvotes:</b> 36</p><p><b>Summary:</b> Improving Multi-modal Large Language Models (MLLMs) in the post-training stage typically relies on supervised fine-tuning (SFT) or reinforcement learning (RL). However, these supervised methods require expensive and manually annotated multi-modal data--an ultimately unsustainable resource. While recent efforts have explored unsupervised post-training, their methods are complex and difficult to iterate. In this work, we are the first to investigate the use of GRPO, a stable and scalable online RL algorithm, for enabling continual self-improvement without any external supervision. We propose MM-UPT, a simple yet effective framework for unsupervised post-training of MLLMs. MM-UPT builds upon GRPO, replacing traditional reward signals with a self-rewarding mechanism based on majority voting over multiple sampled responses. Our experiments demonstrate that MM-UPT significantly improves the reasoning ability of Qwen2.5-VL-7B (e.g., 66.3 %rightarrow72.9 % on MathVista, 62.9 %rightarrow68.7 % on We-Math), using standard dataset without ground truth labels. MM-UPT also outperforms prior unsupervised baselines and even approaches the results of supervised GRPO. Furthermore, we show that incorporating synthetic questions, generated solely by MLLM itself, can boost performance as well, highlighting a promising approach for scalable self-improvement. Overall, MM-UPT offers a new paradigm for continual, autonomous enhancement of MLLMs in the absence of external supervision. Our code is available at https://github.com/waltonfuture/MM-UPT.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22453</guid>
|
||||
<pubDate>Wed, 28 May 2025 15:11:16 +0000</pubDate>
|
||||
<title>Table-R1: Inference-Time Scaling for Table Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.23621</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23621.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zheyuan Yang, Lyuhao Chen, Arman Cohan, Yilun Zhao</p><p><b>Upvotes:</b> 65</p><p><b>Summary:</b> In this work, we present the first study to explore inference-time scaling on table reasoning tasks. We develop and evaluate two post-training strategies to enable inference-time scaling: distillation from frontier model reasoning traces and reinforcement learning with verifiable rewards (RLVR). For distillation, we introduce a large-scale dataset of reasoning traces generated by DeepSeek-R1, which we use to fine-tune LLMs into the Table-R1-SFT model. For RLVR, we propose task-specific verifiable reward functions and apply the GRPO algorithm to obtain the Table-R1-Zero model. We evaluate our Table-R1-series models across diverse table reasoning tasks, including short-form QA, fact verification, and free-form QA. Notably, the Table-R1-Zero model matches or exceeds the performance of GPT-4.1 and DeepSeek-R1, while using only a 7B-parameter LLM. It also demonstrates strong generalization to out-of-domain datasets. Extensive ablation and qualitative analyses reveal the benefits of instruction tuning, model architecture choices, and cross-task generalization, as well as emergence of essential table reasoning skills during RL training.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23621</guid>
|
||||
<pubDate>Thu, 29 May 2025 16:28:50 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Fostering Video Reasoning via Next-Event Prediction</title>
|
||||
<link>https://arxiv.org/abs/2505.22457</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22457.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haonan Wang, Hongfu Liu, Xiangyan Liu, Chao Du, Kenji Kawaguchi, Ye Wang, Tianyu Pang</p><p><b>Upvotes:</b> 26</p><p><b>Summary:</b> Next-token prediction serves as the foundational learning task enabling reasoning in LLMs. But what should the learning task be when aiming to equip MLLMs with temporal reasoning capabilities over video inputs? Existing tasks such as video question answering often rely on annotations from humans or much stronger MLLMs, while video captioning tends to entangle temporal reasoning with spatial information. To address this gap, we propose next-event prediction (NEP), a learning task that harnesses future video segments as a rich, self-supervised signal to foster temporal reasoning. We segment each video into past and future frames: the MLLM takes the past frames as input and predicts a summary of events derived from the future frames, thereby encouraging the model to reason temporally in order to complete the task. To support this task, we curate V1-33K, a dataset comprising 33,000 automatically extracted video segments spanning diverse real-world scenarios. We further explore a range of video instruction-tuning strategies to study their effects on temporal reasoning. To evaluate progress, we introduce FutureBench to assess coherence in predicting unseen future events. Experiments validate that NEP offers a scalable and effective training paradigm for fostering temporal reasoning in MLLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22457</guid>
|
||||
<pubDate>Wed, 28 May 2025 15:13:34 +0000</pubDate>
|
||||
<title>ZeroSep: Separate Anything in Audio with Zero Training</title>
|
||||
<link>https://arxiv.org/abs/2505.23625</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23625.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chao Huang, Yuesheng Ma, Junxuan Huang, Susan Liang, Yunlong Tang, Jing Bi, Wenqiang Liu, Nima Mesgarani, Chenliang Xu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Audio source separation is fundamental for machines to understand complex acoustic environments and underpins numerous audio applications. Current supervised deep learning approaches, while powerful, are limited by the need for extensive, task-specific labeled data and struggle to generalize to the immense variability and open-set nature of real-world acoustic scenes. Inspired by the success of generative foundation models, we investigate whether pre-trained text-guided audio diffusion models can overcome these limitations. We make a surprising discovery: zero-shot source separation can be achieved purely through a pre-trained text-guided audio diffusion model under the right configuration. Our method, named ZeroSep, works by inverting the mixed audio into the diffusion model's latent space and then using text conditioning to guide the denoising process to recover individual sources. Without any task-specific training or fine-tuning, ZeroSep repurposes the generative diffusion model for a discriminative separation task and inherently supports open-set scenarios through its rich textual priors. ZeroSep is compatible with a variety of pre-trained text-guided audio diffusion backbones and delivers strong separation performance on multiple separation benchmarks, surpassing even supervised methods.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23625</guid>
|
||||
<pubDate>Thu, 29 May 2025 16:31:45 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>PrismLayers: Open Data for High-Quality Multi-Layer Transparent Image Generative Models</title>
|
||||
<link>https://arxiv.org/abs/2505.22523</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22523.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junwen Chen, Heyang Jiang, Yanbin Wang, Keming Wu, Ji Li, Chao Zhang, Keiji Yanai, Dong Chen, Yuhui Yuan</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Generating high-quality, multi-layer transparent images from text prompts can unlock a new level of creative control, allowing users to edit each layer as effortlessly as editing text outputs from LLMs. However, the development of multi-layer generative models lags behind that of conventional text-to-image models due to the absence of a large, high-quality corpus of multi-layer transparent data. In this paper, we address this fundamental challenge by: (i) releasing the first open, ultra-high-fidelity PrismLayers (PrismLayersPro) dataset of 200K (20K) multilayer transparent images with accurate alpha mattes, (ii) introducing a trainingfree synthesis pipeline that generates such data on demand using off-the-shelf diffusion models, and (iii) delivering a strong, open-source multi-layer generation model, ART+, which matches the aesthetics of modern text-to-image generation models. The key technical contributions include: LayerFLUX, which excels at generating high-quality single transparent layers with accurate alpha mattes, and MultiLayerFLUX, which composes multiple LayerFLUX outputs into complete images, guided by human-annotated semantic layout. To ensure higher quality, we apply a rigorous filtering stage to remove artifacts and semantic mismatches, followed by human selection. Fine-tuning the state-of-the-art ART model on our synthetic PrismLayersPro yields ART+, which outperforms the original ART in 60% of head-to-head user study comparisons and even matches the visual quality of images generated by the FLUX.1-[dev] model. We anticipate that our work will establish a solid dataset foundation for the multi-layer transparent image generation task, enabling research and applications that require precise, editable, and visually compelling layered imagery.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22523</guid>
|
||||
<pubDate>Wed, 28 May 2025 16:09:33 +0000</pubDate>
|
||||
<title>Are Reasoning Models More Prone to Hallucination?</title>
|
||||
<link>https://arxiv.org/abs/2505.23646</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23646.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zijun Yao, Yantao Liu, Yanxu Chen, Jianhui Chen, Junfeng Fang, Lei Hou, Juanzi Li, Tat-Seng Chua</p><p><b>Upvotes:</b> 19</p><p><b>Summary:</b> Recently evolved large reasoning models (LRMs) show powerful performance in solving complex tasks with long chain-of-thought (CoT) reasoning capability. As these LRMs are mostly developed by post-training on formal reasoning tasks, whether they generalize the reasoning capability to help reduce hallucination in fact-seeking tasks remains unclear and debated. For instance, DeepSeek-R1 reports increased performance on SimpleQA, a fact-seeking benchmark, while OpenAI-o3 observes even severer hallucination. This discrepancy naturally raises the following research question: Are reasoning models more prone to hallucination? This paper addresses the question from three perspectives. (1) We first conduct a holistic evaluation for the hallucination in LRMs. Our analysis reveals that LRMs undergo a full post-training pipeline with cold start supervised fine-tuning (SFT) and verifiable reward RL generally alleviate their hallucination. In contrast, both distillation alone and RL training without cold start fine-tuning introduce more nuanced hallucinations. (2) To explore why different post-training pipelines alters the impact on hallucination in LRMs, we conduct behavior analysis. We characterize two critical cognitive behaviors that directly affect the factuality of a LRM: Flaw Repetition, where the surface-level reasoning attempts repeatedly follow the same underlying flawed logic, and Think-Answer Mismatch, where the final answer fails to faithfully match the previous CoT process. (3) Further, we investigate the mechanism behind the hallucination of LRMs from the perspective of model uncertainty. We find that increased hallucination of LRMs is usually associated with the misalignment between model uncertainty and factual accuracy. Our work provides an initial understanding of the hallucination in LRMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23646</guid>
|
||||
<pubDate>Thu, 29 May 2025 16:53:41 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Thinking with Generated Images</title>
|
||||
<link>https://arxiv.org/abs/2505.22525</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22525.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, Pengfei Liu</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> We present Thinking with Generated Images, a novel paradigm that fundamentally transforms how large multimodal models (LMMs) engage with visual reasoning by enabling them to natively think across text and vision modalities through spontaneous generation of intermediate visual thinking steps. Current visual reasoning with LMMs is constrained to either processing fixed user-provided images or reasoning solely through text-based chain-of-thought (CoT). Thinking with Generated Images unlocks a new dimension of cognitive capability where models can actively construct intermediate visual thoughts, critique their own visual hypotheses, and refine them as integral components of their reasoning process. We demonstrate the effectiveness of our approach through two complementary mechanisms: (1) vision generation with intermediate visual subgoals, where models decompose complex visual tasks into manageable components that are generated and integrated progressively, and (2) vision generation with self-critique, where models generate an initial visual hypothesis, analyze its shortcomings through textual reasoning, and produce refined outputs based on their own critiques. Our experiments on vision generation benchmarks show substantial improvements over baseline approaches, with our models achieving up to 50% (from 38% to 57%) relative improvement in handling complex multi-object scenarios. From biochemists exploring novel protein structures, and architects iterating on spatial designs, to forensic analysts reconstructing crime scenes, and basketball players envisioning strategic plays, our approach enables AI models to engage in the kind of visual imagination and iterative refinement that characterizes human creative, analytical, and strategic thinking. We release our open-source suite at https://github.com/GAIR-NLP/thinking-with-generated-images.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22525</guid>
|
||||
<pubDate>Wed, 28 May 2025 16:12:45 +0000</pubDate>
|
||||
<title>D-AR: Diffusion via Autoregressive Models</title>
|
||||
<link>https://arxiv.org/abs/2505.23660</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23660.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ziteng Gao, Mike Zheng Shou</p><p><b>Upvotes:</b> 20</p><p><b>Summary:</b> This paper presents Diffusion via Autoregressive models (D-AR), a new paradigm recasting the image diffusion process as a vanilla autoregressive procedure in the standard next-token-prediction fashion. We start by designing the tokenizer that converts images into sequences of discrete tokens, where tokens in different positions can be decoded into different diffusion denoising steps in the pixel space. Thanks to the diffusion properties, these tokens naturally follow a coarse-to-fine order, which directly lends itself to autoregressive modeling. Therefore, we apply standard next-token prediction on these tokens, without modifying any underlying designs (either causal masks or training/inference strategies), and such sequential autoregressive token generation directly mirrors the diffusion procedure in image space. That is, once the autoregressive model generates an increment of tokens, we can directly decode these tokens into the corresponding diffusion denoising step in the streaming manner. Our pipeline naturally reveals several intriguing properties, for example, it supports consistent previews when generating only a subset of tokens and enables zero-shot layout-controlled synthesis. On the standard ImageNet benchmark, our method achieves 2.09 FID using a 775M Llama backbone with 256 discrete tokens. We hope our work can inspire future research on unified autoregressive architectures of visual synthesis, especially with large language models. Code and models will be available at https://github.com/showlab/D-AR</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23660</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:09:25 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Precise In-Parameter Concept Erasure in Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.22586</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22586.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yoav Gur-Arieh, Clara Suslik, Yihuai Hong, Fazl Barez, Mor Geva</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large language models (LLMs) often acquire knowledge during pretraining that is undesirable in downstream deployments, e.g., sensitive information or copyrighted content. Existing approaches for removing such knowledge rely on fine-tuning, training low-rank adapters or fact-level editing, but these are either too coarse, too shallow, or ineffective. In this work, we propose PISCES (Precise In-parameter Suppression for Concept EraSure), a novel framework for precisely erasing entire concepts from model parameters by directly editing directions that encode them in parameter space. PISCES uses a disentangler model to decompose MLP vectors into interpretable features, identifies those associated with a target concept using automated interpretability techniques, and removes them from model parameters. Experiments on Gemma 2 and Llama 3.1 over various concepts show that PISCES achieves modest gains in efficacy over leading erasure methods, reducing accuracy on the target concept to as low as 7.7%, while dramatically improving erasure specificity (by up to 31%) and robustness (by up to 38%). Overall, these results demonstrate that feature-based in-parameter editing enables a more precise and reliable approach for removing conceptual knowledge in language models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22586</guid>
|
||||
<pubDate>Wed, 28 May 2025 16:58:23 +0000</pubDate>
|
||||
<title>Grounded Reinforcement Learning for Visual Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.23678</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23678.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, Katerina Fragkiadaki</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23678</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:20:26 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction</title>
|
||||
<link>https://arxiv.org/abs/2505.22613</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22613.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuchi Wang, Yishuo Cai, Shuhuai Ren, Sihan Yang, Linli Yao, Yuanxin Liu, Yuanxing Zhang, Pengfei Wan, Xu Sun</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Image recaptioning is widely used to generate training datasets with enhanced quality for various multimodal tasks. Existing recaptioning methods typically rely on powerful multimodal large language models (MLLMs) to enhance textual descriptions, but often suffer from inaccuracies due to hallucinations and incompleteness caused by missing fine-grained details. To address these limitations, we propose RICO, a novel framework that refines captions through visual reconstruction. Specifically, we leverage a text-to-image model to reconstruct a caption into a reference image, and prompt an MLLM to identify discrepancies between the original and reconstructed images to refine the caption. This process is performed iteratively, further progressively promoting the generation of more faithful and comprehensive descriptions. To mitigate the additional computational cost induced by the iterative process, we introduce RICO-Flash, which learns to generate captions like RICO using DPO. Extensive experiments demonstrate that our approach significantly improves caption accuracy and completeness, outperforms most baselines by approximately 10% on both CapsBench and CompreCap. Code released at https://github.com/wangyuchi369/RICO.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22613</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:29:34 +0000</pubDate>
|
||||
<title>VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos</title>
|
||||
<link>https://arxiv.org/abs/2505.23693</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23693.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao</p><p><b>Upvotes:</b> 51</p><p><b>Summary:</b> MLLMs have been widely studied for video question answering recently. However, most existing assessments focus on natural videos, overlooking synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in video generation rely on MLLMs to evaluate the quality of generated videos, but the capabilities of MLLMs on interpreting AIGC videos remain largely underexplored. To address this, we propose a new benchmark, VF-Eval, which introduces four tasks-coherence validation, error awareness, error type detection, and reasoning evaluation-to comprehensively evaluate the abilities of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that even the best-performing model, GPT-4.1, struggles to achieve consistently good performance across all tasks. This highlights the challenging nature of our benchmark. Additionally, to investigate the practical applications of VF-Eval in improving video generation, we conduct an experiment, RePrompt, demonstrating that aligning MLLMs more closely with human feedback can benefit video generation.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23693</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:31:13 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.22617</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22617.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, Ning Ding</p><p><b>Upvotes:</b> 83</p><p><b>Summary:</b> This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy. Such phenomenon is consistently observed across vast RL runs without entropy intervention, where the policy entropy dropped sharply at the early training stage, this diminished exploratory ability is always accompanied with the saturation of policy performance. In practice, we establish a transformation equation R=-a*e^H+b between entropy H and downstream performance R. This empirical law strongly indicates that, the policy performance is traded from policy entropy, thus bottlenecked by its exhaustion, and the ceiling is fully predictable H=0, R=-a+b. Our finding necessitates entropy management for continuous exploration toward scaling compute for RL. To this end, we investigate entropy dynamics both theoretically and empirically. Our derivation highlights that, the change in policy entropy is driven by the covariance between action probability and the change in logits, which is proportional to its advantage when using Policy Gradient-like algorithms. Empirical study shows that, the values of covariance term and entropy differences matched exactly, supporting the theoretical conclusion. Moreover, the covariance term stays mostly positive throughout training, further explaining why policy entropy would decrease monotonically. Through understanding the mechanism behind entropy dynamics, we motivate to control entropy by restricting the update of high-covariance tokens. Specifically, we propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively. Experiments show that these methods encourage exploration, thus helping policy escape entropy collapse and achieve better downstream performance.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22617</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:38:45 +0000</pubDate>
|
||||
<title>AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views</title>
|
||||
<link>https://arxiv.org/abs/2505.23716</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23716.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lihan Jiang, Yucheng Mao, Linning Xu, Tao Lu, Kerui Ren, Yichen Jin, Xudong Xu, Mulin Yu, Jiangmiao Pang, Feng Zhao, Dahua Lin, Bo Dai</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> We introduce AnySplat, a feed forward network for novel view synthesis from uncalibrated image collections. In contrast to traditional neural rendering pipelines that demand known camera poses and per scene optimization, or recent feed forward methods that buckle under the computational weight of dense views, our model predicts everything in one shot. A single forward pass yields a set of 3D Gaussian primitives encoding both scene geometry and appearance, and the corresponding camera intrinsics and extrinsics for each input image. This unified design scales effortlessly to casually captured, multi view datasets without any pose annotations. In extensive zero shot evaluations, AnySplat matches the quality of pose aware baselines in both sparse and dense view scenarios while surpassing existing pose free approaches. Moreover, it greatly reduce rendering latency compared to optimization based neural fields, bringing real time novel view synthesis within reach for unconstrained capture settings.Project page: https://city-super.github.io/anysplat/</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23716</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:49:56 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control</title>
|
||||
<link>https://arxiv.org/abs/2505.22642</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22642.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, Pieter Abbeel</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Reinforcement learning (RL) has driven significant progress in robotics, but its complexity and long training times remain major bottlenecks. In this report, we introduce FastTD3, a simple, fast, and capable RL algorithm that significantly speeds up training for humanoid robots in popular suites such as HumanoidBench, IsaacLab, and MuJoCo Playground. Our recipe is remarkably simple: we train an off-policy TD3 agent with several modifications -- parallel simulation, large-batch updates, a distributional critic, and carefully tuned hyperparameters. FastTD3 solves a range of HumanoidBench tasks in under 3 hours on a single A100 GPU, while remaining stable during training. We also provide a lightweight and easy-to-use implementation of FastTD3 to accelerate RL research in robotics.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22642</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:55:26 +0000</pubDate>
|
||||
<title>ZPressor: Bottleneck-Aware Compression for Scalable Feed-Forward 3DGS</title>
|
||||
<link>https://arxiv.org/abs/2505.23734</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23734.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Weijie Wang, Donny Y. Chen, Zeyu Zhang, Duochao Shi, Akide Liu, Bohan Zhuang</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Feed-forward 3D Gaussian Splatting (3DGS) models have recently emerged as a promising solution for novel view synthesis, enabling one-pass inference without the need for per-scene 3DGS optimization. However, their scalability is fundamentally constrained by the limited capacity of their encoders, leading to degraded performance or excessive memory consumption as the number of input views increases. In this work, we analyze feed-forward 3DGS frameworks through the lens of the Information Bottleneck principle and introduce ZPressor, a lightweight architecture-agnostic module that enables efficient compression of multi-view inputs into a compact latent state Z that retains essential scene information while discarding redundancy. Concretely, ZPressor enables existing feed-forward 3DGS models to scale to over 100 input views at 480P resolution on an 80GB GPU, by partitioning the views into anchor and support sets and using cross attention to compress the information from the support views into anchor views, forming the compressed latent state Z. We show that integrating ZPressor into several state-of-the-art feed-forward 3DGS models consistently improves performance under moderate input views and enhances robustness under dense view settings on two large-scale benchmarks DL3DV-10K and RealEstate10K. The video results, code and trained models are available on our project page: https://lhmd.top/zpressor.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23734</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:57:04 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese</title>
|
||||
<link>https://arxiv.org/abs/2505.22645</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22645.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hanjia Lyu, Jiebo Luo, Jian Kang, Allison Koenecke</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> While the capabilities of Large Language Models (LLMs) have been studied in both Simplified and Traditional Chinese, it is yet unclear whether LLMs exhibit differential performance when prompted in these two variants of written Chinese. This understanding is critical, as disparities in the quality of LLM responses can perpetuate representational harms by ignoring the different cultural contexts underlying Simplified versus Traditional Chinese, and can exacerbate downstream harms in LLM-facilitated decision-making in domains such as education or hiring. To investigate potential LLM performance disparities, we design two benchmark tasks that reflect real-world scenarios: regional term choice (prompting the LLM to name a described item which is referred to differently in Mainland China and Taiwan), and regional name choice (prompting the LLM to choose who to hire from a list of names in both Simplified and Traditional Chinese). For both tasks, we audit the performance of 11 leading commercial LLM services and open-sourced models -- spanning those primarily trained on English, Simplified Chinese, or Traditional Chinese. Our analyses indicate that biases in LLM responses are dependent on both the task and prompting language: while most LLMs disproportionately favored Simplified Chinese responses in the regional term choice task, they surprisingly favored Traditional Chinese names in the regional name choice task. We find that these disparities may arise from differences in training data representation, written character preferences, and tokenization of Simplified and Traditional Chinese. These findings highlight the need for further analysis of LLM biases; as such, we provide an open-sourced benchmark dataset to foster reproducible evaluations of future LLM behavior across Chinese language variants (https://github.com/brucelyu17/SC-TC-Bench).</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22645</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:56:49 +0000</pubDate>
|
||||
<title>ATLAS: Learning to Optimally Memorize the Context at Test Time</title>
|
||||
<link>https://arxiv.org/abs/2505.23735</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23735.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Transformers have been established as the most popular backbones in sequence modeling, mainly due to their effectiveness in in-context retrieval tasks and the ability to learn at scale. Their quadratic memory and time complexity, however, bound their applicability in longer sequences and so has motivated researchers to explore effective alternative architectures such as modern recurrent neural networks (a.k.a long-term recurrent memory module). Despite their recent success in diverse downstream tasks, they struggle in tasks that requires long context understanding and extrapolation to longer sequences. We observe that these shortcomings come from three disjoint aspects in their design: (1) limited memory capacity that is bounded by the architecture of memory and feature mapping of the input; (2) online nature of update, i.e., optimizing the memory only with respect to the last input; and (3) less expressive management of their fixed-size memory. To enhance all these three aspects, we present ATLAS, a long-term memory module with high capacity that learns to memorize the context by optimizing the memory based on the current and past tokens, overcoming the online nature of long-term memory models. Building on this insight, we present a new family of Transformer-like architectures, called DeepTransformers, that are strict generalizations of the original Transformer architecture. Our experimental results on language modeling, common-sense reasoning, recall-intensive, and long-context understanding tasks show that ATLAS surpasses the performance of Transformers and recent linear recurrent models. ATLAS further improves the long context performance of Titans, achieving +80\% accuracy in 10M context length of BABILong benchmark.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23735</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:57:16 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>WebDancer: Towards Autonomous Information Seeking Agency</title>
|
||||
<link>https://arxiv.org/abs/2505.22648</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22648.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Addressing intricate real-world problems necessitates in-depth information seeking and multi-step reasoning. Recent progress in agentic systems, exemplified by Deep Research, underscores the potential for autonomous multi-step research. In this work, we present a cohesive paradigm for building end-to-end agentic information seeking agents from a data-centric and training-stage perspective. Our approach consists of four key stages: (1) browsing data construction, (2) trajectories sampling, (3) supervised fine-tuning for effective cold start, and (4) reinforcement learning for enhanced generalisation. We instantiate this framework in a web agent based on the ReAct, WebDancer. Empirical evaluations on the challenging information seeking benchmarks, GAIA and WebWalkerQA, demonstrate the strong performance of WebDancer, achieving considerable results and highlighting the efficacy of our training paradigm. Further analysis of agent training provides valuable insights and actionable, systematic pathways for developing more capable agentic models. The codes and demo will be released in https://github.com/Alibaba-NLP/WebAgent.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22648</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:57:07 +0000</pubDate>
|
||||
<title>How Animals Dance (When You're Not Looking)</title>
|
||||
<link>https://arxiv.org/abs/2505.23738</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23738.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiaojuan Wang, Aleksander Holynski, Brian Curless, Ira Kemelmacher, Steve Seitz</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> We present a keyframe-based framework for generating music-synchronized, choreography aware animal dance videos. Starting from a few keyframes representing distinct animal poses -- generated via text-to-image prompting or GPT-4o -- we formulate dance synthesis as a graph optimization problem: find the optimal keyframe structure that satisfies a specified choreography pattern of beats, which can be automatically estimated from a reference dance video. We also introduce an approach for mirrored pose image generation, essential for capturing symmetry in dance. In-between frames are synthesized using an video diffusion model. With as few as six input keyframes, our method can produce up to 30 second dance videos across a wide range of animals and music tracks.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23738</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:58:02 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Sherlock: Self-Correcting Reasoning in Vision-Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.22651</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22651.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yi Ding, Ruqi Zhang</p><p><b>Upvotes:</b> 44</p><p><b>Summary:</b> Reasoning Vision-Language Models (VLMs) have shown promising performance on complex multimodal tasks. However, they still face significant challenges: they are highly sensitive to reasoning errors, require large volumes of annotated data or accurate verifiers, and struggle to generalize beyond specific domains. To address these limitations, we explore self-correction as a strategy to enhance reasoning VLMs. We first conduct an in-depth analysis of reasoning VLMs' self-correction abilities and identify key gaps. Based on our findings, we introduce Sherlock, a self-correction and self-improvement training framework. Sherlock introduces a trajectory-level self-correction objective, a preference data construction method based on visual perturbation, and a dynamic beta for preference tuning. Once the model acquires self-correction capabilities using only 20k randomly sampled annotated data, it continues to self-improve without external supervision. Built on the Llama3.2-Vision-11B model, Sherlock achieves remarkable results across eight benchmarks, reaching an average accuracy of 64.1 with direct generation and 65.4 after self-correction. It outperforms LLaVA-CoT (63.2), Mulberry (63.9), and LlamaV-o1 (63.4) while using less than 20% of the annotated data.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22651</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:58:03 +0000</pubDate>
|
||||
<title>MAGREF: Masked Guidance for Any-Reference Video Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.23742</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23742.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yufan Deng, Xun Guo, Yuanyang Yin, Jacob Zhiyuan Fang, Yiding Yang, Yizhi Wang, Shenghai Yuan, Angtian Wang, Bo Liu, Haibin Huang, Chongyang Ma</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Video generation has made substantial strides with the emergence of deep generative models, especially diffusion-based approaches. However, video generation based on multiple reference subjects still faces significant challenges in maintaining multi-subject consistency and ensuring high generation quality. In this paper, we propose MAGREF, a unified framework for any-reference video generation that introduces masked guidance to enable coherent multi-subject video synthesis conditioned on diverse reference images and a textual prompt. Specifically, we propose (1) a region-aware dynamic masking mechanism that enables a single model to flexibly handle various subject inference, including humans, objects, and backgrounds, without architectural changes, and (2) a pixel-wise channel concatenation mechanism that operates on the channel dimension to better preserve appearance features. Our model delivers state-of-the-art video generation quality, generalizing from single-subject training to complex multi-subject scenarios with coherent synthesis and precise control over individual subjects, outperforming existing open-source and commercial baselines. To facilitate evaluation, we also introduce a comprehensive multi-subject video benchmark. Extensive experiments demonstrate the effectiveness of our approach, paving the way for scalable, controllable, and high-fidelity multi-subject video synthesis. Code and model can be found at: https://github.com/MAGREF-Video/MAGREF</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23742</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:58:15 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Zero-Shot Vision Encoder Grafting via LLM Surrogates</title>
|
||||
<link>https://arxiv.org/abs/2505.22664</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.22664.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kaiyu Yue, Vasu Singla, Menglin Jia, John Kirchenbauer, Rifaa Qadri, Zikui Cai, Abhinav Bhatele, Furong Huang, Tom Goldstein</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Vision language models (VLMs) typically pair a modestly sized vision encoder with a large language model (LLM), e.g., Llama-70B, making the decoder the primary computational burden during training. To reduce costs, a potential promising strategy is to first train the vision encoder using a small language model before transferring it to the large one. We construct small "surrogate models" that share the same embedding space and representation language as the large target LLM by directly inheriting its shallow layers. Vision encoders trained on the surrogate can then be directly transferred to the larger model, a process we call zero-shot grafting -- when plugged directly into the full-size target LLM, the grafted pair surpasses the encoder-surrogate pair and, on some benchmarks, even performs on par with full decoder training with the target LLM. Furthermore, our surrogate training approach reduces overall VLM training costs by ~45% when using Llama-70B as the decoder.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.22664</guid>
|
||||
<pubDate>Wed, 28 May 2025 17:59:59 +0000</pubDate>
|
||||
<title>To Trust Or Not To Trust Your Vision-Language Model's Prediction</title>
|
||||
<link>https://arxiv.org/abs/2505.23745</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23745.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hao Dong, Moru Liu, Jian Liang, Eleni Chatzi, Olga Fink</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Vision-Language Models (VLMs) have demonstrated strong capabilities in aligning visual and textual modalities, enabling a wide range of applications in multimodal understanding and generation. While they excel in zero-shot and transfer learning scenarios, VLMs remain susceptible to misclassification, often yielding confident yet incorrect predictions. This limitation poses a significant risk in safety-critical domains, where erroneous predictions can lead to severe consequences. In this work, we introduce TrustVLM, a training-free framework designed to address the critical challenge of estimating when VLM's predictions can be trusted. Motivated by the observed modality gap in VLMs and the insight that certain concepts are more distinctly represented in the image embedding space, we propose a novel confidence-scoring function that leverages this space to improve misclassification detection. We rigorously evaluate our approach across 17 diverse datasets, employing 4 architectures and 2 VLMs, and demonstrate state-of-the-art performance, with improvements of up to 51.87% in AURC, 9.14% in AUROC, and 32.42% in FPR95 compared to existing baselines. By improving the reliability of the model without requiring retraining, TrustVLM paves the way for safer deployment of VLMs in real-world applications. The code will be available at https://github.com/EPFL-IMOS/TrustVLM.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23745</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:01 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence</title>
|
||||
<link>https://arxiv.org/abs/2505.23747</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23747.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Diankun Wu, Fangfu Liu, Yi-Hsin Hung, Yueqi Duan</p><p><b>Upvotes:</b> 53</p><p><b>Summary:</b> Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced performance on 2D visual tasks. However, improving their spatial intelligence remains a challenge. Existing 3D MLLMs always rely on additional 3D or 2.5D data to incorporate spatial awareness, restricting their utility in scenarios with only 2D inputs, such as images or videos. In this paper, we present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations. Unlike conventional video MLLMs which rely on CLIP-based visual encoders optimized for semantic understanding, our key insight is to unleash the strong structure prior from the feed-forward visual geometry foundation model. Specifically, we propose a dual-encoder architecture: a pretrained 2D visual encoder to extract semantic features, and a spatial encoder-initialized from the backbone of the visual geometry model-to extract 3D structure features. A connector then integrates both features into unified visual tokens for enhanced spatial understanding. Furthermore, we propose a space-aware frame sampling strategy at inference time, which selects the spatially informative frames of a video sequence, ensuring that even under limited token length, the model focuses on frames critical for spatial reasoning. Beyond architecture improvements, we construct the Spatial-MLLM-120k dataset and train the model on it using supervised fine-tuning and GRPO. Extensive experiments on various real-world datasets demonstrate that our spatial-MLLM achieves state-of-the-art performance in a wide range of visual-based spatial understanding and reasoning tasks. Project page: https://diankun-wu.github.io/Spatial-MLLM/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23747</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:04 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>REOrdering Patches Improves Vision Models</title>
|
||||
<link>https://arxiv.org/abs/2505.23751</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23751.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Declan Kutscher, David M. Chan, Yutong Bai, Trevor Darrell, Ritwik Gupta</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Sequence models such as transformers require inputs to be represented as one-dimensional sequences. In vision, this typically involves flattening images using a fixed row-major (raster-scan) order. While full self-attention is permutation-equivariant, modern long-sequence transformers increasingly rely on architectural approximations that break this invariance and introduce sensitivity to patch ordering. We show that patch order significantly affects model performance in such settings, with simple alternatives like column-major or Hilbert curves yielding notable accuracy shifts. Motivated by this, we propose REOrder, a two-stage framework for discovering task-optimal patch orderings. First, we derive an information-theoretic prior by evaluating the compressibility of various patch sequences. Then, we learn a policy over permutations by optimizing a Plackett-Luce policy using REINFORCE. This approach enables efficient learning in a combinatorial permutation space. REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to 3.01% and Functional Map of the World by 13.35%.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23751</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:30 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>DeepTheorem: Advancing LLM Reasoning for Theorem Proving Through Natural Language and Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.23754</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23754.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ziyin Zhang, Jiahao Xu, Zhiwei He, Tian Liang, Qiuzhi Liu, Yansi Li, Linfeng Song, Zhengwen Liang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Theorem proving serves as a major testbed for evaluating complex reasoning abilities in large language models (LLMs). However, traditional automated theorem proving (ATP) approaches rely heavily on formal proof systems that poorly align with LLMs' strength derived from informal, natural language knowledge acquired during pre-training. In this work, we propose DeepTheorem, a comprehensive informal theorem-proving framework exploiting natural language to enhance LLM mathematical reasoning. DeepTheorem includes a large-scale benchmark dataset consisting of 121K high-quality IMO-level informal theorems and proofs spanning diverse mathematical domains, rigorously annotated for correctness, difficulty, and topic categories, accompanied by systematically constructed verifiable theorem variants. We devise a novel reinforcement learning strategy (RL-Zero) explicitly tailored to informal theorem proving, leveraging the verified theorem variants to incentivize robust mathematical inference. Additionally, we propose comprehensive outcome and process evaluation metrics examining proof correctness and the quality of reasoning steps. Extensive experimental analyses demonstrate DeepTheorem significantly improves LLM theorem-proving performance compared to existing datasets and supervised fine-tuning protocols, achieving state-of-the-art accuracy and reasoning quality. Our findings highlight DeepTheorem's potential to fundamentally advance automated informal theorem proving and mathematical exploration.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23754</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:39 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>LoRAShop: Training-Free Multi-Concept Image Generation and Editing with Rectified Flow Transformers</title>
|
||||
<link>https://arxiv.org/abs/2505.23758</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23758.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yusuf Dalva, Hidir Yesiltepe, Pinar Yanardag</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> We introduce LoRAShop, the first framework for multi-concept image editing with LoRA models. LoRAShop builds on a key observation about the feature interaction patterns inside Flux-style diffusion transformers: concept-specific transformer features activate spatially coherent regions early in the denoising process. We harness this observation to derive a disentangled latent mask for each concept in a prior forward pass and blend the corresponding LoRA weights only within regions bounding the concepts to be personalized. The resulting edits seamlessly integrate multiple subjects or styles into the original scene while preserving global context, lighting, and fine details. Our experiments demonstrate that LoRAShop delivers better identity preservation compared to baselines. By eliminating retraining and external constraints, LoRAShop turns personalized diffusion models into a practical `photoshop-with-LoRAs' tool and opens new avenues for compositional visual storytelling and rapid creative iteration.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23758</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:46 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint</title>
|
||||
<link>https://arxiv.org/abs/2505.23759</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23759.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Heekyung Lee, Jiaxin Ge, Tsung-Han Wu, Minwoo Kang, Trevor Darrell, David M. Chan</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs). Unlike traditional image captioning or question answering tasks, rebus solving requires multi-modal abstraction, symbolic reasoning, and a grasp of cultural, phonetic and linguistic puns. In this paper, we investigate the capacity of contemporary VLMs to interpret and solve rebus puzzles by constructing a hand-generated and annotated benchmark of diverse English-language rebus puzzles, ranging from simple pictographic substitutions to spatially-dependent cues ("head" over "heels"). We analyze how different VLMs perform, and our findings reveal that while VLMs exhibit some surprising capabilities in decoding simple visual clues, they struggle significantly with tasks requiring abstract reasoning, lateral thinking, and understanding visual metaphors.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23759</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:47 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Differential Information: An Information-Theoretic Perspective on Preference Optimization</title>
|
||||
<link>https://arxiv.org/abs/2505.23761</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23761.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Direct Preference Optimization (DPO) has become a standard technique for aligning language models with human preferences in a supervised manner. Despite its empirical success, the theoretical justification behind its log-ratio reward parameterization remains incomplete. In this work, we address this gap by utilizing the Differential Information Distribution (DID): a distribution over token sequences that captures the information gained during policy updates. First, we show that when preference labels encode the differential information required to transform a reference policy into a target policy, the log-ratio reward in DPO emerges as the uniquely optimal form for learning the target policy via preference optimization. This result naturally yields a closed-form expression for the optimal sampling distribution over rejected responses. Second, we find that the condition for preferences to encode differential information is fundamentally linked to an implicit assumption regarding log-margin ordered policies-an inductive bias widely used in preference optimization yet previously unrecognized. Finally, by analyzing the entropy of the DID, we characterize how learning low-entropy differential information reinforces the policy distribution, while high-entropy differential information induces a smoothing effect, which explains the log-likelihood displacement phenomenon. We validate our theoretical findings in synthetic experiments and extend them to real-world instruction-following datasets. Our results suggest that learning high-entropy differential information is crucial for general instruction-following, while learning low-entropy differential information benefits knowledge-intensive question answering. Overall, our work presents a unifying perspective on the DPO objective, the structure of preference data, and resulting policy behaviors through the lens of differential information.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23761</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:50 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>ZeroGUI: Automating Online GUI Learning at Zero Human Cost</title>
|
||||
<link>https://arxiv.org/abs/2505.23762</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23762.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chenyu Yang, Shiqian Su, Shi Liu, Xuan Dong, Yue Yu, Weijie Su, Xuehui Wang, Zhaoyang Liu, Jinguo Zhu, Hao Li, Wenhai Wang, Yu Qiao, Xizhou Zhu, Jifeng Dai</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> The rapid advancement of large Vision-Language Models (VLMs) has propelled the development of pure-vision-based GUI Agents, capable of perceiving and operating Graphical User Interfaces (GUI) to autonomously fulfill user instructions. However, existing approaches usually adopt an offline learning framework, which faces two core limitations: (1) heavy reliance on high-quality manual annotations for element grounding and action supervision, and (2) limited adaptability to dynamic and interactive environments. To address these limitations, we propose ZeroGUI, a scalable, online learning framework for automating GUI Agent training at Zero human cost. Specifically, ZeroGUI integrates (i) VLM-based automatic task generation to produce diverse training goals from the current environment state, (ii) VLM-based automatic reward estimation to assess task success without hand-crafted evaluation functions, and (iii) two-stage online reinforcement learning to continuously interact with and learn from GUI environments. Experiments on two advanced GUI Agents (UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance across OSWorld and AndroidLab environments. The code is available at https://github.com/OpenGVLab/ZeroGUI.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23762</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:51 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence</title>
|
||||
<link>https://arxiv.org/abs/2505.23764</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23764.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, Jiangmiao Pang</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Spatial intelligence is essential for multimodal large language models (MLLMs) operating in the complex physical world. Existing benchmarks, however, probe only single-image relations and thus fail to assess the multi-image spatial reasoning that real-world deployments demand. We introduce MMSI-Bench, a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision researchers spent more than 300 hours meticulously crafting 1,000 challenging, unambiguous multiple-choice questions from over 120,000 images, each paired with carefully designed distractors and a step-by-step reasoning process. We conduct extensive experiments and thoroughly evaluate 34 open-source and proprietary MLLMs, observing a wide gap: the strongest open-source model attains roughly 30% accuracy and OpenAI's o3 reasoning model reaches 40%, while humans score 97%. These results underscore the challenging nature of MMSI-Bench and the substantial headroom for future research. Leveraging the annotated reasoning processes, we also provide an automated error analysis pipeline that diagnoses four dominant failure modes, including (1) grounding errors, (2) overlap-matching and scene-reconstruction errors, (3) situation-transformation reasoning errors, and (4) spatial-logic errors, offering valuable insights for advancing multi-image spatial intelligence. Project page: https://runsenxu.com/projects/MMSI_Bench .</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23764</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:52 +0000</pubDate>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
|
||||
Reference in New Issue
Block a user