bot: update RSS feed
This commit is contained in:
github-actions[bot]
799
feed.xml
799
feed.xml
@@ -7,482 +7,517 @@
|
||||
<docs>http://www.rssboard.org/rss-specification</docs>
|
||||
<generator>python-feedgen</generator>
|
||||
<language>en</language>
|
||||
<lastBuildDate>Wed, 28 May 2025 00:02:42 +0000</lastBuildDate>
|
||||
<lastBuildDate>Thu, 29 May 2025 00:02:35 +0000</lastBuildDate>
|
||||
<item>
|
||||
<title>InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction</title>
|
||||
<link>https://arxiv.org/abs/2505.10887</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.10887.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bin Lei, Weitai Kang, Zijian Zhang, Winson Chen, Xi Xie, Shan Zuo, Mimi Xie, Ali Payani, Mingyi Hong, Yan Yan, Caiwen Ding</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> This paper introduces InfantAgent-Next, a generalist agent capable of interacting with computers in a multimodal manner, encompassing text, images, audio, and video. Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within a highly modular architecture, enabling different models to collaboratively solve decoupled tasks in a step-by-step manner. Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e., OSWorld), but also more general or tool-intensive benchmarks (e.g., GAIA and SWE-Bench). Specifically, we achieve 7.27% accuracy on OSWorld, higher than Claude-Computer-Use. Codes and evaluation scripts are open-sourced at https://github.com/bin123apple/InfantAgent.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.10887</guid>
|
||||
<pubDate>Fri, 16 May 2025 05:43:27 +0000</pubDate>
|
||||
<title>Search and Refine During Think: Autonomous Retrieval-Augmented Reasoning of LLMs</title>
|
||||
<link>https://arxiv.org/abs/2505.11277</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.11277.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yaorui Shi, Shihan Li, Chang Wu, Zhiyuan Liu, Junfeng Fang, Hengxing Cai, An Zhang, Xiang Wang</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir. Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning. In this paper, we propose AutoRefine, a reinforcement learning post-training framework that adopts a new ``search-and-refine-during-think'' paradigm. AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer. Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization. Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.11277</guid>
|
||||
<pubDate>Fri, 16 May 2025 14:11:29 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.12737</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.12737.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongjoon Ahn, Heewoong Choi, Jisu Han, Taesup Moon</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Offline goal-conditioned reinforcement learning (GCRL) offers a practical learning paradigm where goal-reaching policies are trained from abundant unlabeled (reward-free) datasets without additional environment interaction. However, offline GCRL still struggles with long-horizon tasks, even with recent advances that employ hierarchical policy structures, such as HIQL. By identifying the root cause of this challenge, we observe the following insights: First, performance bottlenecks mainly stem from the high-level policy's inability to generate appropriate subgoals. Second, when learning the high-level policy in the long-horizon regime, the sign of the advantage signal frequently becomes incorrect. Thus, we argue that improving the value function to produce a clear advantage signal for learning the high-level policy is essential. In this paper, we propose a simple yet effective solution: Option-aware Temporally Abstracted value learning, dubbed OTA, which incorporates temporal abstraction into the temporal-difference learning process. By modifying the value update to be option-aware, the proposed learning scheme contracts the effective horizon length, enabling better advantage estimates even in long-horizon regimes. We experimentally show that the high-level policy extracted using the OTA value function achieves strong performance on complex tasks from OGBench, a recently proposed offline GCRL benchmark, including maze navigation and visual robotic manipulation environments.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.12737</guid>
|
||||
<pubDate>Mon, 19 May 2025 05:51:11 +0000</pubDate>
|
||||
<title>NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI</title>
|
||||
<link>https://arxiv.org/abs/2505.14064</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.14064.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Cosmin I. Bercea, Jun Li, Philipp Raffler, Evamaria O. Riedel, Lena Schmitzer, Angela Kurz, Felix Bitzer, Paula Roßmüller, Julian Canisius, Mirjam L. Beyrle, Che Liu, Wenjia Bai, Bernhard Kainz, Julia A. Schnabel, Benedikt Wiestler</p><p><b>Upvotes:</b> 16</p><p><b>Summary:</b> In many real-world applications, deployed models encounter inputs that differ from the data seen during training. Out-of-distribution detection identifies whether an input stems from an unseen distribution, while open-world recognition flags such inputs to ensure the system remains robust as ever-emerging, previously unknown categories appear and must be addressed without retraining. Foundation and vision-language models are pre-trained on large and diverse datasets with the expectation of broad generalization across domains, including medical imaging. However, benchmarking these models on test sets with only a few common outlier types silently collapses the evaluation back to a closed-set problem, masking failures on rare or truly novel conditions encountered in clinical use. We therefore present NOVA, a challenging, real-life evaluation-only benchmark of sim900 brain MRI scans that span 281 rare pathologies and heterogeneous acquisition protocols. Each case includes rich clinical narratives and double-blinded expert bounding-box annotations. Together, these enable joint assessment of anomaly localisation, visual captioning, and diagnostic reasoning. Because NOVA is never used for training, it serves as an extreme stress-test of out-of-distribution generalisation: models must bridge a distribution gap both in sample appearance and in semantic space. Baseline results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B) reveal substantial performance drops across all tasks, establishing NOVA as a rigorous testbed for advancing models that can detect, localize, and reason about truly unknown anomalies.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.14064</guid>
|
||||
<pubDate>Tue, 20 May 2025 08:10:57 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>ModernGBERT: German-only 1B Encoder Model Trained from Scratch</title>
|
||||
<link>https://arxiv.org/abs/2505.13136</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.13136.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Anton Ehrmanntraut, Julia Wunderle, Jan Pfister, Fotis Jannidis, Andreas Hotho</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Despite the prominence of decoder-only language models, encoders remain crucial for resource-constrained applications. We introduce ModernGBERT (134M, 1B), a fully transparent family of German encoder models trained from scratch, incorporating architectural innovations from ModernBERT. To evaluate the practical trade-offs of training encoders from scratch, we also present LL\"aMmlein2Vec (120M, 1B, 7B), a family of encoders derived from German decoder-only models via LLM2Vec. We benchmark all models on natural language understanding, text embedding, and long-context reasoning tasks, enabling a controlled comparison between dedicated encoders and converted decoders. Our results show that ModernGBERT 1B outperforms prior state-of-the-art German encoders as well as encoders adapted via LLM2Vec, with regard to performance and parameter-efficiency. All models, training data, checkpoints and code are publicly available, advancing the German NLP ecosystem with transparent, high-performance encoder models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.13136</guid>
|
||||
<pubDate>Mon, 19 May 2025 14:07:20 +0000</pubDate>
|
||||
<title>Do RAG Systems Suffer From Positional Bias?</title>
|
||||
<link>https://arxiv.org/abs/2505.15561</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.15561.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Florin Cuconasu, Simone Filice, Guy Horowitz, Yoelle Maarek, Fabrizio Silvestri</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Retrieval Augmented Generation enhances LLM accuracy by adding passages retrieved from an external corpus to the LLM prompt. This paper investigates how positional bias - the tendency of LLMs to weight information differently based on its position in the prompt - affects not only the LLM's capability to capitalize on relevant passages, but also its susceptibility to distracting passages. Through extensive experiments on three benchmarks, we show how state-of-the-art retrieval pipelines, while attempting to retrieve relevant passages, systematically bring highly distracting ones to the top ranks, with over 60% of queries containing at least one highly distracting passage among the top-10 retrieved passages. As a result, the impact of the LLM positional bias, which in controlled settings is often reported as very prominent by related works, is actually marginal in real scenarios since both relevant and distracting passages are, in turn, penalized. Indeed, our findings reveal that sophisticated strategies that attempt to rearrange the passages based on LLM positional preferences do not perform better than random shuffling.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.15561</guid>
|
||||
<pubDate>Wed, 21 May 2025 14:18:01 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.13426</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.13426.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Liang Chen, Hongcheng Gao, Tianyu Liu, Zhiqi Huang, Flood Sung, Xinyu Zhou, Yuxin Wu, Baobao Chang</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Vision-Language Models (VLMs) excel in many direct multimodal tasks but struggle to translate this prowess into effective decision-making within interactive, visually rich environments like games. This ``knowing-doing'' gap significantly limits their potential as autonomous agents, as leading VLMs often performing badly in simple games. To address this, we introduce VLM-Gym, a curated reinforcement learning (RL) environment featuring diverse visual games with unified interfaces and adjustable, compositional difficulty, specifically designed for scalable multi-game parallel training. Leveraging VLM-Gym, we train G0 models using pure RL-driven self-evolution, which demonstrate emergent perception and reasoning patterns. To further mitigate challenges arising from game diversity, we develop G1 models. G1 incorporates a perception-enhanced cold start prior to RL fine-tuning. Our resulting G1 models consistently surpass their teacher across all games and outperform leading proprietary models like Claude-3.7-Sonnet-Thinking. Systematic analysis reveals an intriguing finding: perception and reasoning abilities mutually bootstrap each other throughout the RL training process. Source code including VLM-Gym and RL training are released at https://github.com/chenllliang/G1 to foster future research in advancing VLMs as capable interactive agents.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.13426</guid>
|
||||
<pubDate>Mon, 19 May 2025 17:54:39 +0000</pubDate>
|
||||
<title>Improving Chemical Understanding of LLMs via SMILES Parsing</title>
|
||||
<link>https://arxiv.org/abs/2505.16340</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16340.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yunhui Jang, Jaehyung Kim, Sungsoo Ahn</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large language models (LLMs) are increasingly recognized as powerful tools for scientific discovery, particularly in molecular science. A fundamental requirement for these models is the ability to accurately understand molecular structures, commonly encoded in the SMILES representation. However, current LLMs struggle to interpret SMILES, even failing to carry out basic tasks such as counting molecular rings. To address this limitation, we introduce CLEANMOL, a novel framework that formulates SMILES parsing into a suite of clean and deterministic tasks explicitly designed to promote graph-level molecular comprehension. These tasks span from subgraph matching to global graph matching, providing structured supervision aligned with molecular structural properties. We construct a molecular pretraining dataset with adaptive difficulty scoring and pre-train open-source LLMs on these tasks. Our results show that CLEANMOL not only enhances structural comprehension but also achieves the best or competes with the baseline on the Mol-Instructions benchmark.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16340</guid>
|
||||
<pubDate>Thu, 22 May 2025 07:54:39 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.14071</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.14071.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Dani Yogatama, Vatsal Sharan, Robin Jia, Willie Neiswanger</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Steering methods have emerged as effective and targeted tools for guiding large language models' (LLMs) behavior without modifying their parameters. Multimodal large language models (MLLMs), however, do not currently enjoy the same suite of techniques, due in part to their recency and architectural diversity. Inspired by this gap, we investigate whether MLLMs can be steered using vectors derived from their text-only LLM backbone, via sparse autoencoders (SAEs), mean shift, and linear probing. We find that text-derived steering consistently enhances multimodal accuracy across diverse MLLM architectures and visual tasks. In particular, mean shift boosts spatial relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to +3.3%, outperforming prompting and exhibiting strong generalization to out-of-distribution datasets. These results highlight textual steering vectors as a powerful, efficient mechanism for enhancing grounding in MLLMs with minimal additional data collection and computational overhead.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.14071</guid>
|
||||
<pubDate>Tue, 20 May 2025 08:23:08 +0000</pubDate>
|
||||
<title>MMMR: Benchmarking Massive Multi-Modal Reasoning Tasks</title>
|
||||
<link>https://arxiv.org/abs/2505.16459</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16459.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, Lichao Sun</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> Recent advances in Multi-Modal Large Language Models (MLLMs) have enabled unified processing of language, vision, and structured inputs, opening the door to complex tasks such as logical deduction, spatial reasoning, and scientific analysis. Despite their promise, the reasoning capabilities of MLLMs, particularly those augmented with intermediate thinking traces (MLLMs-T), remain poorly understood and lack standardized evaluation benchmarks. Existing work focuses primarily on perception or final answer correctness, offering limited insight into how models reason or fail across modalities. To address this gap, we introduce the MMMR, a new benchmark designed to rigorously evaluate multi-modal reasoning with explicit thinking. The MMMR comprises 1) a high-difficulty dataset of 1,083 questions spanning six diverse reasoning types with symbolic depth and multi-hop demands and 2) a modular Reasoning Trace Evaluation Pipeline (RTEP) for assessing reasoning quality beyond accuracy through metrics like relevance, consistency, and structured error annotations. Empirical results show that MLLMs-T overall outperform non-thinking counterparts, but even top models like Claude-3.7-Sonnet and Gemini-2.5 Pro suffer from reasoning pathologies such as inconsistency and overthinking. This benchmark reveals persistent gaps between accuracy and reasoning quality and provides an actionable evaluation pipeline for future model development. Overall, the MMMR offers a scalable foundation for evaluating, comparing, and improving the next generation of multi-modal reasoning systems.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16459</guid>
|
||||
<pubDate>Thu, 22 May 2025 09:41:55 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMs</title>
|
||||
<link>https://arxiv.org/abs/2505.15804</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.15804.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zongzhao Li, Zongyang Ma, Mingze Li, Songyou Li, Yu Rong, Tingyang Xu, Ziqi Zhang, Deli Zhao, Wenbing Huang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across diverse tasks, yet they lag significantly behind humans in spatial reasoning. We investigate this gap through Transformation-Driven Visual Reasoning (TVR), a challenging task requiring identification of object transformations across images under varying viewpoints. While traditional Supervised Fine-Tuning (SFT) fails to generate coherent reasoning paths in cross-view settings, sparse-reward Reinforcement Learning (RL) suffers from inefficient exploration and slow convergence. To address these limitations, we propose STAR-R1, a novel framework that integrates a single-stage RL paradigm with a fine-grained reward mechanism tailored for TVR. Specifically, STAR-R1 rewards partial correctness while penalizing excessive enumeration and passive inaction, enabling efficient exploration and precise reasoning. Comprehensive evaluations demonstrate that STAR-R1 achieves state-of-the-art performance across all 11 metrics, outperforming SFT by 23% in cross-view scenarios. Further analysis reveals STAR-R1's anthropomorphic behavior and highlights its unique ability to compare all objects for improving spatial reasoning. Our work provides critical insights in advancing the research of MLLMs and reasoning models. The codes, model weights, and data will be publicly available at https://github.com/zongzhao23/STAR-R1.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.15804</guid>
|
||||
<pubDate>Wed, 21 May 2025 17:57:38 +0000</pubDate>
|
||||
<title>R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO</title>
|
||||
<link>https://arxiv.org/abs/2505.16673</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16673.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, Jiaxing Huang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an effective approach that mitigates the sparse reward and advantage vanishing issues during RL. To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over six widely-used reasoning benchmarks showcase the superior performance of our method. Code will be available at https://github.com/HJYao00/R1-ShareVL.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16673</guid>
|
||||
<pubDate>Thu, 22 May 2025 13:39:32 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey</title>
|
||||
<link>https://arxiv.org/abs/2505.15957</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.15957.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chih-Kai Yang, Neo S. Ho, Hung-yi Lee</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> With advancements in large audio-language models (LALMs), which enhance large language models (LLMs) with auditory capabilities, these models are expected to demonstrate universal proficiency across various auditory tasks. While numerous benchmarks have emerged to assess LALMs' performance, they remain fragmented and lack a structured taxonomy. To bridge this gap, we conduct a comprehensive survey and propose a systematic taxonomy for LALM evaluations, categorizing them into four dimensions based on their objectives: (1) General Auditory Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented Ability, and (4) Fairness, Safety, and Trustworthiness. We provide detailed overviews within each category and highlight challenges in this field, offering insights into promising future directions. To the best of our knowledge, this is the first survey specifically focused on the evaluations of LALMs, providing clear guidelines for the community. We will release the collection of the surveyed papers and actively maintain it to support ongoing advancements in the field.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.15957</guid>
|
||||
<pubDate>Wed, 21 May 2025 19:17:29 +0000</pubDate>
|
||||
<title>Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks</title>
|
||||
<link>https://arxiv.org/abs/2505.16901</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16901.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongyuan Tao, Ying Zhang, Zhenhao Tang, Hongen Peng, Xukun Zhu, Bingchang Liu, Yingguang Yang, Ziyin Zhang, Zhaogui Xu, Haipeng Zhang, Linchao Zhu, Rui Wang, Hang Yu, Jianguo Li, Peng Di</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Recent advances in Large Language Models (LLMs) have shown promise in function-level code generation, yet repository-level software engineering tasks remain challenging. Current solutions predominantly rely on proprietary LLM agents, which introduce unpredictability and limit accessibility, raising concerns about data privacy and model customization. This paper investigates whether open-source LLMs can effectively address repository-level tasks without requiring agent-based approaches. We demonstrate this is possible by enabling LLMs to comprehend functions and files within codebases through their semantic information and structural dependencies. To this end, we introduce Code Graph Models (CGMs), which integrate repository code graph structures into the LLM's attention mechanism and map node attributes to the LLM's input space using a specialized adapter. When combined with an agentless graph RAG framework, our approach achieves a 43.00% resolution rate on the SWE-bench Lite benchmark using the open-source Qwen2.5-72B model. This performance ranks first among open weight models, second among methods with open-source systems, and eighth overall, surpassing the previous best open-source model-based method by 12.33%.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16901</guid>
|
||||
<pubDate>Thu, 22 May 2025 17:00:55 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning</title>
|
||||
<link>https://arxiv.org/abs/2505.16312</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16312.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiawei Liu, Qisi Chen, Jianshu Zhang, Quan Liu, Defu Lian</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Large Language Models (LLMs) excel at complex reasoning through search algorithms, yet current strategies often suffer from massive token consumption due to redundant exploration of semantically equivalent steps. Existing semantic similarity methods struggle to accurately identify such equivalence in domain-specific contexts like mathematical reasoning. To address this, we propose EquivPruner, a simple yet effective approach that identifies and prunes semantically equivalent actions during LLM reasoning search. We also introduce MathEquiv, the first dataset we created for mathematical statement equivalence, which enables the training of a lightweight equivalence detector. Extensive experiments across various models and tasks demonstrate that EquivPruner significantly reduces token consumption, improving searching efficiency and often bolstering reasoning accuracy. For instance, when applied to Qwen2.5-Math-7B-Instruct on GSM8K, EquivPruner reduced token consumption by 48.1\% while also improving accuracy. Our code is available at https://github.com/Lolo1222/EquivPruner.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16312</guid>
|
||||
<pubDate>Thu, 22 May 2025 07:07:43 +0000</pubDate>
|
||||
<title>R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.17005</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17005.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the internal knowledge of the model. In this paper, we introduce R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. R1-Searcher++ employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model's internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++ outperforms previous RAG and reasoning methods and achieves efficient retrieval. The code is available at https://github.com/RUCAIBox/R1-Searcher-plus.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17005</guid>
|
||||
<pubDate>Thu, 22 May 2025 17:58:26 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Embodied Agents Meet Personalization: Exploring Memory Utilization for Personalized Assistance</title>
|
||||
<link>https://arxiv.org/abs/2505.16348</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16348.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Taeyoon Kwon, Dongwook Choi, Sunghwan Kim, Hyojun Kim, Seungjun Moon, Beong-woo Kwak, Kuan-Hao Huang, Jinyoung Yeo</p><p><b>Upvotes:</b> 42</p><p><b>Summary:</b> Embodied agents empowered by large language models (LLMs) have shown strong performance in household object rearrangement tasks. However, these tasks primarily focus on single-turn interactions with simplified instructions, which do not truly reflect the challenges of providing meaningful assistance to users. To provide personalized assistance, embodied agents must understand the unique semantics that users assign to the physical world (e.g., favorite cup, breakfast routine) by leveraging prior interaction history to interpret dynamic, real-world instructions. Yet, the effectiveness of embodied agents in utilizing memory for personalized assistance remains largely underexplored. To address this gap, we present MEMENTO, a personalized embodied agent evaluation framework designed to comprehensively assess memory utilization capabilities to provide personalized assistance. Our framework consists of a two-stage memory evaluation process design that enables quantifying the impact of memory utilization on task performance. This process enables the evaluation of agents' understanding of personalized knowledge in object rearrangement tasks by focusing on its role in goal interpretation: (1) the ability to identify target objects based on personal meaning (object semantics), and (2) the ability to infer object-location configurations from consistent user patterns, such as routines (user patterns). Our experiments across various LLMs reveal significant limitations in memory utilization, with even frontier models like GPT-4o experiencing a 30.5% performance drop when required to reference multiple memories, particularly in tasks involving user patterns. These findings, along with our detailed analyses and case studies, provide valuable insights for future research in developing more effective personalized embodied agents. Project website: https://connoriginal.github.io/MEMENTO</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16348</guid>
|
||||
<pubDate>Thu, 22 May 2025 08:00:10 +0000</pubDate>
|
||||
<title>Tropical Attention: Neural Algorithmic Reasoning for Combinatorial Algorithms</title>
|
||||
<link>https://arxiv.org/abs/2505.17190</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17190.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Baran Hashemi, Kurt Pasque, Chris Teska, Ruriko Yoshida</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Dynamic programming (DP) algorithms for combinatorial optimization problems work with taking maximization, minimization, and classical addition in their recursion algorithms. The associated value functions correspond to convex polyhedra in the max plus semiring. Existing Neural Algorithmic Reasoning models, however, rely on softmax-normalized dot-product attention where the smooth exponential weighting blurs these sharp polyhedral structures and collapses when evaluated on out-of-distribution (OOD) settings. We introduce Tropical attention, a novel attention function that operates natively in the max-plus semiring of tropical geometry. We prove that Tropical attention can approximate tropical circuits of DP-type combinatorial algorithms. We then propose that using Tropical transformers enhances empirical OOD performance in both length generalization and value generalization, on algorithmic reasoning tasks, surpassing softmax baselines while remaining stable under adversarial attacks. We also present adversarial-attack generalization as a third axis for Neural Algorithmic Reasoning benchmarking. Our results demonstrate that Tropical attention restores the sharp, scale-invariant reasoning absent from softmax.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17190</guid>
|
||||
<pubDate>Thu, 22 May 2025 18:01:25 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Don't "Overthink" Passage Reranking: Is Reasoning Truly Necessary?</title>
|
||||
<link>https://arxiv.org/abs/2505.16886</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16886.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nour Jedidi, Yung-Sung Chuang, James Glass, Jimmy Lin</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> With the growing success of reasoning models across complex natural language tasks, researchers in the Information Retrieval (IR) community have begun exploring how similar reasoning capabilities can be integrated into passage rerankers built on Large Language Models (LLMs). These methods typically employ an LLM to produce an explicit, step-by-step reasoning process before arriving at a final relevance prediction. But, does reasoning actually improve reranking accuracy? In this paper, we dive deeper into this question, studying the impact of the reasoning process by comparing reasoning-based pointwise rerankers (ReasonRR) to standard, non-reasoning pointwise rerankers (StandardRR) under identical training conditions, and observe that StandardRR generally outperforms ReasonRR. Building on this observation, we then study the importance of reasoning to ReasonRR by disabling its reasoning process (ReasonRR-NoReason), and find that ReasonRR-NoReason is surprisingly more effective than ReasonRR. Examining the cause of this result, our findings reveal that reasoning-based rerankers are limited by the LLM's reasoning process, which pushes it toward polarized relevance scores and thus fails to consider the partial relevance of passages, a key factor for the accuracy of pointwise rerankers.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16886</guid>
|
||||
<pubDate>Thu, 22 May 2025 16:41:37 +0000</pubDate>
|
||||
<title>SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use</title>
|
||||
<link>https://arxiv.org/abs/2505.17332</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17332.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hitesh Laxmichand Patel, Amit Agarwal, Arion Das, Bhargava Kumar, Srikant Panda, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, Dong-Kyu Chae</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Enterprise customers are increasingly adopting Large Language Models (LLMs) for critical communication tasks, such as drafting emails, crafting sales pitches, and composing casual messages. Deploying such models across different regions requires them to understand diverse cultural and linguistic contexts and generate safe and respectful responses. For enterprise applications, it is crucial to mitigate reputational risks, maintain trust, and ensure compliance by effectively identifying and handling unsafe or offensive language. To address this, we introduce SweEval, a benchmark simulating real-world scenarios with variations in tone (positive or negative) and context (formal or informal). The prompts explicitly instruct the model to include specific swear words while completing the task. This benchmark evaluates whether LLMs comply with or resist such inappropriate instructions and assesses their alignment with ethical frameworks, cultural nuances, and language comprehension capabilities. In order to advance research in building ethically aligned AI systems for enterprise use and beyond, we release the dataset and code: https://github.com/amitbcp/multilingual_profanity.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17332</guid>
|
||||
<pubDate>Thu, 22 May 2025 22:56:58 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition</title>
|
||||
<link>https://arxiv.org/abs/2505.16972</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16972.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tianduo Wang, Lu Xu, Wei Lu, Shanbo Cheng</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> Recent advances in Automatic Speech Recognition (ASR) have been largely fueled by massive speech corpora. However, extending coverage to diverse languages with limited resources remains a formidable challenge. This paper introduces Speech Back-Translation, a scalable pipeline that improves multilingual ASR models by converting large-scale text corpora into synthetic speech via off-the-shelf text-to-speech (TTS) models. We demonstrate that just tens of hours of real transcribed speech can effectively train TTS models to generate synthetic speech at hundreds of times the original volume while maintaining high quality. To evaluate synthetic speech quality, we develop an intelligibility-based assessment framework and establish clear thresholds for when synthetic data benefits ASR training. Using Speech Back-Translation, we generate more than 500,000 hours of synthetic speech in ten languages and continue pre-training Whisper-large-v3, achieving average transcription error reductions of over 30\%. These results highlight the scalability and effectiveness of Speech Back-Translation for enhancing multilingual ASR systems.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16972</guid>
|
||||
<pubDate>Thu, 22 May 2025 17:51:05 +0000</pubDate>
|
||||
<title>MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask Multimodal Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.17613</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17613.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jihan Yao, Yushi Hu, Yujie Yi, Bin Han, Shangbin Feng, Guang Yang, Bingbing Wen, Ranjay Krishna, Lucy Lu Wang, Yulia Tsvetkov, Noah A. Smith, Banghua Zhu</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align reliably with human evaluation, especially for complex tasks that involve multiple modalities. To address this, we present MMMG, a comprehensive and human-aligned benchmark for multimodal generation across 4 modality combinations (image, audio, interleaved text and image, interleaved text and audio), with a focus on tasks that present significant challenges for generation models, while still enabling reliable automatic evaluation through a combination of models and programs. MMMG encompasses 49 tasks (including 29 newly developed ones), each with a carefully designed evaluation pipeline, and 937 instructions to systematically assess reasoning, controllability, and other key capabilities of multimodal generation models. Extensive validation demonstrates that MMMG is highly aligned with human evaluation, achieving an average agreement of 94.3%. Benchmarking results on 24 multimodal generation models reveal that even though the state-of-the-art model, GPT Image, achieves 78.3% accuracy for image generation, it falls short on multimodal reasoning and interleaved generation. Furthermore, results suggest considerable headroom for improvement in audio generation, highlighting an important direction for future research.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17613</guid>
|
||||
<pubDate>Fri, 23 May 2025 08:21:28 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>UFT: Unifying Supervised and Reinforcement Fine-Tuning</title>
|
||||
<link>https://arxiv.org/abs/2505.16984</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.16984.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mingyang Liu, Gabriele Farina, Asuman Ozdaglar</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Post-training has demonstrated its importance in enhancing the reasoning capabilities of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but it may lead to overfitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorporating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless of model sizes. Furthermore, we theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-horizon reasoning tasks.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.16984</guid>
|
||||
<pubDate>Thu, 22 May 2025 17:53:57 +0000</pubDate>
|
||||
<title>PreMoe: Lightening MoEs on Constrained Memory by Expert Pruning and Retrieval</title>
|
||||
<link>https://arxiv.org/abs/2505.17639</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17639.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zehua Pei, Ying Zhang, Hui-Ling Zhen, Xianzhi Yu, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Mixture-of-experts (MoE) architectures enable scaling large language models (LLMs) to vast parameter counts without a proportional rise in computational costs. However, the significant memory demands of large MoE models hinder their deployment across various computational environments, from cloud servers to consumer devices. This study first demonstrates pronounced task-specific specialization in expert activation patterns within MoE layers. Building on this, we introduce PreMoe, a novel framework that enables efficient deployment of massive MoE models in memory-constrained environments. PreMoe features two main components: probabilistic expert pruning (PEP) and task-adaptive expert retrieval (TAER). PEP employs a new metric, the task-conditioned expected selection score (TCESS), derived from router logits to quantify expert importance for specific tasks, thereby identifying a minimal set of critical experts. TAER leverages these task-specific expert importance profiles for efficient inference. It pre-computes and stores compact expert patterns for diverse tasks. When a user query is received, TAER rapidly identifies the most relevant stored task pattern and reconstructs the model by loading only the small subset of experts crucial for that task. This approach dramatically reduces the memory footprint across all deployment scenarios. DeepSeek-R1 671B maintains 97.2\% accuracy on MATH500 when pruned to 8/128 configuration (50\% expert reduction), and still achieves 72.0\% with aggressive 8/32 pruning (87.5\% expert reduction). Pangu-Ultra-MoE 718B achieves 97.15\% on MATH500 and 81.3\% on AIME24 with 8/128 pruning, while even more aggressive pruning to 4/64 (390GB memory) preserves 96.95\% accuracy on MATH500. We make our code publicly available at https://github.com/JarvisPei/PreMoe.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17639</guid>
|
||||
<pubDate>Fri, 23 May 2025 08:59:16 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective</title>
|
||||
<link>https://arxiv.org/abs/2505.17652</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17652.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Deyang Kong, Qi Guo, Xiangyu Xi, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Reinforcement learning exhibits potential in enhancing the reasoning abilities of large language models, yet it is hard to scale for the low sample efficiency during the rollout phase. Existing methods attempt to improve efficiency by scheduling problems based on problem difficulties. However, these approaches suffer from unstable and biased estimations of problem difficulty and fail to capture the alignment between model competence and problem difficulty in RL training, leading to suboptimal results. To tackle these limitations, this paper introduces Competence-Difficulty Alignment Sampling (CDAS), which enables accurate and stable estimation of problem difficulties by aggregating historical performance discrepancies of problems. Then the model competence is quantified to adaptively select problems whose difficulty is in alignment with the model's current competence using a fixed-point system. Experimental results across a range of challenging mathematical benchmarks show that CDAS achieves great improvements in both accuracy and efficiency. CDAS attains the highest average accuracy against baselines and exhibits significant speed advantages compared to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33 times slower than CDAS.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17652</guid>
|
||||
<pubDate>Fri, 23 May 2025 09:15:26 +0000</pubDate>
|
||||
<title>Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.17813</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17813.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Michael Hassid, Gabriel Synnaeve, Yossi Adi, Roy Schwartz</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> Reasoning large language models (LLMs) heavily rely on scaling test-time compute to perform complex reasoning tasks by generating extensive "thinking" chains. While demonstrating impressive results, this approach incurs significant computational costs and inference time. In this work, we challenge the assumption that long thinking chains results in better reasoning capabilities. We first demonstrate that shorter reasoning chains within individual questions are significantly more likely to yield correct answers - up to 34.5% more accurate than the longest chain sampled for the same question. Based on these results, we suggest short-m@k, a novel reasoning LLM inference method. Our method executes k independent generations in parallel and halts computation once the first m thinking processes are done. The final answer is chosen using majority voting among these m chains. Basic short-1@k demonstrates similar or even superior performance over standard majority voting in low-compute settings - using up to 40% fewer thinking tokens. short-3@k, while slightly less efficient than short-1@k, consistently surpasses majority voting across all compute budgets, while still being substantially faster (up to 33% wall time reduction). Inspired by our results, we finetune an LLM using short, long, and randomly selected reasoning chains. We then observe that training on the shorter ones leads to better performance. Our findings suggest rethinking current methods of test-time compute in reasoning LLMs, emphasizing that longer "thinking" does not necessarily translate to improved performance and can, counter-intuitively, lead to degraded results.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17813</guid>
|
||||
<pubDate>Fri, 23 May 2025 12:29:06 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Mutarjim: Advancing Bidirectional Arabic-English Translation with a Small Language Model</title>
|
||||
<link>https://arxiv.org/abs/2505.17894</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17894.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Khalil Hennara, Muhammad Hreden, Mohamed Motaism Hamed, Zeina Aldallal, Sara Chrouf, Safwan AlModhayan</p><p><b>Upvotes:</b> 174</p><p><b>Summary:</b> We introduce Mutarjim, a compact yet powerful language model for bidirectional Arabic-English translation. While large-scale LLMs have shown impressive progress in natural language processing tasks, including machine translation, smaller models. Leveraging this insight, we developed Mutarjim based on Kuwain-1.5B , a language model tailored for both Arabic and English. Despite its modest size, Mutarjim outperforms much larger models on several established benchmarks, achieved through an optimized two-phase training approach and a carefully curated, high-quality training corpus.. Experimental results show that Mutarjim rivals models up to 20 times larger while significantly reducing computational costs and training requirements. We also introduce Tarjama-25, a new benchmark designed to overcome limitations in existing Arabic-English benchmarking datasets, such as domain narrowness, short sentence lengths, and English-source bias. Tarjama-25 comprises 5,000 expert-reviewed sentence pairs and spans a wide range of domains, offering a more comprehensive and balanced evaluation framework. Notably, Mutarjim achieves state-of-the-art performance on the English-to-Arabic task in Tarjama-25, surpassing even significantly larger and proprietary models like GPT-4o mini. We publicly release Tarjama-25 to support future research and advance the evaluation of Arabic-English translation systems.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17894</guid>
|
||||
<pubDate>Fri, 23 May 2025 13:42:21 +0000</pubDate>
|
||||
<title>Explaining Sources of Uncertainty in Automated Fact-Checking</title>
|
||||
<link>https://arxiv.org/abs/2505.17855</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17855.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingyi Sun, Greta Warren, Irina Shklovski, Isabelle Augenstein</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Understanding sources of a model's uncertainty regarding its predictions is crucial for effective human-AI collaboration. Prior work proposes using numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain uncertainty that arises from conflicting evidence, leaving users unable to resolve disagreements or rely on the output. We introduce CLUE (Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the first framework to generate natural language explanations of model uncertainty by (i) identifying relationships between spans of text that expose claim-evidence or inter-evidence conflicts and agreements that drive the model's predictive uncertainty in an unsupervised way, and (ii) generating explanations via prompting and attention steering that verbalize these critical interactions. Across three language models and two fact-checking datasets, we show that CLUE produces explanations that are more faithful to the model's uncertainty and more consistent with fact-checking decisions than prompting for uncertainty explanations without span-interaction guidance. Human evaluators judge our explanations to be more helpful, more informative, less redundant, and more logically consistent with the input than this baseline. CLUE requires no fine-tuning or architectural changes, making it plug-and-play for any white-box language model. By explicitly linking uncertainty to evidence conflicts, it offers practical support for fact-checking and generalises readily to other tasks that require reasoning over complex information.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17855</guid>
|
||||
<pubDate>Fri, 23 May 2025 13:06:43 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Bridging Supervised Learning and Reinforcement Learning in Math Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.18116</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18116.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Yin Cui, Haotian Ye, Tsung-Yi Lin, Ming-Yu Liu, Jun Zhu, Haoxiang Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Reinforcement Learning (RL) has played a central role in the recent surge of LLMs' math abilities by enabling self-improvement through binary verifier signals. In contrast, Supervised Learning (SL) is rarely considered for such verification-driven training, largely due to its heavy reliance on reference answers and inability to reflect on mistakes. In this work, we challenge the prevailing notion that self-improvement is exclusive to RL and propose Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to reflect on their failures and improve autonomously with no external teachers. In online training, instead of throwing away self-generated negative answers, NFT constructs an implicit negative policy to model them. This implicit policy is parameterized with the same positive LLM we target to optimize on positive data, enabling direct policy optimization on all LLMs' generations. We conduct experiments on 7B and 32B models in math reasoning tasks. Results consistently show that through the additional leverage of negative feedback, NFT significantly improves over SL baselines like Rejection sampling Fine-Tuning, matching or even surpassing leading RL algorithms like GRPO and DAPO. Furthermore, we demonstrate that NFT and GRPO are actually equivalent in strict-on-policy training, even though they originate from entirely different theoretical foundations. Our experiments and theoretical findings bridge the gap between SL and RL methods in binary-feedback learning systems.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18116</guid>
|
||||
<pubDate>Fri, 23 May 2025 17:17:40 +0000</pubDate>
|
||||
<title>ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback</title>
|
||||
<link>https://arxiv.org/abs/2505.17908</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17908.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Litao Guo, Xinli Xu, Luozhou Wang, Jiantao Lin, Jinsong Zhou, Zixin Zhang, Bolan Su, Ying-Cong Chen</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> With the rapid advancement of generative models, general-purpose generation has gained increasing attention as a promising approach to unify diverse tasks across modalities within a single system. Despite this progress, existing open-source frameworks often remain fragile and struggle to support complex real-world applications due to the lack of structured workflow planning and execution-level feedback. To address these limitations, we present ComfyMind, a collaborative AI system designed to enable robust and scalable general-purpose generation, built on the ComfyUI platform. ComfyMind introduces two core innovations: Semantic Workflow Interface (SWI) that abstracts low-level node graphs into callable functional modules described in natural language, enabling high-level composition and reducing structural errors; Search Tree Planning mechanism with localized feedback execution, which models generation as a hierarchical decision process and allows adaptive correction at each stage. Together, these components improve the stability and flexibility of complex generative workflows. We evaluate ComfyMind on three public benchmarks: ComfyBench, GenEval, and Reason-Edit, which span generation, editing, and reasoning tasks. Results show that ComfyMind consistently outperforms existing open-source baselines and achieves performance comparable to GPT-Image-1. ComfyMind paves a promising path for the development of open-source general-purpose generative AI systems. Project page: https://github.com/LitaoGuo/ComfyMind</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17908</guid>
|
||||
<pubDate>Fri, 23 May 2025 13:53:03 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification</title>
|
||||
<link>https://arxiv.org/abs/2505.18283</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18283.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jianghao Wu, Feilong Tang, Yulong Li, Ming Hu, Haochen Xue, Shoaib Jameel, Yutong Xie, Imran Razzak</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent advances such as Chain-of-Thought prompting have significantly improved large language models (LLMs) in zero-shot medical reasoning. However, prompting-based methods often remain shallow and unstable, while fine-tuned medical LLMs suffer from poor generalization under distribution shifts and limited adaptability to unseen clinical scenarios. To address these limitations, we present TAGS, a test-time framework that combines a broadly capable generalist with a domain-specific specialist to offer complementary perspectives without any model fine-tuning or parameter updates. To support this generalist-specialist reasoning process, we introduce two auxiliary modules: a hierarchical retrieval mechanism that provides multi-scale exemplars by selecting examples based on both semantic and rationale-level similarity, and a reliability scorer that evaluates reasoning consistency to guide final answer aggregation. TAGS achieves strong performance across nine MedQA benchmarks, boosting GPT-4o accuracy by 13.8%, DeepSeek-R1 by 16.8%, and improving a vanilla 7B model from 14.1% to 23.9%. These results surpass several fine-tuned medical LLMs, without any parameter updates. The code will be available at https://github.com/JianghaoWu/TAGS.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18283</guid>
|
||||
<pubDate>Fri, 23 May 2025 18:28:59 +0000</pubDate>
|
||||
<title>Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL</title>
|
||||
<link>https://arxiv.org/abs/2505.17952</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.17952.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert, Rossella Arcucci</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Improving performance on complex tasks and enabling interpretable decision making in large language models (LLMs), especially for clinical applications, requires effective reasoning. Yet this remains challenging without supervised fine-tuning (SFT) on costly chain-of-thought (CoT) data distilled from closed-source models (e.g., GPT-4o). In this work, we present AlphaMed, the first medical LLM to show that reasoning capability can emerge purely through reinforcement learning (RL), using minimalist rule-based rewards on public multiple-choice QA datasets, without relying on SFT or distilled CoT data. AlphaMed achieves state-of-the-art results on six medical QA benchmarks, outperforming models trained with conventional SFT+RL pipelines. On challenging benchmarks (e.g., MedXpert), AlphaMed even surpasses larger or closed-source models such as DeepSeek-V3-671B and Claude-3.5-Sonnet. To understand the factors behind this success, we conduct a comprehensive data-centric analysis guided by three questions: (i) Can minimalist rule-based RL incentivize reasoning without distilled CoT supervision? (ii) How do dataset quantity and diversity impact reasoning? (iii) How does question difficulty shape the emergence and generalization of reasoning? Our findings show that dataset informativeness is a key driver of reasoning performance, and that minimalist RL on informative, multiple-choice QA data is effective at inducing reasoning without CoT supervision. We also observe divergent trends across benchmarks, underscoring limitations in current evaluation and the need for more challenging, reasoning-oriented medical QA benchmarks.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.17952</guid>
|
||||
<pubDate>Fri, 23 May 2025 14:27:37 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.18291</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18291.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zifu Wan, Yaqi Xie, Ce Zhang, Zhiqiu Lin, Zihan Wang, Simon Stepputtis, Deva Ramanan, Katia Sycara</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large multimodal foundation models, particularly in the domains of language and vision, have significantly advanced various tasks, including robotics, autonomous driving, information retrieval, and grounding. However, many of these models perceive objects as indivisible, overlooking the components that constitute them. Understanding these components and their associated affordances provides valuable insights into an object's functionality, which is fundamental for performing a wide range of tasks. In this work, we introduce a novel real-world benchmark, InstructPart, comprising hand-labeled part segmentation annotations and task-oriented instructions to evaluate the performance of current models in understanding and executing part-level tasks within everyday contexts. Through our experiments, we demonstrate that task-oriented part segmentation remains a challenging problem, even for state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark, we introduce a simple baseline that achieves a twofold performance improvement through fine-tuning with our dataset. With our dataset and benchmark, we aim to facilitate research on task-oriented part segmentation and enhance the applicability of VLMs across various domains, including robotics, virtual reality, information retrieval, and other related fields. Project website: https://zifuwan.github.io/InstructPart/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18291</guid>
|
||||
<pubDate>Fri, 23 May 2025 18:36:13 +0000</pubDate>
|
||||
<title>VideoGameBench: Can Vision-Language Models complete popular video games?</title>
|
||||
<link>https://arxiv.org/abs/2505.18134</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18134.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, Ofir Press</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Vision-language models (VLMs) have achieved strong results on coding and math benchmarks that are challenging for humans, yet their ability to perform tasks that come naturally to humans--such as perception, spatial navigation, and memory management--remains understudied. Real video games are crafted to be intuitive for humans to learn and master by leveraging innate inductive biases, making them an ideal testbed for evaluating such capabilities in VLMs. To this end, we introduce VideoGameBench, a benchmark consisting of 10 popular video games from the 1990s that VLMs directly interact with in real-time. VideoGameBench challenges models to complete entire games with access to only raw visual inputs and a high-level description of objectives and controls, a significant departure from existing setups that rely on game-specific scaffolding and auxiliary information. We keep three of the games secret to encourage solutions that generalize to unseen environments. Our experiments show that frontier vision-language models struggle to progress beyond the beginning of each game. We find inference latency to be a major limitation of frontier models in the real-time setting; therefore, we introduce VideoGameBench Lite, a setting where the game pauses while waiting for the LM's next action. The best performing model, Gemini 2.5 Pro, completes only 0.48% of VideoGameBench and 1.6% of VideoGameBench Lite. We hope that the formalization of the human skills mentioned above into this benchmark motivates progress in these research directions.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18134</guid>
|
||||
<pubDate>Fri, 23 May 2025 17:43:27 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Architectural Backdoors for Within-Batch Data Stealing and Model Inference Manipulation</title>
|
||||
<link>https://arxiv.org/abs/2505.18323</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18323.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nicolas Küchler, Ivan Petrov, Conrad Grobler, Ilia Shumailov</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> For nearly a decade the academic community has investigated backdoors in neural networks, primarily focusing on classification tasks where adversaries manipulate the model prediction. While demonstrably malicious, the immediate real-world impact of such prediction-altering attacks has remained unclear. In this paper we introduce a novel and significantly more potent class of backdoors that builds upon recent advancements in architectural backdoors. We demonstrate how these backdoors can be specifically engineered to exploit batched inference, a common technique for hardware utilization, enabling large-scale user data manipulation and theft. By targeting the batching process, these architectural backdoors facilitate information leakage between concurrent user requests and allow attackers to fully control model responses directed at other users within the same batch. In other words, an attacker who can change the model architecture can set and steal model inputs and outputs of other users within the same batch. We show that such attacks are not only feasible but also alarmingly effective, can be readily injected into prevalent model architectures, and represent a truly malicious threat to user privacy and system integrity. Critically, to counteract this new class of vulnerabilities, we propose a deterministic mitigation strategy that provides formal guarantees against this new attack vector, unlike prior work that relied on Large Language Models to find the backdoors. Our mitigation strategy employs a novel Information Flow Control mechanism that analyzes the model graph and proves non-interference between different user inputs within the same batch. Using our mitigation strategy we perform a large scale analysis of models hosted through Hugging Face and find over 200 models that introduce (unintended) information leakage between batch entries due to the use of dynamic quantization.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18323</guid>
|
||||
<pubDate>Fri, 23 May 2025 19:28:45 +0000</pubDate>
|
||||
<title>BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases</title>
|
||||
<link>https://arxiv.org/abs/2505.20321</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20321.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mathew J. Koretsky, Maya Willey, Adi Asija, Owen Bianchi, Chelsea X. Alvarado, Tanay Nayak, Nicole Kuznetsov, Sungwon Kim, Mike A. Nalls, Daniel Khashabi, Faraz Faghri</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Biomedical researchers increasingly rely on large-scale structured databases for complex analytical tasks. However, current text-to-SQL systems often struggle to map qualitative scientific questions into executable SQL, particularly when implicit domain reasoning is required. We introduce BiomedSQL, the first benchmark explicitly designed to evaluate scientific reasoning in text-to-SQL generation over a real-world biomedical knowledge base. BiomedSQL comprises 68,000 question/SQL query/answer triples grounded in a harmonized BigQuery knowledge base that integrates gene-disease associations, causal inference from omics data, and drug approval records. Each question requires models to infer domain-specific criteria, such as genome-wide significance thresholds, effect directionality, or trial phase filtering, rather than rely on syntactic translation alone. We evaluate a range of open- and closed-source LLMs across prompting strategies and interaction paradigms. Our results reveal a substantial performance gap: GPT-o3-mini achieves 59.0% execution accuracy, while our custom multi-step agent, BMSQL, reaches 62.6%, both well below the expert baseline of 90.0%. BiomedSQL provides a new foundation for advancing text-to-SQL systems capable of supporting scientific discovery through robust reasoning over structured biomedical knowledge bases. Our dataset is publicly available at https://huggingface.co/datasets/NIH-CARD/BiomedSQL, and our code is open-source at https://github.com/NIH-CARD/biomedsql.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20321</guid>
|
||||
<pubDate>Fri, 23 May 2025 17:58:07 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Dynamic Risk Assessments for Offensive Cybersecurity Agents</title>
|
||||
<link>https://arxiv.org/abs/2505.18384</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18384.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Boyi Wei, Benedikt Stroebl, Jiacen Xu, Joie Zhang, Zhou Li, Peter Henderson</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Foundation models are increasingly becoming better autonomous programmers, raising the prospect that they could also automate dangerous offensive cyber-operations. Current frontier model audits probe the cybersecurity risks of such agents, but most fail to account for the degrees of freedom available to adversaries in the real world. In particular, with strong verifiers and financial incentives, agents for offensive cybersecurity are amenable to iterative improvement by would-be adversaries. We argue that assessments should take into account an expanded threat model in the context of cybersecurity, emphasizing the varying degrees of freedom that an adversary may possess in stateful and non-stateful environments within a fixed compute budget. We show that even with a relatively small compute budget (8 H100 GPU Hours in our study), adversaries can improve an agent's cybersecurity capability on InterCode CTF by more than 40\% relative to the baseline -- without any external assistance. These results highlight the need to evaluate agents' cybersecurity risk in a dynamic manner, painting a more representative picture of risk.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18384</guid>
|
||||
<pubDate>Fri, 23 May 2025 21:18:59 +0000</pubDate>
|
||||
<title>Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms</title>
|
||||
<link>https://arxiv.org/abs/2505.20322</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20322.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mengru Wang, Ziwen Xu, Shengyu Mao, Shumin Deng, Zhaopeng Tu, Huajun Chen, Ningyu Zhang</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Precise control over language model generation is vital for ensuring both safety and reliability. Although prompt engineering and steering are commonly used to intervene in model behaviors, the vast number of parameters in models often results in highly intertwined internal representations. This interdependency can limit control precision and sometimes lead to unintended side effects. Recent research has explored the use of sparse autoencoders (SAE) to disentangle knowledge in high-dimensional spaces for steering. However, these applications have been limited to toy tasks owing to the nontrivial issue of locating atomic knowledge components. In this paper, we propose Steering Target Atoms (STA), a novel method that isolates and manipulates disentangled knowledge components to enhance safety. Comprehensive experiments demonstrate the effectiveness of our approach. Further analysis reveals that steering exhibits superior robustness and flexibility, particularly in adversarial scenarios. We also apply the steering strategy to the large reasoning model, confirming its effectiveness in precise reasoning control.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20322</guid>
|
||||
<pubDate>Fri, 23 May 2025 17:59:18 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Hybrid Latent Reasoning via Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.18454</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18454.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhenrui Yue, Bowen Jin, Huimin Zeng, Honglei Zhuang, Zhen Qin, Jinsung Yoon, Lanyu Shang, Jiawei Han, Dong Wang</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Recent advances in large language models (LLMs) have introduced latent reasoning as a promising alternative to autoregressive reasoning. By performing internal computation with hidden states from previous steps, latent reasoning benefit from more informative features rather than sampling a discrete chain-of-thought (CoT) path. Yet latent reasoning approaches are often incompatible with LLMs, as their continuous paradigm conflicts with the discrete nature of autoregressive generation. Moreover, these methods rely on CoT traces for training and thus fail to exploit the inherent reasoning patterns of LLMs. In this work, we explore latent reasoning by leveraging the intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid latent reasoning approach that (1) integrates prior hidden states into sampled tokens with a learnable gating mechanism, and (2) initializes training with predominantly token embeddings while progressively incorporating more hidden features. This design maintains LLMs' generative capabilities and incentivizes hybrid reasoning using both discrete and continuous representations. In addition, the hybrid HRPO introduces stochasticity into latent reasoning via token sampling, thereby enabling RL-based optimization without requiring CoT trajectories. Extensive evaluations across diverse benchmarks show that HRPO outperforms prior methods in both knowledge- and reasoning-intensive tasks. Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing behaviors like cross-lingual patterns and shorter completion lengths, highlighting the potential of our RL-based approach and offer insights for future work in latent reasoning.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18454</guid>
|
||||
<pubDate>Sat, 24 May 2025 01:26:16 +0000</pubDate>
|
||||
<title>Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidence</title>
|
||||
<link>https://arxiv.org/abs/2505.20325</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20325.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Amirhosein Ghasemabadi, Keith G. Mills, Baochun Li, Di Niu</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> Test-Time Scaling (TTS) methods for enhancing Large Language Model (LLM) reasoning often incur substantial computational costs, primarily due to extensive reliance on external Process Reward Models (PRMs) or sampling methods like Best-of-N (BoN). This paper introduces Guided by Gut (GG), an efficient self-guided TTS framework that achieves PRM-level performance without costly external verifier models. Our method employs a lightweight tree search guided solely by intrinsic LLM signals, token-level confidence and step novelty. One critical innovation is improving the reliability of internal confidence estimates via a targeted reinforcement learning fine-tuning phase. Empirical evaluations on challenging mathematical reasoning benchmarks demonstrate that GG enables smaller models (e.g., 1.5B parameters) to achieve accuracy matching or surpassing significantly larger models (e.g., 32B-70B parameters), while reducing GPU memory usage by up to 10x. Compared to PRM-based methods, GG achieves comparable accuracy with 8x faster inference speeds and 4-5x lower memory usage. Additionally, GG reduces KV cache memory usage by approximately 50% compared to the BoN strategy, facilitating more efficient and practical deployment of TTS techniques.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20325</guid>
|
||||
<pubDate>Fri, 23 May 2025 18:19:09 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.18536</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18536.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haoyuan Sun, Jiaqi Wu, Bo Xia, Yifu Luo, Yifei Zhao, Kai Qin, Xufei Lv, Tiantian Zhang, Yongzhe Chang, Xueqian Wang</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Standing in 2025, at a critical juncture in the pursuit of Artificial General Intelligence (AGI), reinforcement fine-tuning (RFT) has demonstrated significant potential in enhancing the reasoning capability of large language models (LLMs) and has led to the development of cutting-edge AI models such as OpenAI-o1 and DeepSeek-R1. Moreover, the efficient application of RFT to enhance the reasoning capability of multimodal large language models (MLLMs) has attracted widespread attention from the community. In this position paper, we argue that reinforcement fine-tuning powers the reasoning capability of multimodal large language models. To begin with, we provide a detailed introduction to the fundamental background knowledge that researchers interested in this field should be familiar with. Furthermore, we meticulously summarize the improvements of RFT in powering reasoning capability of MLLMs into five key points: diverse modalities, diverse tasks and domains, better training algorithms, abundant benchmarks and thriving engineering frameworks. Finally, we propose five promising directions for future research that the community might consider. We hope that this position paper will provide valuable insights to the community at this pivotal stage in the advancement toward AGI. Summary of works done on RFT for MLLMs is available at https://github.com/Sun-Haoyuan23/Awesome-RL-based-Reasoning-MLLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18536</guid>
|
||||
<pubDate>Sat, 24 May 2025 06:01:48 +0000</pubDate>
|
||||
<title>OmniConsistency: Learning Style-Agnostic Consistency from Paired Stylization Data</title>
|
||||
<link>https://arxiv.org/abs/2505.18445</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18445.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yiren Song, Cheng Liu, Mike Zheng Shou</p><p><b>Upvotes:</b> 57</p><p><b>Summary:</b> Diffusion models have advanced image stylization significantly, yet two core challenges persist: (1) maintaining consistent stylization in complex scenes, particularly identity, composition, and fine details, and (2) preventing style degradation in image-to-image pipelines with style LoRAs. GPT-4o's exceptional stylization consistency highlights the performance gap between open-source methods and proprietary models. To bridge this gap, we propose OmniConsistency, a universal consistency plugin leveraging large-scale Diffusion Transformers (DiTs). OmniConsistency contributes: (1) an in-context consistency learning framework trained on aligned image pairs for robust generalization; (2) a two-stage progressive learning strategy decoupling style learning from consistency preservation to mitigate style degradation; and (3) a fully plug-and-play design compatible with arbitrary style LoRAs under the Flux framework. Extensive experiments show that OmniConsistency significantly enhances visual coherence and aesthetic quality, achieving performance comparable to commercial state-of-the-art model GPT-4o.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18445</guid>
|
||||
<pubDate>Sat, 24 May 2025 01:00:20 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>B-score: Detecting biases in large language models using response history</title>
|
||||
<link>https://arxiv.org/abs/2505.18545</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18545.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> An Vo, Mohammad Reza Taesiri, Daeyoung Kim, Anh Totti Nguyen</p><p><b>Upvotes:</b> 25</p><p><b>Summary:</b> Large language models (LLMs) often exhibit strong biases, e.g, against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves in a multi-turn conversation in response to questions that seek an Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases to Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e, accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: https://b-score.github.io.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18545</guid>
|
||||
<pubDate>Sat, 24 May 2025 06:23:52 +0000</pubDate>
|
||||
<title>MLLMs are Deeply Affected by Modality Bias</title>
|
||||
<link>https://arxiv.org/abs/2505.18657</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18657.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xu Zheng, Chenfei Liao, Yuqian Fu, Kaiyu Lei, Yuanhuiyi Lyu, Lutao Jiang, Bin Ren, Jialei Chen, Jiawen Wang, Chengxin Li, Linfeng Zhang, Danda Pani Paudel, Xuanjing Huang, Yu-Gang Jiang, Nicu Sebe, Dacheng Tao, Luc Van Gool, Xuming Hu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Recent advances in Multimodal Large Language Models (MLLMs) have shown promising results in integrating diverse modalities such as texts and images. MLLMs are heavily influenced by modality bias, often relying on language while under-utilizing other modalities like visual inputs. This position paper argues that MLLMs are deeply affected by modality bias. Firstly, we diagnose the current state of modality bias, highlighting its manifestations across various tasks. Secondly, we propose a systematic research road-map related to modality bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and offer actionable suggestions for future research to mitigate it. To substantiate these findings, we conduct experiments that demonstrate the influence of each factor: 1. Data Characteristics: Language data is compact and abstract, while visual data is redundant and complex, creating an inherent imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The dominance of pretrained language models in MLLMs leads to overreliance on language and neglect of visual information. 3. Training Objectives: Current objectives often fail to promote balanced cross-modal alignment, resulting in shortcut learning biased toward language. These findings highlight the need for balanced training strategies and model architectures to better integrate multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle these challenges and drive innovation in MLLM research. Our work provides a fresh perspective on modality bias in MLLMs and offers insights for developing more robust and generalizable multimodal systems-advancing progress toward Artificial General Intelligence.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18657</guid>
|
||||
<pubDate>Sat, 24 May 2025 11:49:31 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Flex-Judge: Think Once, Judge Anywhere</title>
|
||||
<link>https://arxiv.org/abs/2505.18601</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18601.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jongwoo Ko, Sungnyun Kim, Sungwoo Cho, Se-Young Yun</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Human-generated reward signals are critical for aligning generative models with human preferences, guiding both training and inference-time evaluations. While large language models (LLMs) employed as proxy evaluators, i.e., LLM-as-a-Judge, significantly reduce the costs associated with manual annotations, they typically require extensive modality-specific training data and fail to generalize well across diverse multimodal tasks. In this paper, we propose Flex-Judge, a reasoning-guided multimodal judge model that leverages minimal textual reasoning data to robustly generalize across multiple modalities and evaluation formats. Our core intuition is that structured textual reasoning explanations inherently encode generalizable decision-making patterns, enabling an effective transfer to multimodal judgments, e.g., with images or videos. Empirical results demonstrate that Flex-Judge, despite being trained on significantly fewer text data, achieves competitive or superior performance compared to state-of-the-art commercial APIs and extensively trained multimodal evaluators. Notably, Flex-Judge presents broad impact in modalities like molecule, where comprehensive evaluation benchmarks are scarce, underscoring its practical value in resource-constrained domains. Our framework highlights reasoning-based text supervision as a powerful, cost-effective alternative to traditional annotation-intensive approaches, substantially advancing scalable multimodal model-as-a-judge.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18601</guid>
|
||||
<pubDate>Sat, 24 May 2025 08:50:53 +0000</pubDate>
|
||||
<title>Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation</title>
|
||||
<link>https://arxiv.org/abs/2505.18875</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18875.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, Jianfei Chen, Song Han, Kurt Keutzer, Ion Stoica</p><p><b>Upvotes:</b> 34</p><p><b>Summary:</b> Diffusion Transformers (DiTs) are essential for video generation but suffer from significant latency due to the quadratic complexity of attention. By computing only critical tokens, sparse attention reduces computational costs and offers a promising acceleration approach. However, we identify that existing methods fail to approach optimal generation quality under the same computation budget for two reasons: (1) Inaccurate critical token identification: current methods cluster tokens based on position rather than semantics, leading to imprecise aggregated representations. (2) Excessive computation waste: critical tokens are scattered among non-critical ones, leading to wasted computation on GPUs, which are optimized for processing contiguous tokens. In this paper, we propose SVG2, a training-free framework that maximizes identification accuracy and minimizes computation waste, achieving a Pareto frontier trade-off between generation quality and efficiency. The core of SVG2 is semantic-aware permutation, which clusters and reorders tokens based on semantic similarity using k-means. This approach ensures both a precise cluster representation, improving identification accuracy, and a densified layout of critical tokens, enabling efficient computation without padding. Additionally, SVG2 integrates top-p dynamic budget control and customized kernel implementations, achieving up to 2.30x and 1.89x speedup while maintaining a PSNR of up to 30 and 26 on HunyuanVideo and Wan 2.1, respectively.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18875</guid>
|
||||
<pubDate>Sat, 24 May 2025 21:30:29 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps</title>
|
||||
<link>https://arxiv.org/abs/2505.18675</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18675.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sicheng Feng, Song Wang, Shuyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang, Xinchao Wang</p><p><b>Upvotes:</b> 22</p><p><b>Summary:</b> Multimodal large language models (MLLMs) have recently achieved significant progress in visual tasks, including semantic scene understanding and text-image alignment, with reasoning variants enhancing performance on complex tasks involving mathematics and logic. However, their capacity for reasoning tasks involving fine-grained visual understanding remains insufficiently evaluated. To address this gap, we introduce ReasonMap, a benchmark designed to assess the fine-grained visual understanding and spatial reasoning abilities of MLLMs. ReasonMap encompasses high-resolution transit maps from 30 cities across 13 countries and includes 1,008 question-answer pairs spanning two question types and three templates. Furthermore, we design a two-level evaluation pipeline that properly assesses answer correctness and quality. Comprehensive evaluations of 15 popular MLLMs, including both base and reasoning variants, reveal a counterintuitive pattern: among open-source models, base models outperform reasoning ones, while the opposite trend is observed in closed-source models. Additionally, performance generally degrades when visual inputs are masked, indicating that while MLLMs can leverage prior knowledge to answer some questions, fine-grained visual reasoning tasks still require genuine visual perception for strong performance. Our benchmark study offers new insights into visual reasoning and contributes to investigating the gap between open-source and closed-source models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18675</guid>
|
||||
<pubDate>Sat, 24 May 2025 12:33:52 +0000</pubDate>
|
||||
<title>MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems</title>
|
||||
<link>https://arxiv.org/abs/2505.18943</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18943.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xuanming Zhang, Yuxuan Chen, Min-Hsuan Yeh, Yixuan Li</p><p><b>Upvotes:</b> 21</p><p><b>Summary:</b> Human social interactions depend on the ability to infer others' unspoken intentions, emotions, and beliefs-a cognitive skill grounded in the psychological concept of Theory of Mind (ToM). While large language models (LLMs) excel in semantic understanding tasks, they struggle with the ambiguity and contextual nuance inherent in human communication. To bridge this gap, we introduce MetaMind, a multi-agent framework inspired by psychological theories of metacognition, designed to emulate human-like social reasoning. MetaMind decomposes social understanding into three collaborative stages: (1) a Theory-of-Mind Agent generates hypotheses user mental states (e.g., intent, emotion), (2) a Domain Agent refines these hypotheses using cultural norms and ethical constraints, and (3) a Response Agent generates contextually appropriate responses while validating alignment with inferred intent. Our framework achieves state-of-the-art performance across three challenging benchmarks, with 35.7% improvement in real-world social scenarios and 6.2% gain in ToM reasoning. Notably, it enables LLMs to match human-level performance on key ToM tasks for the first time. Ablation studies confirm the necessity of all components, which showcase the framework's ability to balance contextual plausibility, social appropriateness, and user adaptation. This work advances AI systems toward human-like social intelligence, with applications in empathetic dialogue and culturally sensitive interactions. Code is available at https://github.com/XMZhangAI/MetaMind.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18943</guid>
|
||||
<pubDate>Sun, 25 May 2025 02:32:57 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation</title>
|
||||
<link>https://arxiv.org/abs/2505.18759</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18759.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ruichen Zhang, Rana Muhammad Shahroz Khan, Zhen Tan, Dawei Li, Song Wang, Tianlong Chen</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Data-centric distillation, including data augmentation, selection, and mixing, offers a promising path to creating smaller, more efficient student Large Language Models (LLMs) that retain strong reasoning abilities. However, there still lacks a comprehensive benchmark to systematically assess the effect of each distillation approach. This paper introduces DC-CoT, the first data-centric benchmark that investigates data manipulation in chain-of-thought (CoT) distillation from method, model and data perspectives. Utilizing various teacher models (e.g., o4-mini, Gemini-Pro, Claude-3.5) and student architectures (e.g., 3B, 7B parameters), we rigorously evaluate the impact of these data manipulations on student model performance across multiple reasoning datasets, with a focus on in-distribution (IID) and out-of-distribution (OOD) generalization, and cross-domain transfer. Our findings aim to provide actionable insights and establish best practices for optimizing CoT distillation through data-centric techniques, ultimately facilitating the development of more accessible and capable reasoning models. The dataset can be found at https://huggingface.co/datasets/rana-shahroz/DC-COT, while our code is shared in https://anonymous.4open.science/r/DC-COT-FF4C/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18759</guid>
|
||||
<pubDate>Sat, 24 May 2025 15:54:19 +0000</pubDate>
|
||||
<title>VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization</title>
|
||||
<link>https://arxiv.org/abs/2505.19000</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19000.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yunxin Li, Xinyu Chen, Zitao Li, Zhenyu Liu, Longyue Wang, Wenhan Luo, Baotian Hu, Min Zhang</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> Applying Reinforcement Learning (RL) to Video Large Language Models (Video-LLMs) shows significant promise for complex video reasoning. However, popular Reinforcement Fine-Tuning (RFT) methods, such as outcome-based Group Relative Policy Optimization (GRPO), are limited by data preparation bottlenecks (e.g., noise or high cost) and exhibit unstable improvements in the quality of long chain-of-thoughts (CoTs) and downstream performance.To address these limitations, we propose VerIPO, a Verifier-guided Iterative Policy Optimization method designed to gradually improve video LLMs' capacity for generating deep, long-term reasoning chains. The core component is Rollout-Aware Verifier, positioned between the GRPO and Direct Preference Optimization (DPO) training phases to form the GRPO-Verifier-DPO training loop. This verifier leverages small LLMs as a judge to assess the reasoning logic of rollouts, enabling the construction of high-quality contrastive data, including reflective and contextually consistent CoTs. These curated preference samples drive the efficient DPO stage (7x faster than GRPO), leading to marked improvements in reasoning chain quality, especially in terms of length and contextual consistency. This training loop benefits from GRPO's expansive search and DPO's targeted optimization. Experimental results demonstrate: 1) Significantly faster and more effective optimization compared to standard GRPO variants, yielding superior performance; 2) Our trained models exceed the direct inference of large-scale instruction-tuned Video-LLMs, producing long and contextually consistent CoTs on diverse video reasoning tasks; and 3) Our model with one iteration outperforms powerful LMMs (e.g., Kimi-VL) and long reasoning models (e.g., Video-R1), highlighting its effectiveness and stability.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19000</guid>
|
||||
<pubDate>Sun, 25 May 2025 06:41:28 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.18773</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18773.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jamie Hayes, Ilia Shumailov, Christopher A. Choquette-Choo, Matthew Jagielski, George Kaissis, Katherine Lee, Milad Nasr, Sahra Ghalebikesabi, Niloofar Mireshghallah, Meenatchi Sundaram Mutu Selva Annamalai, Igor Shilov, Matthieu Meeus, Yves-Alexandre de Montjoye, Franziska Boenisch, Adam Dziedzic, A. Feder Cooper</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> State-of-the-art membership inference attacks (MIAs) typically require training many reference models, making it difficult to scale these attacks to large pre-trained language models (LLMs). As a result, prior research has either relied on weaker attacks that avoid training reference models (e.g., fine-tuning attacks), or on stronger attacks applied to small-scale models and datasets. However, weaker attacks have been shown to be brittle - achieving close-to-arbitrary success - and insights from strong attacks in simplified settings do not translate to today's LLMs. These challenges have prompted an important question: are the limitations observed in prior work due to attack design choices, or are MIAs fundamentally ineffective on LLMs? We address this question by scaling LiRA - one of the strongest MIAs - to GPT-2 architectures ranging from 10M to 1B parameters, training reference models on over 20B tokens from the C4 dataset. Our results advance the understanding of MIAs on LLMs in three key ways: (1) strong MIAs can succeed on pre-trained LLMs; (2) their effectiveness, however, remains limited (e.g., AUC<0.7) in practical settings; and, (3) the relationship between MIA success and related privacy metrics is not as straightforward as prior work has suggested.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18773</guid>
|
||||
<pubDate>Sat, 24 May 2025 16:23:43 +0000</pubDate>
|
||||
<title>SATORI-R1: Incentivizing Multimodal Reasoning with Spatial Grounding and Verifiable Rewards</title>
|
||||
<link>https://arxiv.org/abs/2505.19094</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19094.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chuming Shen, Wei Wei, Xiaoye Qu, Yu Cheng</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> DeepSeek-R1 has demonstrated powerful reasoning capabilities in the text domain through stable reinforcement learning (RL). Recently, in the multimodal domain, works have begun to directly apply RL to generate R1-like free-form reasoning for Visual Question Answering (VQA) tasks. However, multimodal tasks share an intrinsically different nature from textual tasks, which heavily rely on the understanding of the input image to solve the problem. Therefore, such free-form reasoning faces two critical limitations in the VQA task: (1) Extended reasoning chains diffuse visual focus away from task-critical regions, degrading answer accuracy. (2) Unverifiable intermediate steps amplify policy-gradient variance and computational costs overhead. To address these issues, in this paper, we introduce SATORI (Spatially Anchored Task Optimization with ReInforcement Learning), which decomposes VQA into three verifiable stages, including global image captioning, region localization, and answer prediction, each supplying explicit reward signals. Furthermore, we also introduce VQA-Verify, a 12k dataset annotated with answer-aligned captions and bounding-boxes to facilitate training. Experiments demonstrate consistent performance improvements across seven VQA benchmarks, achieving up to 15.7% improvement in accuracy in accuracy compared to the R1-like baseline. Our analysis of the attention map confirms enhanced focus on critical regions, which brings improvements in accuracy. Our code is available at https://github.com/justairr/SATORI-R1.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19094</guid>
|
||||
<pubDate>Sun, 25 May 2025 11:11:06 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgeting</title>
|
||||
<link>https://arxiv.org/abs/2505.18822</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18822.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shijue Huang, Hongru Wang, Wanjun Zhong, Zhaochen Su, Jiazhan Feng, Bowen Cao, Yi R. Fung</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Modern large reasoning models demonstrate impressive problem-solving capabilities by employing sophisticated reasoning strategies. However, they often struggle to balance efficiency and effectiveness, frequently generating unnecessarily lengthy reasoning chains for simple problems. In this work, we propose AdaCtrl, a novel framework to support both difficulty-aware adaptive reasoning budget allocation and explicit user control over reasoning depth. AdaCtrl dynamically adjusts its reasoning length based on self-assessed problem difficulty, while also allowing users to manually control the budget to prioritize either efficiency or effectiveness. This is achieved through a two-stage training pipeline: an initial cold-start fine-tuning phase to instill the ability to self-aware difficulty and adjust reasoning budget, followed by a difficulty-aware reinforcement learning (RL) stage that refines the model's adaptive reasoning strategies and calibrates its difficulty assessments based on its evolving capabilities during online training. To enable intuitive user interaction, we design explicit length-triggered tags that function as a natural interface for budget control. Empirical results show that AdaCtrl adapts reasoning length based on estimated difficulty, compared to the standard training baseline that also incorporates fine-tuning and RL, it yields performance improvements and simultaneously reduces response length by 10.06% and 12.14% on the more challenging AIME2024 and AIME2025 datasets, which require elaborate reasoning, and by 62.05% and 91.04% on the MATH500 and GSM8K datasets, where more concise responses are sufficient. Furthermore, AdaCtrl enables precise user control over the reasoning budget, allowing for tailored responses to meet specific needs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18822</guid>
|
||||
<pubDate>Sat, 24 May 2025 18:46:50 +0000</pubDate>
|
||||
<title>SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.19099</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19099.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han, Hang Xu, Hanhui Li, Mrinmaya Sachan, Xiaodan Liang</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> We present SeePhys, a large-scale multimodal benchmark for LLM reasoning grounded in physics questions ranging from middle school to PhD qualifying exams. The benchmark covers 7 fundamental domains spanning the physics discipline, incorporating 21 categories of highly heterogeneous diagrams. In contrast to prior works where visual elements mainly serve auxiliary purposes, our benchmark features a substantial proportion of vision-essential problems (75\%) that mandate visual information extraction for correct solutions. Through extensive evaluation, we observe that even the most advanced visual reasoning models (e.g., Gemini-2.5-pro and o4-mini) achieve sub-60\% accuracy on our benchmark. These results reveal fundamental challenges in current large language models' visual understanding capabilities, particularly in: (i) establishing rigorous coupling between diagram interpretation and physics reasoning, and (ii) overcoming their persistent reliance on textual cues as cognitive shortcuts.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19099</guid>
|
||||
<pubDate>Sun, 25 May 2025 11:28:34 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Hybrid Neural-MPM for Interactive Fluid Simulations in Real-Time</title>
|
||||
<link>https://arxiv.org/abs/2505.18926</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.18926.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingxuan Xu, Hong Huang, Chuhang Zou, Manolis Savva, Yunchao Wei, Wuyang Chen</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> We propose a neural physics system for real-time, interactive fluid simulations. Traditional physics-based methods, while accurate, are computationally intensive and suffer from latency issues. Recent machine-learning methods reduce computational costs while preserving fidelity; yet most still fail to satisfy the latency constraints for real-time use and lack support for interactive applications. To bridge this gap, we introduce a novel hybrid method that integrates numerical simulation, neural physics, and generative control. Our neural physics jointly pursues low-latency simulation and high physical fidelity by employing a fallback safeguard to classical numerical solvers. Furthermore, we develop a diffusion-based controller that is trained using a reverse modeling strategy to generate external dynamic force fields for fluid manipulation. Our system demonstrates robust performance across diverse 2D/3D scenarios, material types, and obstacle interactions, achieving real-time simulations at high frame rates (11~29% latency) while enabling fluid control guided by user-friendly freehand sketches. We present a significant step towards practical, controllable, and physically plausible fluid simulations for real-time interactive applications. We promise to release both models and data upon acceptance.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.18926</guid>
|
||||
<pubDate>Sun, 25 May 2025 01:27:18 +0000</pubDate>
|
||||
<title>CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.19235</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19235.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qinsi Wang, Hancheng Ye, Ming-Yu Chung, Yudong Liu, Yueqian Lin, Martin Kuo, Mingyuan Ma, Jianyi Zhang, Yiran Chen</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Vision-Language Models (VLMs) excel across diverse tasks but suffer from high inference costs in time and memory. Token sparsity mitigates inefficiencies in token usage, while neuron sparsity reduces high-dimensional computations, both offering promising solutions to enhance efficiency. Recently, these two sparsity paradigms have evolved largely in parallel, fostering the prevailing assumption that they function independently. However, a fundamental yet underexplored question remains: Do they truly operate in isolation, or is there a deeper underlying interplay that has yet to be uncovered? In this paper, we conduct the first comprehensive investigation into this question. By introducing and analyzing the matching mechanism between Core Neurons and Core Tokens, we found that key neurons and tokens for inference mutually influence and reinforce each other. Building on this insight, we propose CoreMatching, a co-adaptive sparse inference framework, which leverages the synergy between token and neuron sparsity to enhance inference efficiency. Through theoretical analysis and efficiency evaluations, we demonstrate that the proposed method surpasses state-of-the-art baselines on ten image understanding tasks and three hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs reduction and a 10x overall speedup. Code is released at https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19235</guid>
|
||||
<pubDate>Sun, 25 May 2025 17:16:34 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>An Embarrassingly Simple Defense Against LLM Abliteration Attacks</title>
|
||||
<link>https://arxiv.org/abs/2505.19056</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19056.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Harethah Abu Shairah, Hasan Abed Al Kader Hammoud, Bernard Ghanem, George Turkiyyah</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Large language models (LLMs) are typically aligned to comply with safety guidelines by refusing harmful instructions. A recent attack, termed abliteration, isolates and suppresses the single latent direction most responsible for refusal behavior, enabling the model to generate unethical content. We propose a defense that modifies how models generate refusals. We construct an extended-refusal dataset that contains harmful prompts with a full response that justifies the reason for refusal. We then fine-tune Llama-2-7B-Chat and Qwen2.5-Instruct (1.5B and 3B parameters) on our extended-refusal dataset, and evaluate the resulting systems on a set of harmful prompts. In our experiments, extended-refusal models maintain high refusal rates, dropping at most by 10%, whereas baseline models' refusal rates drop by 70-80% after abliteration. A broad evaluation of safety and utility shows that extended-refusal fine-tuning neutralizes the abliteration attack while preserving general performance.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19056</guid>
|
||||
<pubDate>Sun, 25 May 2025 09:18:24 +0000</pubDate>
|
||||
<title>SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline</title>
|
||||
<link>https://arxiv.org/abs/2505.19314</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19314.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Helin Wang, Jiarui Hai, Dongchao Yang, Chen Chen, Kai Li, Junyi Peng, Thomas Thebaud, Laureano Moro Velazquez, Jesus Villalba, Najim Dehak</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Target Speech Extraction (TSE) aims to isolate a target speaker's voice from a mixture of multiple speakers by leveraging speaker-specific cues, typically provided as auxiliary audio (a.k.a. cue audio). Although recent advancements in TSE have primarily employed discriminative models that offer high perceptual quality, these models often introduce unwanted artifacts, reduce naturalness, and are sensitive to discrepancies between training and testing environments. On the other hand, generative models for TSE lag in perceptual quality and intelligibility. To address these challenges, we present SoloSpeech, a novel cascaded generative pipeline that integrates compression, extraction, reconstruction, and correction processes. SoloSpeech features a speaker-embedding-free target extractor that utilizes conditional information from the cue audio's latent space, aligning it with the mixture audio's latent space to prevent mismatches. Evaluated on the widely-used Libri2Mix dataset, SoloSpeech achieves the new state-of-the-art intelligibility and quality in target speech extraction and speech separation tasks while demonstrating exceptional generalization on out-of-domain data and real-world scenarios.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19314</guid>
|
||||
<pubDate>Sun, 25 May 2025 21:00:48 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Jodi: Unification of Visual Generation and Understanding via Joint Modeling</title>
|
||||
<link>https://arxiv.org/abs/2505.19084</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19084.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifeng Xu, Zhenliang He, Meina Kan, Shiguang Shan, Xilin Chen</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19084</guid>
|
||||
<pubDate>Sun, 25 May 2025 10:40:52 +0000</pubDate>
|
||||
<title>Absolute Coordinates Make Motion Generation Easy</title>
|
||||
<link>https://arxiv.org/abs/2505.19377</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19377.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zichong Meng, Zeyu Han, Xiaogang Peng, Yiming Xie, Huaizu Jiang</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> State-of-the-art text-to-motion generation models rely on the kinematic-aware, local-relative motion representation popularized by HumanML3D, which encodes motion relative to the pelvis and to the previous frame with built-in redundancy. While this design simplifies training for earlier generation models, it introduces critical limitations for diffusion models and hinders applicability to downstream tasks. In this work, we revisit the motion representation and propose a radically simplified and long-abandoned alternative for text-to-motion generation: absolute joint coordinates in global space. Through systematic analysis of design choices, we show that this formulation achieves significantly higher motion fidelity, improved text alignment, and strong scalability, even with a simple Transformer backbone and no auxiliary kinematic-aware losses. Moreover, our formulation naturally supports downstream tasks such as text-driven motion control and temporal/spatial editing without additional task-specific reengineering and costly classifier guidance generation from control signals. Finally, we demonstrate promising generalization to directly generate SMPL-H mesh vertices in motion from text, laying a strong foundation for future research and motion-related applications.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19377</guid>
|
||||
<pubDate>Mon, 26 May 2025 00:36:00 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>WHISTRESS: Enriching Transcriptions with Sentence Stress Detection</title>
|
||||
<link>https://arxiv.org/abs/2505.19103</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19103.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Iddo Yosha, Dorin Shteyman, Yossi Adi</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Spoken language conveys meaning not only through words but also through intonation, emotion, and emphasis. Sentence stress, the emphasis placed on specific words within a sentence, is crucial for conveying speaker intent and has been extensively studied in linguistics. In this work, we introduce WHISTRESS, an alignment-free approach for enhancing transcription systems with sentence stress detection. To support this task, we propose TINYSTRESS-15K, a scalable, synthetic training data for the task of sentence stress detection which resulted from a fully automated dataset creation process. We train WHISTRESS on TINYSTRESS-15K and evaluate it against several competitive baselines. Our results show that WHISTRESS outperforms existing methods while requiring no additional input priors during training or inference. Notably, despite being trained on synthetic data, WHISTRESS demonstrates strong zero-shot generalization across diverse benchmarks. Project page: https://pages.cs.huji.ac.il/adiyoss-lab/whistress.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19103</guid>
|
||||
<pubDate>Sun, 25 May 2025 11:45:08 +0000</pubDate>
|
||||
<title>Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression</title>
|
||||
<link>https://arxiv.org/abs/2505.19433</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19433.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Peijie Dong, Zhenheng Tang, Xiang Liu, Lujun Li, Xiaowen Chu, Bo Li</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Post-training compression reduces the computational and memory costs of large language models (LLMs), enabling resource-efficient deployment. However, existing compression benchmarks only focus on language modeling (e.g., perplexity) and natural language understanding tasks (e.g., GLUE accuracy), ignoring the agentic capabilities - workflow, tool use/function call, long-context understanding and real-world application. We introduce the Agent Compression Benchmark (ACBench), the first comprehensive benchmark for evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1) 12 tasks across 4 capabilities (e.g., WorfBench for workflow generation, Needle-in-Haystack for long-context retrieval), (2) quantization (GPTQ, AWQ) and pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B), standard (Qwen2.5 7B-32B), and distilled reasoning LLMs (DeepSeek-R1-Distill). Our experiments reveal compression tradeoffs: 4-bit quantization preserves workflow generation and tool use (1%-3% drop) but degrades real-world application accuracy by 10%-15%. We introduce ERank, Top-k Ranking Correlation and Energy to systematize analysis. ACBench provides actionable insights for optimizing LLM compression in agentic scenarios. The code can be found in https://github.com/pprp/ACBench.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19433</guid>
|
||||
<pubDate>Mon, 26 May 2025 02:49:07 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Shifting AI Efficiency From Model-Centric to Data-Centric Compression</title>
|
||||
<link>https://arxiv.org/abs/2505.19147</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19147.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xuyang Liu, Zichen Wen, Shaobo Wang, Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, Xu Zheng, Honggang Chen, Weijia Li, Xuming Hu, Conghui He, Linfeng Zhang</p><p><b>Upvotes:</b> 121</p><p><b>Summary:</b> The rapid advancement of large language models (LLMs) and multi-modal LLMs (MLLMs) has historically relied on model-centric scaling through increasing parameter counts from millions to hundreds of billions to drive performance gains. However, as we approach hardware limits on model size, the dominant computational bottleneck has fundamentally shifted to the quadratic cost of self-attention over long token sequences, now driven by ultra-long text contexts, high-resolution images, and extended videos. In this position paper, we argue that the focus of research for efficient AI is shifting from model-centric compression to data-centric compression. We position token compression as the new frontier, which improves AI efficiency via reducing the number of tokens during model training or inference. Through comprehensive analysis, we first examine recent developments in long-context AI across various domains and establish a unified mathematical framework for existing model efficiency strategies, demonstrating why token compression represents a crucial paradigm shift in addressing long-context overhead. Subsequently, we systematically review the research landscape of token compression, analyzing its fundamental benefits and identifying its compelling advantages across diverse scenarios. Furthermore, we provide an in-depth analysis of current challenges in token compression research and outline promising future directions. Ultimately, our work aims to offer a fresh perspective on AI efficiency, synthesize existing research, and catalyze innovative developments to address the challenges that increasing context lengths pose to the AI community's advancement.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19147</guid>
|
||||
<pubDate>Sun, 25 May 2025 13:51:17 +0000</pubDate>
|
||||
<title>GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning</title>
|
||||
<link>https://arxiv.org/abs/2505.20355</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20355.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park</p><p><b>Upvotes:</b> 28</p><p><b>Summary:</b> Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient fine-tuning (PEFT) of generative models, valued for its simplicity and effectiveness. Despite recent enhancements, LoRA still suffers from a fundamental limitation: overfitting when the bottleneck is widened. It performs best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks, still falling short of full fine-tuning (FFT) performance. We identify the root cause as LoRA's structural bottleneck, which introduces gradient entanglement to the unrelated input channels and distorts gradient propagation. To address this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA) that partitions weight matrices into sub-blocks, each with its own low-rank adapter. With negligible computational or storage cost, GraLoRA overcomes LoRA's limitations, effectively increases the representational capacity, and more closely approximates FFT behavior. Experiments on code generation and commonsense reasoning benchmarks show that GraLoRA consistently outperforms LoRA and other baselines, achieving up to +8.5% absolute gain in Pass@1 on HumanEval+. These improvements hold across model sizes and rank settings, making GraLoRA a scalable and robust solution for PEFT. Code, data, and scripts are available at https://github.com/SqueezeBits/GraLoRA.git</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20355</guid>
|
||||
<pubDate>Mon, 26 May 2025 06:48:20 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search</title>
|
||||
<link>https://arxiv.org/abs/2505.19209</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19209.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zonglin Yang, Wanhao Liu, Ben Gao, Yujie Liu, Wei Li, Tong Xie, Lidong Bing, Wanli Ouyang, Erik Cambria, Dongzhan Zhou</p><p><b>Upvotes:</b> 22</p><p><b>Summary:</b> Large language models (LLMs) have shown promise in automating scientific hypothesis generation, yet existing approaches primarily yield coarse-grained hypotheses lacking critical methodological and experimental details. We introduce and formally define the novel task of fine-grained scientific hypothesis discovery, which entails generating detailed, experimentally actionable hypotheses from coarse initial research directions. We frame this as a combinatorial optimization problem and investigate the upper limits of LLMs' capacity to solve it when maximally leveraged. Specifically, we explore four foundational questions: (1) how to best harness an LLM's internal heuristics to formulate the fine-grained hypothesis it itself would judge as the most promising among all the possible hypotheses it might generate, based on its own internal scoring-thus defining a latent reward landscape over the hypothesis space; (2) whether such LLM-judged better hypotheses exhibit stronger alignment with ground-truth hypotheses; (3) whether shaping the reward landscape using an ensemble of diverse LLMs of similar capacity yields better outcomes than defining it with repeated instances of the strongest LLM among them; and (4) whether an ensemble of identical LLMs provides a more reliable reward landscape than a single LLM. To address these questions, we propose a hierarchical search method that incrementally proposes and integrates details into the hypothesis, progressing from general concepts to specific experimental configurations. We show that this hierarchical process smooths the reward landscape and enables more effective optimization. Empirical evaluations on a new benchmark of expert-annotated fine-grained hypotheses from recent chemistry literature show that our method consistently outperforms strong baselines.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19209</guid>
|
||||
<pubDate>Sun, 25 May 2025 16:13:46 +0000</pubDate>
|
||||
<title>SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond</title>
|
||||
<link>https://arxiv.org/abs/2505.19641</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19641.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, Yunan Huang, Mozhi Zhang, Pengyu Zhao, Junjie Yan, Junxian He</p><p><b>Upvotes:</b> 42</p><p><b>Summary:</b> Recent advances such as OpenAI-o1 and DeepSeek R1 have demonstrated the potential of Reinforcement Learning (RL) to enhance reasoning abilities in Large Language Models (LLMs). While open-source replication efforts have primarily focused on mathematical and coding domains, methods and resources for developing general reasoning capabilities remain underexplored. This gap is partly due to the challenge of collecting diverse and verifiable reasoning data suitable for RL. We hypothesize that logical reasoning is critical for developing general reasoning capabilities, as logic forms a fundamental building block of reasoning. In this work, we present SynLogic, a data synthesis framework and dataset that generates diverse logical reasoning data at scale, encompassing 35 diverse logical reasoning tasks. The SynLogic approach enables controlled synthesis of data with adjustable difficulty and quantity. Importantly, all examples can be verified by simple rules, making them ideally suited for RL with verifiable rewards. In our experiments, we validate the effectiveness of RL training on the SynLogic dataset based on 7B and 32B models. SynLogic leads to state-of-the-art logical reasoning performance among open-source datasets, surpassing DeepSeek-R1-Distill-Qwen-32B by 6 points on BBEH. Furthermore, mixing SynLogic data with mathematical and coding tasks improves the training efficiency of these domains and significantly enhances reasoning generalization. Notably, our mixed training model outperforms DeepSeek-R1-Zero-Qwen-32B across multiple benchmarks. These findings position SynLogic as a valuable resource for advancing the broader reasoning capabilities of LLMs. We open-source both the data synthesis pipeline and the SynLogic dataset at https://github.com/MiniMax-AI/SynLogic.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19641</guid>
|
||||
<pubDate>Mon, 26 May 2025 07:59:36 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models</title>
|
||||
<link>https://arxiv.org/abs/2505.19223</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19223.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, Chongxuan Li</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> While Masked Diffusion Models (MDMs), such as LLaDA, present a promising paradigm for language modeling, there has been relatively little effort in aligning these models with human preferences via reinforcement learning. The challenge primarily arises from the high variance in Evidence Lower Bound (ELBO)-based likelihood estimates required for preference optimization. To address this issue, we propose Variance-Reduced Preference Optimization (VRPO), a framework that formally analyzes the variance of ELBO estimators and derives bounds on both the bias and variance of preference optimization gradients. Building on this theoretical foundation, we introduce unbiased variance reduction strategies, including optimal Monte Carlo budget allocation and antithetic sampling, that significantly improve the performance of MDM alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA, and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor consistently and significantly across mathematical (GSM8K +4.7), code (HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard +4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical performance compared to strong language MDMs and ARMs. Project page: https://ml-gsai.github.io/LLaDA-1.5-Demo/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19223</guid>
|
||||
<pubDate>Sun, 25 May 2025 16:36:20 +0000</pubDate>
|
||||
<title>Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval</title>
|
||||
<link>https://arxiv.org/abs/2505.19650</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19650.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Victoria W., Fuzheng Zhang, Guorui Zhou</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Multimodal information retrieval (MIR) faces inherent challenges due to the heterogeneity of data sources and the complexity of cross-modal alignment. While previous studies have identified modal gaps in feature spaces, a systematic approach to address these challenges remains unexplored. In this work, we introduce UNITE, a universal framework that tackles these challenges through two critical yet underexplored aspects: data curation and modality-aware training configurations. Our work provides the first comprehensive analysis of how modality-specific data properties influence downstream task performance across diverse scenarios. Moreover, we propose Modal-Aware Masked Contrastive Learning (MAMCL) to mitigate the competitive relationships among the instances of different modalities. Our framework achieves state-of-the-art results on multiple multimodal retrieval benchmarks, outperforming existing methods by notable margins. Through extensive experiments, we demonstrate that strategic modality curation and tailored training protocols are pivotal for robust cross-modal representation learning. This work not only advances MIR performance but also provides a foundational blueprint for future research in multimodal systems. Our project is available at https://friedrichor.github.io/projects/UNITE.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19650</guid>
|
||||
<pubDate>Mon, 26 May 2025 08:09:44 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>PATS: Process-Level Adaptive Thinking Mode Switching</title>
|
||||
<link>https://arxiv.org/abs/2505.19250</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19250.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yi Wang, Junxiao Liu, Shimao Zhang, Jiajun Chen, Shujian Huang</p><p><b>Upvotes:</b> 43</p><p><b>Summary:</b> Current large-language models (LLMs) typically adopt a fixed reasoning strategy, either simple or complex, for all questions, regardless of their difficulty. This neglect of variation in task and reasoning process complexity leads to an imbalance between performance and efficiency. Existing methods attempt to implement training-free fast-slow thinking system switching to handle problems of varying difficulty, but are limited by coarse-grained solution-level strategy adjustments. To address this issue, we propose a novel reasoning paradigm: Process-Level Adaptive Thinking Mode Switching (PATS), which enables LLMs to dynamically adjust their reasoning strategy based on the difficulty of each step, optimizing the balance between accuracy and computational efficiency. Our approach integrates Process Reward Models (PRMs) with Beam Search, incorporating progressive mode switching and bad-step penalty mechanisms. Experiments on diverse mathematical benchmarks demonstrate that our methodology achieves high accuracy while maintaining moderate token usage. This study emphasizes the significance of process-level, difficulty-aware reasoning strategy adaptation, offering valuable insights into efficient inference for LLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19250</guid>
|
||||
<pubDate>Sun, 25 May 2025 17:58:50 +0000</pubDate>
|
||||
<title>ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows</title>
|
||||
<link>https://arxiv.org/abs/2505.19897</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19897.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Xiangru Tang, Tianbao Xie, Xiachong Feng, Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi, Lingpeng Kong, Zhiyong Wu</p><p><b>Upvotes:</b> 82</p><p><b>Summary:</b> Large Language Models (LLMs) have extended their impact beyond Natural Language Processing, substantially fostering the development of interdisciplinary research. Recently, various LLM-based agents have been developed to assist scientific discovery progress across multiple aspects and domains. Among these, computer-using agents, capable of interacting with operating systems as humans do, are paving the way to automated scientific problem-solving and addressing routines in researchers' workflows. Recognizing the transformative potential of these agents, we introduce ScienceBoard, which encompasses two complementary contributions: (i) a realistic, multi-domain environment featuring dynamic and visually rich scientific workflows with integrated professional software, where agents can autonomously interact via different interfaces to accelerate complex research tasks and experiments; and (ii) a challenging benchmark of 169 high-quality, rigorously validated real-world tasks curated by humans, spanning scientific-discovery workflows in domains such as biochemistry, astronomy, and geoinformatics. Extensive evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude 3.7, UI-TARS) show that, despite some promising results, they still fall short of reliably assisting scientists in complex workflows, achieving only a 15% overall success rate. In-depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents for scientific discovery. Our code, environment, and benchmark are at https://qiushisun.github.io/ScienceBoard-Home/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19897</guid>
|
||||
<pubDate>Mon, 26 May 2025 12:27:27 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Alchemist: Turning Public Text-to-Image Data into Generative Gold</title>
|
||||
<link>https://arxiv.org/abs/2505.19297</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19297.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Valerii Startsev, Alexander Ustyuzhanin, Alexey Kirillov, Dmitry Baranchuk, Sergey Kastryulin</p><p><b>Upvotes:</b> 56</p><p><b>Summary:</b> Pre-training equips text-to-image (T2I) models with broad world knowledge, but this alone is often insufficient to achieve high aesthetic quality and alignment. Consequently, supervised fine-tuning (SFT) is crucial for further refinement. However, its effectiveness highly depends on the quality of the fine-tuning dataset. Existing public SFT datasets frequently target narrow domains (e.g., anime or specific art styles), and the creation of high-quality, general-purpose SFT datasets remains a significant challenge. Current curation methods are often costly and struggle to identify truly impactful samples. This challenge is further complicated by the scarcity of public general-purpose datasets, as leading models often rely on large, proprietary, and poorly documented internal data, hindering broader research progress. This paper introduces a novel methodology for creating general-purpose SFT datasets by leveraging a pre-trained generative model as an estimator of high-impact training samples. We apply this methodology to construct and release Alchemist, a compact (3,350 samples) yet highly effective SFT dataset. Experiments demonstrate that Alchemist substantially improves the generative quality of five public T2I models while preserving diversity and style. Additionally, we release the fine-tuned models' weights to the public.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19297</guid>
|
||||
<pubDate>Sun, 25 May 2025 20:08:20 +0000</pubDate>
|
||||
<title>An Explainable Diagnostic Framework for Neurodegenerative Dementias via Reinforcement-Optimized LLM Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.19954</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19954.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Andrew Zamai, Nathanael Fijalkow, Boris Mansencal, Laurent Simon, Eloi Navet, Pierrick Coupe</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> The differential diagnosis of neurodegenerative dementias is a challenging clinical task, mainly because of the overlap in symptom presentation and the similarity of patterns observed in structural neuroimaging. To improve diagnostic efficiency and accuracy, deep learning-based methods such as Convolutional Neural Networks and Vision Transformers have been proposed for the automatic classification of brain MRIs. However, despite their strong predictive performance, these models find limited clinical utility due to their opaque decision making. In this work, we propose a framework that integrates two core components to enhance diagnostic transparency. First, we introduce a modular pipeline for converting 3D T1-weighted brain MRIs into textual radiology reports. Second, we explore the potential of modern Large Language Models (LLMs) to assist clinicians in the differential diagnosis between Frontotemporal dementia subtypes, Alzheimer's disease, and normal aging based on the generated reports. To bridge the gap between predictive accuracy and explainability, we employ reinforcement learning to incentivize diagnostic reasoning in LLMs. Without requiring supervised reasoning traces or distillation from larger models, our approach enables the emergence of structured diagnostic rationales grounded in neuroimaging findings. Unlike post-hoc explainability methods that retrospectively justify model decisions, our framework generates diagnostic rationales as part of the inference process-producing causally grounded explanations that inform and guide the model's decision-making process. In doing so, our framework matches the diagnostic performance of existing deep learning methods while offering rationales that support its diagnostic conclusions.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19954</guid>
|
||||
<pubDate>Mon, 26 May 2025 13:18:32 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals</title>
|
||||
<link>https://arxiv.org/abs/2505.19386</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19386.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nate Gillman, Charles Herrmann, Michael Freeman, Daksh Aggarwal, Evan Luo, Deqing Sun, Chen Sun</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments. While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied. In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the visual and motion prior in the original pretrained model, without using any 3D asset or physics simulator at inference. The primary challenge of force prompting is the difficulty in obtaining high quality paired force-video training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity of physics simulators. Our key finding is that video generation models can generalize remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects. Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try to understand the source of this generalization and perform ablations that reveal two key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for a single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. We release all datasets, code, weights, and interactive video demos at our project page.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19386</guid>
|
||||
<pubDate>Mon, 26 May 2025 01:04:02 +0000</pubDate>
|
||||
<title>DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Response</title>
|
||||
<link>https://arxiv.org/abs/2505.19973</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19973.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bilel Cherif, Tamas Bisztray, Richard A. Dubniczky, Aaesha Aldahmani, Saeed Alshehhi, Norbert Tihanyi</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Digital Forensics and Incident Response (DFIR) involves analyzing digital evidence to support legal investigations. Large Language Models (LLMs) offer new opportunities in DFIR tasks such as log analysis and memory forensics, but their susceptibility to errors and hallucinations raises concerns in high-stakes contexts. Despite growing interest, there is no comprehensive benchmark to evaluate LLMs across both theoretical and practical DFIR domains. To address this gap, we present DFIR-Metric, a benchmark with three components: (1) Knowledge Assessment: a set of 700 expert-reviewed multiple-choice questions sourced from industry-standard certifications and official documentation; (2) Realistic Forensic Challenges: 150 CTF-style tasks testing multi-step reasoning and evidence correlation; and (3) Practical Analysis: 500 disk and memory forensics cases from the NIST Computer Forensics Tool Testing Program (CFTT). We evaluated 14 LLMs using DFIR-Metric, analyzing both their accuracy and consistency across trials. We also introduce a new metric, the Task Understanding Score (TUS), designed to more effectively evaluate models in scenarios where they achieve near-zero accuracy. This benchmark offers a rigorous, reproducible foundation for advancing AI in digital forensics. All scripts, artifacts, and results are available on the project website at https://github.com/DFIR-Metric.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19973</guid>
|
||||
<pubDate>Mon, 26 May 2025 13:35:37 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models</title>
|
||||
<link>https://arxiv.org/abs/2505.19415</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19415.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hang Hua, Ziyun Zeng, Yizhi Song, Yunlong Tang, Liu He, Daniel Aliaga, Wei Xiong, Jiebo Luo</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Recent multimodal image generators such as GPT-4o, Gemini 2.0 Flash, and Gemini 2.5 Pro excel at following complex instructions, editing images and maintaining concept consistency. However, they are still evaluated by disjoint toolkits: text-to-image (T2I) benchmarks that lacks multi-modal conditioning, and customized image generation benchmarks that overlook compositional semantics and common knowledge. We propose MMIG-Bench, a comprehensive Multi-Modal Image Generation Benchmark that unifies these tasks by pairing 4,850 richly annotated text prompts with 1,750 multi-view reference images across 380 subjects, spanning humans, animals, objects, and artistic styles. MMIG-Bench is equipped with a three-level evaluation framework: (1) low-level metrics for visual artifacts and identity preservation of objects; (2) novel Aspect Matching Score (AMS): a VQA-based mid-level metric that delivers fine-grained prompt-image alignment and shows strong correlation with human judgments; and (3) high-level metrics for aesthetics and human preference. Using MMIG-Bench, we benchmark 17 state-of-the-art models, including Gemini 2.5 Pro, FLUX, DreamBooth, and IP-Adapter, and validate our metrics with 32k human ratings, yielding in-depth insights into architecture and data design. We will release the dataset and evaluation code to foster rigorous, unified evaluation and accelerate future innovations in multi-modal image generation.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19415</guid>
|
||||
<pubDate>Mon, 26 May 2025 02:07:24 +0000</pubDate>
|
||||
<title>Beyond Simple Concatenation: Fairly Assessing PLM Architectures for Multi-Chain Protein-Protein Interactions Prediction</title>
|
||||
<link>https://arxiv.org/abs/2505.20036</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20036.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hazem Alsamkary, Mohamed Elshaffei, Mohamed Soudy, Sara Ossman, Abdallah Amr, Nehal Adel Abdelsalam, Mohamed Elkerdawy, Ahmed Elnaggar</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Protein-protein interactions (PPIs) are fundamental to numerous cellular processes, and their characterization is vital for understanding disease mechanisms and guiding drug discovery. While protein language models (PLMs) have demonstrated remarkable success in predicting protein structure and function, their application to sequence-based PPI binding affinity prediction remains relatively underexplored. This gap is often attributed to the scarcity of high-quality, rigorously refined datasets and the reliance on simple strategies for concatenating protein representations. In this work, we address these limitations. First, we introduce a meticulously curated version of the PPB-Affinity dataset of a total of 8,207 unique protein-protein interaction entries, by resolving annotation inconsistencies and duplicate entries for multi-chain protein interactions. This dataset incorporates a stringent, less than or equal to 30%, sequence identity threshold to ensure robust splitting into training, validation, and test sets, minimizing data leakage. Second, we propose and systematically evaluate four architectures for adapting PLMs to PPI binding affinity prediction: embeddings concatenation (EC), sequences concatenation (SC), hierarchical pooling (HP), and pooled attention addition (PAD). These architectures were assessed using two training methods: full fine-tuning and a lightweight approach employing ConvBERT heads over frozen PLM features. Our comprehensive experiments across multiple leading PLMs (ProtT5, ESM2, Ankh, Ankh2, and ESM3) demonstrated that the HP and PAD architectures consistently outperform conventional concatenation methods, achieving up to 12% increase in terms of Spearman correlation. These results highlight the necessity of sophisticated architectural designs to fully exploit the capabilities of PLMs for nuanced PPI binding affinity prediction.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20036</guid>
|
||||
<pubDate>Mon, 26 May 2025 14:23:08 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference</title>
|
||||
<link>https://arxiv.org/abs/2505.19427</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19427.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sihan Chen, Dan Zhao, Jongwoo Ko, Colby Banbury, Huiping Zhuang, Luming Liang, Tianyi Chen</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> The growing computational demands of large language models (LLMs) make efficient inference and activation strategies increasingly critical. While recent approaches, such as Mixture-of-Experts (MoE), leverage selective activation but require specialized training, training-free sparse activation methods offer broader applicability and superior resource efficiency through their plug-and-play design. However, many existing methods rely solely on hidden state magnitudes to determine activation, resulting in high approximation errors and suboptimal inference accuracy. To address these limitations, we propose WINA (Weight Informed Neuron Activation), a novel, simple, and training-free sparse activation framework that jointly considers hidden state magnitudes and the column-wise ell_2-norms of weight matrices. We show that this leads to a sparsification strategy that obtains optimal approximation error bounds with theoretical guarantees tighter than existing techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g., TEAL) by up to 2.94% in average performance at the same sparsity levels, across a diverse set of LLM architectures and datasets. These results position WINA as a new performance frontier for training-free sparse activation in LLM inference, advancing training-free sparse activation methods and setting a robust baseline for efficient inference. The source code is available at https://github.com/microsoft/wina.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19427</guid>
|
||||
<pubDate>Mon, 26 May 2025 02:37:32 +0000</pubDate>
|
||||
<title>Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations</title>
|
||||
<link>https://arxiv.org/abs/2505.20052</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20052.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hazem Alsamkary, Mohamed Elshaffei, Mohamed Elkerdawy, Ahmed Elnaggar</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Protein language models (PLMs) have emerged as powerful tools to detect complex patterns of protein sequences. However, the capability of PLMs to fully capture information on protein sequences might be limited by focusing on single pre-training tasks. Although adding data modalities or supervised objectives can improve the performance of PLMs, pre-training often remains focused on denoising corrupted sequences. To push the boundaries of PLMs, our research investigated a multi-task pre-training strategy. We developed Ankh3, a model jointly optimized on two objectives: masked language modeling with multiple masking probabilities and protein sequence completion relying only on protein sequences as input. This multi-task pre-training demonstrated that PLMs can learn richer and more generalizable representations solely from protein sequences. The results demonstrated improved performance in downstream tasks, such as secondary structure prediction, fluorescence, GB1 fitness, and contact prediction. The integration of multiple tasks gave the model a more comprehensive understanding of protein properties, leading to more robust and accurate predictions.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20052</guid>
|
||||
<pubDate>Mon, 26 May 2025 14:41:10 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers</title>
|
||||
<link>https://arxiv.org/abs/2505.19439</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19439.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Rihui Xin, Han Liu, Zecheng Wang, Yupeng Zhang, Dianbo Sui, Xiaolin Hu, Bingning Wang</p><p><b>Upvotes:</b> 20</p><p><b>Summary:</b> Large Language Models have achieved remarkable success in natural language processing tasks, with Reinforcement Learning playing a key role in adapting them to specific applications. However, obtaining ground truth answers for training LLMs in mathematical problem-solving is often challenging, costly, and sometimes unfeasible. This research delves into the utilization of format and length as surrogate signals to train LLMs for mathematical problem-solving, bypassing the need for traditional ground truth answers.Our study shows that a reward function centered on format correctness alone can yield performance improvements comparable to the standard GRPO algorithm in early phases. Recognizing the limitations of format-only rewards in the later phases, we incorporate length-based rewards. The resulting GRPO approach, leveraging format-length surrogate signals, not only matches but surpasses the performance of the standard GRPO algorithm relying on ground truth answers in certain scenarios, achieving 40.0\% accuracy on AIME2024 with a 7B base model. Through systematic exploration and experimentation, this research not only offers a practical solution for training LLMs to solve mathematical problems and reducing the dependence on extensive ground truth data collection, but also reveals the essence of why our label-free approach succeeds: base model is like an excellent student who has already mastered mathematical and logical reasoning skills, but performs poorly on the test paper, it simply needs to develop good answering habits to achieve outstanding results in exams , in other words, to unlock the capabilities it already possesses.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19439</guid>
|
||||
<pubDate>Mon, 26 May 2025 02:56:22 +0000</pubDate>
|
||||
<title>Capability-Based Scaling Laws for LLM Red-Teaming</title>
|
||||
<link>https://arxiv.org/abs/2505.20162</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20162.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Alexander Panfilov, Paul Kassianik, Maksym Andriushchenko, Jonas Geiping</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> As large language models grow in capability and agency, identifying vulnerabilities through red-teaming becomes vital for safe deployment. However, traditional prompt-engineering approaches may prove ineffective once red-teaming turns into a weak-to-strong problem, where target models surpass red-teamers in capabilities. To study this shift, we frame red-teaming through the lens of the capability gap between attacker and target. We evaluate more than 500 attacker-target pairs using LLM-based jailbreak attacks that mimic human red-teamers across diverse families, sizes, and capability levels. Three strong trends emerge: (i) more capable models are better attackers, (ii) attack success drops sharply once the target's capability exceeds the attacker's, and (iii) attack success rates correlate with high performance on social science splits of the MMLU-Pro benchmark. From these trends, we derive a jailbreaking scaling law that predicts attack success for a fixed target based on attacker-target capability gap. These findings suggest that fixed-capability attackers (e.g., humans) may become ineffective against future models, increasingly capable open-source models amplify risks for existing systems, and model providers must accurately measure and control models' persuasive and manipulative abilities to limit their effectiveness as attackers.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20162</guid>
|
||||
<pubDate>Mon, 26 May 2025 16:05:41 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.19440</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19440.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shashata Sawmya, Micah Adler, Nir Shavit</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> This paper studies the emergence of interpretable categorical features within large language models (LLMs), analyzing their behavior across training checkpoints (time), transformer layers (space), and varying model sizes (scale). Using sparse autoencoders for mechanistic interpretability, we identify when and where specific semantic concepts emerge within neural activations. Results indicate clear temporal and scale-specific thresholds for feature emergence across multiple domains. Notably, spatial analysis reveals unexpected semantic reactivation, with early-layer features re-emerging at later layers, challenging standard assumptions about representational dynamics in transformer models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19440</guid>
|
||||
<pubDate>Mon, 26 May 2025 02:59:54 +0000</pubDate>
|
||||
<title>ImgEdit: A Unified Image Editing Dataset and Benchmark</title>
|
||||
<link>https://arxiv.org/abs/2505.20275</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20275.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, Li Yuan</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Recent advancements in generative models have enabled high-fidelity text-to-image generation. However, open-source image-editing models still lag behind their proprietary counterparts, primarily due to limited high-quality data and insufficient benchmarks. To overcome these limitations, we introduce ImgEdit, a large-scale, high-quality image-editing dataset comprising 1.2 million carefully curated edit pairs, which contain both novel and complex single-turn edits, as well as challenging multi-turn tasks. To ensure the data quality, we employ a multi-stage pipeline that integrates a cutting-edge vision-language model, a detection model, a segmentation model, alongside task-specific in-painting procedures and strict post-processing. ImgEdit surpasses existing datasets in both task novelty and data quality. Using ImgEdit, we train ImgEdit-E1, an editing model using Vision Language Model to process the reference image and editing prompt, which outperforms existing open-source models on multiple tasks, highlighting the value of ImgEdit and model design. For comprehensive evaluation, we introduce ImgEdit-Bench, a benchmark designed to evaluate image editing performance in terms of instruction adherence, editing quality, and detail preservation. It includes a basic testsuite, a challenging single-turn suite, and a dedicated multi-turn suite. We evaluate both open-source and proprietary models, as well as ImgEdit-E1, providing deep analysis and actionable insights into the current behavior of image-editing models. The source data are publicly available on https://github.com/PKU-YuanGroup/ImgEdit.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20275</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:53:33 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI</title>
|
||||
<link>https://arxiv.org/abs/2505.19443</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19443.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ranjan Sapkota, Konstantinos I. Roumeliotis, Manoj Karkee</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> This review presents a comprehensive analysis of two emerging paradigms in AI-assisted software development: vibe coding and agentic coding. While both leverage large language models (LLMs), they differ fundamentally in autonomy, architectural design, and the role of the developer. Vibe coding emphasizes intuitive, human-in-the-loop interaction through prompt-based, conversational workflows that support ideation, experimentation, and creative exploration. In contrast, agentic coding enables autonomous software development through goal-driven agents capable of planning, executing, testing, and iterating tasks with minimal human intervention. We propose a detailed taxonomy spanning conceptual foundations, execution models, feedback loops, safety mechanisms, debugging strategies, and real-world tool ecosystems. Through comparative workflow analysis and 20 detailed use cases, we illustrate how vibe systems thrive in early-stage prototyping and education, while agentic systems excel in enterprise-grade automation, codebase refactoring, and CI/CD integration. We further examine emerging trends in hybrid architectures, where natural language interfaces are coupled with autonomous execution pipelines. Finally, we articulate a future roadmap for agentic AI, outlining the infrastructure needed for trustworthy, explainable, and collaborative systems. Our findings suggest that successful AI software engineering will rely not on choosing one paradigm, but on harmonizing their strengths within a unified, human-centered development lifecycle.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19443</guid>
|
||||
<pubDate>Mon, 26 May 2025 03:00:21 +0000</pubDate>
|
||||
<title>VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction</title>
|
||||
<link>https://arxiv.org/abs/2505.20279</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20279.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Tianlong Chen, Jiachen Li, Zhengzhong Tu, Zhangyang Wang, Rakesh Ranjan</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> The rapid advancement of Large Multimodal Models (LMMs) for 2D images and videos has motivated extending these models to understand 3D scenes, aiming for human-like visual-spatial intelligence. Nevertheless, achieving deep spatial understanding comparable to human capabilities poses significant challenges in model encoding and data acquisition. Existing methods frequently depend on external depth sensors for geometry capture or utilize off-the-shelf algorithms for pre-constructing 3D maps, thereby limiting their scalability, especially with prevalent monocular video inputs and for time-sensitive applications. In this work, we introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding. Leveraging our Spatial-Visual-View Fusion and over 200K curated 3D reconstructive instruction tuning question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial context with language instructions. This enables monocular 3D spatial assistance and embodied reasoning. To facilitate the evaluation of temporal reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark, featuring over 138.6K QA pairs across five distinct tasks focused on evolving spatial relationships. Extensive experiments demonstrate that our model, VLM-3R, not only facilitates robust visual-spatial reasoning but also enables the understanding of temporal 3D context changes, excelling in both accuracy and scalability.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20279</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:56:30 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs</title>
|
||||
<link>https://arxiv.org/abs/2505.19457</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19457.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guilong Lu, Xuntao Guo, Rongjunchen Zhang, Wenqiao Zhu, Ji Liu</p><p><b>Upvotes:</b> 55</p><p><b>Summary:</b> Large language models excel in general tasks, yet assessing their reliability in logic-heavy, precision-critical domains like finance, law, and healthcare remains challenging. To address this, we introduce BizFinBench, the first benchmark specifically designed to evaluate LLMs in real-world financial applications. BizFinBench consists of 6,781 well-annotated queries in Chinese, spanning five dimensions: numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering, grouped into nine fine-grained categories. The benchmark includes both objective and subjective metrics. We also introduce IteraJudge, a novel LLM evaluation method that reduces bias when LLMs serve as evaluators in objective metrics. We benchmark 25 models, including both proprietary and open-source systems. Extensive experiments show that no model dominates across all tasks. Our evaluation reveals distinct capability patterns: (1) In Numerical Calculation, Claude-3.5-Sonnet (63.18) and DeepSeek-R1 (64.04) lead, while smaller models like Qwen2.5-VL-3B (15.92) lag significantly; (2) In Reasoning, proprietary models dominate (ChatGPT-o3: 83.58, Gemini-2.0-Flash: 81.15), with open-source models trailing by up to 19.49 points; (3) In Information Extraction, the performance spread is the largest, with DeepSeek-R1 scoring 71.46, while Qwen3-1.7B scores 11.23; (4) In Prediction Recognition, performance variance is minimal, with top models scoring between 39.16 and 50.00. We find that while current LLMs handle routine finance queries competently, they struggle with complex scenarios requiring cross-concept reasoning. BizFinBench offers a rigorous, business-aligned benchmark for future research. The code and dataset are available at https://github.com/HiThink-Research/BizFinBench.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19457</guid>
|
||||
<pubDate>Mon, 26 May 2025 03:23:02 +0000</pubDate>
|
||||
<title>Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolution</title>
|
||||
<link>https://arxiv.org/abs/2505.20286</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20286.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, Xing Zhou, Dongrui Liu, Ling Yang, Yue Wu, Kaixuan Huang, Shilong Liu, Hongru Wang, Mengdi Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent advances in large language models (LLMs) have enabled agents to autonomously perform complex, open-ended tasks. However, many existing frameworks depend heavily on manually predefined tools and workflows, which hinder their adaptability, scalability, and generalization across domains. In this work, we introduce Alita--a generalist agent designed with the principle of "Simplicity is the ultimate sophistication," enabling scalable agentic reasoning through minimal predefinition and maximal self-evolution. For minimal predefinition, Alita is equipped with only one component for direct problem-solving, making it much simpler and neater than previous approaches that relied heavily on hand-crafted, elaborate tools and workflows. This clean design enhances its potential to generalize to challenging questions, without being limited by tools. For Maximal self-evolution, we enable the creativity of Alita by providing a suite of general-purpose components to autonomously construct, refine, and reuse external capabilities by generating task-related model context protocols (MCPs) from open source, which contributes to scalable agentic reasoning. Notably, Alita achieves 75.15% pass@1 and 87.27% pass@3 accuracy, which is top-ranking among general-purpose agents, on the GAIA benchmark validation dataset, 74.00% and 52.00% pass@1, respectively, on Mathvista and PathVQA, outperforming many agent systems with far greater complexity. More details will be updated at https://github.com/CharlesQ9/Alita{https://github.com/CharlesQ9/Alita}.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20286</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:58:53 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Learning to Reason without External Rewards</title>
|
||||
<link>https://arxiv.org/abs/2505.19590</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19590.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, Dawn Song</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision. We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data. We propose Intuitor, an RLIF method that uses a model's own confidence, termed self-certainty, as its sole reward signal. Intuitor replaces external rewards in Group Relative Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised learning. Experiments demonstrate that Intuitor matches GRPO's performance on mathematical benchmarks while achieving superior generalization to out-of-domain tasks like code generation, without requiring gold solutions or test cases. Our findings show that intrinsic model signals can drive effective learning across domains, offering a scalable alternative to RLVR for autonomous AI systems where verifiable rewards are unavailable. Code is available at https://github.com/sunblaze-ucb/Intuitor</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19590</guid>
|
||||
<pubDate>Mon, 26 May 2025 07:01:06 +0000</pubDate>
|
||||
<title>MotionPro: A Precise Motion Controller for Image-to-Video Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.20287</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20287.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhongwei Zhang, Fuchen Long, Zhaofan Qiu, Yingwei Pan, Wu Liu, Ting Yao, Tao Mei</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Animating images with interactive motion control has garnered popularity for image-to-video (I2V) generation. Modern approaches typically rely on large Gaussian kernels to extend motion trajectories as condition without explicitly defining movement region, leading to coarse motion control and failing to disentangle object and camera moving. To alleviate these, we present MotionPro, a precise motion controller that novelly leverages region-wise trajectory and motion mask to regulate fine-grained motion synthesis and identify target motion category (i.e., object or camera moving), respectively. Technically, MotionPro first estimates the flow maps on each training video via a tracking model, and then samples the region-wise trajectories to simulate inference scenario. Instead of extending flow through large Gaussian kernels, our region-wise trajectory approach enables more precise control by directly utilizing trajectories within local regions, thereby effectively characterizing fine-grained movements. A motion mask is simultaneously derived from the predicted flow maps to capture the holistic motion dynamics of the movement regions. To pursue natural motion control, MotionPro further strengthens video denoising by incorporating both region-wise trajectories and motion mask through feature modulation. More remarkably, we meticulously construct a benchmark, i.e., MC-Bench, with 1.1K user-annotated image-trajectory pairs, for the evaluation of both fine-grained and object-level I2V motion control. Extensive experiments conducted on WebVid-10M and MC-Bench demonstrate the effectiveness of MotionPro. Please refer to our project page for more results: https://zhw-zhang.github.io/MotionPro-page/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20287</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:59:03 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression</title>
|
||||
<link>https://arxiv.org/abs/2505.19602</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19602.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kunjun Li, Zigeng Chen, Cheng-Yen Yang, Jenq-Neng Hwang</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Visual Autoregressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction approach, which yields substantial improvements in efficiency, scalability, and zero-shot generalization. Nevertheless, the coarse-to-fine methodology inherent in VAR results in exponential growth of the KV cache during inference, causing considerable memory consumption and computational redundancy. To address these bottlenecks, we introduce ScaleKV, a novel KV cache compression framework tailored for VAR architectures. ScaleKV leverages two critical observations: varying cache demands across transformer layers and distinct attention patterns at different scales. Based on these insights, ScaleKV categorizes transformer layers into two functional groups: drafters and refiners. Drafters exhibit dispersed attention across multiple scales, thereby requiring greater cache capacity. Conversely, refiners focus attention on the current token map to process local details, consequently necessitating substantially reduced cache capacity. ScaleKV optimizes the multi-scale inference pipeline by identifying scale-specific drafters and refiners, facilitating differentiated cache management tailored to each scale. Evaluation on the state-of-the-art text-to-image VAR model family, Infinity, demonstrates that our approach effectively reduces the required KV cache memory to 10% while preserving pixel-level fidelity.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19602</guid>
|
||||
<pubDate>Mon, 26 May 2025 07:11:42 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>DoctorAgent-RL: A Multi-Agent Collaborative Reinforcement Learning System for Multi-Turn Clinical Dialogue</title>
|
||||
<link>https://arxiv.org/abs/2505.19630</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19630.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yichun Feng, Jiawei Wang, Lu Zhou, Yixue Li</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Large language models (LLMs) have demonstrated excellent capabilities in the field of biomedical question answering, but their application in real-world clinical consultations still faces core challenges. Existing systems rely on a one-way information transmission mode where patients must fully describe their symptoms in a single round, leading to nonspecific diagnostic recommendations when complaints are vague. Traditional multi-turn dialogue methods based on supervised learning are constrained by static data-driven paradigms, lacking generalizability and struggling to intelligently extract key clinical information. To address these limitations, we propose DoctorAgent-RL, a reinforcement learning (RL)-based multi-agent collaborative framework that models medical consultations as a dynamic decision-making process under uncertainty. The doctor agent continuously optimizes its questioning strategy within the RL framework through multi-turn interactions with the patient agent, dynamically adjusting its information-gathering path based on comprehensive rewards from the Consultation Evaluator. This RL fine-tuning mechanism enables LLMs to autonomously develop interaction strategies aligned with clinical reasoning logic, rather than superficially imitating patterns in existing dialogue data. Notably, we constructed MTMedDialog, the first English multi-turn medical consultation dataset capable of simulating patient interactions. Experiments demonstrate that DoctorAgent-RL outperforms existing models in both multi-turn reasoning capability and final diagnostic performance, demonstrating practical value in assisting clinical consultations. https://github.com/JarvisUSTC/DoctorAgent-RL</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19630</guid>
|
||||
<pubDate>Mon, 26 May 2025 07:48:14 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Interleaved Reasoning for Large Language Models via Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.19640</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19640.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Roy Xie, David Qiu, Deepak Gopinath, Dong Lin, Yanchao Sun, Chong Wang, Saloni Potdar, Bhuwan Dhingra</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Long chain-of-thought (CoT) significantly enhances large language models' (LLM) reasoning capabilities. However, the extensive reasoning traces lead to inefficiencies and an increased time-to-first-token (TTFT). We propose a novel training paradigm that uses reinforcement learning (RL) to guide reasoning LLMs to interleave thinking and answering for multi-hop questions. We observe that models inherently possess the ability to perform interleaved reasoning, which can be further enhanced through RL. We introduce a simple yet effective rule-based reward to incentivize correct intermediate steps, which guides the policy model toward correct reasoning paths by leveraging intermediate signals generated during interleaved reasoning. Extensive experiments conducted across five diverse datasets and three RL algorithms (PPO, GRPO, and REINFORCE++) demonstrate consistent improvements over traditional think-answer reasoning, without requiring external tools. Specifically, our approach reduces TTFT by over 80% on average and improves up to 19.3% in Pass@1 accuracy. Furthermore, our method, trained solely on question answering and logical reasoning datasets, exhibits strong generalization ability to complex reasoning datasets such as MATH, GPQA, and MMLU. Additionally, we conduct in-depth analysis to reveal several valuable insights into conditional reward modeling.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19640</guid>
|
||||
<pubDate>Mon, 26 May 2025 07:58:17 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision</title>
|
||||
<link>https://arxiv.org/abs/2505.19706</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19706.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tej Deep Pala, Panshul Sharma, Amir Zadeh, Chuan Li, Soujanya Poria</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Large Language Models (LLMs) are prone to hallucination, especially during multi-hop and reasoning-intensive tasks such as mathematical problem solving. While Outcome Reward Models verify only final answers, Process Reward Models (PRMs) score each intermediate step to steer generation toward coherent solutions. We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM that first classifies math and consistency errors at each step, then combines these fine-grained signals to estimate step correctness. To train PathFinder-PRM, we construct a 400K-sample dataset by enriching the human-annotated PRM800K corpus and RLHFlow Mistral traces with three-dimensional step-level labels. On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3 times less data. When applied to reward guided greedy search, our model yields prm@8 48.3, a +1.5 point gain over the strongest baseline. These results demonstrate that decoupled error detection and reward estimation not only boost fine-grained error detection but also substantially improve end-to-end, reward-guided mathematical reasoning with greater data efficiency.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19706</guid>
|
||||
<pubDate>Mon, 26 May 2025 08:56:36 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Accelerating Nash Learning from Human Feedback via Mirror Prox</title>
|
||||
<link>https://arxiv.org/abs/2505.19731</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19731.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daniil Tiapkin, Daniele Calandriello, Denis Belomestny, Eric Moulines, Alexey Naumov, Kashif Rasul, Michal Valko, Pierre Menard</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Traditional Reinforcement Learning from Human Feedback (RLHF) often relies on reward models, frequently assuming preference structures like the Bradley-Terry model, which may not accurately capture the complexities of real human preferences (e.g., intransitivity). Nash Learning from Human Feedback (NLHF) offers a more direct alternative by framing the problem as finding a Nash equilibrium of a game defined by these preferences. In this work, we introduce Nash Mirror Prox (Nash-MP), an online NLHF algorithm that leverages the Mirror Prox optimization scheme to achieve fast and stable convergence to the Nash equilibrium. Our theoretical analysis establishes that Nash-MP exhibits last-iterate linear convergence towards the beta-regularized Nash equilibrium. Specifically, we prove that the KL-divergence to the optimal policy decreases at a rate of order (1+2beta)^{-N/2}, where N is a number of preference queries. We further demonstrate last-iterate linear convergence for the exploitability gap and uniformly for the span semi-norm of log-probabilities, with all these rates being independent of the size of the action space. Furthermore, we propose and analyze an approximate version of Nash-MP where proximal steps are estimated using stochastic policy gradients, making the algorithm closer to applications. Finally, we detail a practical implementation strategy for fine-tuning large language models and present experiments that demonstrate its competitive performance and compatibility with existing methods.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19731</guid>
|
||||
<pubDate>Mon, 26 May 2025 09:17:32 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Discrete Markov Bridge</title>
|
||||
<link>https://arxiv.org/abs/2505.19752</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19752.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hengli Li, Yuxuan Wang, Song-Chun Zhu, Ying Nian Wu, Zilong Zheng</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> Discrete diffusion has recently emerged as a promising paradigm in discrete data modeling. However, existing methods typically rely on a fixed rate transition matrix during training, which not only limits the expressiveness of latent representations, a fundamental strength of variational methods, but also constrains the overall design space. To address these limitations, we propose Discrete Markov Bridge, a novel framework specifically designed for discrete representation learning. Our approach is built upon two key components: Matrix Learning and Score Learning. We conduct a rigorous theoretical analysis, establishing formal performance guarantees for Matrix Learning and proving the convergence of the overall framework. Furthermore, we analyze the space complexity of our method, addressing practical constraints identified in prior studies. Extensive empirical evaluations validate the effectiveness of the proposed Discrete Markov Bridge, which achieves an Evidence Lower Bound (ELBO) of 1.38 on the Text8 dataset, outperforming established baselines. Moreover, the proposed model demonstrates competitive performance on the CIFAR-10 dataset, achieving results comparable to those obtained by image-specific generation approaches.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19752</guid>
|
||||
<pubDate>Mon, 26 May 2025 09:32:12 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Done Is Better than Perfect: Unlocking Efficient Reasoning by Structured Multi-Turn Decomposition</title>
|
||||
<link>https://arxiv.org/abs/2505.19788</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19788.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zihao Zeng, Xuyao Huang, Boxiu Li, Hao Zhang, Zhijie Deng</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Large Reasoning Models (LRMs) are criticized for the excessively lengthy Chain-of-Thought (CoT) to derive the final answer, suffering from high first-token and overall latency. Typically, the CoT of LRMs mixes multiple thinking units; each unit attempts to produce a candidate answer to the original query. Hence, a natural idea to improve efficiency is to reduce the unit number. Yet, the fact that the thinking units in vanilla CoT cannot be explicitly managed renders doing so challenging. This paper introduces Multi-Turn Decomposition (MinD) to decode conventional CoT into a sequence of explicit, structured, and turn-wise interactions to bridge the gap. In MinD, the model provides a multi-turn response to the query, where each turn embraces a thinking unit and yields a corresponding answer. The subsequent turns can reflect, verify, revise, or explore alternative approaches to both the thinking and answer parts of earlier ones. This not only makes the answer delivered more swiftly, but also enables explicit controls over the iterative reasoning process (i.e., users may halt or continue at any turn). We follow a supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm to realize MinD. We first rephrase the outputs of an LRM into multi-turn formats by prompting another LLM, and then tune the LRM with such data. Observing that the tuned model tends to consume even more tokens than the original one (probably due to that the multi-turn formats introduce additional answer tokens), we advocate leveraging RL algorithms like GRPO to prioritize correct outputs with fewer turns. Trained on the MATH dataset using R1-Distill models, MinD can achieve up to ~70% reduction in both output token usage and time to first token (TTFT), while maintaining competitive performance on reasoning benchmarks such as MATH-500, AIME24, AMC23, and GPQA-Diamond.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19788</guid>
|
||||
<pubDate>Mon, 26 May 2025 10:18:57 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs</title>
|
||||
<link>https://arxiv.org/abs/2505.19800</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19800.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zaid Alyafeai, Maged S. Al-Shaibani, Bernard Ghanem</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Metadata extraction is essential for cataloging and preserving datasets, enabling effective research discovery and reproducibility, especially given the current exponential growth in scientific research. While Masader (Alyafeai et al.,2021) laid the groundwork for extracting a wide range of metadata attributes from Arabic NLP datasets' scholarly articles, it relies heavily on manual annotation. In this paper, we present MOLE, a framework that leverages Large Language Models (LLMs) to automatically extract metadata attributes from scientific papers covering datasets of languages other than Arabic. Our schema-driven methodology processes entire documents across multiple input formats and incorporates robust validation mechanisms for consistent output. Additionally, we introduce a new benchmark to evaluate the research progress on this task. Through systematic analysis of context length, few-shot learning, and web browsing integration, we demonstrate that modern LLMs show promising results in automating this task, highlighting the need for further future work improvements to ensure consistent and reliable performance. We release the code: https://github.com/IVUL-KAUST/MOLE and dataset: https://huggingface.co/datasets/IVUL-KAUST/MOLE for the research community.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19800</guid>
|
||||
<pubDate>Mon, 26 May 2025 10:31:26 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective</title>
|
||||
<link>https://arxiv.org/abs/2505.19815</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19815.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junnan Liu, Hongwei Liu, Linchen Xiao, Shudong Liu, Taolin Zhang, Zihan Ma, Songyang Zhang, Kai Chen</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> We propose a novel framework for comprehending the reasoning capabilities of large language models (LLMs) through the perspective of meta-learning. By conceptualizing reasoning trajectories as pseudo-gradient descent updates to the LLM's parameters, we identify parallels between LLM reasoning and various meta-learning paradigms. We formalize the training process for reasoning tasks as a meta-learning setup, with each question treated as an individual task, and reasoning trajectories serving as the inner loop optimization for adapting model parameters. Once trained on a diverse set of questions, the LLM develops fundamental reasoning capabilities that can generalize to previously unseen questions. Extensive empirical evaluations substantiate the strong connection between LLM reasoning and meta-learning, exploring several issues of significant interest from a meta-learning standpoint. Our work not only enhances the understanding of LLM reasoning but also provides practical insights for improving these models through established meta-learning techniques.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19815</guid>
|
||||
<pubDate>Mon, 26 May 2025 10:52:17 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles</title>
|
||||
<link>https://arxiv.org/abs/2505.19914</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19914.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, Hongli Yu, Qiying Yu, Xuefeng Li, Jiaze Chen, Hao Zhou, Mingxuan Wang</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> Large Language Models (LLMs), such as OpenAI's o1 and DeepSeek's R1, excel at advanced reasoning tasks like math and coding via Reinforcement Learning with Verifiable Rewards (RLVR), but still struggle with puzzles solvable by humans without domain knowledge. We introduce Enigmata, the first comprehensive suite tailored for improving LLMs with puzzle reasoning skills. It includes 36 tasks across seven categories, each with 1) a generator that produces unlimited examples with controllable difficulty and 2) a rule-based verifier for automatic evaluation. This generator-verifier design supports scalable, multi-task RL training, fine-grained analysis, and seamless RLVR integration. We further propose Enigmata-Eval, a rigorous benchmark, and develop optimized multi-task RLVR strategies. Our trained model, Qwen2.5-32B-Enigmata, consistently surpasses o3-mini-high and o1 on the puzzle reasoning benchmarks like Enigmata-Eval, ARC-AGI (32.8%), and ARC-AGI 2 (0.6%). It also generalizes well to out-of-domain puzzle benchmarks and mathematical reasoning, with little multi-tasking trade-off. When trained on larger models like Seed1.5-Thinking (20B activated parameters and 200B total parameters), puzzle data from Enigmata further boosts SoTA performance on advanced math and STEM reasoning tasks such as AIME (2024-2025), BeyondAIME and GPQA (Diamond), showing nice generalization benefits of Enigmata. This work offers a unified, controllable framework for advancing logical reasoning in LLMs. Resources of this work can be found at https://seed-enigmata.github.io.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19914</guid>
|
||||
<pubDate>Mon, 26 May 2025 12:40:31 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions</title>
|
||||
<link>https://arxiv.org/abs/2505.19949</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19949.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siqi Kou, Qingyuan Tian, Hanwen Xu, Zihao Zeng, Zhijie Deng</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Large language models (LLMs) have demonstrated remarkable reasoning capabilities in math and coding, often bolstered by post-training on the chain-of-thoughts (CoTs) generated by stronger models. However, existing strategies for curating such training data predominantly rely on heuristics, limiting generalizability and failing to capture subtleties underlying in data. To address these limitations, we leverage influence functions to systematically attribute LLMs' reasoning ability on math and coding to individual training examples, sequences, and tokens, enabling deeper insights into effective data characteristics. Our Influence-based Reasoning Attribution (Infra) uncovers nontrivial cross-domain effects across math and coding tasks: high-difficulty math examples improve both math and code reasoning, while low-difficulty code tasks most effectively benefit code reasoning. Based on these findings, we introduce a simple yet effective dataset reweighting strategy by flipping task difficulty, which doubles AIME24 accuracy from 10\% to 20\% and boosts LiveCodeBench accuracy from 33.8\% to 35.3\% for Qwen2.5-7B-Instruct. Moreover, our fine-grained attribution reveals that the sequence-level exploratory behaviors enhance reasoning performance in both math and code, and the token-level influence patterns are distinct for math and code reasoning: the former prefers natural language logic connectors and the latter emphasizes structural syntax.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19949</guid>
|
||||
<pubDate>Mon, 26 May 2025 13:15:26 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research</title>
|
||||
<link>https://arxiv.org/abs/2505.19955</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.19955.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, Bryan Hooi</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Recent advancements in AI agents have demonstrated their growing potential to drive and support scientific discovery. In this work, we introduce MLR-Bench, a comprehensive benchmark for evaluating AI agents on open-ended machine learning research. MLR-Bench includes three key components: (1) 201 research tasks sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2) MLR-Judge, an automated evaluation framework combining LLM-based reviewers with carefully designed review rubrics to assess research quality; and (3) MLR-Agent, a modular agent scaffold capable of completing research tasks through four stages: idea generation, proposal formulation, experimentation, and paper writing. Our framework supports both stepwise assessment across these distinct research stages, and end-to-end evaluation of the final research paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced coding agent, finding that while LLMs are effective at generating coherent ideas and well-structured papers, current coding agents frequently (e.g., in 80% of the cases) produce fabricated or invalidated experimental results--posing a major barrier to scientific reliability. We validate MLR-Judge through human evaluation, showing high agreement with expert reviewers, supporting its potential as a scalable tool for research evaluation. We open-source MLR-Bench to help the community benchmark, diagnose, and improve AI research agents toward trustworthy and transparent scientific discovery.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.19955</guid>
|
||||
<pubDate>Mon, 26 May 2025 13:18:37 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>REARANK: Reasoning Re-ranking Agent via Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.20046</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20046.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, Aishwarya Agrawal</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> We present REARANK, a large language model (LLM)-based listwise reasoning reranking agent. REARANK explicitly reasons before reranking, significantly improving both performance and interpretability. Leveraging reinforcement learning and data augmentation, REARANK achieves substantial improvements over baseline models across popular information retrieval benchmarks, notably requiring only 179 annotated samples. Built on top of Qwen2.5-7B, our REARANK-7B demonstrates performance comparable to GPT-4 on both in-domain and out-of-domain benchmarks and even surpasses GPT-4 on reasoning-intensive BRIGHT benchmarks. These results underscore the effectiveness of our approach and highlight how reinforcement learning can enhance LLM reasoning capabilities in reranking.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20046</guid>
|
||||
<pubDate>Mon, 26 May 2025 14:31:48 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs</title>
|
||||
<link>https://arxiv.org/abs/2505.20139</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20139.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu, Quy Duc Do, Ziyan Jiang, Ping Nie, Wenhu Chen</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and 2) conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps, even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20139</guid>
|
||||
<pubDate>Mon, 26 May 2025 15:40:42 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Hard Negative Contrastive Learning for Fine-Grained Geometric Understanding in Large Multimodal Models</title>
|
||||
<link>https://arxiv.org/abs/2505.20152</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20152.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kai Sun, Yushi Bai, Zhen Yang, Jiajie Zhang, Ji Qi, Lei Hou, Juanzi Li</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Benefiting from contrastively trained visual encoders on large-scale natural scene images, Large Multimodal Models (LMMs) have achieved remarkable performance across various visual perception tasks. However, the inherent limitations of contrastive learning upon summarized descriptions fundamentally restrict the capabilities of models in meticulous reasoning, particularly in crucial scenarios of geometric problem-solving. To enhance geometric understanding, we propose a novel hard negative contrastive learning framework for the vision encoder, which combines image-based contrastive learning using generation-based hard negatives created by perturbing diagram generation code, and text-based contrastive learning using rule-based negatives derived from modified geometric descriptions and retrieval-based negatives selected based on caption similarity. We train CLIP using our strong negative learning method, namely MMCLIP (Multimodal Math CLIP), and subsequently train an LMM for geometric problem-solving. Experiments show that our trained model, MMGeoLM, significantly outperforms other open-source models on three geometric reasoning benchmarks. Even with a size of 7B, it can rival powerful closed-source models like GPT-4o. We further study the impact of different negative sample construction methods and the number of negative samples on the geometric reasoning performance of LMM, yielding fruitful conclusions. The code and dataset are available at https://github.com/THU-KEG/MMGeoLM.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20152</guid>
|
||||
<pubDate>Mon, 26 May 2025 15:55:28 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.20225</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20225.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hao Kang, Zichun Yu, Chenyan Xiong</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Recent large language models such as Gemini-1.5, DeepSeek-V3, and Llama-4 increasingly adopt Mixture-of-Experts (MoE) architectures, which offer strong efficiency-performance trade-offs by activating only a fraction of the model per token. Yet academic researchers still lack a fully open, end-to-end MoE platform for investigating scaling, routing, and expert behavior. We release FLAME-MoE, a completely open-source research suite composed of seven decoder-only models, ranging from 38M to 1.7B active parameters, whose architecture--64 experts with top-8 gating and 2 shared experts--closely reflects modern production LLMs. All training data pipelines, scripts, logs, and checkpoints are publicly available to enable reproducible experimentation. Across six evaluation tasks, FLAME-MoE improves average accuracy by up to 3.4 points over dense baselines trained with identical FLOPs. Leveraging full training trace transparency, we present initial analyses showing that (i) experts increasingly specialize on distinct token subsets, (ii) co-activation matrices remain sparse, reflecting diverse expert usage, and (iii) routing behavior stabilizes early in training. All code, training logs, and model checkpoints are available at https://github.com/cmu-flame/FLAME-MoE.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20225</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:06:25 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs</title>
|
||||
<link>https://arxiv.org/abs/2505.20254</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20254.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiangchen Song, Aashiq Muhamed, Yujia Zheng, Lingjing Kong, Zeyu Tang, Mona T. Diab, Virginia Smith, Kun Zhang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Sparse Autoencoders (SAEs) are a prominent tool in mechanistic interpretability (MI) for decomposing neural network activations into interpretable features. However, the aspiration to identify a canonical set of features is challenged by the observed inconsistency of learned SAE features across different training runs, undermining the reliability and efficiency of MI research. This position paper argues that mechanistic interpretability should prioritize feature consistency in SAEs -- the reliable convergence to equivalent feature sets across independent runs. We propose using the Pairwise Dictionary Mean Correlation Coefficient (PW-MCC) as a practical metric to operationalize consistency and demonstrate that high levels are achievable (0.80 for TopK SAEs on LLM activations) with appropriate architectural choices. Our contributions include detailing the benefits of prioritizing consistency; providing theoretical grounding and synthetic validation using a model organism, which verifies PW-MCC as a reliable proxy for ground-truth recovery; and extending these findings to real-world LLM data, where high feature consistency strongly correlates with the semantic similarity of learned feature explanations. We call for a community-wide shift towards systematically measuring feature consistency to foster robust cumulative progress in MI.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20254</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:31:36 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration</title>
|
||||
<link>https://arxiv.org/abs/2505.20256</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20256.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hao Zhong, Muzhi Zhu, Zongze Du, Zheng Huang, Canyu Zhao, Mingyu Liu, Wen Wang, Hao Chen, Chunhua Shen</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Long-horizon video-audio reasoning and fine-grained pixel understanding impose conflicting requirements on omnimodal models: dense temporal coverage demands many low-resolution frames, whereas precise grounding calls for high-resolution inputs. We tackle this trade-off with a two-system architecture: a Global Reasoning System selects informative keyframes and rewrites the task at low spatial cost, while a Detail Understanding System performs pixel-level grounding on the selected high-resolution snippets. Because ``optimal'' keyframe selection and reformulation are ambiguous and hard to supervise, we formulate them as a reinforcement learning (RL) problem and present Omni-R1, an end-to-end RL framework built on Group Relative Policy Optimization. Omni-R1 trains the Global Reasoning System through hierarchical rewards obtained via online collaboration with the Detail Understanding System, requiring only one epoch of RL on small task splits. Experiments on two challenging benchmarks, namely Referring Audio-Visual Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show that Omni-R1 not only surpasses strong supervised baselines but also outperforms specialized state-of-the-art models, while substantially improving out-of-domain generalization and mitigating multimodal hallucination. Our results demonstrate the first successful application of RL to large-scale omnimodal reasoning and highlight a scalable path toward universally foundation models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20256</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:34:06 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>ARM: Adaptive Reasoning Model</title>
|
||||
<link>https://arxiv.org/abs/2505.20258</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20258.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, Yanghua Xiao</p><p><b>Upvotes:</b> 40</p><p><b>Summary:</b> While large reasoning models demonstrate strong performance on complex tasks, they lack the ability to adjust reasoning token usage based on task difficulty. This often leads to the "overthinking" problem -- excessive and unnecessary reasoning -- which, although potentially mitigated by human intervention to control the token budget, still fundamentally contradicts the goal of achieving fully autonomous AI. In this work, we propose Adaptive Reasoning Model (ARM), a reasoning model capable of adaptively selecting appropriate reasoning formats based on the task at hand. These formats include three efficient ones -- Direct Answer, Short CoT, and Code -- as well as a more elaborate format, Long CoT. To train ARM, we introduce Ada-GRPO, an adaptation of Group Relative Policy Optimization (GRPO), which addresses the format collapse issue in traditional GRPO. Ada-GRPO enables ARM to achieve high token efficiency, reducing tokens by an average of 30%, and up to 70%, while maintaining performance comparable to the model that relies solely on Long CoT. Furthermore, not only does it improve inference efficiency through reduced token generation, but it also brings a 2x speedup in training. In addition to the default Adaptive Mode, ARM supports two additional reasoning modes: 1) Instruction-Guided Mode, which allows users to explicitly specify the reasoning format via special tokens -- ideal when the appropriate format is known for a batch of tasks. 2) Consensus-Guided Mode, which aggregates the outputs of the three efficient formats and resorts to Long CoT in case of disagreement, prioritizing performance with higher token usage.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20258</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:38:50 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Lifelong Safety Alignment for Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.20259</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20259.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haoyu Wang, Zeyu Qin, Yifei Zhao, Chao Du, Min Lin, Xueqian Wang, Tianyu Pang</p><p><b>Upvotes:</b> 22</p><p><b>Summary:</b> LLMs have made impressive progress, but their growing capabilities also expose them to highly flexible jailbreaking attacks designed to bypass safety alignment. While many existing defenses focus on known types of attacks, it is more critical to prepare LLMs for unseen attacks that may arise during deployment. To address this, we propose a lifelong safety alignment framework that enables LLMs to continuously adapt to new and evolving jailbreaking strategies. Our framework introduces a competitive setup between two components: a Meta-Attacker, trained to actively discover novel jailbreaking strategies, and a Defender, trained to resist them. To effectively warm up the Meta-Attacker, we first leverage the GPT-4o API to extract key insights from a large collection of jailbreak-related research papers. Through iterative training, the first iteration Meta-Attacker achieves a 73% attack success rate (ASR) on RR and a 57% transfer ASR on LAT using only single-turn attacks. Meanwhile, the Defender progressively improves its robustness and ultimately reduces the Meta-Attacker's success rate to just 7%, enabling safer and more reliable deployment of LLMs in open-ended environments. The code is available at https://github.com/sail-sg/LifelongSafetyAlignment.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20259</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:40:40 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>The Coverage Principle: A Framework for Understanding Compositional Generalization</title>
|
||||
<link>https://arxiv.org/abs/2505.20278</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20278.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hoyeon Chang, Jinho Park, Hanseul Cho, Sohee Yang, Miyoung Ko, Hyeonbin Hwang, Seungpil Won, Dohaeng Lee, Youbin Ahn, Minjoon Seo</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Large language models excel at pattern matching, yet often fall short in systematic compositional generalization. We propose the coverage principle: a data-centric framework showing that models relying primarily on pattern matching for compositional tasks cannot reliably generalize beyond substituting fragments that yield identical results when used in the same contexts. We demonstrate that this framework has a strong predictive power for the generalization capabilities of Transformers. First, we derive and empirically confirm that the training data required for two-hop generalization grows at least quadratically with the token set size, and the training data efficiency does not improve with 20x parameter scaling. Second, for compositional tasks with path ambiguity where one variable affects the output through multiple computational paths, we show that Transformers learn context-dependent state representations that undermine both performance and interoperability. Third, Chain-of-Thought supervision improves training data efficiency for multi-hop tasks but still struggles with path ambiguity. Finally, we outline a mechanism-based taxonomy that distinguishes three ways neural networks can generalize: structure-based (bounded by coverage), property-based (leveraging algebraic invariances), and shared-operator (through function reuse). This conceptual lens contextualizes our results and highlights where new architectural ideas are needed to achieve systematic compositionally. Overall, the coverage principle provides a unified lens for understanding compositional reasoning, and underscores the need for fundamental architectural or training innovations to achieve truly systematic compositionality.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20278</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:55:15 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>EgoZero: Robot Learning from Smart Glasses</title>
|
||||
<link>https://arxiv.org/abs/2505.20290</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20290.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Vincent Liu, Ademi Adeniji, Haotian Zhan, Raunaq Bhirangi, Pieter Abbeel, Lerrel Pinto</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Despite recent progress in general purpose robotics, robot policies still lag far behind basic human capabilities in the real world. Humans interact constantly with the physical world, yet this rich data resource remains largely untapped in robot learning. We propose EgoZero, a minimal system that learns robust manipulation policies from human demonstrations captured with Project Aria smart glasses, and zero robot data. EgoZero enables: (1) extraction of complete, robot-executable actions from in-the-wild, egocentric, human demonstrations, (2) compression of human visual observations into morphology-agnostic state representations, and (3) closed-loop policy learning that generalizes morphologically, spatially, and semantically. We deploy EgoZero policies on a gripper Franka Panda robot and demonstrate zero-shot transfer with 70% success rate over 7 manipulation tasks and only 20 minutes of data collection per task. Our results suggest that in-the-wild human data can serve as a scalable foundation for real-world robot learning - paving the way toward a future of abundant, diverse, and naturalistic training data for robots. Code and videos are available at https://egozero-robot.github.io.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20290</guid>
|
||||
<title>VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection</title>
|
||||
<link>https://arxiv.org/abs/2505.20289</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20289.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zeyi Huang, Yuyang Ji, Anirudh Sundara Rajan, Zefan Cai, Wen Xiao, Junjie Hu, Yong Jae Lee</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> We introduce VisTA, a new reinforcement learning framework that empowers visual agents to dynamically explore, select, and combine tools from a diverse library based on empirical performance. Existing methods for tool-augmented reasoning either rely on training-free prompting or large-scale fine-tuning; both lack active tool exploration and typically assume limited tool diversity, and fine-tuning methods additionally demand extensive human supervision. In contrast, VisTA leverages end-to-end reinforcement learning to iteratively refine sophisticated, query-specific tool selection strategies, using task outcomes as feedback signals. Through Group Relative Policy Optimization (GRPO), our framework enables an agent to autonomously discover effective tool-selection pathways without requiring explicit reasoning supervision. Experiments on the ChartQA, Geometry3K, and BlindTest benchmarks demonstrate that VisTA achieves substantial performance gains over training-free baselines, especially on out-of-distribution examples. These results highlight VisTA's ability to enhance generalization, adaptively utilize diverse tools, and pave the way for flexible, experience-driven visual reasoning systems.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20289</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:59:17 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>GLEAM: Learning Generalizable Exploration Policy for Active Mapping in Complex 3D Indoor Scenes</title>
|
||||
<link>https://arxiv.org/abs/2505.20294</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20294.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiao Chen, Tai Wang, Quanyi Li, Tao Huang, Jiangmiao Pang, Tianfan Xue</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Generalizable active mapping in complex unknown environments remains a critical challenge for mobile robots. Existing methods, constrained by insufficient training data and conservative exploration strategies, exhibit limited generalizability across scenes with diverse layouts and complex connectivity. To enable scalable training and reliable evaluation, we introduce GLEAM-Bench, the first large-scale benchmark designed for generalizable active mapping with 1,152 diverse 3D scenes from synthetic and real-scan datasets. Building upon this foundation, we propose GLEAM, a unified generalizable exploration policy for active mapping. Its superior generalizability comes mainly from our semantic representations, long-term navigable goals, and randomized strategies. It significantly outperforms state-of-the-art methods, achieving 66.50% coverage (+9.49%) with efficient trajectories and improved mapping accuracy on 128 unseen complex scenes. Project page: https://xiao-chen.tech/gleam/.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20294</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:59:52 +0000</pubDate>
|
||||
<title>OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.20292</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20292.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shenghai Yuan, Xianyi He, Yufan Deng, Yang Ye, Jinfa Huang, Bin Lin, Chongyang Ma, Jiebo Luo, Li Yuan</p><p><b>Upvotes:</b> 49</p><p><b>Summary:</b> Subject-to-Video (S2V) generation aims to create videos that faithfully incorporate reference content, providing enhanced flexibility in the production of videos. To establish the infrastructure for S2V generation, we propose OpenS2V-Nexus, consisting of (i) OpenS2V-Eval, a fine-grained benchmark, and (ii) OpenS2V-5M, a million-scale dataset. In contrast to existing S2V benchmarks inherited from VBench that focus on global and coarse-grained assessment of generated videos, OpenS2V-Eval focuses on the model's ability to generate subject-consistent videos with natural subject appearance and identity fidelity. For these purposes, OpenS2V-Eval introduces 180 prompts from seven major categories of S2V, which incorporate both real and synthetic test data. Furthermore, to accurately align human preferences with S2V benchmarks, we propose three automatic metrics, NexusScore, NaturalScore and GmeScore, to separately quantify subject consistency, naturalness, and text relevance in generated videos. Building on this, we conduct a comprehensive evaluation of 16 representative S2V models, highlighting their strengths and weaknesses across different content. Moreover, we create the first open-source large-scale S2V generation dataset OpenS2V-5M, which consists of five million high-quality 720P subject-text-video triples. Specifically, we ensure subject-information diversity in our dataset by (1) segmenting subjects and building pairing information via cross-video associations and (2) prompting GPT-Image-1 on raw frames to synthesize multi-view representations. Through OpenS2V-Nexus, we deliver a robust infrastructure to accelerate future S2V generation research.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20292</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:59:46 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>DiSA: Diffusion Step Annealing in Autoregressive Image Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.20297</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20297.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qinyu Zhao, Jaskirat Singh, Ming Xu, Akshay Asthana, Stephen Gould, Liang Zheng</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> An increasing number of autoregressive models, such as MAR, FlowAR, xAR, and Harmon adopt diffusion sampling to improve the quality of image generation. However, this strategy leads to low inference efficiency, because it usually takes 50 to 100 steps for diffusion to sample a token. This paper explores how to effectively address this issue. Our key motivation is that as more tokens are generated during the autoregressive process, subsequent tokens follow more constrained distributions and are easier to sample. To intuitively explain, if a model has generated part of a dog, the remaining tokens must complete the dog and thus are more constrained. Empirical evidence supports our motivation: at later generation stages, the next tokens can be well predicted by a multilayer perceptron, exhibit low variance, and follow closer-to-straight-line denoising paths from noise to tokens. Based on our finding, we introduce diffusion step annealing (DiSA), a training-free method which gradually uses fewer diffusion steps as more tokens are generated, e.g., using 50 steps at the beginning and gradually decreasing to 5 steps at later stages. Because DiSA is derived from our finding specific to diffusion in autoregressive models, it is complementary to existing acceleration methods designed for diffusion alone. DiSA can be implemented in only a few lines of code on existing models, and albeit simple, achieves 5-10times faster inference for MAR and Harmon and 1.4-2.5times for FlowAR and xAR, while maintaining the generation quality.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20297</guid>
|
||||
<pubDate>Mon, 26 May 2025 17:59:57 +0000</pubDate>
|
||||
<title>MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness</title>
|
||||
<link>https://arxiv.org/abs/2505.20426</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20426.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yunlong Tang, Pinxin Liu, Mingqian Feng, Zhangyun Tan, Rui Mao, Chao Huang, Jing Bi, Yunzhong Xiao, Susan Liang, Hang Hua, Ali Vosoughi, Luchuan Song, Zeliang Zhang, Chenliang Xu</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Understanding perspective is fundamental to human visual perception, yet the extent to which multimodal large language models (MLLMs) internalize perspective geometry remains unclear. We introduce MMPerspective, the first benchmark specifically designed to systematically evaluate MLLMs' understanding of perspective through 10 carefully crafted tasks across three complementary dimensions: Perspective Perception, Reasoning, and Robustness. Our benchmark comprises 2,711 real-world and synthetic image instances with 5,083 question-answer pairs that probe key capabilities, such as vanishing point perception and counting, perspective type reasoning, line relationship understanding in 3D space, invariance to perspective-preserving transformations, etc. Through a comprehensive evaluation of 43 state-of-the-art MLLMs, we uncover significant limitations: while models demonstrate competence on surface-level perceptual tasks, they struggle with compositional reasoning and maintaining spatial consistency under perturbations. Our analysis further reveals intriguing patterns between model architecture, scale, and perspective capabilities, highlighting both robustness bottlenecks and the benefits of chain-of-thought prompting. MMPerspective establishes a valuable testbed for diagnosing and advancing spatial understanding in vision-language systems. Resources available at: https://yunlong10.github.io/MMPerspective/</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20426</guid>
|
||||
<pubDate>Mon, 26 May 2025 18:20:22 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2505.20561</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20561.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shenao Zhang, Yaqing Wang, Yinxiao Liu, Tianqi Liu, Peter Grabowski, Eugene Ie, Zhaoran Wang, Yunxuan Li</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Large Language Models (LLMs) trained via Reinforcement Learning (RL) have exhibited strong reasoning capabilities and emergent reflective behaviors, such as backtracking and error correction. However, conventional Markovian RL confines exploration to the training phase to learn an optimal deterministic policy and depends on the history contexts only through the current state. Therefore, it remains unclear whether reflective reasoning will emerge during Markovian RL training, or why they are beneficial at test time. To remedy this, we recast reflective exploration within the Bayes-Adaptive RL framework, which explicitly optimizes the expected return under a posterior distribution over Markov decision processes. This Bayesian formulation inherently incentivizes both reward-maximizing exploitation and information-gathering exploration via belief updates. Our resulting algorithm, BARL, instructs the LLM to stitch and switch strategies based on the observed outcomes, offering principled guidance on when and how the model should reflectively explore. Empirical results on both synthetic and mathematical reasoning tasks demonstrate that BARL outperforms standard Markovian RL approaches at test time, achieving superior token efficiency with improved exploration effectiveness. Our code is available at https://github.com/shenao-zhang/BARL.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20561</guid>
|
||||
<pubDate>Mon, 26 May 2025 22:51:00 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>FinTagging: An LLM-ready Benchmark for Extracting and Structuring Financial Information</title>
|
||||
<link>https://arxiv.org/abs/2505.20650</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20650.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Xiao-Yang Liu, Jimin Huang, Qianqian Xie</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> We introduce FinTagging, the first full-scope, table-aware XBRL benchmark designed to evaluate the structured information extraction and semantic alignment capabilities of large language models (LLMs) in the context of XBRL-based financial reporting. Unlike prior benchmarks that oversimplify XBRL tagging as flat multi-class classification and focus solely on narrative text, FinTagging decomposes the XBRL tagging problem into two subtasks: FinNI for financial entity extraction and FinCL for taxonomy-driven concept alignment. It requires models to jointly extract facts and align them with the full 10k+ US-GAAP taxonomy across both unstructured text and structured tables, enabling realistic, fine-grained evaluation. We assess a diverse set of LLMs under zero-shot settings, systematically analyzing their performance on both subtasks and overall tagging accuracy. Our results reveal that, while LLMs demonstrate strong generalization in information extraction, they struggle with fine-grained concept alignment, particularly in disambiguating closely related taxonomy entries. These findings highlight the limitations of existing LLMs in fully automating XBRL tagging and underscore the need for improved semantic reasoning and schema-aware modeling to meet the demands of accurate financial disclosure. Code is available at our GitHub repository and data is at our Hugging Face repository.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20650</guid>
|
||||
<pubDate>Tue, 27 May 2025 02:55:53 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Rendering-Aware Reinforcement Learning for Vector Graphics Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.20793</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20793.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Juan A. Rodriguez, Haotian Zhang, Abhay Puri, Aarash Feizi, Rishav Pramanik, Pascal Wichmann, Arnab Mondal, Mohammad Reza Samsami, Rabiul Awal, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Scalable Vector Graphics (SVG) offer a powerful format for representing visual designs as interpretable code. Recent advances in vision-language models (VLMs) have enabled high-quality SVG generation by framing the problem as a code generation task and leveraging large-scale pretraining. VLMs are particularly suitable for this task as they capture both global semantics and fine-grained visual patterns, while transferring knowledge across vision, natural language, and code domains. However, existing VLM approaches often struggle to produce faithful and efficient SVGs because they never observe the rendered images during training. Although differentiable rendering for autoregressive SVG code generation remains unavailable, rendered outputs can still be compared to original inputs, enabling evaluative feedback suitable for reinforcement learning (RL). We introduce RLRF(Reinforcement Learning from Rendering Feedback), an RL method that enhances SVG generation in autoregressive VLMs by leveraging feedback from rendered SVG outputs. Given an input image, the model generates SVG roll-outs that are rendered and compared to the original image to compute a reward. This visual fidelity feedback guides the model toward producing more accurate, efficient, and semantically coherent SVGs. RLRF significantly outperforms supervised fine-tuning, addressing common failure modes and enabling precise, high-quality SVG generation with strong structural understanding and generalization.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20793</guid>
|
||||
<pubDate>Tue, 27 May 2025 06:56:00 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individuals</title>
|
||||
<link>https://arxiv.org/abs/2505.21062</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21062.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Davide Lobba, Fulvio Sanguigni, Bin Ren, Marcella Cornia, Rita Cucchiara, Nicu Sebe</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> While virtual try-on (VTON) systems aim to render a garment onto a target person image, this paper tackles the novel task of virtual try-off (VTOFF), which addresses the inverse problem: generating standardized product images of garments from real-world photos of clothed individuals. Unlike VTON, which must resolve diverse pose and style variations, VTOFF benefits from a consistent and well-defined output format -- typically a flat, lay-down-style representation of the garment -- making it a promising tool for data generation and dataset enhancement. However, existing VTOFF approaches face two major limitations: (i) difficulty in disentangling garment features from occlusions and complex poses, often leading to visual artifacts, and (ii) restricted applicability to single-category garments (e.g., upper-body clothes only), limiting generalization. To address these challenges, we present Text-Enhanced MUlti-category Virtual Try-Off (TEMU-VTOFF), a novel architecture featuring a dual DiT-based backbone with a modified multimodal attention mechanism for robust garment feature extraction. Our architecture is designed to receive garment information from multiple modalities like images, text, and masks to work in a multi-category setting. Finally, we propose an additional alignment module to further refine the generated visual details. Experiments on VITON-HD and Dress Code datasets show that TEMU-VTOFF sets a new state-of-the-art on the VTOFF task, significantly improving both visual quality and fidelity to the target garments.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21062</guid>
|
||||
<pubDate>Tue, 27 May 2025 11:47:51 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Minute-Long Videos with Dual Parallelisms</title>
|
||||
<link>https://arxiv.org/abs/2505.21070</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21070.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zeqing Wang, Bowen Zheng, Xingyi Yang, Yuecong Xu, Xinchao Wang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos. To address this, we propose a novel distributed inference strategy, termed DualParal. The core idea is that, instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs. However, a naive implementation of this division faces a key limitation: since diffusion models require synchronized noise levels across frames, this implementation leads to the serialization of original parallelisms. We leverage a block-wise denoising scheme to handle this. Namely, we process a sequence of frame blocks through the pipeline with progressively decreasing noise levels. Each GPU handles a specific block and layer subset while passing previous results to the next GPU, enabling asynchronous computation and communication. To further optimize performance, we incorporate two key enhancements. Firstly, a feature cache is implemented on each GPU to store and reuse features from the prior block as context, minimizing inter-GPU communication and redundant computation. Secondly, we employ a coordinated noise initialization strategy, ensuring globally consistent temporal dynamics by sharing initial noise patterns across GPUs without extra resource costs. Together, these enable fast, artifact-free, and infinitely long video generation. Applied to the latest diffusion transformer video generator, our method efficiently produces 1,025-frame videos with up to 6.54times lower latency and 1.48times lower memory cost on 8timesRTX 4090 GPUs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21070</guid>
|
||||
<pubDate>Tue, 27 May 2025 11:55:22 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Thinker: Learning to Think Fast and Slow</title>
|
||||
<link>https://arxiv.org/abs/2505.21097</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21097.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Stephen Chung, Wenyu Du, Jie Fu</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Recent studies show that the reasoning capabilities of Large Language Models (LLMs) can be improved by applying Reinforcement Learning (RL) to question-answering (QA) tasks in areas such as math and coding. With a long context length, LLMs may learn to perform search, as indicated by the self-correction behavior observed in DeepSeek R1. However, this search behavior is often imprecise and lacks confidence, resulting in long, redundant responses and highlighting deficiencies in intuition and verification. Inspired by the Dual Process Theory in psychology, we introduce a simple modification to the QA task that includes four stages: Fast Thinking, where the LLM must answer within a strict token budget; Verification, where the model evaluates its initial response; Slow Thinking, where it refines the initial response with more deliberation; and Summarization, where it distills the refinement from the previous stage into precise steps. Our proposed task improves average accuracy from 24.9% to 27.9% for Qwen2.5-1.5B, and from 45.9% to 49.8% for DeepSeek-R1-Qwen-1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone achieves 26.8% accuracy using fewer than 1000 tokens, demonstrating substantial inference efficiency gains. These findings suggest that intuition and deliberative reasoning are distinct, complementary systems benefiting from targeted training.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21097</guid>
|
||||
<pubDate>Tue, 27 May 2025 12:22:46 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning</title>
|
||||
<link>https://arxiv.org/abs/2505.21178</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21178.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mingyang Song, Mao Zheng</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> As test-time scaling becomes a pivotal research frontier in Large Language Models (LLMs) development, contemporary and advanced post-training methodologies increasingly focus on extending the generation length of long Chain-of-Thought (CoT) responses to enhance reasoning capabilities toward DeepSeek R1-like performance. However, recent studies reveal a persistent overthinking phenomenon in state-of-the-art reasoning models, manifesting as excessive redundancy or repetitive thinking patterns in long CoT responses. To address this issue, in this paper, we propose a simple yet effective two-stage reinforcement learning framework for achieving concise reasoning in LLMs, named ConciseR. Specifically, the first stage, using more training steps, aims to incentivize the model's reasoning capabilities via Group Relative Policy Optimization with clip-higher and dynamic sampling components (GRPO++), and the second stage, using fewer training steps, explicitly enforces conciseness and improves efficiency via Length-aware Group Relative Policy Optimization (L-GRPO). Significantly, ConciseR only optimizes response length once all rollouts of a sample are correct, following the "walk before you run" principle. Extensive experimental results demonstrate that our ConciseR model, which generates more concise CoT reasoning responses, outperforms recent state-of-the-art reasoning models with zero RL paradigm across AIME 2024, MATH-500, AMC 2023, Minerva, and Olympiad benchmarks.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21178</guid>
|
||||
<pubDate>Tue, 27 May 2025 13:29:51 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Exploring the Latent Capacity of LLMs for One-Step Text Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.21189</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21189.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Gleb Mezentsev, Ivan Oseledets</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts - up to thousands of tokens - via autoregressive generation from just one specially trained input embedding. In this work, we explore whether such reconstruction is possible without autoregression. We show that frozen LLMs can generate hundreds of accurate tokens in just one forward pass, when provided with only two learned embeddings. This reveals a surprising and underexplored capability of LLMs - multi-token generation without iterative decoding. We investigate the behaviour of these embeddings and provide insight into the type of information they encode. We also empirically show that although these representations are not unique for a given text, they form connected and local regions in embedding space - a property that suggests the potential of learning a dedicated encoder into that space.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21189</guid>
|
||||
<pubDate>Tue, 27 May 2025 13:39:24 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Sci-Fi: Symmetric Constraint for Frame Inbetweening</title>
|
||||
<link>https://arxiv.org/abs/2505.21205</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21205.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Liuhan Chen, Xiaodong Cun, Xiaoyu Li, Xianyi He, Shenghai Yuan, Jie Chen, Ying Shan, Li Yuan</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Frame inbetweening aims to synthesize intermediate video sequences conditioned on the given start and end frames. Current state-of-the-art methods mainly extend large-scale pre-trained Image-to-Video Diffusion models (I2V-DMs) by incorporating end-frame constraints via directly fine-tuning or omitting training. We identify a critical limitation in their design: Their injections of the end-frame constraint usually utilize the same mechanism that originally imposed the start-frame (single image) constraint. However, since the original I2V-DMs are adequately trained for the start-frame condition in advance, naively introducing the end-frame constraint by the same mechanism with much less (even zero) specialized training probably can't make the end frame have a strong enough impact on the intermediate content like the start frame. This asymmetric control strength of the two frames over the intermediate content likely leads to inconsistent motion or appearance collapse in generated frames. To efficiently achieve symmetric constraints of start and end frames, we propose a novel framework, termed Sci-Fi, which applies a stronger injection for the constraint of a smaller training scale. Specifically, it deals with the start-frame constraint as before, while introducing the end-frame constraint by an improved mechanism. The new mechanism is based on a well-designed lightweight module, named EF-Net, which encodes only the end frame and expands it into temporally adaptive frame-wise features injected into the I2V-DM. This makes the end-frame constraint as strong as the start-frame constraint, enabling our Sci-Fi to produce more harmonious transitions in various scenarios. Extensive experiments prove the superiority of our Sci-Fi compared with other baselines.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21205</guid>
|
||||
<pubDate>Tue, 27 May 2025 13:53:50 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset</title>
|
||||
<link>https://arxiv.org/abs/2505.21297</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21297.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, Mao Yang</p><p><b>Upvotes:</b> 22</p><p><b>Summary:</b> Advancing code reasoning in large language models (LLMs) is fundamentally limited by the scarcity of high-difficulty datasets, especially those with verifiable input-output test cases necessary for rigorous solution validation at scale. We introduce rStar-Coder, which significantly improves LLM code reasoning capabilities by constructing a large-scale, verified dataset of 418K competition-level code problems, 580K long-reasoning solutions along with rich test cases of varying difficulty. This is achieved through three core contributions: (1) we curate competitive programming code problems and oracle solutions to synthesize new, solvable problems; (2) we introduce a reliable input-output test case synthesis pipeline that decouples the generation into a three-step input generation method and a mutual verification mechanism for effective output labeling; (3) we augment problems with high-quality, test-case-verified long-reasoning solutions. Extensive experiments on Qwen models (1.5B-14B) across various code reasoning benchmarks demonstrate the superiority of rStar-Coder dataset, achieving leading performance comparable to frontier reasoning LLMs with much smaller model sizes. On LiveCodeBench, rStar-Coder improves Qwen2.5-7B from 17.4% to an impressive 57.3%, and Qwen2.5-14B from 23.3% to 62.5%, surpassing o3-mini (low) by3.1%. On the more challenging USA Computing Olympiad, our 7B model achieves an average pass@1 accuracy of 16.15%, outperforming the frontier-level QWQ-32B. Code and the dataset will be released at https://github.com/microsoft/rStar.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21297</guid>
|
||||
<pubDate>Tue, 27 May 2025 15:00:57 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs</title>
|
||||
<link>https://arxiv.org/abs/2505.21327</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21327.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiakang Yuan, Tianshuo Peng, Yilei Jiang, Yiting Lu, Renrui Zhang, Kaituo Feng, Chaoyou Fu, Tao Chen, Lei Bai, Bo Zhang, Xiangyu Yue</p><p><b>Upvotes:</b> 72</p><p><b>Summary:</b> Logical reasoning is a fundamental aspect of human intelligence and an essential capability for multimodal large language models (MLLMs). Despite the significant advancement in multimodal reasoning, existing benchmarks fail to comprehensively evaluate their reasoning abilities due to the lack of explicit categorization for logical reasoning types and an unclear understanding of reasoning. To address these issues, we introduce MME-Reasoning, a comprehensive benchmark designed to evaluate the reasoning ability of MLLMs, which covers all three types of reasoning (i.e., inductive, deductive, and abductive) in its questions. We carefully curate the data to ensure that each question effectively evaluates reasoning ability rather than perceptual skills or knowledge breadth, and extend the evaluation protocols to cover the evaluation of diverse questions. Our evaluation reveals substantial limitations of state-of-the-art MLLMs when subjected to holistic assessments of logical reasoning capabilities. Even the most advanced MLLMs show limited performance in comprehensive logical reasoning, with notable performance imbalances across reasoning types. In addition, we conducted an in-depth analysis of approaches such as ``thinking mode'' and Rule-based RL, which are commonly believed to enhance reasoning abilities. These findings highlight the critical limitations and performance imbalances of current MLLMs in diverse logical reasoning scenarios, providing comprehensive and systematic insights into the understanding and evaluation of reasoning capabilities.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21327</guid>
|
||||
<pubDate>Tue, 27 May 2025 15:23:23 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios</title>
|
||||
<link>https://arxiv.org/abs/2505.21333</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21333.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Shi, Huanqian Wang, Wulin Xie, Huanyao Zhang, Lijie Zhao, Yi-Fan Zhang, Xinfeng Li, Chaoyou Fu, Zhuoer Wen, Wenting Liu, Zhuoran Zhang, Xinlong Chen, Bohan Zeng, Sihan Yang, Yuanxing Zhang, Pengfei Wan, Haotian Wang, Wenjing Yang</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) have achieved considerable accuracy in Optical Character Recognition (OCR) from static images. However, their efficacy in video OCR is significantly diminished due to factors such as motion blur, temporal variations, and visual effects inherent in video content. To provide clearer guidance for training practical MLLMs, we introduce the MME-VideoOCR benchmark, which encompasses a comprehensive range of video OCR application scenarios. MME-VideoOCR features 10 task categories comprising 25 individual tasks and spans 44 diverse scenarios. These tasks extend beyond text recognition to incorporate deeper comprehension and reasoning of textual content within videos. The benchmark consists of 1,464 videos with varying resolutions, aspect ratios, and durations, along with 2,000 meticulously curated, manually annotated question-answer pairs. We evaluate 18 state-of-the-art MLLMs on MME-VideoOCR, revealing that even the best-performing model (Gemini-2.5 Pro) achieves an accuracy of only 73.7%. Fine-grained analysis indicates that while existing MLLMs demonstrate strong performance on tasks where relevant texts are contained within a single or few frames, they exhibit limited capability in effectively handling tasks that demand holistic video comprehension. These limitations are especially evident in scenarios that require spatio-temporal reasoning, cross-frame information integration, or resistance to language prior bias. Our findings also highlight the importance of high-resolution visual input and sufficient temporal coverage for reliable OCR in dynamic video scenarios.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21333</guid>
|
||||
<pubDate>Tue, 27 May 2025 15:27:46 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>HoliTom: Holistic Token Merging for Fast Video Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.21334</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21334.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, Huan Wang</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio-temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of combining these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training-free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatial-temporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust inner-LLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a 1.32x acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21334</guid>
|
||||
<pubDate>Tue, 27 May 2025 15:28:45 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?</title>
|
||||
<link>https://arxiv.org/abs/2505.21374</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21374.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, Ying Shan</p><p><b>Upvotes:</b> 26</p><p><b>Summary:</b> Recent advances in CoT reasoning and RL post-training have been reported to enhance video reasoning capabilities of MLLMs. This progress naturally raises a question: can these models perform complex video reasoning in a manner comparable to human experts? However, existing video benchmarks primarily evaluate visual perception and grounding abilities, with questions that can be answered based on explicit prompts or isolated visual cues. Such benchmarks do not fully capture the intricacies of real-world reasoning, where humans must actively search for, integrate, and analyze multiple clues before reaching a conclusion. To address this issue, we present Video-Holmes, a benchmark inspired by the reasoning process of Sherlock Holmes, designed to evaluate the complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837 questions derived from 270 manually annotated suspense short films, which spans seven carefully designed tasks. Each task is constructed by first identifying key events and causal relationships within films, and then designing questions that require models to actively locate and connect multiple relevant visual clues scattered across different video segments. Our comprehensive evaluation of state-of-the-art MLLMs reveals that, while these models generally excel at visual perception, they encounter substantial difficulties with integrating information and often miss critical clues. For example, the best-performing model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for multimodal reasoning, motivating models to reason more like humans and emphasizing the ongoing challenges in this field. The benchmark is released in https://github.com/TencentARC/Video-Holmes.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21374</guid>
|
||||
<pubDate>Tue, 27 May 2025 16:05:01 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Active-O3: Empowering Multimodal Large Language Models with Active Perception via GRPO</title>
|
||||
<link>https://arxiv.org/abs/2505.21457</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21457.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Muzhi Zhu, Hao Zhong, Canyu Zhao, Zongze Du, Zheng Huang, Mingyu Liu, Hao Chen, Cheng Zou, Jingdong Chen, Ming Yang, Chunhua Shen</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Active vision, also known as active perception, refers to the process of actively selecting where and how to look in order to gather task-relevant information. It is a critical component of efficient perception and decision-making in humans and advanced embodied agents. Recently, the use of Multimodal Large Language Models (MLLMs) as central planning and decision-making modules in robotic systems has gained extensive attention. However, despite the importance of active perception in embodied intelligence, there is little to no exploration of how MLLMs can be equipped with or learn active perception capabilities. In this paper, we first provide a systematic definition of MLLM-based active perception tasks. We point out that the recently proposed GPT-o3 model's zoom-in search strategy can be regarded as a special case of active perception; however, it still suffers from low search efficiency and inaccurate region selection. To address these issues, we propose ACTIVE-O3, a purely reinforcement learning based training framework built on top of GRPO, designed to equip MLLMs with active perception capabilities. We further establish a comprehensive benchmark suite to evaluate ACTIVE-O3 across both general open-world tasks, such as small-object and dense object grounding, and domain-specific scenarios, including small object detection in remote sensing and autonomous driving, as well as fine-grained interactive segmentation. In addition, ACTIVE-O3 also demonstrates strong zero-shot reasoning abilities on the V* Benchmark, without relying on any explicit reasoning data. We hope that our work can provide a simple codebase and evaluation protocol to facilitate future research on active perception in MLLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21457</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:29:31 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration</title>
|
||||
<link>https://arxiv.org/abs/2505.21471</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21471.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zijun Liu, Zhennan Wan, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> With the rapid advancement of post-training techniques for reasoning and information seeking, large language models (LLMs) can incorporate a large quantity of retrieved knowledge to solve complex tasks. However, the limited context window of LLMs obstructs scaling the amount of external knowledge input, prohibiting further improvement, especially for tasks requiring significant amount of external knowledge. Existing context window extension methods inevitably cause information loss. LLM-based multi-agent methods emerge as a new paradigm to handle massive input in a distributional manner, where we identify two core bottlenecks in existing knowledge synchronization and reasoning processes. In this work, we develop a multi-agent framework, ExtAgents, to overcome the bottlenecks and enable better scalability in inference-time knowledge integration without longer-context training. Benchmarked with our enhanced multi-hop question answering test, $boldsymbol{inftyBench+}, and other public test sets including long survey generation, ExtAgents significantly enhances the performance over existing non-training methods with the same amount of external knowledge input, regardless of whether it falls within or exceeds the context window$. Moreover, the method maintains high efficiency due to high parallelism. Further study in the coordination of LLM agents on increasing external knowledge input could benefit real-world applications.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21471</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:45:04 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>DetailFlow: 1D Coarse-to-Fine Autoregressive Image Generation via Next-Detail Prediction</title>
|
||||
<link>https://arxiv.org/abs/2505.21473</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21473.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yiheng Liu, Liao Qu, Huichao Zhang, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Xian Li, Shuai Wang, Daniel K. Du, Shu Cheng, Zehuan Yuan, Xinglong Wu</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> This paper presents DetailFlow, a coarse-to-fine 1D autoregressive (AR) image generation method that models images through a novel next-detail prediction strategy. By learning a resolution-aware token sequence supervised with progressively degraded images, DetailFlow enables the generation process to start from the global structure and incrementally refine details. This coarse-to-fine 1D token sequence aligns well with the autoregressive inference mechanism, providing a more natural and efficient way for the AR model to generate complex visual content. Our compact 1D AR model achieves high-quality image synthesis with significantly fewer tokens than previous approaches, i.e. VAR/VQGAN. We further propose a parallel inference mechanism with self-correction that accelerates generation speed by approximately 8x while reducing accumulation sampling error inherent in teacher-forcing supervision. On the ImageNet 256x256 benchmark, our method achieves 2.96 gFID with 128 tokens, outperforming VAR (3.3 FID) and FlexVAR (3.05 FID), which both require 680 tokens in their AR models. Moreover, due to the significantly reduced token count and parallel inference mechanism, our method runs nearly 2x faster inference speed compared to VAR and FlexVAR. Extensive experimental results demonstrate DetailFlow's superior generation quality and efficiency compared to existing state-of-the-art methods.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21473</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:45:21 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Frame In-N-Out: Unbounded Controllable Image-to-Video Generation</title>
|
||||
<link>https://arxiv.org/abs/2505.21491</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21491.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Boyang Wang, Xuweiyi Chen, Matheus Gadelha, Zezhou Cheng</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Controllability, temporal coherence, and detail synthesis remain the most critical challenges in video generation. In this paper, we focus on a commonly used yet underexplored cinematic technique known as Frame In and Frame Out. Specifically, starting from image-to-video generation, users can control the objects in the image to naturally leave the scene or provide breaking new identity references to enter the scene, guided by user-specified motion trajectory. To support this task, we introduce a new dataset curated semi-automatically, a comprehensive evaluation protocol targeting this setting, and an efficient identity-preserving motion-controllable video Diffusion Transformer architecture. Our evaluation shows that our proposed approach significantly outperforms existing baselines.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21491</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:56:07 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Reinforcing General Reasoning without Verifiers</title>
|
||||
<link>https://arxiv.org/abs/2505.21493</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21493.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, Chao Du</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method (VeriFree) that bypasses answer verification and instead uses RL to directly maximize the probability of generating the reference answer. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks. Moreover, we provide insights into this method from multiple perspectives: as an elegant integration of training both the policy and implicit verifier in a unified model, and as a variational optimization approach. Code is available at https://github.com/sail-sg/VeriFree.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21493</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:56:27 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment</title>
|
||||
<link>https://arxiv.org/abs/2505.21494</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21494.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, Yang Liu</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples. While existing methods typically achieve targeted attacks by aligning global features-such as CLIP's [CLS] token-between adversarial and target samples, they often overlook the rich local information encoded in patch tokens. This leads to suboptimal alignment and limited transferability, particularly for closed-source models. To address this limitation, we propose a targeted transferable adversarial attack method based on feature optimal alignment, called FOA-Attack, to improve adversarial transfer capability. Specifically, at the global level, we introduce a global feature loss based on cosine similarity to align the coarse-grained features of adversarial samples with those of target samples. At the local level, given the rich local representations within Transformers, we leverage clustering techniques to extract compact local patterns to alleviate redundant local features. We then formulate local feature alignment between adversarial and target samples as an optimal transport (OT) problem and propose a local clustering optimal transport loss to refine fine-grained feature alignment. Additionally, we propose a dynamic ensemble model weighting strategy to adaptively balance the influence of multiple models during adversarial example generation, thereby further improving transferability. Extensive experiments across various models demonstrate the superiority of the proposed method, outperforming state-of-the-art methods, especially in transferring to closed-source MLLMs. The code is released at https://github.com/jiaxiaojunQAQ/FOA-Attack.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21494</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:56:57 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents</title>
|
||||
<link>https://arxiv.org/abs/2505.21496</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21496.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Aojun Zhou, Hongsheng Li</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> In this paper, we introduce UI-Genie, a self-improving framework addressing two key challenges in GUI agents: verification of trajectory outcome is challenging and high-quality training data are not scalable. These challenges are addressed by a reward model and a self-improving pipeline, respectively. The reward model, UI-Genie-RM, features an image-text interleaved architecture that efficiently pro- cesses historical context and unifies action-level and task-level rewards. To sup- port the training of UI-Genie-RM, we develop deliberately-designed data genera- tion strategies including rule-based verification, controlled trajectory corruption, and hard negative mining. To address the second challenge, a self-improvement pipeline progressively expands solvable complex GUI tasks by enhancing both the agent and reward models through reward-guided exploration and outcome verification in dynamic environments. For training the model, we generate UI- Genie-RM-517k and UI-Genie-Agent-16k, establishing the first reward-specific dataset for GUI agents while demonstrating high-quality synthetic trajectory gen- eration without manual annotation. Experimental results show that UI-Genie achieves state-of-the-art performance across multiple GUI agent benchmarks with three generations of data-model self-improvement. We open-source our complete framework implementation and generated datasets to facilitate further research in https://github.com/Euphoria16/UI-Genie.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21496</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:58:06 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers</title>
|
||||
<link>https://arxiv.org/abs/2505.21497</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21497.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Wei Pang, Kevin Qinghong Lin, Xiangru Jian, Xi He, Philip Torr</p><p><b>Upvotes:</b> 66</p><p><b>Summary:</b> Academic poster generation is a crucial yet challenging task in scientific communication, requiring the compression of long-context interleaved documents into a single, visually coherent page. To address this challenge, we introduce the first benchmark and metric suite for poster generation, which pairs recent conference papers with author-designed posters and evaluates outputs on (i)Visual Quality-semantic alignment with human posters, (ii)Textual Coherence-language fluency, (iii)Holistic Assessment-six fine-grained aesthetic and informational criteria scored by a VLM-as-judge, and notably (iv)PaperQuiz-the poster's ability to convey core paper content as measured by VLMs answering generated quizzes. Building on this benchmark, we propose PosterAgent, a top-down, visual-in-the-loop multi-agent pipeline: the (a)Parser distills the paper into a structured asset library; the (b)Planner aligns text-visual pairs into a binary-tree layout that preserves reading order and spatial balance; and the (c)Painter-Commenter loop refines each panel by executing rendering code and using VLM feedback to eliminate overflow and ensure alignment. In our comprehensive evaluation, we find that GPT-4o outputs-though visually appealing at first glance-often exhibit noisy text and poor PaperQuiz scores, and we find that reader engagement is the primary aesthetic bottleneck, as human-designed posters rely largely on visual semantics to convey meaning. Our fully open-source variants (e.g. based on the Qwen-2.5 series) outperform existing 4o-driven multi-agent systems across nearly all metrics, while using 87% fewer tokens. It transforms a 22-page paper into a finalized yet editable .pptx poster - all for just $0.005. These findings chart clear directions for the next generation of fully automated poster-generation models. The code and datasets are available at https://github.com/Paper2Poster/Paper2Poster.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21497</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:58:49 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery</title>
|
||||
<link>https://arxiv.org/abs/2505.21499</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21499.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haowei Wang, Junjie Wang, Xiaojun Jia, Rupeng Zhang, Mingyang Li, Zhe Liu, Yang Liu, Qing Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Vision-Language Model (VLM) based Web Agents represent a significant step towards automating complex tasks by simulating human-like interaction with websites. However, their deployment in uncontrolled web environments introduces significant security vulnerabilities. Existing research on adversarial environmental injection attacks often relies on unrealistic assumptions, such as direct HTML manipulation, knowledge of user intent, or access to agent model parameters, limiting their practical applicability. In this paper, we propose AdInject, a novel and real-world black-box attack method that leverages the internet advertising delivery to inject malicious content into the Web Agent's environment. AdInject operates under a significantly more realistic threat model than prior work, assuming a black-box agent, static malicious content constraints, and no specific knowledge of user intent. AdInject includes strategies for designing malicious ad content aimed at misleading agents into clicking, and a VLM-based ad content optimization technique that infers potential user intents from the target website's context and integrates these intents into the ad content to make it appear more relevant or critical to the agent's task, thus enhancing attack effectiveness. Experimental evaluations demonstrate the effectiveness of AdInject, attack success rates exceeding 60% in most scenarios and approaching 100% in certain cases. This strongly demonstrates that prevalent advertising delivery constitutes a potent and real-world vector for environment injection attacks against Web Agents. This work highlights a critical vulnerability in Web Agent security arising from real-world environment manipulation channels, underscoring the urgent need for developing robust defense mechanisms against such threats. Our code is available at https://github.com/NicerWang/AdInject.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21499</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:59:05 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models</title>
|
||||
<link>https://arxiv.org/abs/2505.21500</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21500.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Dingming Li, Hongxing Li, Zixuan Wang, Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, Weiming Lu, Yueting Zhuang</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content, but significant challenges persist in tasks requiring cross-viewpoint understanding and spatial reasoning. We identify a critical limitation: current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints when required to adopt another entity's spatial frame of reference. We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation across five distinct task types, supported by an automated 3D annotation pipeline that generates precise directional labels. Comprehensive evaluation of diverse VLMs on ViewSpatial-Bench reveals a significant performance disparity: models demonstrate reasonable performance on camera-perspective tasks but exhibit reduced accuracy when reasoning from a human viewpoint. By fine-tuning VLMs on our multi-perspective spatial dataset, we achieve an overall performance improvement of 46.24% across tasks, highlighting the efficacy of our approach. Our work establishes a crucial benchmark for spatial intelligence in embodied AI systems and provides empirical evidence that modeling 3D spatial relationships enhances VLMs' corresponding spatial comprehension capabilities.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21500</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:59:26 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Vision Transformers with Self-Distilled Registers</title>
|
||||
<link>https://arxiv.org/abs/2505.21501</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21501.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yinjie Chen, Zipeng Yan, Chong Zhou, Bo Dai, Andrew F. Luo</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Vision Transformers (ViTs) have emerged as the dominant architecture for visual processing tasks, demonstrating excellent scalability with increased training data and model size. However, recent work has identified the emergence of artifact tokens in ViTs that are incongruous with the local semantics. These anomalous tokens degrade ViT performance in tasks that require fine-grained localization or structural coherence. An effective mitigation of this issue is to the addition of register tokens to ViTs, which implicitly "absorb" the artifact term during training. Given the availability of various large-scale pre-trained ViTs, in this paper we aim at equipping them with such register tokens without the need of re-training them from scratch, which is infeasible considering their size. Specifically, we propose Post Hoc Registers (PH-Reg), an efficient self-distillation method that integrates registers into an existing ViT without requiring additional labeled data and full retraining. PH-Reg initializes both teacher and student networks from the same pre-trained ViT. The teacher remains frozen and unmodified, while the student is augmented with randomly initialized register tokens. By applying test-time augmentation to the teacher's inputs, we generate denoised dense embeddings free of artifacts, which are then used to optimize only a small subset of unlocked student weights. We show that our approach can effectively reduce the number of artifact tokens, improving the segmentation and depth prediction of the student ViT under zero-shot and linear probing.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21501</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:59:41 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>How does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective</title>
|
||||
<link>https://arxiv.org/abs/2505.21505</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21505.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shimao Zhang, Zhejian Lai, Xiang Liu, Shuaijie She, Xiao Liu, Yeyun Gong, Shujian Huang, Jiajun Chen</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> Multilingual Alignment is an effective and representative paradigm to enhance LLMs' multilingual capabilities, which transfers the capabilities from the high-resource languages to the low-resource languages. Meanwhile, some researches on language-specific neurons reveal that there are language-specific neurons that are selectively activated in LLMs when processing different languages. This provides a new perspective to analyze and understand LLMs' mechanisms more specifically in multilingual scenarios. In this work, we propose a new finer-grained neuron identification algorithm, which detects language neurons~(including language-specific neurons and language-related neurons) and language-agnostic neurons. Furthermore, based on the distributional characteristics of different types of neurons, we divide the LLMs' internal process for multilingual inference into four parts: (1) multilingual understanding, (2) shared semantic space reasoning, (3) multilingual output space transformation, and (4) vocabulary space outputting. Additionally, we systematically analyze the models before and after alignment with a focus on different types of neurons. We also analyze the phenomenon of ''Spontaneous Multilingual Alignment''. Overall, our work conducts a comprehensive investigation based on different types of neurons, providing empirical results and valuable insights for better understanding multilingual alignment and multilingual capabilities of LLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21505</guid>
|
||||
<pubDate>Tue, 27 May 2025 17:59:52 +0000</pubDate>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
|
||||
Reference in New Issue
Block a user