From 181d36626c0b7497d1c4c69841708cde4db5fa7c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 19 Jun 2025 00:02:33 +0000 Subject: [PATCH] bot: update RSS feed --- feed.xml | 356 ++++++++++++++++++++++++++----------------------------- 1 file changed, 171 insertions(+), 185 deletions(-) diff --git a/feed.xml b/feed.xml index 75d253f..4b2b23e 100644 --- a/feed.xml +++ b/feed.xml @@ -7,258 +7,244 @@ http://www.rssboard.org/rss-specification python-feedgen en - Wed, 18 Jun 2025 00:02:27 +0000 + Thu, 19 Jun 2025 00:02:33 +0000 - AI Agent Behavioral Science - https://arxiv.org/abs/2506.06366 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06366.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lin Chen, Yunke Zhang, Jie Feng, Haoye Chai, Honglin Zhang, Bingbing Fan, Yibo Ma, Shiyuan Zhang, Nian Li, Tianhui Liu, Nicholas Sukiennik, Keyu Zhao, Yu Li, Ziyi Liu, Fengli Xu, Yong Li</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Recent advances in large language models (LLMs) have enabled the development of AI agents that exhibit increasingly human-like behaviors, including planning, adaptation, and social dynamics across diverse, interactive, and open-ended scenarios. These behaviors are not solely the product of the internal architectures of the underlying models, but emerge from their integration into agentic systems operating within specific contexts, where environmental factors, social cues, and interaction feedbacks shape behavior over time. This evolution necessitates a new scientific perspective: AI Agent Behavioral Science. Rather than focusing only on internal mechanisms, this perspective emphasizes the systematic observation of behavior, design of interventions to test hypotheses, and theory-guided interpretation of how AI agents act, adapt, and interact over time. We systematize a growing body of research across individual agent, multi-agent, and human-agent interaction settings, and further demonstrate how this perspective informs responsible AI by treating fairness, safety, interpretability, accountability, and privacy as behavioral properties. By unifying recent findings and laying out future directions, we position AI Agent Behavioral Science as a necessary complement to traditional model-centric approaches, providing essential tools for understanding, evaluating, and governing the real-world behavior of increasingly autonomous AI systems.</p> - https://arxiv.org/abs/2506.06366 - Wed, 04 Jun 2025 08:12:32 +0000 + Graph Counselor: Adaptive Graph Exploration via Multi-Agent Synergy to Enhance LLM Reasoning + https://arxiv.org/abs/2506.03939 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.03939.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junqi Gao, Xiang Zou, YIng Ai, Dong Li, Yichen Niu, Biqing Qi, Jianxing Liu</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Graph Retrieval Augmented Generation (GraphRAG) effectively enhances external knowledge integration capabilities by explicitly modeling knowledge relationships, thereby improving the factual accuracy and generation quality of Large Language Models (LLMs) in specialized domains. However, existing methods suffer from two inherent limitations: 1) Inefficient Information Aggregation: They rely on a single agent and fixed iterative patterns, making it difficult to adaptively capture multi-level textual, structural, and degree information within graph data. 2) Rigid Reasoning Mechanism: They employ preset reasoning schemes, which cannot dynamically adjust reasoning depth nor achieve precise semantic correction. To overcome these limitations, we propose Graph Counselor, an GraphRAG method based on multi-agent collaboration. This method uses the Adaptive Graph Information Extraction Module (AGIEM), where Planning, Thought, and Execution Agents work together to precisely model complex graph structures and dynamically adjust information extraction strategies, addressing the challenges of multi-level dependency modeling and adaptive reasoning depth. Additionally, the Self-Reflection with Multiple Perspectives (SR) module improves the accuracy and semantic consistency of reasoning results through self-reflection and backward reasoning mechanisms. Experiments demonstrate that Graph Counselor outperforms existing methods in multiple graph reasoning tasks, exhibiting higher reasoning accuracy and generalization ability. Our code is available at https://github.com/gjq100/Graph-Counselor.git.</p> + https://arxiv.org/abs/2506.03939 + Wed, 04 Jun 2025 13:31:21 +0000 - From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding - https://arxiv.org/abs/2506.03968 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.03968.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chiwei Zhu, Benfeng Xu, Xiaorui Wang, Zhendong Mao</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> The pursuit of diverse, complex, and large-scale instruction data is crucial for automatically aligning large language models (LLMs). While there are methods capable of generating synthetic instructions at scale, they either suffer from limited grounding sources, leading to a narrow distribution, or rely on trivial extensions that fail to produce meaningful trajectories in terms of complexity. In contrast, instructions that benefit efficient alignment are typically crafted with cognitive insights and grounded in real-world use cases. In this paper, we synthesize such instructions using attributed grounding, which involves 1) a top-down attribution process that grounds a selective set of real instructions to situated users, and 2) a bottom-up synthesis process that leverages web documents to first generate a situation, then a meaningful instruction. This framework allows us to harvest diverse and complex instructions at scale, utilizing the vast range of web documents. Specifically, we construct a dataset of 1 million instructions, called SynthQuestions, and demonstrate that models trained on it achieve leading performance on several common benchmarks, with improvements that continually scale with more web corpora. Data, models and codes will be available at https://github.com/Ignoramus0817/SynthQuestions.</p> - https://arxiv.org/abs/2506.03968 - Wed, 04 Jun 2025 14:00:47 +0000 + Mixture-of-Experts Meets In-Context Reinforcement Learning + https://arxiv.org/abs/2506.05426 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05426.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Wenhao Wu, Fuhong Liu, Haoru Li, Zican Hu, Daoyi Dong, Chunlin Chen, Zhi Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> In-context reinforcement learning (ICRL) has emerged as a promising paradigm for adapting RL agents to downstream tasks through prompt conditioning. However, two notable challenges remain in fully harnessing in-context learning within RL domains: the intrinsic multi-modality of the state-action-reward data and the diverse, heterogeneous nature of decision tasks. To tackle these challenges, we propose T2MIR (Token- and Task-wise MoE for In-context RL), an innovative framework that introduces architectural advances of mixture-of-experts (MoE) into transformer-based decision models. T2MIR substitutes the feedforward layer with two parallel layers: a token-wise MoE that captures distinct semantics of input tokens across multiple modalities, and a task-wise MoE that routes diverse tasks to specialized experts for managing a broad task distribution with alleviated gradient conflicts. To enhance task-wise routing, we introduce a contrastive learning method that maximizes the mutual information between the task and its router representation, enabling more precise capture of task-relevant information. The outputs of two MoE components are concatenated and fed into the next layer. Comprehensive experiments show that T2MIR significantly facilitates in-context learning capacity and outperforms various types of baselines. We bring the potential and promise of MoE to ICRL, offering a simple and scalable architectural enhancement to advance ICRL one step closer toward achievements in language and vision communities. Our code is available at https://github.com/NJU-RL/T2MIR.</p> + https://arxiv.org/abs/2506.05426 + Thu, 05 Jun 2025 06:29:14 +0000 - LETS Forecast: Learning Embedology for Time Series Forecasting - https://arxiv.org/abs/2506.06454 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06454.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Abrar Majeedi, Viswanatha Reddy Gajjala, Satya Sai Srinath Namburi GNVV, Nada Magdi Elkordi, Yin Li</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Real-world time series are often governed by complex nonlinear dynamics. Understanding these underlying dynamics is crucial for precise future prediction. While deep learning has achieved major success in time series forecasting, many existing approaches do not explicitly model the dynamics. To bridge this gap, we introduce DeepEDM, a framework that integrates nonlinear dynamical systems modeling with deep neural networks. Inspired by empirical dynamic modeling (EDM) and rooted in Takens' theorem, DeepEDM presents a novel deep model that learns a latent space from time-delayed embeddings, and employs kernel regression to approximate the underlying dynamics, while leveraging efficient implementation of softmax attention and allowing for accurate prediction of future time steps. To evaluate our method, we conduct comprehensive experiments on synthetic data of nonlinear dynamical systems as well as real-world time series across domains. Our results show that DeepEDM is robust to input noise, and outperforms state-of-the-art methods in forecasting accuracy. Our code is available at: https://abrarmajeedi.github.io/deep_edm.</p> - https://arxiv.org/abs/2506.06454 - Fri, 06 Jun 2025 18:24:12 +0000 + VideoMolmo: Spatio-Temporal Grounding Meets Pointing + https://arxiv.org/abs/2506.05336 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05336.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ghazi Shazan Ahmad, Ahmed Heakl, Hanan Gani, Abdelrahman Shaker, Zhiqiang Shen, Ranjay Krishna, Fahad Shahbaz Khan, Salman Khan</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Spatio-temporal localization is vital for precise interactions across diverse domains, from biological research to autonomous navigation and interactive interfaces. Current video-based approaches, while proficient in tracking, lack the sophisticated reasoning capabilities of large language models, limiting their contextual understanding and generalization. We introduce VideoMolmo, a large multimodal model tailored for fine-grained spatio-temporal pointing conditioned on textual descriptions. Building upon the Molmo architecture, VideoMolmo incorporates a temporal module utilizing an attention mechanism to condition each frame on preceding frames, ensuring temporal consistency. Additionally, our novel temporal mask fusion pipeline employs SAM2 for bidirectional point propagation, significantly enhancing coherence across video sequences. This two-step decomposition, i.e., first using the LLM to generate precise pointing coordinates, then relying on a sequential mask-fusion module to produce coherent segmentation, not only simplifies the task for the language model but also enhances interpretability. Due to the lack of suitable datasets, we curate a comprehensive dataset comprising 72k video-caption pairs annotated with 100k object points. To evaluate the generalization of VideoMolmo, we introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving, Video-GUI Interaction, and Robotics. We also evaluate our model on Referring Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to existing models, VideoMolmo substantially improves spatio-temporal pointing accuracy and reasoning capability. Our code and models are publicly available at https://github.com/mbzuai-oryx/VideoMolmo.</p> + https://arxiv.org/abs/2506.05336 + Thu, 05 Jun 2025 17:59:29 +0000 - AR-RAG: Autoregressive Retrieval Augmentation for Image Generation - https://arxiv.org/abs/2506.06962 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06962.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingyuan Qi, Zhiyang Xu, Qifan Wang, Lifu Huang</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> We introduce Autoregressive Retrieval Augmentation (AR-RAG), a novel paradigm that enhances image generation by autoregressively incorporating knearest neighbor retrievals at the patch level. Unlike prior methods that perform a single, static retrieval before generation and condition the entire generation on fixed reference images, AR-RAG performs context-aware retrievals at each generation step, using prior-generated patches as queries to retrieve and incorporate the most relevant patch-level visual references, enabling the model to respond to evolving generation needs while avoiding limitations (e.g., over-copying, stylistic bias, etc.) prevalent in existing methods. To realize AR-RAG, we propose two parallel frameworks: (1) Distribution-Augmentation in Decoding (DAiD), a training-free plug-and-use decoding strategy that directly merges the distribution of model-predicted patches with the distribution of retrieved patches, and (2) Feature-Augmentation in Decoding (FAiD), a parameter-efficient fine-tuning method that progressively smooths the features of retrieved patches via multi-scale convolution operations and leverages them to augment the image generation process. We validate the effectiveness of AR-RAG on widely adopted benchmarks, including Midjourney-30K, GenEval and DPG-Bench, demonstrating significant performance gains over state-of-the-art image generation models.</p> - https://arxiv.org/abs/2506.06962 - Sun, 08 Jun 2025 01:33:05 +0000 + Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning + https://arxiv.org/abs/2506.09033 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.09033.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haozhen Zhang, Tao Feng, Jiaxuan You</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> The rapid emergence of diverse large language models (LLMs) has spurred the development of LLM routers that assign user queries to the most suitable model. However, existing LLM routers typically perform a single-round, one-to-one mapping (i.e., assigning each query to a single model in isolation), which limits their capability to tackle complex tasks that demand the complementary strengths of multiple LLMs. In this paper, we present Router-R1, a reinforcement learning (RL)-based framework that formulates multi-LLM routing and aggregation as a sequential decision process. Router-R1 instantiates the router itself as a capable LLM, leveraging its reasoning ability to interleave "think" actions (internal deliberation) with "route" actions (dynamic model invocation), and integrates each response into its evolving context. To guide learning, we employ a lightweight rule-based reward comprising format rewards, final outcome rewards, and a novel cost reward for performance and cost trade-off optimization, opening a pathway toward optimizing performance-cost tradeoffs via RL. Router-R1 also conditions only on simple model descriptors such as pricing, latency, and example performance, enabling strong generalization to unseen model selection. Experiments on seven general and multi-hop QA benchmarks show that Router-R1 outperforms over several strong baselines, achieving superior performance while maintaining robust generalization and cost management.Code is available at https://github.com/ulab-uiuc/Router-R1.</p> + https://arxiv.org/abs/2506.09033 + Tue, 10 Jun 2025 17:56:45 +0000 - Incorporating Domain Knowledge into Materials Tokenization - https://arxiv.org/abs/2506.11115 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.11115.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yerim Oh, Jun-Hyung Park, Junho Kim, SungHo Kim, SangKeun Lee</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> While language models are increasingly utilized in materials science, typical models rely on frequency-centric tokenization methods originally developed for natural language processing. However, these methods frequently produce excessive fragmentation and semantic loss, failing to maintain the structural and semantic integrity of material concepts. To address this issue, we propose MATTER, a novel tokenization approach that integrates material knowledge into tokenization. Based on MatDetector trained on our materials knowledge base and a re-ranking method prioritizing material concepts in token merging, MATTER maintains the structural integrity of identified material concepts and prevents fragmentation during tokenization, ensuring their semantic meaning remains intact. The experimental results demonstrate that MATTER outperforms existing tokenization methods, achieving an average performance gain of 4% and 2% in the generation and classification tasks, respectively. These results underscore the importance of domain knowledge for tokenization strategies in scientific text processing. Our code is available at https://github.com/yerimoh/MATTER</p> - https://arxiv.org/abs/2506.11115 - Mon, 09 Jun 2025 04:59:13 +0000 + Ambient Diffusion Omni: Training Good Models with Bad Data + https://arxiv.org/abs/2506.10038 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.10038.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Giannis Daras, Adrian Rodriguez-Munoz, Adam Klivans, Antonio Torralba, Constantinos Daskalakis</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> We show how to use low-quality, synthetic, and out-of-distribution images to improve the quality of a diffusion model. Typically, diffusion models are trained on curated datasets that emerge from highly filtered data pools from the Web and other sources. We show that there is immense value in the lower-quality images that are often discarded. We present Ambient Diffusion Omni, a simple, principled framework to train diffusion models that can extract signal from all available images during training. Our framework exploits two properties of natural images -- spectral power law decay and locality. We first validate our framework by successfully training diffusion models with images synthetically corrupted by Gaussian blur, JPEG compression, and motion blur. We then use our framework to achieve state-of-the-art ImageNet FID, and we show significant improvements in both image quality and diversity for text-to-image generative modeling. The core insight is that noise dampens the initial skew between the desired high-quality distribution and the mixed distribution we actually observe. We provide rigorous theoretical justification for our approach by analyzing the trade-off between learning from biased data versus limited unbiased data across diffusion times.</p> + https://arxiv.org/abs/2506.10038 + Tue, 10 Jun 2025 22:37:39 +0000 - BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models - https://arxiv.org/abs/2506.07961 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07961.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Peiyan Li, Yixiang Chen, Hongtao Wu, Xiao Ma, Xiangnan Wu, Yan Huang, Liang Wang, Tao Kong, Tieniu Tan</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Recently, leveraging pre-trained vision-language models (VLMs) for building vision-language-action (VLA) models has emerged as a promising approach to effective robot manipulation learning. However, only few methods incorporate 3D signals into VLMs for action prediction, and they do not fully leverage the spatial structure inherent in 3D data, leading to low sample efficiency. In this paper, we introduce BridgeVLA, a novel 3D VLA model that (1) projects 3D inputs to multiple 2D images, ensuring input alignment with the VLM backbone, and (2) utilizes 2D heatmaps for action prediction, unifying the input and output spaces within a consistent 2D image space. In addition, we propose a scalable pre-training method that equips the VLM backbone with the capability to predict 2D heatmaps before downstream policy learning. Extensive experiments show the proposed method is able to learn 3D manipulation efficiently and effectively. BridgeVLA outperforms state-of-the-art baseline methods across three simulation benchmarks. In RLBench, it improves the average success rate from 81.4% to 88.2%. In COLOSSEUM, it demonstrates significantly better performance in challenging generalization settings, boosting the average success rate from 56.7% to 64.0%. In GemBench, it surpasses all the comparing baseline methods in terms of average success rate. In real-robot experiments, BridgeVLA outperforms a state-of-the-art baseline method by 32% on average. It generalizes robustly in multiple out-of-distribution settings, including visual disturbances and unseen instructions. Remarkably, it is able to achieve a success rate of 96.8% on 10+ tasks with only 3 trajectories per task, highlighting its extraordinary sample efficiency. Project Website:https://bridgevla.github.io/</p> - https://arxiv.org/abs/2506.07961 - Mon, 09 Jun 2025 17:36:34 +0000 + V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning + https://arxiv.org/abs/2506.09985 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.09985.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, Nicolas Ballas</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> A major challenge for modern AI is to learn to understand the world and learn to act largely by observation. This paper explores a self-supervised approach that combines internet-scale video data with a small amount of interaction data (robot trajectories), to develop models capable of understanding, predicting, and planning in the physical world. We first pre-train an action-free joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset comprising over 1 million hours of internet video. V-JEPA 2 achieves strong performance on motion understanding (77.3 top-1 accuracy on Something-Something v2) and state-of-the-art performance on human action anticipation (39.7 recall-at-5 on Epic-Kitchens-100) surpassing previous task-specific models. Additionally, after aligning V-JEPA 2 with a large language model, we demonstrate state-of-the-art performance on multiple video question-answering tasks at the 8 billion parameter scale (e.g., 84.0 on PerceptionTest, 76.9 on TempCompass). Finally, we show how self-supervised learning can be applied to robotic planning tasks by post-training a latent action-conditioned world model, V-JEPA 2-AC, using less than 62 hours of unlabeled robot videos from the Droid dataset. We deploy V-JEPA 2-AC zero-shot on Franka arms in two different labs and enable picking and placing of objects using planning with image goals. Notably, this is achieved without collecting any data from the robots in these environments, and without any task-specific training or reward. This work demonstrates how self-supervised learning from web-scale data and a small amount of robot interaction data can yield a world model capable of planning in the physical world.</p> + https://arxiv.org/abs/2506.09985 + Wed, 11 Jun 2025 17:57:09 +0000 - Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency - https://arxiv.org/abs/2506.08343 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08343.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chenlong Wang, Yuanning Feng, Dongping Chen, Zhaoyang Chu, Ranjay Krishna, Tianyi Zhou</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Recent advances in large reasoning models have enabled complex, step-by-step reasoning but often introduce significant overthinking, resulting in verbose and redundant outputs that hinder efficiency. In this study, we examine whether explicit self-reflection, signaled by tokens such as "Wait" and "Hmm", is necessary for advanced reasoning. We propose NoWait, a simple yet effective approach that disables explicit self-reflection by suppressing these tokens during inference. Extensive experiments on ten benchmarks across textual, visual, and video reasoning tasks show that NoWait reduces chain-of-thought trajectory length by up to 27%-51% in five R1-style model series, without compromising model utility. NoWait thus offers a plug-and-play solution for efficient and utility-preserving multimodal reasoning.</p> - https://arxiv.org/abs/2506.08343 - Tue, 10 Jun 2025 01:54:04 +0000 + CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios + https://arxiv.org/abs/2506.13977 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13977.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shiting Huang, Zhen Fang, Zehui Chen, Siyu Yuan, Junjie Ye, Yu Zeng, Lin Chen, Qi Mao, Feng Zhao</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> The ability of large language models (LLMs) to utilize external tools has enabled them to tackle an increasingly diverse range of tasks. However, as the tasks become more complex and long-horizon, the intricate tool utilization process may trigger various unexpected errors. Therefore, how to effectively handle such errors, including identifying, diagnosing, and recovering from them, has emerged as a key research direction for advancing tool learning. In this work, we first extensively analyze the types of errors encountered during the function-calling process on several competitive tool evaluation benchmarks. Based on it, we introduce CRITICTOOL, a comprehensive critique evaluation benchmark specialized for tool learning. Building upon a novel evolutionary strategy for dataset construction, CRITICTOOL holds diverse tool-use errors with varying complexities, which better reflects real-world scenarios. We conduct extensive experiments on CRITICTOOL, and validate the generalization and effectiveness of our constructed benchmark strategy. We also provide an in-depth analysis of the tool reflection ability on various LLMs, offering a new perspective on the field of tool learning in LLMs. The code is available at https://github.com/Shellorley0513/CriticTool{https://github.com/Shellorley0513/CriticTool}.</p> + https://arxiv.org/abs/2506.13977 + Wed, 11 Jun 2025 17:59:18 +0000 - ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering - https://arxiv.org/abs/2506.09050 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.09050.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, Takuya Akiba</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.</p> - https://arxiv.org/abs/2506.09050 - Tue, 10 Jun 2025 17:59:56 +0000 + EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models + https://arxiv.org/abs/2506.10100 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.10100.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yantai Yang, Yuhao Wang, Zichen Wen, Luo Zhongwei, Chang Zou, Zhipeng Zhang, Chuan Wen, Linfeng Zhang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Vision-Language-Action (VLA) models, particularly diffusion-based architectures, demonstrate transformative potential for embodied intelligence but are severely hampered by high computational and memory demands stemming from extensive inherent and inference-time redundancies. While existing acceleration efforts often target isolated inefficiencies, such piecemeal solutions typically fail to holistically address the varied computational and memory bottlenecks across the entire VLA pipeline, thereby limiting practical deployability. We introduce EfficientVLA, a structured and training-free inference acceleration framework that systematically eliminates these barriers by cohesively exploiting multifaceted redundancies. EfficientVLA synergistically integrates three targeted strategies: (1) pruning of functionally inconsequential layers from the language module, guided by an analysis of inter-layer redundancies; (2) optimizing the visual processing pathway through a task-aware strategy that selects a compact, diverse set of visual tokens, balancing task-criticality with informational coverage; and (3) alleviating temporal computational redundancy within the iterative diffusion-based action head by strategically caching and reusing key intermediate features. We apply our method to a standard VLA model CogACT, yielding a 1.93X inference speedup and reduces FLOPs to 28.9%, with only a 0.6% success rate drop in the SIMPLER benchmark.</p> + https://arxiv.org/abs/2506.10100 + Wed, 11 Jun 2025 18:34:57 +0000 - Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression - https://arxiv.org/abs/2506.09482 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.09482.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Dingcheng Zhen, Qian Qiao, Tan Yu, Kangxi Wu, Ziwei Zhang, Siyuan Liu, Shunshun Yin, Ming Tao</p><p><b>Upvotes:</b> 23</p><p><b>Summary:</b> We introduce TransDiff, the first image generation model that marries Autoregressive (AR) Transformer with diffusion models. In this joint modeling framework, TransDiff encodes labels and images into high-level semantic features and employs a diffusion model to estimate the distribution of image samples. On the ImageNet 256x256 benchmark, TransDiff significantly outperforms other image generation models based on standalone AR Transformer or diffusion models. Specifically, TransDiff achieves a Fr\'echet Inception Distance (FID) of 1.61 and an Inception Score (IS) of 293.4, and further provides x2 faster inference latency compared to state-of-the-art methods based on AR Transformer and x112 faster inference compared to diffusion-only models. Furthermore, building on the TransDiff model, we introduce a novel image generation paradigm called Multi-Reference Autoregression (MRAR), which performs autoregressive generation by predicting the next image. MRAR enables the model to reference multiple previously generated images, thereby facilitating the learning of more diverse representations and improving the quality of generated images in subsequent iterations. By applying MRAR, the performance of TransDiff is improved, with the FID reduced from 1.61 to 1.42. We expect TransDiff to open up a new frontier in the field of image generation.</p> - https://arxiv.org/abs/2506.09482 - Wed, 11 Jun 2025 07:50:31 +0000 + EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction + https://arxiv.org/abs/2506.12015 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12015.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hsi-Che Lin, Yu-Chu Yu, Kai-Po Chang, Yu-Chiang Frank Wang</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Open-source foundation models have seen rapid adoption and development, enabling powerful general-purpose capabilities across diverse domains. However, fine-tuning large foundation models for domain-specific or personalized tasks remains prohibitively expensive for most users due to the significant memory overhead beyond that of inference. We introduce EMLoC, an Emulator-based Memory-efficient fine-tuning framework with LoRA Correction, which enables model fine-tuning within the same memory budget required for inference. EMLoC constructs a task-specific light-weight emulator using activation-aware singular value decomposition (SVD) on a small downstream calibration set. Fine-tuning then is performed on this lightweight emulator via LoRA. To tackle the misalignment between the original model and the compressed emulator, we propose a novel compensation algorithm to correct the fine-tuned LoRA module, which thus can be merged into the original model for inference. EMLoC supports flexible compression ratios and standard training pipelines, making it adaptable to a wide range of applications. Extensive experiments demonstrate that EMLoC outperforms other baselines across multiple datasets and modalities. Moreover, without quantization, EMLoC enables fine-tuning of a 38B model on a single 24GB consumer GPU-bringing efficient and practical model adaptation to individual users.</p> + https://arxiv.org/abs/2506.12015 + Fri, 13 Jun 2025 17:59:58 +0000 - SRLAgent: Enhancing Self-Regulated Learning Skills through Gamification and LLM Assistance - https://arxiv.org/abs/2506.09968 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.09968.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Wentao Ge, Yuqing Sun, Ziyan Wang, Haoyue Zheng, Weiyang He, Piaohong Wang, Qianyu Zhu, Benyou Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Self-regulated learning (SRL) is crucial for college students navigating increased academic demands and independence. Insufficient SRL skills can lead to disorganized study habits, low motivation, and poor time management, undermining learners ability to thrive in challenging environments. Through a formative study involving 59 college students, we identified key challenges students face in developing SRL skills, including difficulties with goal-setting, time management, and reflective learning. To address these challenges, we introduce SRLAgent, an LLM-assisted system that fosters SRL skills through gamification and adaptive support from large language models (LLMs). Grounded in Zimmermans three-phase SRL framework, SRLAgent enables students to engage in goal-setting, strategy execution, and self-reflection within an interactive game-based environment. The system offers real-time feedback and scaffolding powered by LLMs to support students independent study efforts. We evaluated SRLAgent using a between-subjects design, comparing it to a baseline system (SRL without Agent features) and a traditional multimedia learning condition. Results showed significant improvements in SRL skills within the SRLAgent group (p < .001, Cohens d = 0.234) and higher engagement compared to the baselines. This work highlights the value of embedding SRL scaffolding and real-time AI support within gamified environments, offering design implications for educational technologies that aim to promote deeper learning and metacognitive skill development.</p> - https://arxiv.org/abs/2506.09968 - Wed, 11 Jun 2025 17:45:03 +0000 + Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure + https://arxiv.org/abs/2506.12278 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12278.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zheyuan Yang, Zexi Kuang, Xue Xia, Yilun Zhao</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs in test-case generation. TestCase-Eval includes 500 algorithm problems and 100,000 human-crafted solutions from the Codeforces platform. It focuses on two pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test sets probe diverse input scenarios and cover a wide range of potential failure modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored test input that reveals a specific incorrect code implementation. We provide a comprehensive assessment of 19 state-of-the-art open-source and proprietary LLMs on TestCase-Eval, offering insights into their strengths and limitations in generating effective test cases for algorithm problems.</p> + https://arxiv.org/abs/2506.12278 + Fri, 13 Jun 2025 23:56:17 +0000 - TaskCraft: Automated Generation of Agentic Tasks - https://arxiv.org/abs/2506.10055 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.10055.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Yang, Jian Yang, Ge Zhang, Jiaheng Liu, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, Wangchunshu Zhou</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> Agentic tasks, which require multi-step problem solving with autonomy, tool use, and adaptive reasoning, are becoming increasingly central to the advancement of NLP and AI. However, existing instruction data lacks tool interaction, and current agentic benchmarks rely on costly human annotation, limiting their scalability. We introduce TaskCraft, an automated workflow for generating difficulty-scalable, multi-tool, and verifiable agentic tasks with execution trajectories. TaskCraft expands atomic tasks using depth-based and width-based extensions to create structurally and hierarchically complex challenges. Empirical results show that these tasks improve prompt optimization in the generation workflow and enhance supervised fine-tuning of agentic foundation models. We present a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.</p> - https://arxiv.org/abs/2506.10055 - Wed, 11 Jun 2025 17:58:14 +0000 + CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following + https://arxiv.org/abs/2506.12285 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12285.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yinghao Ma, Siyou Li, Juntao Yu, Emmanouil Benetos, Akira Maezawa</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> Recent advances in audio-text large language models (LLMs) have opened new possibilities for music understanding and generation. However, existing benchmarks are limited in scope, often relying on simplified tasks or multi-choice evaluations that fail to reflect the complexity of real-world music analysis. We reinterpret a broad range of traditional MIR annotations as instruction-following formats and introduce CMI-Bench, a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks. These include genre classification, emotion regression, emotion tagging, instrument classification, pitch estimation, key detection, lyrics transcription, melody extraction, vocal technique recognition, instrument performance technique detection, music tagging, music captioning, and (down)beat tracking: reflecting core challenges in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models, ensuring direct comparability with supervised approaches. We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant performance gaps between LLMs and supervised models, along with their culture, chronological and gender bias, highlighting the potential and limitations of current models in addressing MIR tasks. CMI-Bench establishes a unified foundation for evaluating music instruction following, driving progress in music-aware LLMs.</p> + https://arxiv.org/abs/2506.12285 + Sat, 14 Jun 2025 00:18:44 +0000 - Provably Learning from Language Feedback - https://arxiv.org/abs/2506.10341 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.10341.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Wanqiao Xu, Allen Nie, Ruijie Zheng, Aditya Modi, Adith Swaminathan, Ching-An Cheng</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Interactively learning from observation and language feedback is an increasingly studied area driven by the emergence of large language model (LLM) agents. While impressive empirical demonstrations have been shown, so far a principled framing of these decision problems remains lacking. In this paper, we formalize the Learning from Language Feedback (LLF) problem, assert sufficient assumptions to enable learning despite latent rewards, and introduce transfer eluder dimension as a complexity measure to characterize the hardness of LLF problems. We show that transfer eluder dimension captures the intuition that information in the feedback changes the learning complexity of the LLF problem. We demonstrate cases where learning from rich language feedback can be exponentially faster than learning from reward. We develop a no-regret algorithm, called HELiX, that provably solves LLF problems through sequential interactions, with performance guarantees that scale with the transfer eluder dimension of the problem. Across several empirical domains, we show that HELiX performs well even when repeatedly prompting LLMs does not work reliably. Our contributions mark a first step towards designing principled interactive learning algorithms from generic language feedback.</p> - https://arxiv.org/abs/2506.10341 - Thu, 12 Jun 2025 04:35:02 +0000 + QFFT, Question-Free Fine-Tuning for Adaptive Reasoning + https://arxiv.org/abs/2506.12860 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12860.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Wanlong Liu, Junxiao Xu, Fei Yu, Yukang Lin, Ke Ji, Wenyu Chen, Yan Xu, Yasheng Wang, Lifeng Shang, Benyou Wang</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Recent advancements in Long Chain-of-Thought (CoT) reasoning models have improved performance on complex tasks, but they suffer from overthinking, which generates redundant reasoning steps, especially for simple questions. This paper revisits the reasoning patterns of Long and Short CoT models, observing that the Short CoT patterns offer concise reasoning efficiently, while the Long CoT patterns excel in challenging scenarios where the Short CoT patterns struggle. To enable models to leverage both patterns, we propose Question-Free Fine-Tuning (QFFT), a fine-tuning approach that removes the input question during training and learns exclusively from Long CoT responses. This approach enables the model to adaptively employ both reasoning patterns: it prioritizes the Short CoT patterns and activates the Long CoT patterns only when necessary. Experiments on various mathematical datasets demonstrate that QFFT reduces average response length by more than 50\%, while achieving performance comparable to Supervised Fine-Tuning (SFT). Additionally, QFFT exhibits superior performance compared to SFT in noisy, out-of-domain, and low-resource scenarios.</p> + https://arxiv.org/abs/2506.12860 + Sun, 15 Jun 2025 14:21:28 +0000 - Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning - https://arxiv.org/abs/2506.10521 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.10521.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuhao Zhou, Yiheng Wang, Xuming He, Ruoyao Xiao, Zhiwei Li, Qiantai Feng, Zijie Guo, Yuejin Yang, Hao Wu, Wenxuan Huang, Jiaqi Wei, Dan Si, Xiuqi Yao, Jia Bu, Haiwen Huang, Tianfan Fu, Shixiang Tang, Ben Fei, Dongzhan Zhou, Fenghua Ling, Yan Lu, Siqi Sun, Chenhui Li, Guanjie Zheng, Jiancheng Lv, Wenlong Zhang, Lei Bai</p><p><b>Upvotes:</b> 60</p><p><b>Summary:</b> Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries.</p> - https://arxiv.org/abs/2506.10521 - Thu, 12 Jun 2025 09:29:16 +0000 + Universal Jailbreak Suffixes Are Strong Attention Hijackers + https://arxiv.org/abs/2506.12880 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12880.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Matan Ben-Tov, Mor Geva, Mahmood Sharif</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> We study suffix-based jailbreaksx2013a powerful family of attacks against large language models (LLMs) that optimize adversarial suffixes to circumvent safety alignment. Focusing on the widely used foundational GCG attack (Zou et al., 2023), we observe that suffixes vary in efficacy: some markedly more universalx2013generalizing to many unseen harmful instructionsx2013than others. We first show that GCG's effectiveness is driven by a shallow, critical mechanism, built on the information flow from the adversarial suffix to the final chat template tokens before generation. Quantifying the dominance of this mechanism during generation, we find GCG irregularly and aggressively hijacks the contextualization process. Crucially, we tie hijacking to the universality phenomenon, with more universal suffixes being stronger hijackers. Subsequently, we show that these insights have practical implications: GCG universality can be efficiently enhanced (up to times5 in some cases) at no additional computational cost, and can also be surgically mitigated, at least halving attack success with minimal utility loss. We release our code and data at http://github.com/matanbt/interp-jailbreak.</p> + https://arxiv.org/abs/2506.12880 + Sun, 15 Jun 2025 15:20:37 +0000 - DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents - https://arxiv.org/abs/2506.11763 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.11763.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, Zhendong Mao</p><p><b>Upvotes:</b> 43</p><p><b>Summary:</b> Deep Research Agents are a prominent category of LLM-based agents. By autonomously orchestrating multistep web exploration, targeted retrieval, and higher-order synthesis, they transform vast amounts of online information into analyst-grade, citation-rich reports--compressing hours of manual desk research into minutes. However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we present DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. Evaluating DRAs is inherently complex and labor-intensive. We therefore propose two novel methodologies that achieve strong alignment with human judgment. The first is a reference-based method with adaptive criteria to assess the quality of generated research reports. The other framework is introduced to evaluate DRA's information retrieval and collection capabilities by assessing its effective citation count and overall citation accuracy. We have open-sourced DeepResearch Bench and key components of these frameworks at https://github.com/Ayanami0730/deep_research_bench to accelerate the development of practical LLM-based agents.</p> - https://arxiv.org/abs/2506.11763 - Fri, 13 Jun 2025 13:17:32 +0000 + Scaling Test-time Compute for LLM Agents + https://arxiv.org/abs/2506.12928 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12928.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, Wangchunshu Zhou</p><p><b>Upvotes:</b> 38</p><p><b>Summary:</b> Scaling test time compute has shown remarkable success in improving the reasoning abilities of large language models (LLMs). In this work, we conduct the first systematic exploration of applying test-time scaling methods to language agents and investigate the extent to which it improves their effectiveness. Specifically, we explore different test-time scaling strategies, including: (1) parallel sampling algorithms; (2) sequential revision strategies; (3) verifiers and merging methods; (4)strategies for diversifying rollouts.We carefully analyze and ablate the impact of different design strategies on applying test-time scaling on language agents, and have follow findings: 1. Scaling test time compute could improve the performance of agents. 2. Knowing when to reflect is important for agents. 3. Among different verification and result merging approaches, the list-wise method performs best. 4. Increasing diversified rollouts exerts a positive effect on the agent's task performance.</p> + https://arxiv.org/abs/2506.12928 + Sun, 15 Jun 2025 17:59:47 +0000 - VGR: Visual Grounded Reasoning - https://arxiv.org/abs/2506.11991 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.11991.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiacong Wang, Zijiang Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng, Jun Xiao</p><p><b>Upvotes:</b> 16</p><p><b>Summary:</b> In the field of multimodal chain-of-thought (CoT) reasoning, existing approaches predominantly rely on reasoning on pure language space, which inherently suffers from language bias and is largely confined to math or science domains. This narrow focus limits their ability to handle complex visual reasoning tasks that demand comprehensive understanding of image details. To address these limitations, this paper introduces VGR, a novel reasoning multimodal large language model (MLLM) with enhanced fine-grained visual perception capabilities. Unlike traditional MLLMs that answer the question or reasoning solely on the language space, our VGR first detects relevant regions that may help to solve problems, and then provides precise answers based on replayed image regions. To achieve this, we conduct a large-scale SFT dataset called VGR -SFT that contains reasoning data with mixed vision grounding and language deduction. The inference pipeline of VGR allows the model to choose bounding boxes for visual reference and a replay stage is introduced to integrates the corresponding regions into the reasoning process, enhancing multimodel comprehension. Experiments on the LLaVA-NeXT-7B baseline show that VGR achieves superior performance on multi-modal benchmarks requiring comprehensive image detail understanding. Compared to the baseline, VGR uses only 30\% of the image token count while delivering scores of +4.1 on MMStar, +7.1 on AI2D, and a +12.9 improvement on ChartQA.</p> - https://arxiv.org/abs/2506.11991 - Fri, 13 Jun 2025 17:47:43 +0000 + Efficient Medical VIE via Reinforcement Learning + https://arxiv.org/abs/2506.13363 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13363.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lijun Liu, Ruiyang Li, Zhaocheng Liu, Chenglin Zhu, Chong Li, Jiehan Cheng, Qiang Ju, Jian Xie</p><p><b>Upvotes:</b> 22</p><p><b>Summary:</b> Visual Information Extraction (VIE) converts unstructured document images into structured formats like JSON, critical for medical applications such as report analysis and online consultations. Traditional methods rely on OCR and language models, while end-to-end multimodal models offer direct JSON generation. However, domain-specific schemas and high annotation costs limit their effectiveness in medical VIE. We base our approach on the Reinforcement Learning with Verifiable Rewards (RLVR) framework to address these challenges using only 100 annotated samples. Our approach ensures dataset diversity, a balanced precision-recall reward mechanism to reduce hallucinations and improve field coverage, and innovative sampling strategies to enhance reasoning capabilities. Fine-tuning Qwen2.5-VL-7B with our RLVR method, we achieve state-of-the-art performance on medical VIE tasks, significantly improving F1, precision, and recall. While our models excel on tasks similar to medical datasets, performance drops on dissimilar tasks, highlighting the need for domain-specific optimization. Case studies further demonstrate the value of reasoning during training and inference for VIE.</p> + https://arxiv.org/abs/2506.13363 + Mon, 16 Jun 2025 11:10:25 +0000 - Hatevolution: What Static Benchmarks Don't Tell Us - https://arxiv.org/abs/2506.12148 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12148.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chiara Di Bonaventura, Barbara McGillivray, Yulan He, Albert Meroño-Peñuela</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Language changes over time, including in the hate speech domain, which evolves quickly following social dynamics and cultural shifts. While NLP research has investigated the impact of language evolution on model training and has proposed several solutions for it, its impact on model benchmarking remains under-explored. Yet, hate speech benchmarks play a crucial role to ensure model safety. In this paper, we empirically evaluate the robustness of 20 language models across two evolving hate speech experiments, and we show the temporal misalignment between static and time-sensitive evaluations. Our findings call for time-sensitive linguistic benchmarks in order to correctly and reliably evaluate language models in the hate speech domain.</p> - https://arxiv.org/abs/2506.12148 - Fri, 13 Jun 2025 18:08:19 +0000 + TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Scale-Oriented Contrast + https://arxiv.org/abs/2506.13387 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13387.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Beilei Cui, Yiming Huang, Long Bai, Hongliang Ren</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> This work presents a generalizable framework to transfer relative depth to metric depth. Current monocular depth estimation methods are mainly divided into metric depth estimation (MMDE) and relative depth estimation (MRDE). MMDEs estimate depth in metric scale but are often limited to a specific domain. MRDEs generalize well across different domains, but with uncertain scales which hinders downstream applications. To this end, we aim to build up a framework to solve scale uncertainty and transfer relative depth to metric depth. Previous methods used language as input and estimated two factors for conducting rescaling. Our approach, TR2M, utilizes both text description and image as inputs and estimates two rescale maps to transfer relative depth to metric depth at pixel level. Features from two modalities are fused with a cross-modality attention module to better capture scale information. A strategy is designed to construct and filter confident pseudo metric depth for more comprehensive supervision. We also develop scale-oriented contrastive learning to utilize depth distribution as guidance to enforce the model learning about intrinsic knowledge aligning with the scale distribution. TR2M only exploits a small number of trainable parameters to train on datasets in various domains and experiments not only demonstrate TR2M's great performance in seen datasets but also reveal superior zero-shot capabilities on five unseen datasets. We show the huge potential in pixel-wise transferring relative depth to metric depth with language assistance. (Code is available at: https://github.com/BeileiCui/TR2M)</p> + https://arxiv.org/abs/2506.13387 + Mon, 16 Jun 2025 11:50:00 +0000 - Supernova Event Dataset: Interpreting Large Language Model's Personality through Critical Event Analysis - https://arxiv.org/abs/2506.12189 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12189.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Pranav Agarwal, Ioana Ciucă</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Large Language Models (LLMs) are increasingly integrated into everyday applications. As their influence grows, understanding their decision making and underlying personality becomes essential. In this work, we interpret model personality using our proposed Supernova Event Dataset, a novel dataset with diverse articles spanning biographies, historical events, news, and scientific discoveries. We use this dataset to benchmark LLMs on extracting and ranking key events from text, a subjective and complex challenge that requires reasoning over long-range context and modeling causal chains. We evaluate small models like Phi-4, Orca 2, and Qwen 2.5, and large, stronger models such as Claude 3.7, Gemini 2.5, and OpenAI o3, and propose a framework where another LLM acts as a judge to infer each model's personality based on its selection and classification of events. Our analysis shows distinct personality traits: for instance, Orca 2 demonstrates emotional reasoning focusing on interpersonal dynamics, while Qwen 2.5 displays a more strategic, analytical style. When analyzing scientific discovery events, Claude Sonnet 3.7 emphasizes conceptual framing, Gemini 2.5 Pro prioritizes empirical validation, and o3 favors step-by-step causal reasoning. This analysis improves model interpretability, making them user-friendly for a wide range of diverse applications.</p> - https://arxiv.org/abs/2506.12189 - Fri, 13 Jun 2025 19:31:52 +0000 + CAMS: A CityGPT-Powered Agentic Framework for Urban Human Mobility Simulation + https://arxiv.org/abs/2506.13599 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13599.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuwei Du, Jie Feng, Jian Yuan, Yong Li</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Human mobility simulation plays a crucial role in various real-world applications. Recently, to address the limitations of traditional data-driven approaches, researchers have explored leveraging the commonsense knowledge and reasoning capabilities of large language models (LLMs) to accelerate human mobility simulation. However, these methods suffer from several critical shortcomings, including inadequate modeling of urban spaces and poor integration with both individual mobility patterns and collective mobility distributions. To address these challenges, we propose CityGPT-Powered Agentic framework for Mobility Simulation (CAMS), an agentic framework that leverages the language based urban foundation model to simulate human mobility in urban space. CAMS comprises three core modules, including MobExtractor to extract template mobility patterns and synthesize new ones based on user profiles, GeoGenerator to generate anchor points considering collective knowledge and generate candidate urban geospatial knowledge using an enhanced version of CityGPT, TrajEnhancer to retrieve spatial knowledge based on mobility patterns and generate trajectories with real trajectory preference alignment via DPO. Experiments on real-world datasets show that CAMS achieves superior performance without relying on externally provided geospatial information. Moreover, by holistically modeling both individual mobility patterns and collective mobility constraints, CAMS generates more realistic and plausible trajectories. In general, CAMS establishes a new paradigm that integrates the agentic framework with urban-knowledgeable LLMs for human mobility simulation.</p> + https://arxiv.org/abs/2506.13599 + Mon, 16 Jun 2025 15:24:07 +0000 - EgoPrivacy: What Your First-Person Camera Says About You? - https://arxiv.org/abs/2506.12258 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12258.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yijiang Li, Genpei Zhang, Jiacheng Cheng, Yi Li, Xiaojun Shan, Dashan Gao, Jiancheng Lyu, Yuan Li, Ning Bi, Nuno Vasconcelos</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> While the rapid proliferation of wearable cameras has raised significant concerns about egocentric video privacy, prior work has largely overlooked the unique privacy threats posed to the camera wearer. This work investigates the core question: How much privacy information about the camera wearer can be inferred from their first-person view videos? We introduce EgoPrivacy, the first large-scale benchmark for the comprehensive evaluation of privacy risks in egocentric vision. EgoPrivacy covers three types of privacy (demographic, individual, and situational), defining seven tasks that aim to recover private information ranging from fine-grained (e.g., wearer's identity) to coarse-grained (e.g., age group). To further emphasize the privacy threats inherent to egocentric vision, we propose Retrieval-Augmented Attack, a novel attack strategy that leverages ego-to-exo retrieval from an external pool of exocentric videos to boost the effectiveness of demographic privacy attacks. An extensive comparison of the different attacks possible under all threat models is presented, showing that private information of the wearer is highly susceptible to leakage. For instance, our findings indicate that foundation models can effectively compromise wearer privacy even in zero-shot settings by recovering attributes such as identity, scene, gender, and race with 70-80% accuracy. Our code and data are available at https://github.com/williamium3000/ego-privacy.</p> - https://arxiv.org/abs/2506.12258 - Fri, 13 Jun 2025 22:19:54 +0000 + Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model + https://arxiv.org/abs/2506.13642 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13642.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shaolei Zhang, Shoutao Guo, Qingkai Fang, Yan Zhou, Yang Feng</p><p><b>Upvotes:</b> 21</p><p><b>Summary:</b> The emergence of GPT-4o-like large multimodal models (LMMs) has raised the exploration of integrating text, vision, and speech modalities to support more flexible multimodal interaction. Existing LMMs typically concatenate representation of modalities along the sequence dimension and feed them into a large language model (LLM) backbone. While sequence-dimension concatenation is straightforward for modality integration, it often relies heavily on large-scale data to learn modality alignments. In this paper, we aim to model the relationships between modalities more purposefully, thereby achieving more efficient and flexible modality alignments. To this end, we propose Stream-Omni, a large language-vision-speech model with efficient modality alignments, which can simultaneously support interactions under various modality combinations. Stream-Omni employs LLM as the backbone and aligns the vision and speech to the text based on their relationships. For vision that is semantically complementary to text, Stream-Omni uses sequence-dimension concatenation to achieve vision-text alignment. For speech that is semantically consistent with text, Stream-Omni introduces a CTC-based layer-dimension mapping to achieve speech-text alignment. In this way, Stream-Omni can achieve modality alignments with less data (especially speech), enabling the transfer of text capabilities to other modalities. Experiments on various benchmarks demonstrate that Stream-Omni achieves strong performance on visual understanding, speech interaction, and vision-grounded speech interaction tasks. Owing to the layer-dimensional mapping, Stream-Omni can simultaneously provide intermediate text outputs (such as ASR transcriptions and model responses) during speech interaction, offering users a comprehensive multimodal experience.</p> + https://arxiv.org/abs/2506.13642 + Mon, 16 Jun 2025 16:06:45 +0000 - QGuard:Question-based Zero-shot Guard for Multi-modal LLM Safety - https://arxiv.org/abs/2506.12299 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12299.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Taegyeong Lee, Jeonghwa Yoo, Hyoungseo Cho, Soo Yong Kim, Yunho Maeng</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> The recent advancements in Large Language Models(LLMs) have had a significant impact on a wide range of fields, from general domains to specialized areas. However, these advancements have also significantly increased the potential for malicious users to exploit harmful and jailbreak prompts for malicious attacks. Although there have been many efforts to prevent harmful prompts and jailbreak prompts, protecting LLMs from such malicious attacks remains an important and challenging task. In this paper, we propose QGuard, a simple yet effective safety guard method, that utilizes question prompting to block harmful prompts in a zero-shot manner. Our method can defend LLMs not only from text-based harmful prompts but also from multi-modal harmful prompt attacks. Moreover, by diversifying and modifying guard questions, our approach remains robust against the latest harmful prompts without fine-tuning. Experimental results show that our model performs competitively on both text-only and multi-modal harmful datasets. Additionally, by providing an analysis of question prompting, we enable a white-box analysis of user inputs. We believe our method provides valuable insights for real-world LLM services in mitigating security risks associated with harmful prompts.</p> - https://arxiv.org/abs/2506.12299 - Sat, 14 Jun 2025 01:23:50 +0000 + xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations + https://arxiv.org/abs/2506.13651 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13651.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, Chen Sun, Han Hou, Hui Yang, James Pan, Jianan Lou, Jiayi Mao, Jizheng Liu, Jinpeng Li, Kangyi Liu, Kenkun Liu, Rui Wang, Run Li, Tong Niu, Wenlong Zhang, Wenqi Yan, Xuanzheng Wang, Yuchen Zhang, Yi-Hsin Hung, Yuan Jiang, Zexuan Liu, Zihan Yin, Zijian Ma, Zhiwen Mo</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity. While existing benchmarks often focus on isolated technical skills, they may not accurately reflect the economic value agents deliver in professional settings. To address this, xbench targets commercially significant domains with evaluation tasks defined by industry professionals. Our framework creates metrics that strongly correlate with productivity value, enables prediction of Technology-Market Fit (TMF), and facilitates tracking of product capabilities over time. As our initial implementations, we present two benchmarks: Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world headhunting business scenarios to evaluate agents' abilities in company mapping, information retrieval, and talent sourcing. For Marketing, we assess agents' ability to match influencers with advertiser needs, evaluating their performance across 50 advertiser requirements using a curated pool of 836 candidate influencers. We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains. Our continuously updated evalsets and evaluations are available at https://xbench.org.</p> + https://arxiv.org/abs/2506.13651 + Mon, 16 Jun 2025 16:16:14 +0000 - Language Surgery in Multilingual Large Language Models - https://arxiv.org/abs/2506.12450 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12450.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Joanito Agili Lopo, Muhammad Ravi Shulthan Habibi, Tack Hwa Wong, Muhammad Ilham Ghozali, Fajri Koto, Genta Indra Winata, Peerat Limkonchotiwat, Alham Fikri Aji, Samuel Cahyawijaya</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages, revolutionizing natural language processing. This paper investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers, and its implications for disentangling language-specific and language-agnostic information. We empirically confirm the existence of this alignment, analyze its behavior in comparison to explicitly designed alignment models, and demonstrate its potential for language-specific manipulation without semantic degradation. Building on these findings, we propose Inference-Time Language Control (ITLC), a novel method that leverages latent injection to enable precise cross-lingual language control and mitigate language confusion in LLMs. Our experiments highlight ITLC's strong cross-lingual control capabilities while preserving semantic integrity in target languages. Furthermore, we demonstrate its effectiveness in alleviating the cross-lingual language confusion problem, which persists even in current large-scale LLMs, leading to inconsistent language generation. This work advances our understanding of representation alignment in LLMs and introduces a practical solution for enhancing their cross-lingual performance.</p> - https://arxiv.org/abs/2506.12450 - Sat, 14 Jun 2025 11:09:50 +0000 + Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise Pooled Representations + https://arxiv.org/abs/2506.13901 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13901.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Abhilekh Borah, Chhavi Sharma, Danush Khanna, Utkarsh Bhatt, Gurpreet Singh, Hasnat Md Abdullah, Raghav Kaushik Ravi, Vinija Jain, Jyoti Patel, Shubham Singh, Vasu Sharma, Arpita Vats, Rahul Raja, Aman Chadha, Amitava Das</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Alignment is no longer a luxury, it is a necessity. As large language models (LLMs) enter high-stakes domains like education, healthcare, governance, and law, their behavior must reliably reflect human-aligned values and safety constraints. Yet current evaluations rely heavily on behavioral proxies such as refusal rates, G-Eval scores, and toxicity classifiers, all of which have critical blind spots. Aligned models are often vulnerable to jailbreaking, stochasticity of generation, and alignment faking. To address this issue, we introduce the Alignment Quality Index (AQI). This novel geometric and prompt-invariant metric empirically assesses LLM alignment by analyzing the separation of safe and unsafe activations in latent space. By combining measures such as the Davies-Bouldin Score (DBS), Dunn Index (DI), Xie-Beni Index (XBI), and Calinski-Harabasz Index (CHI) across various formulations, AQI captures clustering quality to detect hidden misalignments and jailbreak risks, even when outputs appear compliant. AQI also serves as an early warning signal for alignment faking, offering a robust, decoding invariant tool for behavior agnostic safety auditing. Additionally, we propose the LITMUS dataset to facilitate robust evaluation under these challenging conditions. Empirical tests on LITMUS across different models trained under DPO, GRPO, and RLHF conditions demonstrate AQI's correlation with external judges and ability to reveal vulnerabilities missed by refusal metrics. We make our implementation publicly available to foster future research in this area.</p> + https://arxiv.org/abs/2506.13901 + Mon, 16 Jun 2025 18:22:28 +0000 - Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts - https://arxiv.org/abs/2506.12552 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12552.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro, Preslav Nakov</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> In an age characterized by the proliferation of mis- and disinformation online, it is critical to empower readers to understand the content they are reading. Important efforts in this direction rely on manual or automatic fact-checking, which can be challenging for emerging claims with limited information. Such scenarios can be handled by assessing the reliability and the political bias of the source of the claim, i.e., characterizing entire news outlets rather than individual claims or articles. This is an important but understudied research direction. While prior work has looked into linguistic and social contexts, we do not analyze individual articles or information in social media. Instead, we propose a novel methodology that emulates the criteria that professional fact-checkers use to assess the factuality and political bias of an entire outlet. Specifically, we design a variety of prompts based on these criteria and elicit responses from large language models (LLMs), which we aggregate to make predictions. In addition to demonstrating sizable improvements over strong baselines via extensive experiments with multiple LLMs, we provide an in-depth error analysis of the effect of media popularity and region on model performance. Further, we conduct an ablation study to highlight the key components of our dataset that contribute to these improvements. To facilitate future research, we released our dataset and code at https://github.com/mbzuai-nlp/llm-media-profiling.</p> - https://arxiv.org/abs/2506.12552 - Sat, 14 Jun 2025 15:49:20 +0000 + DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance + https://arxiv.org/abs/2506.13922 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13922.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Maximilian Du, Shuran Song</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Deploying large, complex policies in the real world requires the ability to steer them to fit the needs of a situation. Most common steering approaches, like goal-conditioning, require training the robot policy with a distribution of test-time objectives in mind. To overcome this limitation, we present DynaGuide, a steering method for diffusion policies using guidance from an external dynamics model during the diffusion denoising process. DynaGuide separates the dynamics model from the base policy, which gives it multiple advantages, including the ability to steer towards multiple objectives, enhance underrepresented base policy behaviors, and maintain robustness on low-quality objectives. The separate guidance signal also allows DynaGuide to work with off-the-shelf pretrained diffusion policies. We demonstrate the performance and features of DynaGuide against other steering approaches in a series of simulated and real experiments, showing an average steering success of 70% on a set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x when steered with low-quality objectives. We also successfully steer an off-the-shelf real robot policy to express preference for particular objects and even create novel behavior. Videos and more can be found on the project website: https://dynaguide.github.io</p> + https://arxiv.org/abs/2506.13922 + Mon, 16 Jun 2025 19:00:54 +0000 - DoTA-RAG: Dynamic of Thought Aggregation RAG - https://arxiv.org/abs/2506.12571 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12571.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Saksorn Ruangtanusak, Natthapath Rungseesiripak, Peerawat Rojratchadakorn, Monthol Charattrakool, Natapong Nitarach</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> In this paper, we introduce DoTA-RAG (Dynamic-of-Thought Aggregation RAG), a retrieval-augmented generation system optimized for high-throughput, large-scale web knowledge indexes. Traditional RAG pipelines often suffer from high latency and limited accuracy over massive, diverse datasets. DoTA-RAG addresses these challenges with a three-stage pipeline: query rewriting, dynamic routing to specialized sub-indexes, and multi-stage retrieval and ranking. We further enhance retrieval by evaluating and selecting a superior embedding model, re-embedding the large FineWeb-10BT corpus. Moreover, we create a diverse Q&A dataset of 500 questions generated via the DataMorgana setup across a broad range of WebOrganizer topics and formats. DoTA-RAG improves the answer correctness score from 0.752 (baseline, using LiveRAG pre-built vector store) to 1.478 while maintaining low latency, and it achieves a 0.929 correctness score on the Live Challenge Day. These results highlight DoTA-RAG's potential for practical deployment in domains requiring fast, reliable access to large and evolving knowledge sources.</p> - https://arxiv.org/abs/2506.12571 - Sat, 14 Jun 2025 16:56:00 +0000 + Taming Polysemanticity in LLMs: Provable Feature Recovery via Sparse Autoencoders + https://arxiv.org/abs/2506.14002 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14002.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siyu Chen, Heejune Sheen, Xuyuan Xiong, Tianhao Wang, Zhuoran Yang</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> We study the challenge of achieving theoretically grounded feature recovery using Sparse Autoencoders (SAEs) for the interpretation of Large Language Models. Existing SAE training algorithms often lack rigorous mathematical guarantees and suffer from practical limitations such as hyperparameter sensitivity and instability. To address these issues, we first propose a novel statistical framework for the feature recovery problem, which includes a new notion of feature identifiability by modeling polysemantic features as sparse mixtures of underlying monosemantic concepts. Building on this framework, we introduce a new SAE training algorithm based on ``bias adaptation'', a technique that adaptively adjusts neural network bias parameters to ensure appropriate activation sparsity. We theoretically prove that this algorithm correctly recovers all monosemantic features when input data is sampled from our proposed statistical model. Furthermore, we develop an improved empirical variant, Group Bias Adaptation (GBA), and demonstrate its superior performance against benchmark methods when applied to LLMs with up to 1.5 billion parameters. This work represents a foundational step in demystifying SAE training by providing the first SAE algorithm with theoretical recovery guarantees, thereby advancing the development of more transparent and trustworthy AI systems through enhanced mechanistic interpretability.</p> + https://arxiv.org/abs/2506.14002 + Mon, 16 Jun 2025 20:58:05 +0000 - MS4UI: A Dataset for Multi-modal Summarization of User Interface Instructional Videos - https://arxiv.org/abs/2506.12623 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12623.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuan Zang, Hao Tan, Seunghyun Yoon, Franck Dernoncourt, Jiuxiang Gu, Kushal Kafle, Chen Sun, Trung Bui</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> We study multi-modal summarization for instructional videos, whose goal is to provide users an efficient way to learn skills in the form of text instructions and key video frames. We observe that existing benchmarks focus on generic semantic-level video summarization, and are not suitable for providing step-by-step executable instructions and illustrations, both of which are crucial for instructional videos. We propose a novel benchmark for user interface (UI) instructional video summarization to fill the gap. We collect a dataset of 2,413 UI instructional videos, which spans over 167 hours. These videos are manually annotated for video segmentation, text summarization, and video summarization, which enable the comprehensive evaluations for concise and executable video summarization. We conduct extensive experiments on our collected MS4UI dataset, which suggest that state-of-the-art multi-modal summarization methods struggle on UI video summarization, and highlight the importance of new methods for UI instructional video summarization.</p> - https://arxiv.org/abs/2506.12623 - Sat, 14 Jun 2025 20:39:32 +0000 + MultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark for Financial LLM Evaluation + https://arxiv.org/abs/2506.14028 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14028.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang, Shanshan Yang, Yilun Zhao, Zhiwei Liu, Peng Lu, Jerry Huang, Suyuchen Wang, Triantafillos Papadopoulos, Polydoros Giannouris, Efstathia Soufleri, Nuo Chen, Guojun Xiong, Zhiyang Deng, Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Alejandro Lopez-Lira, Xi Chen, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou, Qianqian Xie</p><p><b>Upvotes:</b> 65</p><p><b>Summary:</b> Recent advances in large language models (LLMs) have accelerated progress in financial NLP and applications, yet existing benchmarks remain limited to monolingual and unimodal settings, often over-relying on simple tasks and failing to reflect the complexity of real-world financial communication. We introduce MultiFinBen, the first multilingual and multimodal benchmark tailored to the global financial domain, evaluating LLMs across modalities (text, vision, audio) and linguistic settings (monolingual, bilingual, multilingual) on domain-specific tasks. We introduce two novel tasks, including PolyFiQA-Easy and PolyFiQA-Expert, the first multilingual financial benchmarks requiring models to perform complex reasoning over mixed-language inputs; and EnglishOCR and SpanishOCR, the first OCR-embedded financial QA tasks challenging models to extract and reason over information from visual-text financial documents. Moreover, we propose a dynamic, difficulty-aware selection mechanism and curate a compact, balanced benchmark rather than simple aggregation existing datasets. Extensive evaluation of 22 state-of-the-art models reveals that even the strongest models, despite their general multimodal and multilingual capabilities, struggle dramatically when faced with complex cross-lingual and multimodal tasks in financial domain. MultiFinBen is publicly released to foster transparent, reproducible, and inclusive progress in financial studies and applications.</p> + https://arxiv.org/abs/2506.14028 + Mon, 16 Jun 2025 22:01:49 +0000 - PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization - https://arxiv.org/abs/2506.12915 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12915.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Meiling Tao, Chenghao Zhu, Dongyi Ding, Tiannan Wang, Yuchen Eleanor Jiang, Wangchunshu Zhou</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> With the rapid improvement in the general capabilities of LLMs, LLM personalization, i.e., how to build LLM systems that can generate personalized responses or services that are tailored to distinct user personas, has become an increasingly important research and engineering problem. However, unlike many new challenging benchmarks being released for evaluating the general/reasoning capabilities, the lack of high-quality benchmarks for evaluating LLM personalization greatly hinders progress in this field. To address this, we introduce PersonaFeedback, a new benchmark that directly evaluates LLMs' ability to provide personalized responses given pre-defined user personas and queries. Unlike existing benchmarks that require models to infer implicit user personas from historical interactions, PersonaFeedback decouples persona inference from personalization, focusing on evaluating the model's ability to generate responses tailored to explicit personas. PersonaFeedback consists of 8298 human-annotated test cases, which are categorized into easy, medium, and hard tiers based on the contextual complexity of the user personas and the difficulty in distinguishing subtle differences between two personalized responses. We conduct comprehensive evaluations across a wide range of models. The empirical results reveal that even state-of-the-art LLMs that can solve complex real-world reasoning tasks could fall short on the hard tier of PersonaFeedback where even human evaluators may find the distinctions challenging. Furthermore, we conduct an in-depth analysis of failure modes across various types of systems, demonstrating that the current retrieval-augmented framework should not be seen as a de facto solution for personalization tasks. All benchmark data, annotation protocols, and the evaluation pipeline will be publicly available to facilitate future research on LLM personalization.</p> - https://arxiv.org/abs/2506.12915 - Sun, 15 Jun 2025 17:19:19 +0000 + Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Team + https://arxiv.org/abs/2506.14234 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14234.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Md Tanzib Hosain, Salman Rahman, Md Kishor Morol, Md Rizwan Parvez</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> Despite impressive progress on complex reasoning, current large language models (LLMs) typically operate in isolation - treating each problem as an independent attempt, without accumulating or integrating experiential knowledge. In contrast, expert problem solvers - such as Olympiad or programming contest teams - leverage a rich tapestry of experiences: absorbing mentorship from coaches, developing intuition from past problems, leveraging knowledge of tool usage and library functionality, adapting strategies based on the expertise and experiences of peers, continuously refining their reasoning through trial and error, and learning from other related problems even during competition. We introduce Xolver, a training-free multi-agent reasoning framework that equips a black-box LLM with a persistent, evolving memory of holistic experience. Xolver integrates diverse experience modalities, including external and self-retrieval, tool use, collaborative interactions, agent-driven evaluation, and iterative refinement. By learning from relevant strategies, code fragments, and abstract reasoning patterns at inference time, Xolver avoids generating solutions from scratch - marking a transition from isolated inference toward experience-aware language agents. Built on both open-weight and proprietary models, Xolver consistently outperforms specialized reasoning agents. Even with lightweight backbones (e.g., QWQ-32B), it often surpasses advanced models including Qwen3-235B, Gemini 2.5 Pro, o3, and o4-mini-high. With o3-mini-high, it achieves new best results on GSM8K (98.1%), AIME'24 (94.4%), AIME'25 (93.7%), Math-500 (99.8%), and LiveCodeBench-V5 (91.6%) - highlighting holistic experience learning as a key step toward generalist agents capable of expert-level reasoning. Code and data are available at https://kagnlp.github.io/xolver.github.io/.</p> + https://arxiv.org/abs/2506.14234 + Tue, 17 Jun 2025 06:47:19 +0000 - Forecasting Time Series with LLMs via Patch-Based Prompting and Decomposition - https://arxiv.org/abs/2506.12953 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.12953.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mayank Bumb, Anshul Vemulapalli, Sri Harsha Vardhan Prasad Jella, Anish Gupta, An La, Ryan A. Rossi, Hongjie Chen, Franck Dernoncourt, Nesreen K. Ahmed, Yu Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent advances in Large Language Models (LLMs) have demonstrated new possibilities for accurate and efficient time series analysis, but prior work often required heavy fine-tuning and/or ignored inter-series correlations. In this work, we explore simple and flexible prompt-based strategies that enable LLMs to perform time series forecasting without extensive retraining or the use of a complex external architecture. Through the exploration of specialized prompting methods that leverage time series decomposition, patch-based tokenization, and similarity-based neighbor augmentation, we find that it is possible to enhance LLM forecasting quality while maintaining simplicity and requiring minimal preprocessing of data. To this end, we propose our own method, PatchInstruct, which enables LLMs to make precise and effective predictions.</p> - https://arxiv.org/abs/2506.12953 - Sun, 15 Jun 2025 19:42:58 +0000 + Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs + https://arxiv.org/abs/2506.14245 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14245.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, Jiang Bian, Mao Yang</p><p><b>Upvotes:</b> 25</p><p><b>Summary:</b> Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for advancing the reasoning capabilities of Large Language Models (LLMs). However, a critical paradox clouds its efficacy: RLVR-tuned models often underperform their base models on the Pass@K metric for solution-finding, leading to the hypothesis that RLVR merely re-weights existing reasoning paths at the cost of reasoning diversity. In this work, we resolve this contradiction by identifying the source of the problem: the Pass@K metric itself is a flawed measure of reasoning, as it credits correct final answers that probably arise from inaccurate or incomplete chains of thought (CoTs). To address this, we introduce a more precise evaluation metric, CoT-Pass@K, which mandates that both the reasoning path and the final answer be correct. We provide a new theoretical foundation that formalizes how RLVR, unlike traditional RL, is uniquely structured to incentivize logical integrity. Our empirical results are supportive: using CoT-Pass@K, we observe that RLVR can incentivize the generalization of correct reasoning for all values of K. Furthermore, by analyzing the training dynamics, we find that this enhanced reasoning capability emerges early in the training process and smoothly generalizes. Our work provides a clear perspective on the role of RLVR, offers a more reliable method for its evaluation, and confirms its potential to genuinely advance machine reasoning.</p> + https://arxiv.org/abs/2506.14245 + Tue, 17 Jun 2025 07:06:56 +0000 - Ai-Facilitated Analysis of Abstracts and Conclusions: Flagging Unsubstantiated Claims and Ambiguous Pronouns - https://arxiv.org/abs/2506.13172 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13172.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Evgeny Markhasin</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> We present and evaluate a suite of proof-of-concept (PoC), structured workflow prompts designed to elicit human-like hierarchical reasoning while guiding Large Language Models (LLMs) in high-level semantic and linguistic analysis of scholarly manuscripts. The prompts target two non-trivial analytical tasks: identifying unsubstantiated claims in summaries (informational integrity) and flagging ambiguous pronoun references (linguistic clarity). We conducted a systematic, multi-run evaluation on two frontier models (Gemini Pro 2.5 Pro and ChatGPT Plus o3) under varied context conditions. Our results for the informational integrity task reveal a significant divergence in model performance: while both models successfully identified an unsubstantiated head of a noun phrase (95% success), ChatGPT consistently failed (0% success) to identify an unsubstantiated adjectival modifier that Gemini correctly flagged (95% success), raising a question regarding potential influence of the target's syntactic role. For the linguistic analysis task, both models performed well (80-90% success) with full manuscript context. In a summary-only setting, however, ChatGPT achieved a perfect (100%) success rate, while Gemini's performance was substantially degraded. Our findings suggest that structured prompting is a viable methodology for complex textual analysis but show that prompt performance may be highly dependent on the interplay between the model, task type, and context, highlighting the need for rigorous, model-specific testing.</p> - https://arxiv.org/abs/2506.13172 - Mon, 16 Jun 2025 07:34:31 +0000 + LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs + https://arxiv.org/abs/2506.14429 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14429.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiaoran Liu, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, Xipeng Qiu</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> Large Language Diffusion Models, or diffusion LLMs, have emerged as a significant focus in NLP research, with substantial effort directed toward understanding their scalability and downstream task performance. However, their long-context capabilities remain unexplored, lacking systematic analysis or methods for context extension. In this work, we present the first systematic investigation comparing the long-context performance of diffusion LLMs and traditional auto-regressive LLMs. We first identify a unique characteristic of diffusion LLMs, unlike auto-regressive LLMs, they maintain remarkably \textit{stable perplexity} during direct context extrapolation. Furthermore, where auto-regressive models fail outright during the Needle-In-A-Haystack task with context exceeding their pretrained length, we discover diffusion LLMs exhibit a distinct \textit{local perception} phenomenon, enabling successful retrieval from recent context segments. We explain both phenomena through the lens of Rotary Position Embedding (RoPE) scaling theory. Building on these observations, we propose LongLLaDA, a training-free method that integrates LLaDA with the NTK-based RoPE extrapolation. Our results validate that established extrapolation scaling laws remain effective for extending the context windows of diffusion LLMs. Furthermore, we identify long-context tasks where diffusion LLMs outperform auto-regressive LLMs and others where they fall short. Consequently, this study establishes the first context extrapolation method for diffusion LLMs while providing essential theoretical insights and empirical benchmarks critical for advancing future research on long-context diffusion LLMs.</p> + https://arxiv.org/abs/2506.14429 + Tue, 17 Jun 2025 11:45:37 +0000 - SeqPE: Transformer with Sequential Position Encoding - https://arxiv.org/abs/2506.13277 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13277.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Huyang Li, Yahui Liu, Hongyu Sun, Deng Cai, Leyang Cui, Wei Bi, Peilin Zhao, Taro Watanabe</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Since self-attention layers in Transformers are permutation invariant by design, positional encodings must be explicitly incorporated to enable spatial understanding. However, fixed-size lookup tables used in traditional learnable position embeddings (PEs) limit extrapolation capabilities beyond pre-trained sequence lengths. Expert-designed methods such as ALiBi and RoPE, mitigate this limitation but demand extensive modifications for adapting to new modalities, underscoring fundamental challenges in adaptability and scalability. In this work, we present SeqPE, a unified and fully learnable position encoding framework that represents each n-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings in an end-to-end manner. To regularize SeqPE's embedding space, we introduce two complementary objectives: a contrastive objective that aligns embedding distances with a predefined position-distance function, and a knowledge distillation loss that anchors out-of-distribution position embeddings to in-distribution teacher representations, further enhancing extrapolation performance. Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM), and accuracy--particularly under context length extrapolation--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign. We release our code, data, and checkpoints at https://github.com/ghrua/seqpe.</p> - https://arxiv.org/abs/2506.13277 - Mon, 16 Jun 2025 09:16:40 +0000 + Align Your Flow: Scaling Continuous-Time Flow Map Distillation + https://arxiv.org/abs/2506.14603 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14603.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Amirmojtaba Sabour, Sanja Fidler, Karsten Kreis</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Diffusion- and flow-based models have emerged as state-of-the-art generative modeling approaches, but they require many sampling steps. Consistency models can distill these models into efficient one-step generators; however, unlike flow- and diffusion-based methods, their performance inevitably degrades when increasing the number of steps, which we show both analytically and empirically. Flow maps generalize these approaches by connecting any two noise levels in a single step and remain effective across all step counts. In this paper, we introduce two new continuous-time objectives for training flow maps, along with additional novel training techniques, generalizing existing consistency and flow matching objectives. We further demonstrate that autoguidance can improve performance, using a low-quality model for guidance during distillation, and an additional boost can be achieved by adversarial finetuning, with minimal loss in sample diversity. We extensively validate our flow map models, called Align Your Flow, on challenging image generation benchmarks and achieve state-of-the-art few-step generation performance on both ImageNet 64x64 and 512x512, using small and efficient neural networks. Finally, we show text-to-image flow map models that outperform all existing non-adversarially trained few-step samplers in text-conditioned synthesis.</p> + https://arxiv.org/abs/2506.14603 + Tue, 17 Jun 2025 15:06:07 +0000 - AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy - https://arxiv.org/abs/2506.13284 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13284.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> In this work, we investigate the synergy between supervised fine-tuning (SFT) and reinforcement learning (RL) in developing strong reasoning models. We begin by curating the SFT training data through two scaling strategies: increasing the number of collected prompts and the number of generated responses per prompt. Both approaches yield notable improvements in reasoning performance, with scaling the number of prompts resulting in more substantial gains. We then explore the following questions regarding the synergy between SFT and RL: (i) Does a stronger SFT model consistently lead to better final performance after large-scale RL training? (ii) How can we determine an appropriate sampling temperature during RL training to effectively balance exploration and exploitation for a given SFT initialization? Our findings suggest that (i) holds true, provided effective RL training is conducted, particularly when the sampling temperature is carefully chosen to maintain the temperature-adjusted entropy around 0.3, a setting that strikes a good balance between exploration and exploitation. Notably, the performance gap between initial SFT models narrows significantly throughout the RL process. Leveraging a strong SFT foundation and insights into the synergistic interplay between SFT and RL, our AceReason-Nemotron-1.1 7B model significantly outperforms AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among Qwen2.5-7B-based reasoning models on challenging math and code benchmarks, thereby demonstrating the effectiveness of our post-training recipe. We release the model and data at: https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B</p> - https://arxiv.org/abs/2506.13284 - Mon, 16 Jun 2025 09:27:48 +0000 + Guaranteed Guess: A Language Modeling Approach for CISC-to-RISC Transpilation with Testing Guarantees + https://arxiv.org/abs/2506.14606 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14606.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ahmed Heakl, Sarim Hashmi, Chaimaa Abi, Celine Lee, Abdulrahman Mahmoud</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> The hardware ecosystem is rapidly evolving, with increasing interest in translating low-level programs across different instruction set architectures (ISAs) in a quick, flexible, and correct way to enhance the portability and longevity of existing code. A particularly challenging class of this transpilation problem is translating between complex- (CISC) and reduced- (RISC) hardware architectures, due to fundamental differences in instruction complexity, memory models, and execution paradigms. In this work, we introduce GG (Guaranteed Guess), an ISA-centric transpilation pipeline that combines the translation power of pre-trained large language models (LLMs) with the rigor of established software testing constructs. Our method generates candidate translations using an LLM from one ISA to another, and embeds such translations within a software-testing framework to build quantifiable confidence in the translation. We evaluate our GG approach over two diverse datasets, enforce high code coverage (>98%) across unit tests, and achieve functional/semantic correctness of 99% on HumanEval programs and 49% on BringupBench programs, respectively. Further, we compare our approach to the state-of-the-art Rosetta 2 framework on Apple Silicon, showcasing 1.73x faster runtime performance, 1.47x better energy efficiency, and 2.41x better memory usage for our transpiled code, demonstrating the effectiveness of GG for real-world CISC-to-RISC translation tasks. We will open-source our codes, data, models, and benchmarks to establish a common foundation for ISA-level code translation research.</p> + https://arxiv.org/abs/2506.14606 + Tue, 17 Jun 2025 15:06:54 +0000 - A Technical Study into Small Reasoning Language Models - https://arxiv.org/abs/2506.13404 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13404.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xialie Zhuang, Peixian Ma, Zhikai Jia, Zheng Cao, Shiwei Liu</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> The ongoing evolution of language models has led to the development of large-scale architectures that demonstrate exceptional performance across a wide range of tasks. However, these models come with significant computational and energy demands, as well as potential privacy implications. In this context, Small Reasoning Language Models (SRLMs) with approximately 0.5 billion parameters present a compelling alternative due to their remarkable computational efficiency and cost effectiveness, particularly in resource-constrained environments. Despite these advantages, the limited capacity of 0.5 billion parameter models poses challenges in handling complex tasks such as mathematical reasoning and code generation. This research investigates various training strategies, including supervised fine-tuning (SFT), knowledge distillation (KD), and reinforcement learning (RL), as well as their hybrid implementations, to enhance the performance of 0.5B SRLMs. We analyze effective methodologies to bridge the performance gap between SRLMS and larger models and present insights into optimal training pipelines tailored for these smaller architectures. Through extensive experimental validation and analysis, our work aims to provide actionable recommendations for maximizing the reasoning capabilities of 0.5B models.</p> - https://arxiv.org/abs/2506.13404 - Mon, 16 Jun 2025 12:18:11 +0000 + VisText-Mosquito: A Multimodal Dataset and Benchmark for AI-Based Mosquito Breeding Site Detection and Reasoning + https://arxiv.org/abs/2506.14629 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14629.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Md. Adnanul Islam, Md. Faiyaz Abdullah Sayeedi, Md. Asaduzzaman Shuvo, Muhammad Ziaur Rahman, Shahanur Rahman Bappy, Raiyan Rahman, Swakkhar Shatabda</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Mosquito-borne diseases pose a major global health risk, requiring early detection and proactive control of breeding sites to prevent outbreaks. In this paper, we present VisText-Mosquito, a multimodal dataset that integrates visual and textual data to support automated detection, segmentation, and reasoning for mosquito breeding site analysis. The dataset includes 1,828 annotated images for object detection, 142 images for water surface segmentation, and natural language reasoning texts linked to each image. The YOLOv9s model achieves the highest precision of 0.92926 and mAP@50 of 0.92891 for object detection, while YOLOv11n-Seg reaches a segmentation precision of 0.91587 and mAP@50 of 0.79795. For reasoning generation, our fine-tuned BLIP model achieves a final loss of 0.0028, with a BLEU score of 54.7, BERTScore of 0.91, and ROUGE-L of 0.87. This dataset and model framework emphasize the theme "Prevention is Better than Cure", showcasing how AI-based detection can proactively address mosquito-borne disease risks. The dataset and implementation code are publicly available at GitHub: https://github.com/adnanul-islam-jisun/VisText-Mosquito</p> + https://arxiv.org/abs/2506.14629 + Tue, 17 Jun 2025 15:24:30 +0000 - Uncertainty-Aware Remaining Lifespan Prediction from Images - https://arxiv.org/abs/2506.13430 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13430.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tristan Kenneweg, Philip Kenneweg, Barbara Hammer</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Predicting mortality-related outcomes from images offers the prospect of accessible, noninvasive, and scalable health screening. We present a method that leverages pretrained vision transformer foundation models to estimate remaining lifespan from facial and whole-body images, alongside robust uncertainty quantification. We show that predictive uncertainty varies systematically with the true remaining lifespan, and that this uncertainty can be effectively modeled by learning a Gaussian distribution for each sample. Our approach achieves state-of-the-art mean absolute error (MAE) of 7.48 years on an established Dataset, and further improves to 4.79 and 5.07 years MAE on two new, higher-quality datasets curated and published in this work. Importantly, our models provide well-calibrated uncertainty estimates, as demonstrated by a bucketed expected calibration error of 0.62 years. While not intended for clinical deployment, these results highlight the potential of extracting medically relevant signals from images. We make all code and datasets available to facilitate further research.</p> - https://arxiv.org/abs/2506.13430 - Mon, 16 Jun 2025 12:47:37 +0000 + Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers + https://arxiv.org/abs/2506.14702 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14702.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daniel D'souza, Julia Kreutzer, Adrien Morisot, Ahmet Üstün, Sara Hooker</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> One of the most profound challenges of modern machine learning is performing well on the long-tail of rare and underrepresented features. Large general-purpose models are trained for many tasks, but work best on high-frequency use cases. After training, it is hard to adapt a model to perform well on specific use cases underrepresented in the training corpus. Relying on prompt engineering or few-shot examples to maximize the output quality on a particular test case can be frustrating, as models can be highly sensitive to small changes, react in unpredicted ways or rely on a fixed system prompt for maintaining performance. In this work, we ask: "Can we optimize our training protocols to both improve controllability and performance on underrepresented use cases at inference time?" We revisit the divide between training and inference techniques to improve long-tail performance while providing users with a set of control levers the model is trained to be responsive to. We create a detailed taxonomy of data characteristics and task provenance to explicitly control generation attributes and implicitly condition generations at inference time. We fine-tune a base model to infer these markers automatically, which makes them optional at inference time. This principled and flexible approach yields pronounced improvements in performance, especially on examples from the long tail of the training distribution. While we observe an average lift of 5.7% win rates in open-ended generation quality with our markers, we see over 9.1% gains in underrepresented domains. We also observe relative lifts of up to 14.1% on underrepresented tasks like CodeRepair and absolute improvements of 35.3% on length instruction following evaluations.</p> + https://arxiv.org/abs/2506.14702 + Tue, 17 Jun 2025 16:40:42 +0000 - BOW: Bottlenecked Next Word Exploration - https://arxiv.org/abs/2506.13502 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13502.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ming Shen, Zhikun Xu, Xiao Ye, Jacob Dineen, Ben Zhou</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large language models (LLMs) are typically trained via next-word prediction (NWP), which provides strong surface-level fluency but often lacks support for robust reasoning. We propose BOttlenecked next Word exploration (BOW), a novel RL framework that rethinks NWP by introducing a reasoning bottleneck where a policy model first generates a reasoning path rather than predicting the next token directly, after which a frozen judge model predicts the next token distribution based solely on this reasoning path. We train the policy model using GRPO with rewards that quantify how effectively the reasoning path facilitates next-word recovery. Compared with other continual pretraining baselines, we show that BOW improves both the general and next-word reasoning capabilities of the base model, evaluated on various benchmarks. Our findings show that BOW can serve as an effective and scalable alternative to vanilla NWP.</p> - https://arxiv.org/abs/2506.13502 - Mon, 16 Jun 2025 13:58:54 +0000 + Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs + https://arxiv.org/abs/2506.14731 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14731.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ring Team, Bin Hu, Cai Chen, Deng Zhao, Ding Liu, Dingnan Jin, Feng Zhu, Hao Dai, Hongzhi Luan, Jia Guo, Jiaming Liu, Jiewei Wu, Jun Mei, Jun Zhou, Junbo Zhao, Junwu Xiong, Kaihong Zhang, Kuan Xu, Lei Liang, Liang Jiang, Liangcheng Fu, Longfei Zheng, Qiang Gao, Qing Cui, Quan Wan, Shaomian Zheng, Shuaicheng Li, Tongkai Yang, Wang Ren, Xiaodong Yan, Xiaopei Wan, Xiaoyun Feng, Xin Zhao, Xinxing Yang, Xinyu Kong, Xuemin Yang, Yang Li, Yingting Wu, Yongkang Liu, Zhankai Xu, Zhenduo Zhang, Zhenglei Zhou, Zhenyu Huang, Zhiqiang Zhang, Zihao Wang, Zujie Wen</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> We present Ring-lite, a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL) to achieve efficient and robust reasoning capabilities. Built upon the publicly available Ling-lite model, a 16.8 billion parameter model with 2.75 billion activated parameters, our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks (e.g., AIME, LiveCodeBench, GPQA-Diamond) while activating only one-third of the parameters required by comparable models. To accomplish this, we introduce a joint training pipeline integrating distillation with RL, revealing undocumented challenges in MoE RL training. First, we identify optimization instability during RL training, and we propose Constrained Contextual Computation Policy Optimization(C3PO), a novel approach that enhances training stability and improves computational throughput via algorithm-system co-design methodology. Second, we empirically demonstrate that selecting distillation checkpoints based on entropy loss for RL training, rather than validation metrics, yields superior performance-efficiency trade-offs in subsequent RL training. Finally, we develop a two-stage training paradigm to harmonize multi-domain data integration, addressing domain conflicts that arise in training with mixed dataset. We will release the model, dataset, and code.</p> + https://arxiv.org/abs/2506.14731 + Tue, 17 Jun 2025 17:12:34 +0000 - MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention - https://arxiv.org/abs/2506.13585 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13585.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> MiniMax, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng, Huaiguang Cai, Haichao Zhu, Jian Sun, Jiaqi Zhuang, Jiaren Cai, Jiayuan Song, Jin Zhu, Jingyang Li, Jinhao Tian, Jinli Liu, Junhao Xu, Junjie Yan, Junteng Liu, Junxian He, Kaiyi Feng, Ke Yang, Kecheng Xiao, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Li, Lin Zheng, Linge Du, Lingyu Yang, Lunbin Zeng, Minghui Yu, Mingliang Tao, Mingyuan Chi, Mozhi Zhang, Mujie Lin, Nan Hu, Nongyu Di, Peng Gao, Pengfei Li, Pengyu Zhao, Qibing Ren, Qidi Xu, Qile Li, Qin Wang, Rong Tian, Ruitao Leng, Shaoxiang Chen, Shaoyu Chen, Shengmin Shi, Shitong Weng, Shuchang Guan, Shuqi Yu, Sichen Li, Songquan Zhu, Tengfei Li, Tianchi Cai, Tianrun Liang, Weiyu Cheng, Weize Kong, Wenkai Li, Xiancai Chen, Xiangjun Song, Xiao Luo, Xiao Su, Xiaobo Li, Xiaodong Han, Xinzhu Hou, Xuan Lu, Xun Zou, Xuyang Shen, Yan Gong, Yan Ma, Yang Wang, Yiqi Shi, Yiran Zhong, Yonghong Duan, Yongxiang Fu, Yongyi Hu, Yu Gao, Yuanxiang Fan, Yufeng Yang, Yuhao Li, Yulin Hu, Yunan Huang, Yunji Li, Yunzhi Xu, Yuxin Mao, Yuxuan Shi, Yuze Wenren, Zehan Li, Zelin Li, Zhanxu Tian, Zhengmao Zhu, Zhenhua Fan, Zhenzhen Wu, Zhichao Xu, Zhihang Yu, Zhiheng Lyu, Zhuo Jiang, Zibo Gao, Zijia Wu, Zijian Song, Zijun Sun</p><p><b>Upvotes:</b> 184</p><p><b>Summary:</b> We introduce MiniMax-M1, the world's first open-weight, large-scale hybrid-attention reasoning model. MiniMax-M1 is powered by a hybrid Mixture-of-Experts (MoE) architecture combined with a lightning attention mechanism. The model is developed based on our previous MiniMax-Text-01 model, which contains a total of 456 billion parameters with 45.9 billion parameters activated per token. The M1 model natively supports a context length of 1 million tokens, 8x the context size of DeepSeek R1. Furthermore, the lightning attention mechanism in MiniMax-M1 enables efficient scaling of test-time compute. These properties make M1 particularly suitable for complex tasks that require processing long inputs and thinking extensively. MiniMax-M1 is trained using large-scale reinforcement learning (RL) on diverse problems including sandbox-based, real-world software engineering environments. In addition to M1's inherent efficiency advantage for RL training, we propose CISPO, a novel RL algorithm to further enhance RL efficiency. CISPO clips importance sampling weights rather than token updates, outperforming other competitive RL variants. Combining hybrid-attention and CISPO enables MiniMax-M1's full RL training on 512 H800 GPUs to complete in only three weeks, with a rental cost of just $534,700. We release two versions of MiniMax-M1 models with 40K and 80K thinking budgets respectively, where the 40K model represents an intermediate phase of the 80K training. Experiments on standard benchmarks show that our models are comparable or superior to strong open-weight models such as the original DeepSeek-R1 and Qwen3-235B, with particular strengths in complex software engineering, tool utilization, and long-context tasks. We publicly release MiniMax-M1 at https://github.com/MiniMax-AI/MiniMax-M1.</p> - https://arxiv.org/abs/2506.13585 - Mon, 16 Jun 2025 15:08:02 +0000 + Optimizing Length Compression in Large Reasoning Models + https://arxiv.org/abs/2506.14755 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14755.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhengxiang Cheng, Dongping Chen, Mingyang Fu, Tianyi Zhou</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Large Reasoning Models (LRMs) have achieved remarkable success, yet they often suffer from producing unnecessary and verbose reasoning chains. We identify a core aspect of this issue as "invalid thinking" -- models tend to repeatedly double-check their work after having derived the correct answer. To address this specific inefficiency, we move beyond the general principles of Efficacy and Efficiency to propose two new, fine-grained principles: Brevity, which advocates for eliminating redundancy, and Sufficiency, which ensures critical reasoning steps are preserved. Guided by these principles, we introduce LC-R1, a post-training method based on Group Relative Policy Optimization (GRPO). LC-R1 employs a novel combination of a Length Reward for overall conciseness and a Compress Reward that is specifically designed to remove the invalid portion of the thinking process. Extensive experiments on multiple reasoning benchmarks demonstrate that LC-R1 achieves a significant reduction in sequence length (~50%) with only a marginal (~2%) drop in accuracy, achieving a favorable trade-off point on the Pareto frontier that prioritizes high compression. Our analysis further validates the robustness of LC-R1 and provides valuable insights for developing more powerful yet computationally efficient LRMs. Our code is released at https://github.com/zxiangx/LC-R1.</p> + https://arxiv.org/abs/2506.14755 + Tue, 17 Jun 2025 17:50:16 +0000 - Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning - https://arxiv.org/abs/2506.13654 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13654.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shulin Tian, Ruiqi Wang, Hongming Guo, Penghao Wu, Yuhao Dong, Xiuying Wang, Jingkang Yang, Hao Zhang, Hongyuan Zhu, Ziwei Liu</p><p><b>Upvotes:</b> 32</p><p><b>Summary:</b> We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., in days and weeks) egocentric videos, which leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, CoTT decomposes complex reasoning into modular steps, with the RL agent invoking specific tools, one per step, to iteratively and collaboratively answer sub-questions tackling such tasks as temporal retrieval and multi-modal understanding. We design a two-stage training paradigm involving supervised finetuning (SFT) of a pretrained language model using CoTT data and RL to enable our agent to dynamically propose step-by-step tools for long-range reasoning. To facilitate training, we construct a dataset called Ego-R1 Data, which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark, Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources. Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of understanding ultra-long egocentric videos, significantly extending the time coverage from few hours to a week.</p> - https://arxiv.org/abs/2506.13654 - Mon, 16 Jun 2025 16:17:08 +0000 + Reasoning with Exploration: An Entropy Perspective + https://arxiv.org/abs/2506.14758 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14758.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, Furu Wei</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Balancing exploration and exploitation is a central goal in reinforcement learning (RL). Despite recent advances in enhancing language model (LM) reasoning, most methods lean toward exploitation, and increasingly encounter performance plateaus. In this work, we revisit entropy -- a signal of exploration in RL -- and examine its relationship to exploratory reasoning in LMs. Through empirical analysis, we uncover strong positive correlations between high-entropy regions and three types of exploratory reasoning actions: (1) pivotal tokens that determine or connect logical steps, (2) reflective actions such as self-verification and correction, and (3) rare behaviors under-explored by the base LMs. Motivated by this, we introduce a minimal modification to standard RL with only one line of code: augmenting the advantage function with an entropy-based term. Unlike traditional maximum-entropy methods which encourage exploration by promoting uncertainty, we encourage exploration by promoting longer and deeper reasoning chains. Notably, our method achieves significant gains on the Pass@K metric -- an upper-bound estimator of LM reasoning capabilities -- even when evaluated with extremely large K values, pushing the boundaries of LM reasoning.</p> + https://arxiv.org/abs/2506.14758 + Tue, 17 Jun 2025 17:54:03 +0000 - Test3R: Learning to Reconstruct 3D at Test Time - https://arxiv.org/abs/2506.13750 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13750.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuheng Yuan, Qiuhong Shen, Shizun Wang, Xingyi Yang, Xinchao Wang</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Dense matching methods like DUSt3R regress pairwise pointmaps for 3D reconstruction. However, the reliance on pairwise prediction and the limited generalization capability inherently restrict the global geometric consistency. In this work, we introduce Test3R, a surprisingly simple test-time learning technique that significantly boosts geometric accuracy. Using image triplets (I_1,I_2,I_3), Test3R generates reconstructions from pairs (I_1,I_2) and (I_1,I_3). The core idea is to optimize the network at test time via a self-supervised objective: maximizing the geometric consistency between these two reconstructions relative to the common image I_1. This ensures the model produces cross-pair consistent outputs, regardless of the inputs. Extensive experiments demonstrate that our technique significantly outperforms previous state-of-the-art methods on the 3D reconstruction and multi-view depth estimation tasks. Moreover, it is universally applicable and nearly cost-free, making it easily applied to other models and implemented with minimal test-time training overhead and parameter footprint. Code is available at https://github.com/nopQAQ/Test3R.</p> - https://arxiv.org/abs/2506.13750 - Mon, 16 Jun 2025 17:56:22 +0000 - - - Steering LLM Thinking with Budget Guidance - https://arxiv.org/abs/2506.13752 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13752.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junyan Li, Wenshuo Zhao, Yang Zhang, Chuang Gan</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent deep-thinking large language models often reason extensively to improve performance, but such lengthy reasoning is not always desirable, as it incurs excessive inference costs with disproportionate performance gains. Controlling reasoning length without sacrificing performance is therefore important, but remains challenging, especially under tight thinking budgets. We propose budget guidance, a simple yet effective method for steering the reasoning process of LLMs toward a target budget without requiring any LLM fine-tuning. Our approach introduces a lightweight predictor that models a Gamma distribution over the remaining thinking length during next-token generation. This signal is then used to guide generation in a soft, token-level manner, ensuring that the overall reasoning trace adheres to the specified thinking budget. Budget guidance enables natural control of the thinking length, along with significant token efficiency improvements over baseline methods on challenging math benchmarks. For instance, it achieves up to a 26% accuracy gain on the MATH-500 benchmark under tight budgets compared to baseline methods, while maintaining competitive accuracy with only 63% of the thinking tokens used by the full-thinking model. Budget guidance also generalizes to broader task domains and exhibits emergent capabilities, such as estimating question difficulty. The source code is available at: https://github.com/UMass-Embodied-AGI/BudgetGuidance.</p> - https://arxiv.org/abs/2506.13752 - Mon, 16 Jun 2025 17:57:05 +0000 - - - Discrete Diffusion in Large Language and Multimodal Models: A Survey - https://arxiv.org/abs/2506.13759 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.13759.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Runpeng Yu, Qi Li, Xinchao Wang</p><p><b>Upvotes:</b> 22</p><p><b>Summary:</b> In this work, we provide a systematic survey of Discrete Diffusion Language Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs). Unlike autoregressive (AR) models, dLLMs and dMLLMs adopt a multi-token, parallel decoding paradigm using full attention and a denoising-based generation strategy. This paradigm naturally enables parallel generation, fine-grained output controllability, and dynamic, response-aware perception. These capabilities are previously difficult to achieve with AR models. Recently, a growing number of industrial-scale proprietary d(M)LLMs, as well as a large number of open-source academic d(M)LLMs, have demonstrated performance comparable to their autoregressive counterparts, while achieving up to 10x acceleration in inference speed. The advancement of discrete diffusion LLMs and MLLMs has been largely driven by progress in two domains. The first is the development of autoregressive LLMs and MLLMs, which has accumulated vast amounts of data, benchmarks, and foundational infrastructure for training and inference. The second contributing domain is the evolution of the mathematical models underlying discrete diffusion. Together, these advancements have catalyzed a surge in dLLMs and dMLLMs research in early 2025. In this work, we present a comprehensive overview of the research in the dLLM and dMLLM domains. We trace the historical development of dLLMs and dMLLMs, formalize the underlying mathematical frameworks, and categorize representative models. We further analyze key techniques for training and inference, and summarize emerging applications across language, vision-language, and biological domains. We conclude by discussing future directions for research and deployment. Paper collection: https://github.com/LiQiiiii/DLLM-Survey</p> - https://arxiv.org/abs/2506.13759 - Mon, 16 Jun 2025 17:59:08 +0000 + From Bytes to Ideas: Language Modeling with Autoregressive U-Nets + https://arxiv.org/abs/2506.14761 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.14761.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite, Marc Schoenauer, Olivier Teytaud, David Lopez-Paz</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build a static vocabulary, and leave the model stuck with that choice. We relax this rigidity by introducing an autoregressive U-Net that learns to embed its own tokens as it trains. The network reads raw bytes, pools them into words, then pairs of words, then up to 4 words, giving it a multi-scale view of the sequence. At deeper stages, the model must predict further into the future -- anticipating the next few words rather than the next byte -- so deeper stages focus on broader semantic patterns while earlier stages handle fine details. When carefully tuning and controlling pretraining compute, shallow hierarchies tie strong BPE baselines, and deeper hierarchies have a promising trend. Because tokenization now lives inside the model, the same system can handle character-level tasks and carry knowledge across low-resource languages.</p> + https://arxiv.org/abs/2506.14761 + Tue, 17 Jun 2025 17:55:11 +0000