bot: update RSS feed
This commit is contained in:
github-actions[bot]
400
feed.xml
400
feed.xml
@@ -7,167 +7,335 @@
|
||||
<docs>http://www.rssboard.org/rss-specification</docs>
|
||||
<generator>python-feedgen</generator>
|
||||
<language>en</language>
|
||||
<lastBuildDate>Tue, 10 Jun 2025 00:02:34 +0000</lastBuildDate>
|
||||
<lastBuildDate>Wed, 11 Jun 2025 00:02:30 +0000</lastBuildDate>
|
||||
<item>
|
||||
<title>Sparsified State-Space Models are Efficient Highway Networks</title>
|
||||
<link>https://arxiv.org/abs/2505.20698</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.20698.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Woomin Song, Jihoon Tack, Sangwoo Mo, Seunghyuk Oh, Jinwoo Shin</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> State-space models (SSMs) offer a promising architecture for sequence modeling, providing an alternative to Transformers by replacing expensive self-attention with linear recurrences. In this paper, we propose a simple yet effective trick to enhance SSMs within given computational budgets by sparsifying them. Our intuition is that tokens in SSMs are highly redundant due to gradual recurrent updates, and dense recurrence operations block the delivery of past information. In particular, we observe that upper layers of SSMs tend to be more redundant as they encode global information, while lower layers encode local information. Motivated by this, we introduce Simba, a hierarchical sparsification method for SSMs based on token pruning. Simba sparsifies upper layers more than lower layers, encouraging the upper layers to behave like highways. To achieve this, we propose a novel token pruning criterion for SSMs, measuring the global impact of tokens on the final output by accumulating local recurrences. We demonstrate that Simba outperforms the baseline model, Mamba, with the same FLOPS in various natural language tasks. Moreover, we illustrate the effect of highways, showing that Simba not only enhances efficiency but also improves the information flow across long sequences. Code is available at https://github.com/woominsong/Simba.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.20698</guid>
|
||||
<pubDate>Tue, 27 May 2025 04:07:23 +0000</pubDate>
|
||||
<title>EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions</title>
|
||||
<link>https://arxiv.org/abs/2505.23473</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23473.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xiaorui Wu, Xiaofeng Mao, Xin Zhang, Fei Li, Chong Teng, Yuxiang Peng, Li Zheng, Donghong Ji, Zhuang Li</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large language models (LLMs) frequently refuse to respond to pseudo-malicious instructions: semantically harmless input queries triggering unnecessary LLM refusals due to conservative safety alignment, significantly impairing user experience. Collecting such instructions is crucial for evaluating and mitigating over-refusals, but existing instruction curation methods, like manual creation or instruction rewriting, either lack scalability or fail to produce sufficiently diverse and effective refusal-inducing prompts. To address these limitations, we introduce EVOREFUSE, a prompt optimization approach that generates diverse pseudo-malicious instructions consistently eliciting confident refusals across LLMs. EVOREFUSE employs an evolutionary algorithm exploring the instruction space in more diverse directions than existing methods via mutation strategies and recombination, and iteratively evolves seed instructions to maximize evidence lower bound on LLM refusal probability. Using EVOREFUSE, we create two novel datasets: EVOREFUSE-TEST, a benchmark of 582 pseudo-malicious instructions that outperforms the next-best benchmark with 140.41% higher average refusal triggering rate across 9 LLMs, 34.86% greater lexical diversity, and 40.03% improved LLM response confidence scores; and EVOREFUSE-ALIGN, which provides 3,000 pseudo-malicious instructions with responses for supervised and preference-based alignment training. LLAMA3.1-8B-INSTRUCT supervisedly fine-tuned on EVOREFUSE-ALIGN achieves up to 14.31% fewer over-refusals than models trained on the second-best alignment dataset, without compromising safety. Our analysis with EVOREFUSE-TEST reveals models trigger over-refusals by overly focusing on sensitive keywords while ignoring broader context.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23473</guid>
|
||||
<pubDate>Thu, 29 May 2025 14:26:46 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA</title>
|
||||
<link>https://arxiv.org/abs/2505.21115</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21115.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sergey Pletenev, Maria Marina, Nikolay Ivanov, Daria Galimzianova, Nikita Krayko, Mikhail Salnikov, Vasily Konovalov, Alexander Panchenko, Viktor Moskvoretskii</p><p><b>Upvotes:</b> 83</p><p><b>Summary:</b> Large Language Models (LLMs) often hallucinate in question answering (QA) tasks. A key yet underexplored factor contributing to this is the temporality of questions -- whether they are evergreen (answers remain stable over time) or mutable (answers change). In this work, we introduce EverGreenQA, the first multilingual QA dataset with evergreen labels, supporting both evaluation and training. Using EverGreenQA, we benchmark 12 modern LLMs to assess whether they encode question temporality explicitly (via verbalized judgments) or implicitly (via uncertainty signals). We also train EG-E5, a lightweight multilingual classifier that achieves SoTA performance on this task. Finally, we demonstrate the practical utility of evergreen classification across three applications: improving self-knowledge estimation, filtering QA datasets, and explaining GPT-4o retrieval behavior.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.21115</guid>
|
||||
<pubDate>Tue, 27 May 2025 12:35:13 +0000</pubDate>
|
||||
<title>Model Immunization from a Condition Number Perspective</title>
|
||||
<link>https://arxiv.org/abs/2505.23760</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.23760.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Amber Yijia Zheng, Cedar Site Bai, Brian Bullins, Raymond A. Yeh</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Model immunization aims to pre-train models that are difficult to fine-tune on harmful tasks while retaining their utility on other non-harmful tasks. Though prior work has shown empirical evidence for immunizing text-to-image models, the key understanding of when immunization is possible and a precise definition of an immunized model remain unclear. In this work, we propose a framework, based on the condition number of a Hessian matrix, to analyze model immunization for linear models. Building on this framework, we design an algorithm with regularization terms to control the resulting condition numbers after pre-training. Empirical results on linear models and non-linear deep-nets demonstrate the effectiveness of the proposed algorithm on model immunization. The code is available at https://github.com/amberyzheng/model-immunization-cond-num.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2505.23760</guid>
|
||||
<pubDate>Thu, 29 May 2025 17:59:48 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>GuideX: Guided Synthetic Data Generation for Zero-Shot Information Extraction</title>
|
||||
<link>https://arxiv.org/abs/2506.00649</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.00649.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Neil De La Fuente, Oscar Sainz, Iker García-Ferrero, Eneko Agirre</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Information Extraction (IE) systems are traditionally domain-specific, requiring costly adaptation that involves expert schema design, data annotation, and model training. While Large Language Models have shown promise in zero-shot IE, performance degrades significantly in unseen domains where label definitions differ. This paper introduces GUIDEX, a novel method that automatically defines domain-specific schemas, infers guidelines, and generates synthetically labeled instances, allowing for better out-of-domain generalization. Fine-tuning Llama 3.1 with GUIDEX sets a new state-of-the-art across seven zeroshot Named Entity Recognition benchmarks. Models trained with GUIDEX gain up to 7 F1 points over previous methods without humanlabeled data, and nearly 2 F1 points higher when combined with it. Models trained on GUIDEX demonstrate enhanced comprehension of complex, domain-specific annotation schemas. Code, models, and synthetic datasets are available at neilus03.github.io/guidex.com</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.00649</guid>
|
||||
<pubDate>Sat, 31 May 2025 17:36:18 +0000</pubDate>
|
||||
<title>Hidden in Plain Sight: Probing Implicit Reasoning in Multimodal Language Models</title>
|
||||
<link>https://arxiv.org/abs/2506.00258</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.00258.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qianqi Yan, Hongquan Li, Shan Jiang, Yang Zhao, Xinze Guan, Ching-Chen Kuo, Xin Eric Wang</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Multimodal large language models (MLLMs) are increasingly deployed in open-ended, real-world environments where inputs are messy, underspecified, and not always trustworthy. Unlike curated benchmarks, these settings frequently involve instructions that refer to missing objects or contradictory facts, rely on ambiguous references, or request infeasible actions. In such cases, success hinges not on task execution alone, but on a model's ability to detect when something is silently wrong. This paper presents a systematic analysis of how current MLLMs handle such implicit reasoning scenarios: cases where the flaw is not explicitly stated but must be inferred from context. Using a curated diagnostic suite spanning four categories of real-world failure modes, we evaluate six MLLMs, including o3 and GPT-4o, and find that models frequently fail to surface hidden issues, even when they possess the necessary perceptual and reasoning skills. Explicit prompting reveals that the underlying capabilities exist but are often suppressed in favor of user compliance. We further show that simple inference-time interventions, such as cautious persona prompting and, in particular, requiring a clarifying question, can dramatically recover performance. Our findings highlight a persistent gap between reasoning competence and behavioral compliance in current MLLMs and suggest practical strategies for making these models more trustworthy in underconstrained environments.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.00258</guid>
|
||||
<pubDate>Fri, 30 May 2025 21:47:28 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilization</title>
|
||||
<link>https://arxiv.org/abs/2506.04255</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04255.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kunal Pai, Parth Shah, Harshil Patel</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Rapid Large Language Model (LLM) advancements are fueling autonomous Multi-Agent System (MAS) development. However, current frameworks often lack flexibility, resource awareness, model diversity, and autonomous tool creation. This paper introduces HASHIRU (Hierarchical Agent System for Hybrid Intelligent Resource Utilization), a novel MAS framework enhancing flexibility, resource efficiency, and adaptability. HASHIRU features a "CEO" agent dynamically managing specialized "employee" agents, instantiated based on task needs and resource constraints (cost, memory). Its hybrid intelligence prioritizes smaller, local LLMs (via Ollama) while flexibly using external APIs and larger models when necessary. An economic model with hiring/firing costs promotes team stability and efficient resource allocation. The system also includes autonomous API tool creation and a memory function. Evaluations on tasks like academic paper review (58% success), safety assessments (100% on a JailbreakBench subset), and complex reasoning (outperforming Gemini 2.0 Flash on GSM8K: 96% vs. 61%; JEEBench: 80% vs. 68.3%; SVAMP: 92% vs. 84%) demonstrate HASHIRU's capabilities. Case studies illustrate its self-improvement via autonomous cost model generation, tool integration, and budget management. HASHIRU offers a promising approach for more robust, efficient, and adaptable MAS through dynamic hierarchical control, resource-aware hybrid intelligence, and autonomous functional extension. Source code and benchmarks are available at https://github.com/HASHIRU-AI/HASHIRU and https://github.com/HASHIRU-AI/HASHIRUBench respectively, and a live demo is available at https://hashiruagentx-hashiruai.hf.space upon request.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.04255</guid>
|
||||
<pubDate>Sun, 01 Jun 2025 17:33:16 +0000</pubDate>
|
||||
<title>ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists</title>
|
||||
<link>https://arxiv.org/abs/2506.01241</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.01241.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang, Jasmine Gump, Tessa Bialek, Vivek Sankaran, Margo Schlanger, Lu Wang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> This paper introduces ExpertLongBench, an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications. Beyond question answering, the application-driven tasks in ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and strict adherence to domain-specific requirements. Notably, each task in ExpertLongBench includes a rubric, designed or validated by domain experts, to specify task requirements and guide output evaluation. Furthermore, we propose CLEAR, an evaluation framework that supports accurate evaluation of long-form model outputs in our benchmark. To achieve fine-grained, expert-aligned evaluation, CLEAR derives checklists from both model outputs and references by extracting information corresponding to items in the task-specific rubric. Checklist items for model outputs are then compared with corresponding items for reference outputs to assess their correctness, enabling grounded evaluation. We benchmark 11 large language models (LLMs) and analyze components in CLEAR, showing that (1) existing LLMs, with the top performer achieving only a 26.8% F1 score, require significant improvement for expert-level tasks; (2) models can generate content corresponding to the required aspects, though often not accurately; and (3) accurate checklist extraction and comparison in CLEAR can be achieved by open-weight models for more scalable and low-cost usage.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.01241</guid>
|
||||
<pubDate>Mon, 02 Jun 2025 01:39:02 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion</title>
|
||||
<link>https://arxiv.org/abs/2506.01111</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.01111.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang</p><p><b>Upvotes:</b> 25</p><p><b>Summary:</b> High-quality, large-scale audio captioning is crucial for advancing audio understanding, yet current automated methods often generate captions that lack fine-grained detail and contextual accuracy, primarily due to their reliance on limited unimodal or superficial multimodal information. Drawing inspiration from human auditory perception, which adeptly integrates cross-modal cues and performs sophisticated auditory scene analysis, we introduce a novel two-stage automated pipeline. This pipeline first employs specialized pretrained models to extract diverse contextual cues (e.g., speech, music, general sounds, and visual information from associated video). A large language model (LLM) then synthesizes these rich, multimodal inputs to generate detailed and context-aware audio captions. Key contributions of this work include: (1) the proposed scalable method for fine-grained audio caption generation; (2) FusionAudio, a new large-scale dataset comprising 1.2 million such detailed captions, combined with 6 million QA pairs; and (3) enhanced audio models developed using FusionAudio, specifically a CLAP-based audio encoder with superior audio-text alignment and instruction following. This paper paves the way for more nuanced and accurate automated understanding of complex audio environments. Code and data can be found in https://github.com/satsuki2486441738/FusionAudio.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.01111</guid>
|
||||
<pubDate>Sun, 01 Jun 2025 18:29:17 +0000</pubDate>
|
||||
<title>NetPress: Dynamically Generated LLM Benchmarks for Network Applications</title>
|
||||
<link>https://arxiv.org/abs/2506.03231</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.03231.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yajie Zhou, Jiajun Ruan, Eric S. Wang, Sadjad Fouladi, Francis Y. Yan, Kevin Hsieh, Zaoxing Liu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Despite growing interest in domain-specific benchmarking of large language models (LLMs) and agents, current evaluations remain limited to static, small-scale datasets, especially in high-stakes tasks like network operations that demand reliability for deployments. We present NetPress, an automated benchmark generation framework for evaluating LLM agents in network applications. NetPress introduces a unified abstraction with state and action, enabling dynamic generation of diverse query sets along with corresponding ground truths. At runtime, users can specify benchmark configurations to generate millions of queries on the fly. In addition to dynamic benchmark construction, NetPress integrates with network emulators to provide realistic environment feedback, supporting comprehensive evaluation across correctness, safety, and latency. We instantiate NetPress on three representative applications, revealing interesting fine-grained differences in agent behavior that static, correctness-only benchmarks often miss. NetPress moves LLM evaluation toward realistic, scalable testing in infrastructure-centric domains, helping close the gap between benchmark performance and real-world deployment readiness. Code is available at https://github.com/Froot-NetSys/NetPress.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.03231</guid>
|
||||
<pubDate>Tue, 03 Jun 2025 14:04:22 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Is Extending Modality The Right Path Towards Omni-Modality?</title>
|
||||
<link>https://arxiv.org/abs/2506.01872</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.01872.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tinghui Zhu, Kai Zhang, Muhao Chen, Yu Su</p><p><b>Upvotes:</b> 16</p><p><b>Summary:</b> Omni-modal language models (OLMs) aim to integrate and reason over diverse input modalities--such as text, images, video, and audio--while maintaining strong language capabilities. Despite recent advancements, existing models, especially open-source ones, remain far from true omni-modality, struggling to generalize beyond the specific modality pairs they are trained on or to achieve strong performance when processing multi-modal inputs. We study the effect of extending modality, the dominant technique for training multimodal models, where an off-the-shelf language model is fine-tuned on target-domain and language data. Specifically, we investigate three key questions: (1) Does modality extension compromise core language abilities? (2) Can model merging effectively integrate independently fine-tuned modality-specific models to achieve omni-modality? (3) Does omni-modality extension lead to better knowledge sharing and generalization compared to sequential extension? Through extensive experiments, we analyze these trade-offs and provide insights into the feasibility of achieving true omni-modality using current approaches.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.01872</guid>
|
||||
<pubDate>Mon, 02 Jun 2025 17:01:40 +0000</pubDate>
|
||||
<title>Robust Preference Optimization via Dynamic Target Margins</title>
|
||||
<link>https://arxiv.org/abs/2506.03690</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.03690.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jie Sun, Junkang Wu, Jiancan Wu, Zhibo Zhu, Xingyu Lu, Jun Zhou, Lintao Ma, Xiang Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> The alignment of Large Language Models (LLMs) is crucial for ensuring their safety and reliability in practical applications. Direct Preference Optimization (DPO) has emerged as an efficient method that directly optimizes models using preference pairs, significantly reducing resource demands. However, the effectiveness of DPO heavily depends on the data quality, which is frequently compromised by noise. In this work, we propose gamma-PO, a dynamic target margin preference optimization algorithm that adjust reward margins at the pairwise level. By introducing instance-specific margin calibration, gamma-PO strategically prioritizes high-confidence pairs (those demonstrating higher reward margins) while suppressing potential noise from ambiguous pairs. Moreover, gamma-PO is a plug-and-play method, compatible with variants of DPO that rely on reward margin between preference pairs. Across benchmarks such as AlpacaEval2 and Arena-Hard, gamma-PO achieves an average 4.4\% improvement over other baselines, setting new benchmarks for state-of-the-art performance. Additionally, gamma-PO requires minimal code changes and has a negligible impact on training efficiency, making it a robust solution for enhancing LLMs alignment. Our codes are available at https://github.com/sunjie279/gammaPO{https://github.com/sunjie279/gammaPO}.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.03690</guid>
|
||||
<pubDate>Wed, 04 Jun 2025 08:19:37 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance</title>
|
||||
<link>https://arxiv.org/abs/2506.03828</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.03828.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Dhaval Patel, Shuxin Lin, James Rayfield, Nianjun Zhou, Roman Vaculin, Natalia Martinez, Fearghal O'donncha, Jayant Kalagnanam</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> AI for Industrial Asset Lifecycle Management aims to automate complex operational workflows -- such as condition monitoring, maintenance planning, and intervention scheduling -- to reduce human workload and minimize system downtime. Traditional AI/ML approaches have primarily tackled these problems in isolation, solving narrow tasks within the broader operational pipeline. In contrast, the emergence of AI agents and large language models (LLMs) introduces a next-generation opportunity: enabling end-to-end automation across the entire asset lifecycle. This paper envisions a future where AI agents autonomously manage tasks that previously required distinct expertise and manual coordination. To this end, we introduce AssetOpsBench -- a unified framework and environment designed to guide the development, orchestration, and evaluation of domain-specific agents tailored for Industry 4.0 applications. We outline the key requirements for such holistic systems and provide actionable insights into building agents that integrate perception, reasoning, and control for real-world industrial operations. The software is available at https://github.com/IBM/AssetOpsBench.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.03828</guid>
|
||||
<pubDate>Wed, 04 Jun 2025 10:57:35 +0000</pubDate>
|
||||
<title>Agents of Change: Self-Evolving LLM Agents for Strategic Planning</title>
|
||||
<link>https://arxiv.org/abs/2506.04651</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04651.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nikolas Belle, Dakota Barnes, Alfonso Amayuelas, Ivan Bercovich, Xin Eric Wang, William Wang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Recent advances in LLMs have enabled their use as autonomous agents across a range of tasks, yet they continue to struggle with formulating and adhering to coherent long-term strategies. In this paper, we investigate whether LLM agents can self-improve when placed in environments that explicitly challenge their strategic planning abilities. Using the board game Settlers of Catan, accessed through the open-source Catanatron framework, we benchmark a progression of LLM-based agents, from a simple game-playing agent to systems capable of autonomously rewriting their own prompts and their player agent's code. We introduce a multi-agent architecture in which specialized roles (Analyzer, Researcher, Coder, and Player) collaborate to iteratively analyze gameplay, research new strategies, and modify the agent's logic or prompt. By comparing manually crafted agents to those evolved entirely by LLMs, we evaluate how effectively these systems can diagnose failure and adapt over time. Our results show that self-evolving agents, particularly when powered by models like Claude 3.7 and GPT-4o, outperform static baselines by autonomously adopting their strategies, passing along sample behavior to game-playing agents, and demonstrating adaptive reasoning over multiple iterations.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.04651</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 05:45:24 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data</title>
|
||||
<link>https://arxiv.org/abs/2506.04120</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04120.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ben Moran, Mauro Comi, Steven Bohez, Tom Erez, Zhibin Li, Leonard Hasenclever</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Creating accurate, physical simulations directly from real-world robot motion holds great value for safe, scalable, and affordable robot learning, yet remains exceptionally challenging. Real robot data suffers from occlusions, noisy camera poses, dynamic scene elements, which hinder the creation of geometrically accurate and photorealistic digital twins of unseen objects. We introduce a novel real-to-sim framework tackling all these challenges at once. Our key insight is a hybrid scene representation merging the photorealistic rendering of 3D Gaussian Splatting with explicit object meshes suitable for physics simulation within a single representation. We propose an end-to-end optimization pipeline that leverages differentiable rendering and differentiable physics within MuJoCo to jointly refine all scene components - from object geometry and appearance to robot poses and physical parameters - directly from raw and imprecise robot trajectories. This unified optimization allows us to simultaneously achieve high-fidelity object mesh reconstruction, generate photorealistic novel views, and perform annotation-free robot pose calibration. We demonstrate the effectiveness of our approach both in simulation and on challenging real-world sequences using an ALOHA 2 bi-manual manipulator, enabling more practical and robust real-to-simulation pipelines.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.04120</guid>
|
||||
<pubDate>Wed, 04 Jun 2025 16:14:31 +0000</pubDate>
|
||||
<title>MegaHan97K: A Large-Scale Dataset for Mega-Category Chinese Character Recognition with over 97K Categories</title>
|
||||
<link>https://arxiv.org/abs/2506.04807</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04807.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuyi Zhang, Yongxin Shi, Peirong Zhang, Yixin Zhao, Zhenhua Yang, Lianwen Jin</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Foundational to the Chinese language and culture, Chinese characters encompass extraordinarily extensive and ever-expanding categories, with the latest Chinese GB18030-2022 standard containing 87,887 categories. The accurate recognition of this vast number of characters, termed mega-category recognition, presents a formidable yet crucial challenge for cultural heritage preservation and digital applications. Despite significant advances in Optical Character Recognition (OCR), mega-category recognition remains unexplored due to the absence of comprehensive datasets, with the largest existing dataset containing merely 16,151 categories. To bridge this critical gap, we introduce MegaHan97K, a mega-category, large-scale dataset covering an unprecedented 97,455 categories of Chinese characters. Our work offers three major contributions: (1) MegaHan97K is the first dataset to fully support the latest GB18030-2022 standard, providing at least six times more categories than existing datasets; (2) It effectively addresses the long-tail distribution problem by providing balanced samples across all categories through its three distinct subsets: handwritten, historical and synthetic subsets; (3) Comprehensive benchmarking experiments reveal new challenges in mega-category scenarios, including increased storage demands, morphologically similar character recognition, and zero-shot learning difficulties, while also unlocking substantial opportunities for future research. To the best of our knowledge, the MetaHan97K is likely the dataset with the largest classes not only in the field of OCR but may also in the broader domain of pattern recognition. The dataset is available at https://github.com/SCUT-DLVCLab/MegaHan97K.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.04807</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 09:33:06 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2506.04755</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.04755.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shenshen Li, Kaiyuan Deng, Lei Wang, Hao Yang, Chong Peng, Peng Yan, Fumin Shen, Heng Tao Shen, Xing Xu</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> While multi-modal large language models (MLLMs) have made significant progress in complex reasoning tasks via reinforcement learning, it is commonly believed that extensive training data is necessary for improving multi-modal reasoning ability, inevitably leading to data redundancy and substantial computational costs. However, can smaller high-value datasets match or outperform full corpora for multi-modal reasoning in MLLMs? In this work, we challenge this assumption through a key observation: meaningful multi-modal reasoning is triggered by only a sparse subset of training samples, termed cognitive samples, whereas the majority contribute marginally. Building on this insight, we propose a novel data selection paradigm termed Reasoning Activation Potential (RAP), which identifies cognitive samples by estimating each sample's potential to stimulate genuine multi-modal reasoning by two complementary estimators: 1) Causal Discrepancy Estimator (CDE) based on the potential outcome model principle, eliminates samples that overly rely on language priors by comparing outputs between multi-modal and text-only inputs; 2) Attention Confidence Estimator (ACE), which exploits token-level self-attention to discard samples dominated by irrelevant but over-emphasized tokens in intermediate reasoning stages. Moreover, we introduce a Difficulty-aware Replacement Module (DRM) to substitute trivial instances with cognitively challenging ones, thereby ensuring complexity for robust multi-modal reasoning. Experiments on six datasets show that our RAP method consistently achieves superior performance using only 9.3% of the training data, while reducing computational costs by over 43%. Our code is available at https://github.com/Leo-ssl/RAP.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.04755</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 08:40:24 +0000</pubDate>
|
||||
<title>Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation</title>
|
||||
<link>https://arxiv.org/abs/2506.05062</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05062.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Noy Sternlicht, Ariel Gera, Roy Bar-Haim, Tom Hope, Noam Slonim</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> We introduce Debate Speech Evaluation as a novel and challenging benchmark for assessing LLM judges. Evaluating debate speeches requires a deep understanding of the speech at multiple levels, including argument strength and relevance, the coherence and organization of the speech, the appropriateness of its style and tone, and so on. This task involves a unique set of cognitive abilities that have previously received limited attention in systematic LLM benchmarking. To explore such skills, we leverage a dataset of over 600 meticulously annotated debate speeches and present the first in-depth analysis of how state-of-the-art LLMs compare to human judges on this task. Our findings reveal a nuanced picture: while larger models can approximate individual human judgments in some respects, they differ substantially in their overall judgment behavior. We also investigate the ability of frontier LLMs to generate persuasive, opinionated speeches, showing that models may perform at a human level on this task.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05062</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 14:06:51 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Prefix Grouper: Efficient GRPO Training through Shared-Prefix Forward</title>
|
||||
<link>https://arxiv.org/abs/2506.05433</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05433.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zikang Liu, Tongtian Yue, Yepeng Tang, Longteng Guo, Junxian Cai, Qingbin Liu, Xi Chen, Jing Liu</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Group Relative Policy Optimization (GRPO) enhances policy learning by computing gradients from relative comparisons among candidate outputs that share a common input prefix. Despite its effectiveness, GRPO introduces substantial computational overhead when processing long shared prefixes, which must be redundantly encoded for each group member. This inefficiency becomes a major scalability bottleneck in long-context learning scenarios. We propose Prefix Grouper, an efficient GRPO training algorithm that eliminates redundant prefix computation via a Shared-Prefix Forward strategy. In particular, by restructuring self-attention into two parts, our method enables the shared prefix to be encoded only once, while preserving full differentiability and compatibility with end-to-end training. We provide both theoretical and empirical evidence that Prefix Grouper is training-equivalent to standard GRPO: it yields identical forward outputs and backward gradients, ensuring that the optimization dynamics and final policy performance remain unchanged. Empirically, our experiments confirm that Prefix Grouper achieves consistent results while significantly reducing the computational cost of training, particularly in long-prefix scenarios. The proposed method is fully plug-and-play: it is compatible with existing GRPO-based architectures and can be seamlessly integrated into current training pipelines as a drop-in replacement, requiring no structural modifications and only minimal changes to input construction and attention computation. Prefix Grouper enables the use of larger group sizes under the same computational budget, thereby improving the scalability of GRPO to more complex tasks and larger models. Code is now available at https://github.com/johncaged/PrefixGrouper</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05433</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 09:13:37 +0000</pubDate>
|
||||
<title>SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs</title>
|
||||
<link>https://arxiv.org/abs/2506.05598</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05598.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Michael J Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, Diyi Yang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Recent calls for pluralistic alignment of Large Language Models (LLMs) encourage adapting models to diverse user preferences. However, most prior work on personalized reward models heavily rely on additional identity information, such as demographic details or a predefined set of preference categories. To this end, we introduce SynthesizeMe, an approach to inducing synthetic user personas from user interactions for personalized reward modeling. SynthesizeMe first generates and verifies reasoning to explain user preferences, then induces synthetic user personas from that reasoning, and finally filters to informative prior user interactions in order to build personalized prompts for a particular user. We show that using SynthesizeMe induced prompts improves personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining SynthesizeMe derived prompts with a reward model achieves top performance on PersonalRewardBench: a new curation of user-stratified interactions with chatbots collected from 854 users of Chatbot Arena and PRISM.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05598</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 21:23:16 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Sentinel: SOTA model to protect against prompt injections</title>
|
||||
<link>https://arxiv.org/abs/2506.05446</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05446.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Dror Ivry, Oran Nahum</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Large Language Models (LLMs) are increasingly powerful but remain vulnerable to prompt injection attacks, where malicious inputs cause the model to deviate from its intended instructions. This paper introduces Sentinel, a novel detection model, qualifire/prompt-injection-sentinel, based on the \answerdotai/ModernBERT-large architecture. By leveraging ModernBERT's advanced features and fine-tuning on an extensive and diverse dataset comprising a few open-source and private collections, Sentinel achieves state-of-the-art performance. This dataset amalgamates varied attack types, from role-playing and instruction hijacking to attempts to generate biased content, alongside a broad spectrum of benign instructions, with private datasets specifically targeting nuanced error correction and real-world misclassifications. On a comprehensive, unseen internal test set, Sentinel demonstrates an average accuracy of 0.987 and an F1-score of 0.980. Furthermore, when evaluated on public benchmarks, it consistently outperforms strong baselines like protectai/deberta-v3-base-prompt-injection-v2. This work details Sentinel's architecture, its meticulous dataset curation, its training methodology, and a thorough evaluation, highlighting its superior detection capabilities.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05446</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 14:07:15 +0000</pubDate>
|
||||
<title>Proactive Assistant Dialogue Generation from Streaming Egocentric Videos</title>
|
||||
<link>https://arxiv.org/abs/2506.05904</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05904.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, Seungwhan Moon</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactive, proactive assistance based on streaming visual inputs, yet their development is constrained by the costly and labor-intensive process of data collection and system evaluation. To address these limitations, we present a comprehensive framework with three key contributions. First, we introduce a novel data curation pipeline that synthesizes dialogues from annotated egocentric videos, resulting in \dataset, a large-scale synthetic dialogue dataset spanning multiple domains. Second, we develop a suite of automatic evaluation metrics, validated through extensive human studies. Third, we propose an end-to-end model that processes streaming video inputs to generate contextually appropriate responses, incorporating novel techniques for handling data imbalance and long-duration videos. This work lays the foundation for developing real-time, proactive AI assistants capable of guiding users through diverse tasks. Project page: https://pro-assist.github.io/</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05904</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 09:23:29 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2506.05523</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05523.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zikui Cai, Andrew Wang, Anirudh Satheesh, Ankit Nakhawa, Hyunwoo Jae, Keenan Powell, Minghui Liu, Neel Jay, Sungbin Oh, Xiyao Wang, Yongyuan Liang, Tom Goldstein, Furong Huang</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> Despite rapid advances in vision-language models (VLMs), current benchmarks for multimodal reasoning fall short in three key dimensions. First, they overwhelmingly rely on static images, failing to capture the temporal complexity of real-world environments. Second, they narrowly focus on mathematical problem-solving, neglecting the broader spectrum of reasoning skills -- including abstract, physical, planning, spatial, and temporal capabilities -- required for robust multimodal intelligence. Third, many benchmarks quickly saturate, offering limited headroom for diagnosing failure modes or measuring continued progress. We introduce MORSE-500 (Multimodal Reasoning Stress-test Environment), a video benchmark composed of 500 fully scripted clips with embedded questions spanning six complementary reasoning categories. Each instance is programmatically generated using deterministic Python scripts (via Manim, Matplotlib, MoviePy), generative video models, and curated real footage. This script-driven design allows fine-grained control over visual complexity, distractor density, and temporal dynamics -- enabling difficulty to be scaled systematically as models improve. Unlike static benchmarks that become obsolete once saturated, MORSE-500 is built to evolve: its controllable generation pipeline supports the creation of arbitrarily challenging new instances, making it ideally suited for stress-testing next-generation models. Initial experiments with state-of-the-art systems -- including various Gemini 2.5 Pro and OpenAI o3 which represent the strongest available at the time, alongside strong open-source models -- reveal substantial performance gaps across all categories, with particularly large deficits in abstract and planning tasks. We release the full dataset, generation scripts, and evaluation harness to support transparent, reproducible, and forward-looking multimodal reasoning research.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05523</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 19:12:45 +0000</pubDate>
|
||||
<title>Bootstrapping World Models from Dynamics Models in Multimodal Foundation Models</title>
|
||||
<link>https://arxiv.org/abs/2506.06006</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06006.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. Ponti</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> To what extent do vision-and-language foundation models possess a realistic world model (observation times action rightarrow observation) and a dynamics model (observation times observation rightarrow action), when actions are expressed through language? While open-source foundation models struggle with both, we find that fine-tuning them to acquire a dynamics model through supervision is significantly easier than acquiring a world model. In turn, dynamics models can be used to bootstrap world models through two main strategies: 1) weakly supervised learning from synthetic data and 2) inference time verification. Firstly, the dynamics model can annotate actions for unlabelled pairs of video frame observations to expand the training data. We further propose a new objective, where image tokens in observation pairs are weighted by their importance, as predicted by a recognition model. Secondly, the dynamics models can assign rewards to multiple samples of the world model to score them, effectively guiding search at inference time. We evaluate the world models resulting from both strategies through the task of action-centric image editing on Aurora-Bench. Our best model achieves a performance competitive with state-of-the-art image editing models, improving on them by a margin of 15% on real-world subsets according to GPT4o-as-judge, and achieving the best average human evaluation across all subsets of Aurora-Bench.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06006</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 11:50:18 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding</title>
|
||||
<link>https://arxiv.org/abs/2506.05551</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05551.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yan Shu, Hangui Lin, Yexin Liu, Yan Zhang, Gangyan Zeng, Yan Li, Yu Zhou, Ser-Nam Lim, Harry Yang, Nicu Sebe</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Large Multimodal Models (LMMs) have achieved impressive progress in visual perception and reasoning. However, when confronted with visually ambiguous or non-semantic scene text, they often struggle to accurately spot and understand the content, frequently generating semantically plausible yet visually incorrect answers, which we refer to as semantic hallucination. In this work, we investigate the underlying causes of semantic hallucination and identify a key finding: Transformer layers in LLM with stronger attention focus on scene text regions are less prone to producing semantic hallucinations. Thus, we propose a training-free semantic hallucination mitigation framework comprising two key components: (1) ZoomText, a coarse-to-fine strategy that identifies potential text regions without external detectors; and (2) Grounded Layer Correction, which adaptively leverages the internal representations from layers less prone to hallucination to guide decoding, correcting hallucinated outputs for non-semantic samples while preserving the semantics of meaningful ones. To enable rigorous evaluation, we introduce TextHalu-Bench, a benchmark of over 1,730 samples spanning both semantic and non-semantic cases, with manually curated question-answer pairs designed to probe model hallucinations. Extensive experiments demonstrate that our method not only effectively mitigates semantic hallucination but also achieves strong performance on public benchmarks for scene text spotting and understanding.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05551</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 19:53:19 +0000</pubDate>
|
||||
<title>Astra: Toward General-Purpose Mobile Robots via Hierarchical Multimodal Learning</title>
|
||||
<link>https://arxiv.org/abs/2506.06205</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06205.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sheng Chen, Peiyu He, Jiaxin Hu, Ziyang Liu, Yansheng Wang, Tao Xu, Chi Zhang, Chongchong Zhang, Chao An, Shiyu Cai, Duo Cao, Kangping Chen, Shuai Chu, Tianwei Chu, Mingdi Dan, Min Du, Weiwei Fang, Pengyou Fu, Junkai Hu, Xiaowei Jiang, Zhaodi Jiang, Fuxuan Li, Jun Li, Minghui Li, Mingyao Li, Yanchang Li, Zhibin Li, Guangming Liu, Kairui Liu, Lihao Liu, Weizhi Liu, Xiaoshun Liu, Yufei Liu, Yunfei Liu, Qiang Lu, Yuanfei Luo, Xiang Lv, Hongying Ma, Sai Ma, Lingxian Mi, Sha Sa, Hongxiang Shu, Lei Tian, Chengzhi Wang, Jiayu Wang, Kaijie Wang, Qingyi Wang, Renwen Wang, Tao Wang, Wei Wang, Xirui Wang, Chao Wei, Xuguang Wei, Zijun Xia, Zhaohao Xiao, Tingshuai Yan, Liyan Yang, Yifan Yang, Zhikai Yang, Zhong Yin, Li Yuan, Liuchun Yuan, Chi Zhang, Jinyang Zhang, Junhui Zhang, Linge Zhang, Zhenyi Zhang, Zheyu Zhang, Dongjie Zhu, Hang Li, Yangang Zhang</p><p><b>Upvotes:</b> 20</p><p><b>Summary:</b> Modern robot navigation systems encounter difficulties in diverse and complex indoor environments. Traditional approaches rely on multiple modules with small models or rule-based systems and thus lack adaptability to new environments. To address this, we developed Astra, a comprehensive dual-model architecture, Astra-Global and Astra-Local, for mobile robot navigation. Astra-Global, a multimodal LLM, processes vision and language inputs to perform self and goal localization using a hybrid topological-semantic graph as the global map, and outperforms traditional visual place recognition methods. Astra-Local, a multitask network, handles local path planning and odometry estimation. Its 4D spatial-temporal encoder, trained through self-supervised learning, generates robust 4D features for downstream tasks. The planning head utilizes flow matching and a novel masked ESDF loss to minimize collision risks for generating local trajectories, and the odometry head integrates multi-sensor inputs via a transformer encoder to predict the relative pose of the robot. Deployed on real in-house mobile robots, Astra achieves high end-to-end mission success rate across diverse indoor environments.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06205</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 16:08:47 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers</title>
|
||||
<link>https://arxiv.org/abs/2506.05573</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05573.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuchen Lin, Chenguo Lin, Panwang Pan, Honglei Yan, Yiqiang Feng, Yadong Mu, Katerina Fragkiadaki</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> We introduce PartCrafter, the first structured 3D generative model that jointly synthesizes multiple semantically meaningful and geometrically distinct 3D meshes from a single RGB image. Unlike existing methods that either produce monolithic 3D shapes or follow two-stage pipelines, i.e., first segmenting an image and then reconstructing each segment, PartCrafter adopts a unified, compositional generation architecture that does not rely on pre-segmented inputs. Conditioned on a single image, it simultaneously denoises multiple 3D parts, enabling end-to-end part-aware generation of both individual objects and complex multi-object scenes. PartCrafter builds upon a pretrained 3D mesh diffusion transformer (DiT) trained on whole objects, inheriting the pretrained weights, encoder, and decoder, and introduces two key innovations: (1) A compositional latent space, where each 3D part is represented by a set of disentangled latent tokens; (2) A hierarchical attention mechanism that enables structured information flow both within individual parts and across all parts, ensuring global coherence while preserving part-level detail during generation. To support part-level supervision, we curate a new dataset by mining part-level annotations from large-scale 3D object datasets. Experiments show that PartCrafter outperforms existing approaches in generating decomposable 3D meshes, including parts that are not directly visible in input images, demonstrating the strength of part-aware generative priors for 3D understanding and synthesis. Code and training data will be released.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05573</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 20:30:28 +0000</pubDate>
|
||||
<title>Cartridges: Lightweight and general-purpose long context representations via self-study</title>
|
||||
<link>https://arxiv.org/abs/2506.06266</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06266.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Re</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Large language models are often used to answer queries grounded in large text corpora (e.g. codebases, legal documents, or chat histories) by placing the entire corpus in the context window and leveraging in-context learning (ICL). Although current models support contexts of 100K-1M tokens, this setup is costly to serve because the memory consumption of the KV cache scales with input length. We explore an alternative: training a smaller KV cache offline on each corpus. At inference time, we load this trained KV cache, which we call a Cartridge, and decode a response. Critically, the cost of training a Cartridge can be amortized across all the queries referencing the same corpus. However, we find that the naive approach of training the Cartridge with next-token prediction on the corpus is not competitive with ICL. Instead, we propose self-study, a training recipe in which we generate synthetic conversations about the corpus and train the Cartridge with a context-distillation objective. We find that Cartridges trained with self-study replicate the functionality of ICL, while being significantly cheaper to serve. On challenging long-context benchmarks, Cartridges trained with self-study match ICL performance while using 38.6x less memory and enabling 26.4x higher throughput. Self-study also extends the model's effective context length (e.g. from 128k to 484k tokens on MTOB) and surprisingly, leads to Cartridges that can be composed at inference time without retraining.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06266</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 17:48:23 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration</title>
|
||||
<link>https://arxiv.org/abs/2506.05579</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05579.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Quan Shi, Carlos E. Jimenez, Shunyu Yao, Nick Haber, Diyi Yang, Karthik Narasimhan</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent advancements in AI reasoning have driven substantial improvements across diverse tasks. A critical open question is whether these improvements also yields better knowledge transfer: the ability of models to communicate reasoning in ways humans can understand, apply, and learn from. To investigate this, we introduce Knowledge Integration and Transfer Evaluation (KITE), a conceptual and experimental framework for Human-AI knowledge transfer capabilities and conduct the first large-scale human study (N=118) explicitly designed to measure it. In our two-phase setup, humans first ideate with an AI on problem-solving strategies, then independently implement solutions, isolating model explanations' influence on human understanding. Our findings reveal that although model benchmark performance correlates with collaborative outcomes, this relationship is notably inconsistent, featuring significant outliers, indicating that knowledge transfer requires dedicated optimization. Our analysis identifies behavioral and strategic factors mediating successful knowledge transfer. We release our code, dataset, and evaluation framework to support future work on communicatively aligned models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05579</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 20:48:16 +0000</pubDate>
|
||||
<title>Saffron-1: Towards an Inference Scaling Paradigm for LLM Safety Assurance</title>
|
||||
<link>https://arxiv.org/abs/2506.06444</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06444.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ruizhong Qiu, Gaotang Li, Tianxin Wei, Jingrui He, Hanghang Tong</p><p><b>Upvotes:</b> 60</p><p><b>Summary:</b> Existing safety assurance research has primarily focused on training-phase alignment to instill safe behaviors into LLMs. However, recent studies have exposed these methods' susceptibility to diverse jailbreak attacks. Concurrently, inference scaling has significantly advanced LLM reasoning capabilities but remains unexplored in the context of safety assurance. Addressing this gap, our work pioneers inference scaling for robust and effective LLM safety against emerging threats. We reveal that conventional inference scaling techniques, despite their success in reasoning tasks, perform poorly in safety contexts, even falling short of basic approaches like Best-of-N Sampling. We attribute this inefficiency to a newly identified challenge, the exploration--efficiency dilemma, arising from the high computational overhead associated with frequent process reward model (PRM) evaluations. To overcome this dilemma, we propose SAFFRON, a novel inference scaling paradigm tailored explicitly for safety assurance. Central to our approach is the introduction of a multifurcation reward model (MRM) that significantly reduces the required number of reward model evaluations. To operationalize this paradigm, we further propose: (i) a partial supervision training objective for MRM, (ii) a conservative exploration constraint to prevent out-of-distribution explorations, and (iii) a Trie-based key--value caching strategy that facilitates cache sharing across sequences during tree search. Extensive experiments validate the effectiveness of our method. Additionally, we publicly release our trained multifurcation reward model (Saffron-1) and the accompanying token-level safety reward dataset (Safety4M) to accelerate future research in LLM safety. Our code, model, and data are publicly available at https://github.com/q-rz/saffron , and our project homepage is at https://q-rz.github.io/p/saffron .</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06444</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 18:05:45 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs</title>
|
||||
<link>https://arxiv.org/abs/2506.05629</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05629.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ananth Muppidi, Abhilash Nandy, Sambaran Bandyopadhyay</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> The performance of large language models in domain-specific tasks necessitates fine-tuning, which is computationally expensive and technically challenging. This paper focuses on parameter-efficient fine-tuning using soft prompting, a promising approach that adapts pre-trained models to downstream tasks by learning a small set of parameters. We propose a novel Input Dependent Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that generates soft prompts based on the input tokens and attends different tokens with varying importance. Our method is simple and efficient, keeping the number of trainable parameters small. We show the merits of the proposed approach compared to state-of-the-art techniques on various tasks and show the improved zero shot domain transfer capability.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05629</guid>
|
||||
<pubDate>Thu, 05 Jun 2025 23:13:22 +0000</pubDate>
|
||||
<title>What Is Seen Cannot Be Unseen: The Disruptive Effect of Knowledge Conflict on Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2506.06485</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06485.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kaiser Sun, Fan Bai, Mark Dredze</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Large language models frequently rely on both contextual input and parametric knowledge to perform tasks. However, these sources can come into conflict, especially when retrieved documents contradict the model's parametric knowledge. We propose a diagnostic framework to systematically evaluate LLM behavior under context-memory conflict, where the contextual information diverges from their parametric beliefs. We construct diagnostic data that elicit these conflicts and analyze model performance across multiple task types. Our findings reveal that (1) knowledge conflict has minimal impact on tasks that do not require knowledge utilization, (2) model performance is consistently higher when contextual and parametric knowledge are aligned, (3) models are unable to fully suppress their internal knowledge even when instructed, and (4) providing rationales that explain the conflict increases reliance on contexts. These insights raise concerns about the validity of model-based evaluation and underscore the need to account for knowledge conflict in the deployment of LLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06485</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 19:20:23 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Peer-Ranked Precision: Creating a Foundational Dataset for Fine-Tuning Vision Models from DataSeeds' Annotated Imagery</title>
|
||||
<link>https://arxiv.org/abs/2506.05673</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05673.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sajjad Abdoli, Freeman Lewin, Gediminas Vasiliauskas, Fabian Schonholz</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> The development of modern Artificial Intelligence (AI) models, particularly diffusion-based models employed in computer vision and image generation tasks, is undergoing a paradigmatic shift in development methodologies. Traditionally dominated by a "Model Centric" approach, in which performance gains were primarily pursued through increasingly complex model architectures and hyperparameter optimization, the field is now recognizing a more nuanced "Data-Centric" approach. This emergent framework foregrounds the quality, structure, and relevance of training data as the principal driver of model performance. To operationalize this paradigm shift, we introduce the DataSeeds.AI sample dataset (the "DSD"), initially comprised of approximately 10,610 high-quality human peer-ranked photography images accompanied by extensive multi-tier annotations. The DSD is a foundational computer vision dataset designed to usher in a new standard for commercial image datasets. Representing a small fraction of DataSeed.AI's 100 million-plus image catalog, the DSD provides a scalable foundation necessary for robust commercial and multimodal AI development. Through this in-depth exploratory analysis, we document the quantitative improvements generated by the DSD on specific models against known benchmarks and make the code and the trained models used in our evaluation publicly available.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05673</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 01:50:28 +0000</pubDate>
|
||||
<title>Self-Adapting Improvement Loops for Robotic Learning</title>
|
||||
<link>https://arxiv.org/abs/2506.06658</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06658.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Calvin Luo, Zilai Zeng, Mingxi Jia, Yilun Du, Chen Sun</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Video generative models trained on expert demonstrations have been utilized as performant text-conditioned visual planners for solving robotic tasks. However, generalization to unseen tasks remains a challenge. Whereas improved generalization may be facilitated by leveraging learned prior knowledge from additional pre-collected offline data sources, such as web-scale video datasets, in the era of experience we aim to design agents that can continuously improve in an online manner from self-collected behaviors. In this work we thus propose the Self-Adapting Improvement Loop (SAIL), where an in-domain video model iteratively updates itself on self-produced trajectories, collected through adaptation with an internet-scale pretrained video model, and steadily improves its performance for a specified task of interest. We apply SAIL to a diverse suite of MetaWorld tasks, as well as two manipulation tasks on a real robot arm, and find that performance improvements continuously emerge over multiple iterations for novel tasks initially unseen during original in-domain video model training. Furthermore, we discover that SAIL is surprisingly robust regarding if and how the self-collected experience is filtered, and the quality of the initial in-domain demonstrations. Through adaptation with summarized internet-scale data, and learning through online experience, we thus demonstrate a way to iteratively bootstrap a high-performance video model for solving novel robotic tasks through self-improvement.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06658</guid>
|
||||
<pubDate>Sat, 07 Jun 2025 04:34:37 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>CodeContests+: High-Quality Test Case Generation for Competitive Programming</title>
|
||||
<link>https://arxiv.org/abs/2506.05817</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05817.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, Kai Shen</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Competitive programming, due to its high reasoning difficulty and precise correctness feedback, has become a key task for both training and evaluating the reasoning capabilities of large language models (LLMs). However, while a large amount of public problem data, such as problem statements and solutions, is available, the test cases of these problems are often difficult to obtain. Therefore, test case generation is a necessary task for building large-scale datasets, and the quality of the test cases directly determines the accuracy of the evaluation. In this paper, we introduce an LLM-based agent system that creates high-quality test cases for competitive programming problems. We apply this system to the CodeContests dataset and propose a new version with improved test cases, named CodeContests+. We evaluated the quality of test cases in CodeContestsPlus. First, we used 1.72 million submissions with pass/fail labels to examine the accuracy of these test cases in evaluation. The results indicated that CodeContests+ achieves significantly higher accuracy than CodeContests, particularly with a notably higher True Positive Rate (TPR). Subsequently, our experiments in LLM Reinforcement Learning (RL) further confirmed that improvements in test case quality yield considerable advantages for RL.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05817</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 07:29:01 +0000</pubDate>
|
||||
<title>Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering</title>
|
||||
<link>https://arxiv.org/abs/2506.06905</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06905.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Akash Gupta, Amos Storkey, Mirella Lapata</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, is inconsistent and does not always improve monotonically with increasing examples. We hypothesize that this occurs due to the LMM being overwhelmed by additional information present in the image embeddings, which is not required for the downstream task. To address this, we propose a meta-learning approach that provides an alternative for inducing few-shot capabilities in LMMs, using a fixed set of soft prompts that are distilled from task-relevant image features and can be adapted at test time using a few examples. To facilitate this distillation, we introduce an attention-mapper module that can be easily integrated with the popular LLaVA v1.5 architecture and is jointly learned with soft prompts, enabling task adaptation in LMMs under low-data regimes with just a few gradient steps. Evaluation on the VL-ICL Bench shows that our method consistently outperforms ICL and related prompt-tuning approaches, even under image perturbations, improving task induction and reasoning across visual question answering tasks.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06905</guid>
|
||||
<pubDate>Sat, 07 Jun 2025 19:37:22 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Audio-Aware Large Language Models as Judges for Speaking Styles</title>
|
||||
<link>https://arxiv.org/abs/2506.05984</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05984.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Audio-aware large language models (ALLMs) can understand the textual and non-textual information in the audio input. In this paper, we explore using ALLMs as an automatic judge to assess the speaking styles of speeches. We use ALLM judges to evaluate the speeches generated by SLMs on two tasks: voice style instruction following and role-playing. The speaking style we consider includes emotion, volume, speaking pace, word emphasis, pitch control, and non-verbal elements. We use four spoken language models (SLMs) to complete the two tasks and use humans and ALLMs to judge the SLMs' responses. We compare two ALLM judges, GPT-4o-audio and Gemini-2.5-pro, with human evaluation results and show that the agreement between Gemini and human judges is comparable to the agreement between human evaluators. These promising results show that ALLMs can be used as a judge to evaluate SLMs. Our results also reveal that current SLMs, even GPT-4o-audio, still have room for improvement in controlling the speaking style and generating natural dialogues.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.05984</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 11:05:48 +0000</pubDate>
|
||||
<title>The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity</title>
|
||||
<link>https://arxiv.org/abs/2506.06941</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06941.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Recent generations of language models have introduced Large Reasoning Models (LRMs) that generate detailed thinking processes before providing answers. While these models demonstrate improved performance on reasoning benchmarks, their fundamental capabilities, scaling properties, and limitations remain insufficiently understood. Current evaluations primarily focus on established math and coding benchmarks, emphasizing final answer accuracy. However, this evaluation paradigm often suffers from contamination and does not provide insights into the reasoning traces. In this work, we systematically investigate these gaps with the help of controllable puzzle environments that allow precise manipulation of complexity while maintaining consistent logical structures. This setup enables the analysis of not only final answers but also the internal reasoning traces, offering insights into how LRMs think. Through extensive experiments, we show that LRMs face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counterintuitive scaling limit: their reasoning effort increases with problem complexity up to a point, then declines despite having remaining token budget. By comparing LRMs with their standard LLM counterparts under same inference compute, we identify three performance regimes: (1) low-complexity tasks where standard models outperform LRMs, (2) medium-complexity tasks where LRMs demonstrates advantage, and (3) high-complexity tasks where both models face complete collapse. We found that LRMs have limitations in exact computation: they fail to use explicit algorithms and reason inconsistently across scales. We also investigate the reasoning traces in more depth, studying the patterns of explored solutions and analyzing the models' computational behavior, shedding light on their strengths, limitations, and raising questions about their reasoning capabilities.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06941</guid>
|
||||
<pubDate>Sat, 07 Jun 2025 22:42:29 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MIRIAD: Augmenting LLMs with millions of medical query-response pairs</title>
|
||||
<link>https://arxiv.org/abs/2506.06091</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06091.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qinyue Zheng, Salman Abdullah, Sam Rawal, Cyril Zakka, Sophie Ostmeier, Maximilian Purk, Eduardo Reis, Eric J. Topol, Jure Leskovec, Michael Moor</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> LLMs are bound to transform healthcare with advanced decision support and flexible chat assistants. However, LLMs are prone to generate inaccurate medical content. To ground LLMs in high-quality medical knowledge, LLMs have been equipped with external knowledge via RAG, where unstructured medical knowledge is split into small text chunks that can be selectively retrieved and integrated into the LLMs context. Yet, existing RAG pipelines rely on raw, unstructured medical text, which can be noisy, uncurated and difficult for LLMs to effectively leverage. Systematic approaches to organize medical knowledge to best surface it to LLMs are generally lacking. To address these challenges, we introduce MIRIAD, a large-scale, curated corpus of 5,821,948 medical QA pairs, each rephrased from and grounded in a passage from peer-reviewed medical literature using a semi-automated pipeline combining LLM generation, filtering, grounding, and human annotation. Unlike prior medical corpora, which rely on unstructured text, MIRIAD encapsulates web-scale medical knowledge in an operationalized query-response format, which enables more targeted retrieval. Experiments on challenging medical QA benchmarks show that augmenting LLMs with MIRIAD improves accuracy up to 6.7% compared to unstructured RAG baselines with the same source corpus and with the same amount of retrieved text. Moreover, MIRIAD improved the ability of LLMs to detect medical hallucinations by 22.5 to 37% (increase in F1 score). We further introduce MIRIAD-Atlas, an interactive map of MIRIAD spanning 56 medical disciplines, enabling clinical users to visually explore, search, and refine medical knowledge. MIRIAD promises to unlock a wealth of down-stream applications, including medical information retrievers, enhanced RAG applications, and knowledge-grounded chat interfaces, which ultimately enables more reliable LLM applications in healthcare.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06091</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 13:52:32 +0000</pubDate>
|
||||
<title>Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning</title>
|
||||
<link>https://arxiv.org/abs/2506.07044</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07044.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, Yu Rong</p><p><b>Upvotes:</b> 79</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07044</guid>
|
||||
<pubDate>Sun, 08 Jun 2025 08:47:30 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model</title>
|
||||
<link>https://arxiv.org/abs/2506.06199</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06199.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongyan Zhi, Peihao Chen, Siyuan Zhou, Yubo Dong, Quanxi Wu, Lei Han, Mingkui Tan</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Manipulation has long been a challenging task for robots, while humans can effortlessly perform complex interactions with objects, such as hanging a cup on the mug rack. A key reason is the lack of a large and uniform dataset for teaching robots manipulation skills. Current robot datasets often record robot action in different action spaces within a simple scene. This hinders the robot to learn a unified and robust action representation for different robots within diverse scenes. Observing how humans understand a manipulation task, we find that understanding how the objects should move in the 3D space is a critical clue for guiding actions. This clue is embodiment-agnostic and suitable for both humans and different robots. Motivated by this, we aim to learn a 3D flow world model from both human and robot manipulation data. This model predicts the future movement of the interacting objects in 3D space, guiding action planning for manipulation. Specifically, we synthesize a large-scale 3D optical flow dataset, named ManiFlow-110k, through a moving object auto-detect pipeline. A video diffusion-based world model then learns manipulation physics from these data, generating 3D optical flow trajectories conditioned on language instructions. With the generated 3D object optical flow, we propose a flow-guided rendering mechanism, which renders the predicted final state and leverages GPT-4o to assess whether the predicted flow aligns with the task description. This equips the robot with a closed-loop planning ability. Finally, we consider the predicted 3D optical flow as constraints for an optimization policy to determine a chunk of robot actions for manipulation. Extensive experiments demonstrate strong generalization across diverse robotic manipulation tasks and reliable cross-embodiment adaptation without hardware-specific training.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06199</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 16:00:31 +0000</pubDate>
|
||||
<title>GeometryZero: Improving Geometry Solving for LLM with Group Contrastive Policy Optimization</title>
|
||||
<link>https://arxiv.org/abs/2506.07160</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07160.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yikun Wang, Yibin Wang, Dianyi Wang, Zimian Peng, Qipeng Guo, Dacheng Tao, Jiaqi Wang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent advances in large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, particularly in mathematical reasoning, amid which geometry problem solving remains a challenging area where auxiliary construction plays a enssential role. Existing approaches either achieve suboptimal performance or rely on massive LLMs (e.g., GPT-4o), incurring massive computational costs. We posit that reinforcement learning with verifiable reward (e.g., GRPO) offers a promising direction for training smaller models that effectively combine auxiliary construction with robust geometric reasoning. However, directly applying GRPO to geometric reasoning presents fundamental limitations due to its dependence on unconditional rewards, which leads to indiscriminate and counterproductive auxiliary constructions. To address these challenges, we propose Group Contrastive Policy Optimization (GCPO), a novel reinforcement learning framework featuring two key innovations: (1) Group Contrastive Masking, which adaptively provides positive or negative reward signals for auxiliary construction based on contextual utility, and a (2) length reward that promotes longer reasoning chains. Building on GCPO, we develop GeometryZero, a family of affordable-size geometric reasoning models that judiciously determine when to employ auxiliary construction. Our extensive empirical evaluation across popular geometric benchmarks (Geometry3K, MathVista) demonstrates that GeometryZero models consistently outperform baselines (e.g. GRPO), achieving an average improvement of 4.29% across all benchmarks.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07160</guid>
|
||||
<pubDate>Sun, 08 Jun 2025 14:18:15 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision</title>
|
||||
<link>https://arxiv.org/abs/2506.06253</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06253.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuping He, Yifei Huang, Guo Chen, Lidong Lu, Baoqi Pei, Jilan Xu, Tong Lu, Yoichi Sato</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Perceiving the world from both egocentric (first-person) and exocentric (third-person) perspectives is fundamental to human cognition, enabling rich and complementary understanding of dynamic environments. In recent years, allowing the machines to leverage the synergistic potential of these dual perspectives has emerged as a compelling research direction in video understanding. In this survey, we provide a comprehensive review of video understanding from both exocentric and egocentric viewpoints. We begin by highlighting the practical applications of integrating egocentric and exocentric techniques, envisioning their potential collaboration across domains. We then identify key research tasks to realize these applications. Next, we systematically organize and review recent advancements into three main research directions: (1) leveraging egocentric data to enhance exocentric understanding, (2) utilizing exocentric data to improve egocentric analysis, and (3) joint learning frameworks that unify both perspectives. For each direction, we analyze a diverse set of tasks and relevant works. Additionally, we discuss benchmark datasets that support research in both perspectives, evaluating their scope, diversity, and applicability. Finally, we discuss limitations in current works and propose promising future research directions. By synthesizing insights from both perspectives, our goal is to inspire advancements in video understanding and artificial intelligence, bringing machines closer to perceiving the world in a human-like manner. A GitHub repo of related works can be found at https://github.com/ayiyayi/Awesome-Egocentric-and-Exocentric-Vision.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06253</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 17:25:48 +0000</pubDate>
|
||||
<title>Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs</title>
|
||||
<link>https://arxiv.org/abs/2506.07240</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07240.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Roy Eisenstadt, Itamar Zimerman, Lior Wolf</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recently, techniques such as explicit structured reasoning have demonstrated strong test-time scaling behavior by enforcing a separation between the model's internal "thinking" process and the final response. A key factor influencing answer quality in this setting is the length of the thinking stage. When the reasoning is too short, the model may fail to capture the complexity of the task. Conversely, when it is too long, the model may overthink, leading to unnecessary computation and degraded performance. This paper explores and exploits the underlying mechanisms by which LLMs understand and regulate the length of their reasoning during explicit thought processes. First, we show that LLMs encode their progress through the reasoning process and introduce an interactive progress bar visualization, which is then used to reveal insights on the model's planning dynamics. Second, we manipulate the internal progress encoding during inference to reduce unnecessary steps and generate a more concise and decisive chain of thoughts. Our empirical results demonstrate that this "overclocking" method mitigates overthinking, improves answer accuracy, and reduces inference latency. Our code is publicly available.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07240</guid>
|
||||
<pubDate>Sun, 08 Jun 2025 17:54:33 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis</title>
|
||||
<link>https://arxiv.org/abs/2506.06276</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.06276.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiatao Gu, Tianrong Chen, David Berthelot, Huangjie Zheng, Yuyang Wang, Ruixiang Zhang, Laurent Dinh, Miguel Angel Bautista, Josh Susskind, Shuangfei Zhai</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> We present STARFlow, a scalable generative model based on normalizing flows that achieves strong performance in high-resolution image synthesis. The core of STARFlow is Transformer Autoregressive Flow (TARFlow), which combines the expressive power of normalizing flows with the structured modeling capabilities of Autoregressive Transformers. We first establish the theoretical universality of TARFlow for modeling continuous distributions. Building on this foundation, we introduce several key architectural and algorithmic innovations to significantly enhance scalability: (1) a deep-shallow design, wherein a deep Transformer block captures most of the model representational capacity, complemented by a few shallow Transformer blocks that are computationally efficient yet substantially beneficial; (2) modeling in the latent space of pretrained autoencoders, which proves more effective than direct pixel-level modeling; and (3) a novel guidance algorithm that significantly boosts sample quality. Crucially, our model remains an end-to-end normalizing flow, enabling exact maximum likelihood training in continuous spaces without discretization. STARFlow achieves competitive performance in both class-conditional and text-conditional image generation tasks, approaching state-of-the-art diffusion models in sample quality. To our knowledge, this work is the first successful demonstration of normalizing flows operating effectively at this scale and resolution.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.06276</guid>
|
||||
<pubDate>Fri, 06 Jun 2025 17:58:39 +0000</pubDate>
|
||||
<title>Pre-trained Large Language Models Learn Hidden Markov Models In-context</title>
|
||||
<link>https://arxiv.org/abs/2506.07298</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07298.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yijia Dai, Zhaolin Gao, Yahya Satter, Sarah Dean, Jennifer J. Sun</p><p><b>Upvotes:</b> 16</p><p><b>Summary:</b> Hidden Markov Models (HMMs) are foundational tools for modeling sequential data with latent Markovian structure, yet fitting them to real-world data remains computationally challenging. In this work, we show that pre-trained large language models (LLMs) can effectively model data generated by HMMs via in-context learning (ICL)x2013their ability to infer patterns from examples within a prompt. On a diverse set of synthetic HMMs, LLMs achieve predictive accuracy approaching the theoretical optimum. We uncover novel scaling trends influenced by HMM properties, and offer theoretical conjectures for these empirical observations. We also provide practical guidelines for scientists on using ICL as a diagnostic tool for complex data. On real-world animal decision-making tasks, ICL achieves competitive performance with models designed by human experts. To our knowledge, this is the first demonstration that ICL can learn and predict HMM-generated sequencesx2013an advance that deepens our understanding of in-context learning in LLMs and establishes its potential as a powerful tool for uncovering hidden structure in complex scientific data.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07298</guid>
|
||||
<pubDate>Sun, 08 Jun 2025 21:49:38 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>ConfQA: Answer Only If You Are Confident</title>
|
||||
<link>https://arxiv.org/abs/2506.07309</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07309.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yin Huang, Yifan Ethan Xu, Kai Sun, Vera Yan, Alicia Sun, Haidar Khan, Jimmy Nguyen, Mohammad Kachuee, Zhaojiang Lin, Yue Liu, Aaron Colak, Anuj Kumar, Wen-tau Yih, Xin Luna Dong</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Can we teach Large Language Models (LLMs) to refrain from hallucinating factual statements? In this paper we present a fine-tuning strategy that we call ConfQA, which can reduce hallucination rate from 20-40% to under 5% across multiple factuality benchmarks. The core idea is simple: when the LLM answers a question correctly, it is trained to continue with the answer; otherwise, it is trained to admit "I am unsure". But there are two key factors that make the training highly effective. First, we introduce a dampening prompt "answer only if you are confident" to explicitly guide the behavior, without which hallucination remains high as 15%-25%. Second, we leverage simple factual statements, specifically attribute values from knowledge graphs, to help LLMs calibrate the confidence, resulting in robust generalization across domains and question types. Building on this insight, we propose the Dual Neural Knowledge framework, which seamlessly select between internally parameterized neural knowledge and externally recorded symbolic knowledge based on ConfQA's confidence. The framework enables potential accuracy gains to beyond 95%, while reducing unnecessary external retrievals by over 30%.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07309</guid>
|
||||
<pubDate>Sun, 08 Jun 2025 22:51:46 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding</title>
|
||||
<link>https://arxiv.org/abs/2506.07434</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07434.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Feifan Song, Shaohang Wei, Wen Luo, Yuxuan Fan, Tianyu Liu, Guoyin Wang, Houfeng Wang</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Large Language Models (LLMs) require alignment with human preferences to avoid generating offensive, false, or meaningless content. Recently, low-resource methods for LLM alignment have been popular, while still facing challenges in obtaining both high-quality and aligned content. Motivated by the observation that the difficulty of generating aligned responses is concentrated at the beginning of decoding, we propose a novel framework, Weak-to-Strong Decoding (WSD), to enhance the alignment ability of base models by the guidance of a small aligned model. The small model first drafts well-aligned beginnings, followed by the large base model to continue the rest, controlled by a well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign, to fine-tune a small-sized Pilot-3B as the draft model, which effectively enhances different base models under the WSD framework to outperform all baseline methods, while avoiding degradation on downstream tasks, termed as the alignment tax. Extensive experiments are further conducted to examine the impact of different settings and time efficiency, as well as analyses on the intrinsic mechanisms of WSD in depth.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07434</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 05:21:22 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models</title>
|
||||
<link>https://arxiv.org/abs/2506.07463</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07463.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guang Liu, Liangdong Wang, Jijie Li, Yang Yu, Yao Xu, Jiabei Chen, Yu Bai, Feng Liao, Yonghua Lin</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> We introduce CCI4.0, a large-scale bilingual pre-training dataset engineered for superior data quality and diverse human-like reasoning trajectory. CCI4.0 occupies roughly 35 TB of disk space and comprises two sub-datasets: CCI4.0-M2-Base and CCI4.0-M2-CoT. CCI4.0-M2-Base combines a 5.2 TB carefully curated Chinese web corpus, a 22.5 TB English subset from Nemotron-CC, and diverse sources from math, wiki, arxiv, and code. Although these data are mostly sourced from well-processed datasets, the quality standards of various domains are dynamic and require extensive expert experience and labor to process. So, we propose a novel pipeline justifying data quality mainly based on models through two-stage deduplication, multiclassifier quality scoring, and domain-aware fluency filtering. We extract 4.5 billion pieces of CoT(Chain-of-Thought) templates, named CCI4.0-M2-CoT. Differing from the distillation of CoT from larger models, our proposed staged CoT extraction exemplifies diverse reasoning patterns and significantly decreases the possibility of hallucination. Empirical evaluations demonstrate that LLMs pre-trained in CCI4.0 benefit from cleaner, more reliable training signals, yielding consistent improvements in downstream tasks, especially in math and code reflection tasks. Our results underscore the critical role of rigorous data curation and human thinking templates in advancing LLM performance, shedding some light on automatically processing pretraining corpora.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07463</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 06:14:19 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>SpatialLM: Training Large Language Models for Structured Indoor Modeling</title>
|
||||
<link>https://arxiv.org/abs/2506.07491</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07491.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, Zihan Zhou</p><p><b>Upvotes:</b> 29</p><p><b>Summary:</b> SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07491</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 07:10:58 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions</title>
|
||||
<link>https://arxiv.org/abs/2506.07527</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07527.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, Wentao Zhang</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Recent advances in large language model (LLM) reasoning have shown that sophisticated behaviors such as planning and self-reflection can emerge through reinforcement learning (RL). However, despite these successes, RL in its current form remains insufficient to induce capabilities that exceed the limitations of the base model, as it is primarily optimized based on existing knowledge of the model rather than facilitating the acquisition of new information. To address this limitation, we employ supervised fine-tuning (SFT) to learn what RL cannot, which enables the incorporation of new knowledge and reasoning patterns by leveraging high-quality demonstration data. We analyze the training dynamics of RL and SFT for LLM reasoning and find that RL excels at maintaining and improving performance on questions within the model's original capabilities, while SFT is more effective at enabling progress on questions beyond the current scope of the model. Motivated by the complementary strengths of RL and SFT, we introduce a novel training approach, ReLIFT (Reinforcement Learning Interleaved with Online Fine-Tuning). In ReLIFT, the model is primarily trained using RL, but when it encounters challenging questions, high-quality solutions are collected for fine-tuning, and the training process alternates between RL and fine-tuning to enhance the model's reasoning abilities. ReLIFT achieves an average improvement of over +5.2 points across five competition-level benchmarks and one out-of-distribution benchmark compared to other zero-RL models. Furthermore, we demonstrate that ReLIFT outperforms both RL and SFT while using only 13\% of the detailed demonstration data, highlighting its scalability. These results provide compelling evidence that ReLIFT overcomes the fundamental limitations of RL and underscores the significant potential.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07527</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 08:11:20 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation</title>
|
||||
<link>https://arxiv.org/abs/2506.07530</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07530.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongyu Wang, Chuyan Xiong, Ruiping Wang, Xilin Chen</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Vision-Language-Action (VLA) models have shown impressive capabilities across a wide range of robotics manipulation tasks. However, their growing model size poses significant challenges for deployment on resource-constrained robotic systems. While 1-bit pretraining has proven effective for enhancing the inference efficiency of large language models with minimal performance loss, its application to VLA models remains underexplored. In this work, we present BitVLA, the first 1-bit VLA model for robotics manipulation, in which every parameter is ternary, i.e., {-1, 0, 1}. To further reduce the memory footprint of the vision encoder, we propose the distillation-aware training strategy that compresses the full-precision encoder to 1.58-bit weights. During this process, a full-precision encoder serves as a teacher model to better align latent representations. Despite the lack of large-scale robotics pretraining, BitVLA achieves performance comparable to the state-of-the-art model OpenVLA-OFT with 4-bit post-training quantization on the LIBERO benchmark, while consuming only 29.8% of the memory. These results highlight BitVLA's promise for deployment on memory-constrained edge devices. We release the code and model weights in https://github.com/ustcwhy/BitVLA.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07530</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 08:15:11 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition</title>
|
||||
<link>https://arxiv.org/abs/2506.07553</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07553.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingchao Wang, Haote Yang, Jiang Wu, Yifan He, Xingjian Wei, Yinfan Wang, Chengjin Liu, Lingli Ge, Lijun Wu, Bin Wang, Dahua Lin, Conghui He</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Optical Chemical Structure Recognition (OCSR) is crucial for digitizing chemical knowledge by converting molecular images into machine-readable formats. While recent vision-language models (VLMs) have shown potential in this task, their image-captioning approach often struggles with complex molecular structures and inconsistent annotations. To overcome these challenges, we introduce GTR-Mol-VLM, a novel framework featuring two key innovations: (1) the Graph Traversal as Visual Chain of Thought mechanism that emulates human reasoning by incrementally parsing molecular graphs through sequential atom-bond predictions, and (2) the data-centric principle of Faithfully Recognize What You've Seen, which addresses the mismatch between abbreviated structures in images and their expanded annotations. To support model development, we constructed GTR-CoT-1.3M, a large-scale instruction-tuning dataset with meticulously corrected annotations, and introduced MolRec-Bench, the first benchmark designed for a fine-grained evaluation of graph-parsing accuracy in OCSR. Comprehensive experiments demonstrate that GTR-Mol-VLM achieves superior results compared to specialist models, chemistry-domain VLMs, and commercial general-purpose VLMs. Notably, in scenarios involving molecular images with functional group abbreviations, GTR-Mol-VLM outperforms the second-best baseline by approximately 14 percentage points, both in SMILES-based and graph-based metrics. We hope that this work will drive OCSR technology to more effectively meet real-world needs, thereby advancing the fields of cheminformatics and AI for Science. We will release GTR-CoT at https://github.com/opendatalab/GTR-CoT.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07553</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 08:47:10 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems</title>
|
||||
<link>https://arxiv.org/abs/2506.07564</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07564.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Peiran Li, Xinkai Zou, Zhuohang Wu, Ruifeng Li, Shuo Xing, Hanwen Zheng, Zhikai Hu, Yuping Wang, Haoxi Li, Qin Yuan, Yingmo Zhang, Zhengzhong Tu</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Recent advances in large language models (LLMs) and vision-language models (VLMs) have enabled powerful autonomous agents capable of complex reasoning and multi-modal tool use. Despite their growing capabilities, today's agent frameworks remain fragile, lacking principled mechanisms for secure information flow, reliability, and multi-agent coordination. In this work, we introduce SAFEFLOW, a new protocol-level framework for building trustworthy LLM/VLM-based agents. SAFEFLOW enforces fine-grained information flow control (IFC), precisely tracking provenance, integrity, and confidentiality of all the data exchanged between agents, tools, users, and environments. By constraining LLM reasoning to respect these security labels, SAFEFLOW prevents untrusted or adversarial inputs from contaminating high-integrity decisions. To ensure robustness in concurrent multi-agent settings, SAFEFLOW introduces transactional execution, conflict resolution, and secure scheduling over shared state, preserving global consistency across agents. We further introduce mechanisms, including write-ahead logging, rollback, and secure caches, that further enhance resilience against runtime errors and policy violations. To validate the performances, we built SAFEFLOWBENCH, a comprehensive benchmark suite designed to evaluate agent reliability under adversarial, noisy, and concurrent operational conditions. Extensive experiments demonstrate that agents built with SAFEFLOW maintain impressive task performance and security guarantees even in hostile environments, substantially outperforming state-of-the-art. Together, SAFEFLOW and SAFEFLOWBENCH lay the groundwork for principled, robust, and secure agent ecosystems, advancing the frontier of reliable autonomy.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07564</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 09:04:37 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models</title>
|
||||
<link>https://arxiv.org/abs/2506.07645</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07645.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Maciej Chrabąszcz, Katarzyna Lorenc, Karolina Seweryn</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Large language models (LLMs) have demonstrated impressive capabilities across various natural language processing (NLP) tasks in recent years. However, their susceptibility to jailbreaks and perturbations necessitates additional evaluations. Many LLMs are multilingual, but safety-related training data contains mainly high-resource languages like English. This can leave them vulnerable to perturbations in low-resource languages such as Polish. We show how surprisingly strong attacks can be cheaply created by altering just a few characters and using a small proxy model for word importance calculation. We find that these character and word-level attacks drastically alter the predictions of different LLMs, suggesting a potential vulnerability that can be used to circumvent their internal safety mechanisms. We validate our attack construction methodology on Polish, a low-resource language, and find potential vulnerabilities of LLMs in this language. Additionally, we show how it can be extended to other languages. We release the created datasets and code for further research.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07645</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 11:09:39 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Through the Valley: Path to Effective Long CoT Training for Small Language Models</title>
|
||||
<link>https://arxiv.org/abs/2506.07712</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07712.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Renjie Luo, Jiaxi Li, Chen Huang, Wei Lu</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Long chain-of-thought (CoT) supervision has become a common strategy to enhance reasoning in language models. While effective for large models, we identify a phenomenon we call Long CoT Degradation, in which small language models (SLMs; <=3B parameters) trained on limited long CoT data experience significant performance deterioration. Through extensive experiments on the Qwen2.5, LLaMA3 and Gemma3 families, we demonstrate that this degradation is widespread across SLMs. In some settings, models trained on only 8k long CoT examples lose up to 75% of their original performance before fine-tuning. Strikingly, we further observe that for some particularly small models, even training on 220k long CoT examples fails to recover or surpass their original performance prior to fine-tuning. Our analysis attributes this effect to error accumulation: while longer responses increase the capacity for multi-step reasoning, they also amplify the risk of compounding mistakes. Furthermore, we find that Long CoT Degradation may negatively impacts downstream reinforcement learning (RL), although this can be alleviated by sufficiently scaled supervised fine-tuning (SFT). Our findings challenge common assumptions about the benefits of long CoT training for SLMs and offer practical guidance for building more effective small-scale reasoning models.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07712</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 12:56:41 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Image Reconstruction as a Tool for Feature Analysis</title>
|
||||
<link>https://arxiv.org/abs/2506.07803</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07803.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Eduard Allakhverdov, Dmitrii Tarasov, Elizaveta Goncharova, Andrey Kuznetsov</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> Vision encoders are increasingly used in modern applications, from vision-only models to multimodal systems such as vision-language models. Despite their remarkable success, it remains unclear how these architectures represent features internally. Here, we propose a novel approach for interpreting vision features via image reconstruction. We compare two related model families, SigLIP and SigLIP2, which differ only in their training objective, and show that encoders pre-trained on image-based tasks retain significantly more image information than those trained on non-image tasks such as contrastive learning. We further apply our method to a range of vision encoders, ranking them by the informativeness of their feature representations. Finally, we demonstrate that manipulating the feature space yields predictable changes in reconstructed images, revealing that orthogonal rotations (rather than spatial transformations) control color encoding. Our approach can be applied to any vision encoder, shedding light on the inner structure of its feature space. The code and model weights to reproduce the experiments are available in GitHub.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07803</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 14:32:18 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Improving large language models with concept-aware fine-tuning</title>
|
||||
<link>https://arxiv.org/abs/2506.07833</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07833.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Michael K. Chen, Xikun Zhang, Jiaxing Huang, Dacheng Tao</p><p><b>Upvotes:</b> 0</p><p><b>Summary:</b> Large language models (LLMs) have become the cornerstone of modern AI. However, the existing paradigm of next-token prediction fundamentally limits their ability to form coherent, high-level concepts, making it a critical barrier to human-like understanding and reasoning. Take the phrase "ribonucleic acid" as an example: an LLM will first decompose it into tokens, i.e., artificial text fragments ("rib", "on", ...), then learn each token sequentially, rather than grasping the phrase as a unified, coherent semantic entity. This fragmented representation hinders deeper conceptual understanding and, ultimately, the development of truly intelligent systems. In response, we introduce Concept-Aware Fine-Tuning (CAFT), a novel multi-token training method that redefines how LLMs are fine-tuned. By enabling the learning of sequences that span multiple tokens, this method fosters stronger concept-aware learning. Our experiments demonstrate significant improvements compared to conventional next-token finetuning methods across diverse tasks, including traditional applications like text summarization and domain-specific ones like de novo protein design. Multi-token prediction was previously only possible in the prohibitively expensive pretraining phase; CAFT, to our knowledge, is the first to bring the multi-token setting to the post-training phase, thus effectively democratizing its benefits for the broader community of practitioners and researchers. Finally, the unexpected effectiveness of our proposed method suggests wider implications for the machine learning research community. All code and data are available at https://github.com/michaelchen-lab/caft-llm</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07833</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 14:55:00 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement</title>
|
||||
<link>https://arxiv.org/abs/2506.07848</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07848.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Teng Hu, Zhentao Yu, Zhengguang Zhou, Jiangning Zhang, Yuan Zhou, Qinglin Lu, Ran Yi</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Despite recent advances in video generation, existing models still lack fine-grained controllability, especially for multi-subject customization with consistent identity and interaction. In this paper, we propose PolyVivid, a multi-subject video customization framework that enables flexible and identity-consistent generation. To establish accurate correspondences between subject images and textual entities, we design a VLLM-based text-image fusion module that embeds visual identities into the textual space for precise grounding. To further enhance identity preservation and subject interaction, we propose a 3D-RoPE-based enhancement module that enables structured bidirectional fusion between text and image embeddings. Moreover, we develop an attention-inherited identity injection module to effectively inject fused identity features into the video generation process, mitigating identity drift. Finally, we construct an MLLM-based data pipeline that combines MLLM-based grounding, segmentation, and a clique-based subject consolidation strategy to produce high-quality multi-subject data, effectively enhancing subject distinction and reducing ambiguity in downstream video generation. Extensive experiments demonstrate that PolyVivid achieves superior performance in identity fidelity, video realism, and subject alignment, outperforming existing open-source and commercial baselines.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07848</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 15:11:09 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>MiniCPM4: Ultra-Efficient LLMs on End Devices</title>
|
||||
<link>https://arxiv.org/abs/2506.07900</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07900.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Jie Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, Ning Ding, Shengdan Fan, Yewei Fang, Zixuan Fu, Wenyu Guan, Yitong Guan, Junshao Guo, Yufeng Han, Bingxiang He, Yuxiang Huang, Cunliang Kong, Qiuzuo Li, Siyuan Li, Wenhao Li, Yanghao Li, Yishan Li, Zhen Li, Dan Liu, Biyuan Lin, Yankai Lin, Xiang Long, Quanyu Lu, Yaxi Lu, Peiyan Luo, Hongya Lyu, Litu Ou, Yinxu Pan, Zekai Qu, Qundong Shi, Zijun Song, Jiayuan Su, Zhou Su, Ao Sun, Xianghui Sun, Peijun Tang, Fangzheng Wang, Feng Wang, Shuo Wang, Yudong Wang, Yesai Wu, Zhenyu Xiao, Jie Xie, Zihao Xie, Yukun Yan, Jiarui Yuan, Kaihuo Zhang, Lei Zhang, Linyue Zhang, Xueren Zhang, Yudi Zhang, Hengyu Zhao, Weilin Zhao, Weilun Zhao, Yuanqian Zhao, Zhi Zheng, Ge Zhou, Jie Zhou, Wei Zhou, Zihan Zhou, Zixuan Zhou, Zhiyuan Liu, Guoyang Zeng, Chao Jia, Dahai Li, Maosong Sun</p><p><b>Upvotes:</b> 55</p><p><b>Summary:</b> This paper introduces MiniCPM4, a highly efficient large language model (LLM) designed explicitly for end-side devices. We achieve this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems. Specifically, in terms of model architecture, we propose InfLLM v2, a trainable sparse attention mechanism that accelerates both prefilling and decoding phases for long-context processing. Regarding training data, we propose UltraClean, an efficient and accurate pre-training data filtering and generation strategy, and UltraChat v2, a comprehensive supervised fine-tuning dataset. These datasets enable satisfactory model performance to be achieved using just 8 trillion training tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient pre-training strategy search, and improve existing post-training methods by introducing chunk-wise rollout for load-balanced reinforcement learning and data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose CPM.cu that integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding. To meet diverse on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B parameters, respectively. Sufficient evaluation results show that MiniCPM4 outperforms open-source models of similar size across multiple benchmarks, highlighting both its efficiency and effectiveness. Notably, MiniCPM4-8B demonstrates significant speed improvements over Qwen3-8B when processing long sequences. Through further adaptation, MiniCPM4 successfully powers diverse applications, including trustworthy survey generation and tool use with model context protocol, clearly showcasing its broad usability.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07900</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 16:16:50 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>CyberV: Cybernetics for Test-time Scaling in Video Understanding</title>
|
||||
<link>https://arxiv.org/abs/2506.07971</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07971.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiahao Meng, Shuyang Sun, Yue Tan, Lu Qi, Yunhai Tong, Xiangtai Li, Longyin Wen</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Current Multimodal Large Language Models (MLLMs) may struggle with understanding long or complex videos due to computational demands at test time, lack of robustness, and limited accuracy, primarily stemming from their feed-forward processing nature. These limitations could be more severe for models with fewer parameters. To address these limitations, we propose a novel framework inspired by cybernetic principles, redesigning video MLLMs as adaptive systems capable of self-monitoring, self-correction, and dynamic resource allocation during inference. Our approach, CyberV, introduces a cybernetic loop consisting of an MLLM Inference System, a Sensor, and a Controller. Specifically, the sensor monitors forward processes of the MLLM and collects intermediate interpretations, such as attention drift, then the controller determines when and how to trigger self-correction and generate feedback to guide the next round. This test-time adaptive scaling framework enhances frozen MLLMs without requiring retraining or additional components. Experiments demonstrate significant improvements: CyberV boosts Qwen2.5-VL-7B by 8.3% and InternVL3-8B by 5.5% on VideoMMMU, surpassing the competitive proprietary model GPT-4o. When applied to Qwen2.5-VL-72B, it yields a 10.0% improvement, achieving performance even comparable to human experts. Furthermore, our method demonstrates consistent gains on general-purpose benchmarks, such as VideoMME and WorldSense, highlighting its effectiveness and generalization capabilities in making MLLMs more robust and accurate for dynamic video understanding. The code is released at https://github.com/marinero4972/CyberV.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07971</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:45:18 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation</title>
|
||||
<link>https://arxiv.org/abs/2506.07977</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07977.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingjing Chang, Yixiao Fang, Peng Xing, Shuhan Wu, Wei Cheng, Rui Wang, Xianfang Zeng, Gang Yu, Hai-Bao Chen</p><p><b>Upvotes:</b> 37</p><p><b>Summary:</b> Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts. However, rapid T2I model advancements reveal limitations in early benchmarks, lacking comprehensive evaluations, for example, the evaluation on reasoning, text rendering and style. Notably, recent state-of-the-art models, with their rich knowledge modeling capabilities, show promising results on the image generation problems requiring strong reasoning ability, yet existing evaluation systems have not adequately addressed this frontier. To systematically address these gaps, we introduce OneIG-Bench, a meticulously designed comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions, including prompt-image alignment, text rendering precision, reasoning-generated content, stylization, and diversity. By structuring the evaluation, this benchmark enables in-depth analysis of model performance, helping researchers and practitioners pinpoint strengths and bottlenecks in the full pipeline of image generation. Specifically, OneIG-Bench enables flexible evaluation by allowing users to focus on a particular evaluation subset. Instead of generating images for the entire set of prompts, users can generate images only for the prompts associated with the selected dimension and complete the corresponding evaluation accordingly. Our codebase and dataset are now publicly available to facilitate reproducible evaluation studies and cross-model comparisons within the T2I research community.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07977</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:50:21 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>τ^2-Bench: Evaluating Conversational Agents in a Dual-Control Environment</title>
|
||||
<link>https://arxiv.org/abs/2506.07982</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07982.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider. This differs from real-world scenarios like technical support, where users need to actively participate in modifying the state of the (shared) world. In order to address this gap, we introduce tau^2-bench, with four key contributions: 1) A novel Telecom dual-control domain modeled as a Dec-POMDP, where both agent and user make use of tools to act in a shared, dynamic environment that tests both agent coordination and communication, 2) A compositional task generator that programmatically creates diverse, verifiable tasks from atomic components, ensuring domain coverage and controlled complexity, 3) A reliable user simulator tightly coupled with the environment, whose behavior is constrained by tools and observable states, improving simulation fidelity, 4) Fine-grained analysis of agent performance through multiple ablations including separating errors arising from reasoning vs communication/coordination. In particular, our experiments show significant performance drops when agents shift from no-user to dual-control, highlighting the challenges of guiding users. Overall, tau^2-bench provides a controlled testbed for agents that must both reason effectively and guide user actions.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07982</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:52:18 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers</title>
|
||||
<link>https://arxiv.org/abs/2506.07986</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.07986.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhengyao Lv, Tianlin Pan, Chenyang Si, Zhaoxi Chen, Wangmeng Zuo, Ziwei Liu, Kwan-Yee K. Wong</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress in text-driven visual generation. However, even state-of-the-art MM-DiT models like FLUX struggle with achieving precise alignment between text prompts and generated content. We identify two key issues in the attention mechanism of MM-DiT, namely 1) the suppression of cross-modal attention due to token imbalance between visual and textual modalities and 2) the lack of timestep-aware attention weighting, which hinder the alignment. To address these issues, we propose Temperature-Adjusted Cross-modal Attention (TACA), a parameter-efficient method that dynamically rebalances multimodal interactions through temperature scaling and timestep-dependent adjustment. When combined with LoRA fine-tuning, TACA significantly enhances text-image alignment on the T2I-CompBench benchmark with minimal computational overhead. We tested TACA on state-of-the-art models like FLUX and SD3.5, demonstrating its ability to improve image-text alignment in terms of object appearance, attribute binding, and spatial relationships. Our findings highlight the importance of balancing cross-modal attention in improving semantic fidelity in text-to-image diffusion models. Our codes are publicly available at https://github.com/Vchitect/TACA</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.07986</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:54:04 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Dynamic View Synthesis as an Inverse Problem</title>
|
||||
<link>https://arxiv.org/abs/2506.08004</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08004.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hidir Yesiltepe, Pinar Yanardag</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> In this work, we address dynamic view synthesis from monocular videos as an inverse problem in a training-free setting. By redesigning the noise initialization phase of a pre-trained video diffusion model, we enable high-fidelity dynamic view synthesis without any weight updates or auxiliary modules. We begin by identifying a fundamental obstacle to deterministic inversion arising from zero-terminal signal-to-noise ratio (SNR) schedules and resolve it by introducing a novel noise representation, termed K-order Recursive Noise Representation. We derive a closed form expression for this representation, enabling precise and efficient alignment between the VAE-encoded and the DDIM inverted latents. To synthesize newly visible regions resulting from camera motion, we introduce Stochastic Latent Modulation, which performs visibility aware sampling over the latent space to complete occluded regions. Comprehensive experiments demonstrate that dynamic view synthesis can be effectively performed through structured latent manipulation in the noise initialization phase.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08004</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:59:47 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Dreamland: Controllable World Creation with Simulator and Generative Models</title>
|
||||
<link>https://arxiv.org/abs/2506.08006</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08006.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sicheng Mo, Ziyang Leng, Leon Liu, Weizhen Wang, Honglin He, Bolei Zhou</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Large-scale video generative models can synthesize diverse and realistic visual content for dynamic world creation, but they often lack element-wise controllability, hindering their use in editing scenes and training embodied AI agents. We propose Dreamland, a hybrid world generation framework combining the granular control of a physics-based simulator and the photorealistic content output of large-scale pretrained generative models. In particular, we design a layered world abstraction that encodes both pixel-level and object-level semantics and geometry as an intermediate representation to bridge the simulator and the generative model. This approach enhances controllability, minimizes adaptation cost through early alignment with real-world distributions, and supports off-the-shelf use of existing and future pretrained generative models. We further construct a D3Sim dataset to facilitate the training and evaluation of hybrid generation pipelines. Experiments demonstrate that Dreamland outperforms existing baselines with 50.8% improved image quality, 17.9% stronger controllability, and has great potential to enhance embodied agent training. Code and data will be made available.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08006</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:59:52 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Reinforcement Pre-Training</title>
|
||||
<link>https://arxiv.org/abs/2506.08007</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08007.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, Furu Wei</p><p><b>Upvotes:</b> 143</p><p><b>Summary:</b> In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08007</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:59:53 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior</title>
|
||||
<link>https://arxiv.org/abs/2506.08012</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08012.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08012</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:59:57 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Play to Generalize: Learning to Reason Through Game Play</title>
|
||||
<link>https://arxiv.org/abs/2506.08011</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08011.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yunfei Xie, Yinsong Ma, Shiyi Lan, Alan Yuille, Junfei Xiao, Chen Wei</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Developing generalizable reasoning capabilities in multimodal large language models (MLLMs) remains challenging. Motivated by cognitive science literature suggesting that gameplay promotes transferable cognitive skills, we propose a novel post-training paradigm, Visual Game Learning, or ViGaL, where MLLMs develop out-of-domain generalization of multimodal reasoning through playing arcade-like games. Specifically, we show that post-training a 7B-parameter MLLM via reinforcement learning (RL) on simple arcade-like games, e.g. Snake, significantly enhances its downstream performance on multimodal math benchmarks like MathVista, and on multi-discipline questions like MMMU, without seeing any worked solutions, equations, or diagrams during RL, suggesting the capture of transferable reasoning skills. Remarkably, our model outperforms specialist models tuned on multimodal reasoning data in multimodal reasoning benchmarks, while preserving the base model's performance on general visual benchmarks, a challenge where specialist models often fall short. Our findings suggest a new post-training paradigm: synthetic, rule-based games can serve as controllable and scalable pre-text tasks that unlock generalizable multimodal reasoning abilities in MLLMs.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08011</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:59:57 +0000</pubDate>
|
||||
</item>
|
||||
<item>
|
||||
<title>Vision Transformers Don't Need Trained Registers</title>
|
||||
<link>https://arxiv.org/abs/2506.08010</link>
|
||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.08010.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nick Jiang, Amil Dravid, Alexei Efros, Yossi Gandelsman</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> We investigate the mechanism underlying a previously identified phenomenon in Vision Transformers -- the emergence of high-norm tokens that lead to noisy attention maps. We observe that in multiple models (e.g., CLIP, DINOv2), a sparse set of neurons is responsible for concentrating high-norm activations on outlier tokens, leading to irregular attention patterns and degrading downstream visual processing. While the existing solution for removing these outliers involves retraining models from scratch with additional learned register tokens, we use our findings to create a training-free approach to mitigate these artifacts. By shifting the high-norm activations from our discovered register neurons into an additional untrained token, we can mimic the effect of register tokens on a model already trained without registers. We demonstrate that our method produces cleaner attention and feature maps, enhances performance over base models across multiple downstream visual tasks, and achieves results comparable to models explicitly trained with register tokens. We then extend test-time registers to off-the-shelf vision-language models to improve their interpretability. Our results suggest that test-time registers effectively take on the role of register tokens at test-time, offering a training-free solution for any pre-trained model released without them.</p></description>
|
||||
<guid isPermaLink="false">https://arxiv.org/abs/2506.08010</guid>
|
||||
<pubDate>Mon, 09 Jun 2025 17:59:57 +0000</pubDate>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
|
||||
Reference in New Issue
Block a user