650 lines
194 KiB
XML
650 lines
194 KiB
XML
<?xml version='1.0' encoding='UTF-8'?>
|
|
<rss xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" version="2.0">
|
|
<channel>
|
|
<title>Hugging Face Daily Papers</title>
|
|
<link>https://huggingface.co/papers</link>
|
|
<description>Daily research papers curated by the Hugging Face community.</description>
|
|
<docs>http://www.rssboard.org/rss-specification</docs>
|
|
<generator>python-feedgen</generator>
|
|
<language>en</language>
|
|
<lastBuildDate>Mon, 07 Jul 2025 00:12:41 +0000</lastBuildDate>
|
|
<item>
|
|
<title>Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity</title>
|
|
<link>https://arxiv.org/abs/2505.21411</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2505.21411.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yehui Tang, Xiaosong Li, Fangcheng Liu, Wei Guo, Hang Zhou, Yaoyuan Wang, Kai Han, Xianzhi Yu, Jinpeng Li, Hui Zang, Fei Mi, Xiaojun Meng, Zhicheng Liu, Hanting Chen, Binfan Zheng, Can Chen, Youliang Yan, Ruiming Tang, Peifeng Qin, Xinghao Chen, Dacheng Tao, Yunhe Wang</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> The surgence of Mixture of Experts (MoE) in Large Language Models promises a small price of execution cost for a much larger model parameter count and learning capacity, because only a small fraction of parameters are activated for each input token. However, it is commonly observed that some experts are activated far more often than others, leading to system inefficiency when running the experts on different devices in parallel. Therefore, we introduce Mixture of Grouped Experts (MoGE), which groups the experts during selection and balances the expert workload better than MoE in nature. It constrains tokens to activate an equal number of experts within each predefined expert group. When a model execution is distributed on multiple devices, this architectural design ensures a balanced computational load across devices, significantly enhancing throughput, particularly for the inference phase. Further, we build Pangu Pro MoE on Ascend NPUs, a sparse model based on MoGE with 72 billion total parameters, 16 billion of which are activated for each token. The configuration of Pangu Pro MoE is optimized for Ascend 300I Duo and 800I A2 through extensive system simulation studies. Our experiments indicate that MoGE indeed leads to better expert load balancing and more efficient execution for both model training and inference on Ascend NPUs. The inference performance of Pangu Pro MoE achieves 1148 tokens/s per card and can be further improved to 1528 tokens/s per card by speculative acceleration, outperforming comparable 32B and 72B Dense models. Furthermore, we achieve an excellent cost-to-performance ratio for model inference on Ascend 300I Duo. Our studies show that Ascend NPUs are capable of training Pangu Pro MoE with massive parallelization to make it a leading model within the sub-100B total parameter class, outperforming prominent open-source models like GLM-Z1-32B and Qwen3-32B.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2505.21411</guid>
|
|
<pubDate>Tue, 27 May 2025 16:40:21 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Gazal-R1: Achieving State-of-the-Art Medical Reasoning with Parameter-Efficient Two-Stage Training</title>
|
|
<link>https://arxiv.org/abs/2506.21594</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21594.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ahmed M. Adly, Mostafa Samy, Amr Fawzy</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> We present Gazal-R1, a 32-billion-parameter language model that achieves state-of-the-art performance in medical reasoning while providing transparent, step-by-step explanations for clinical decision-making. Built upon Qwen3 32B, our model demonstrates that strategic training can enable mid-sized models to outperform significantly larger counterparts in specialized domains. We developed a novel two-stage training pipeline: first, supervised fine-tuning on a carefully curated dataset of 107,033 synthetic medical reasoning examples that teaches structured clinical thinking, enhanced by advanced parameter-efficient techniques including Weight-Decomposed Low-Rank Adaptation (DoRA) and Rank-Stabilized LoRA (rsLoRA); second, reinforcement learning using Group Relative Policy Optimization (GRPO) with a sophisticated multi-component reward system that refines accuracy, format adherence, and reasoning quality. Gazal-R1 achieves exceptional performance across medical benchmarks, scoring 87.1% on MedQA, 81.6% on MMLU Pro (Medical), and 79.6% on PubMedQA, surpassing models up to 12x larger. Beyond its strong empirical results, this work provides detailed insights into the challenges of training reasoning-capable models in specialized domains, including issues with reward hacking, training instability, and the fundamental tension between factual recall and detailed reasoning. Our methodology offers a reproducible framework for developing high-capability, domain-specific language models that balance performance, efficiency, and explainability.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21594</guid>
|
|
<pubDate>Wed, 18 Jun 2025 09:44:21 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Fractional Reasoning via Latent Steering Vectors Improves Inference Time Compute</title>
|
|
<link>https://arxiv.org/abs/2506.15882</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.15882.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sheng Liu, Tianlang Chen, Pan Lu, Haotian Ye, Yizheng Chen, Lei Xing, James Zou</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Test-time compute has emerged as a powerful paradigm for improving the performance of large language models (LLMs), where generating multiple outputs or refining individual chains can significantly boost answer accuracy. However, existing methods like Best-of-N, majority voting, and self-reflection typically apply reasoning in a uniform way across inputs, overlooking the fact that different problems may require different levels of reasoning depth. In this work, we propose Fractional Reasoning, a training-free and model-agnostic framework that enables continuous control over reasoning intensity at inference time, going beyond the limitations of fixed instructional prompts. Our method operates by extracting the latent steering vector associated with deeper reasoning and reapplying it with a tunable scaling factor, allowing the model to tailor its reasoning process to the complexity of each input. This supports two key modes of test-time scaling: (1) improving output quality in breadth-based strategies (e.g., Best-of-N, majority voting), and (2) enhancing the correctness of individual reasoning chains in depth-based strategies (e.g., self-reflection). Experiments on GSM8K, MATH500, and GPQA demonstrate that Fractional Reasoning consistently improves performance across diverse reasoning tasks and models.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.15882</guid>
|
|
<pubDate>Wed, 18 Jun 2025 21:15:59 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity</title>
|
|
<link>https://arxiv.org/abs/2506.16500</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.16500.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Samir Khaki, Xiuyu Li, Junxian Guo, Ligeng Zhu, Chenfeng Xu, Konstantinos N. Plataniotis, Amir Yazdanbakhsh, Kurt Keutzer, Song Han, Zhijian Liu</p><p><b>Upvotes:</b> 16</p><p><b>Summary:</b> Fine-tuning LLMs is both computationally and memory-intensive. While parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the number of trainable parameters and lower memory usage, they do not decrease computational cost. In some cases, they may even slow down fine-tuning. In this paper, we introduce SparseLoRA, a method that accelerates LLM fine-tuning through contextual sparsity. We propose a lightweight, training-free SVD sparsity estimator that dynamically selects a sparse subset of weights for loss and gradient computation. Also, we systematically analyze and address sensitivity across layers, tokens, and training steps. Our experimental results show that SparseLoRA reduces computational cost by up to 2.2 times and a measured speedup of up to 1.6 times while maintaining accuracy across various downstream tasks, including commonsense and arithmetic reasoning, code generation, and instruction following.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.16500</guid>
|
|
<pubDate>Thu, 19 Jun 2025 17:53:34 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs</title>
|
|
<link>https://arxiv.org/abs/2506.17080</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.17080.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, André F. T. Martins</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Fine-tuning pretrained LLMs has been shown to be an effective strategy for reaching state-of-the-art performance on specific tasks like machine translation. However, this process of adaptation often implies sacrificing general-purpose capabilities, such as conversational reasoning and instruction-following, hampering the utility of the system in real-world applications that require a mixture of skills. In this paper, we introduce Tower+, a suite of models designed to deliver strong performance across both translation and multilingual general-purpose text capabilities. We achieve a Pareto frontier between translation specialization and multilingual general-purpose capabilities by introducing a novel training recipe that builds on Tower (Alves et al., 2024), comprising continued pretraining, supervised fine-tuning, preference optimization, and reinforcement learning with verifiable rewards. At each stage of training, we carefully generate and curate data to strengthen performance on translation as well as general-purpose tasks involving code generation, mathematics problem solving, and general instruction-following. We develop models at multiple scales: 2B, 9B, and 72B. Our smaller models often outperform larger general-purpose open-weight and proprietary LLMs (e.g., Llama 3.3 70B, GPT-4o). Our largest model delivers best-in-class translation performance for high-resource languages and top results in multilingual Arena Hard evaluations and in IF-MT, a benchmark we introduce for evaluating both translation and instruction-following. Our findings highlight that it is possible to rival frontier models in general capabilities, while optimizing for specific business domains, such as translation and localization.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.17080</guid>
|
|
<pubDate>Fri, 20 Jun 2025 15:30:06 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?</title>
|
|
<link>https://arxiv.org/abs/2506.17417</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.17417.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mingyuan Wu, Meitang Li, Jingcheng Yang, Jize Jiang, Kaizhuo Yan, Zhaoheng Li, Minjia Zhang, Klara Nahrstedt</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Recent advances in large language models (LLMs) have demonstrated that inference-time computation techniques, such as decoding-time scaling and self-refinement, can significantly enhance reasoning capabilities without relying on external knowledge. A key driver of this success is the emergence of self-correction and self-verification behaviors, often elicited through reinforcement learning (RL). In this paper, we investigate whether these inference-time techniques extend effectively to vision-language models (VLMs), particularly those trained with RL. We find that while decoding strategies such as majority voting and best-of-N selection with self-verification all improve VLM reasoning performance, generation-reliant methods such as the former achieve significantly higher gains versus verification-reliant methods such as the latter. Additionally, the self-correction behavior often associated with RL-tuned models, such as aha moment, does not lead to measurable gains. We show via extensive experimentation within the inference-time scaling framework to identify a key root cause: RL-trained VLMs still lack robust self-verification capabilities across both visual and textual modalities.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.17417</guid>
|
|
<pubDate>Fri, 20 Jun 2025 18:23:48 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing</title>
|
|
<link>https://arxiv.org/abs/2506.17450</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.17450.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiacheng Chen, Ramin Mehran, Xuhui Jia, Saining Xie, Sanghyun Woo</p><p><b>Upvotes:</b> 58</p><p><b>Summary:</b> We present BlenderFusion, a generative visual compositing framework that synthesizes new scenes by recomposing objects, camera, and background. It follows a layering-editing-compositing pipeline: (i) segmenting and converting visual inputs into editable 3D entities (layering), (ii) editing them in Blender with 3D-grounded control (editing), and (iii) fusing them into a coherent scene using a generative compositor (compositing). Our generative compositor extends a pre-trained diffusion model to process both the original (source) and edited (target) scenes in parallel. It is fine-tuned on video frames with two key training strategies: (i) source masking, enabling flexible modifications like background replacement; (ii) simulated object jittering, facilitating disentangled control over objects and camera. BlenderFusion significantly outperforms prior methods in complex compositional scene editing tasks.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.17450</guid>
|
|
<pubDate>Fri, 20 Jun 2025 19:38:34 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>In-Context Learning Strategies Emerge Rationally</title>
|
|
<link>https://arxiv.org/abs/2506.17859</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.17859.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daniel Wurgaft, Ekdeep Singh Lubana, Core Francisco Park, Hidenori Tanaka, Gautam Reddy, Noah D. Goodman</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Recent work analyzing in-context learning (ICL) has identified a broad set of strategies that describe model behavior in different experimental conditions. We aim to unify these findings by asking why a model learns these disparate strategies in the first place. Specifically, we start with the observation that when trained to learn a mixture of tasks, as is popular in the literature, the strategies learned by a model for performing ICL can be captured by a family of Bayesian predictors: a memorizing predictor, which assumes a discrete prior on the set of seen tasks, and a generalizing predictor, where the prior matches the underlying task distribution. Adopting the normative lens of rational analysis, where a learner's behavior is explained as an optimal adaptation to data given computational constraints, we develop a hierarchical Bayesian framework that almost perfectly predicts Transformer next-token predictions throughout training -- without assuming access to its weights. Under this framework, pretraining is viewed as a process of updating the posterior probability of different strategies, and inference-time behavior as a posterior-weighted average over these strategies' predictions. Our framework draws on common assumptions about neural network learning dynamics, which make explicit a tradeoff between loss and complexity among candidate strategies: beyond how well it explains the data, a model's preference towards implementing a strategy is dictated by its complexity. This helps explain well-known ICL phenomena, while offering novel predictions: e.g., we show a superlinear trend in the timescale for transitioning from generalization to memorization as task diversity increases. Overall, our work advances an explanatory and predictive account of ICL grounded in tradeoffs between strategy loss and complexity.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.17859</guid>
|
|
<pubDate>Sat, 21 Jun 2025 23:49:08 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective</title>
|
|
<link>https://arxiv.org/abs/2506.17930</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.17930.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jianyu Wang, Zhiqiang Hu, Lidong Bing</p><p><b>Upvotes:</b> 19</p><p><b>Summary:</b> We propose a novel prompt design paradigm that challenges conventional wisdom in large language model (LLM) prompting. While conventional wisdom prioritizes well-crafted instructions and demonstrations for in-context learning (ICL), we show that pruning random demonstrations into seemingly incoherent "gibberish" can remarkably improve performance across diverse tasks. Notably, the "gibberish" always matches or surpasses state-of-the-art automatic prompt optimization techniques, achieving substantial gains regardless of LLM alignment. Nevertheless, discovering an effective pruning strategy is non-trivial, as existing attribution methods and prompt compression algorithms fail to deliver robust results, let alone human intuition. In terms of this, we propose a self-discover prompt optimization framework, PromptQuine, an evolutionary search framework that automatically searches for the pruning strategy by itself using only low-data regimes. Much like the emergent complexity in nature--such as symbiosis and self-organization--arising in response to resource constraints, our framework evolves and refines unconventional yet highly effective prompts by leveraging only the tokens present within the context. We demonstrate its effectiveness across classification, multi-choice question answering, generation and math reasoning tasks across LLMs, while achieving decent runtime efficiency. We hope our findings can guide mechanistic studies on in-context learning, and provide a call to action, to pave the way for more open-ended search algorithms for more effective LLM prompting.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.17930</guid>
|
|
<pubDate>Sun, 22 Jun 2025 07:53:07 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Confucius3-Math: A Lightweight High-Performance Reasoning LLM for Chinese K-12 Mathematics Learning</title>
|
|
<link>https://arxiv.org/abs/2506.18330</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.18330.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lixin Wu, Na Cai, Qiao Cheng, Jiachen Wang, Yitao Duan</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> We introduce Confucius3-Math, an open-source large language model with 14B parameters that (1) runs efficiently on a single consumer-grade GPU; (2) achieves SOTA performances on a range of mathematical reasoning tasks, outperforming many models with significantly larger sizes. In particular, as part of our mission to enhancing education and knowledge dissemination with AI, Confucius3-Math is specifically committed to mathematics learning for Chinese K-12 students and educators. Built via post-training with large-scale reinforcement learning (RL), Confucius3-Math aligns with national curriculum and excels at solving main-stream Chinese K-12 mathematical problems with low cost. In this report we share our development recipe, the challenges we encounter and the techniques we develop to overcome them. In particular, we introduce three technical innovations: Targeted Entropy Regularization, Recent Sample Recovery and Policy-Specific Hardness Weighting. These innovations encompass a new entropy regularization, a novel data scheduling policy, and an improved group-relative advantage estimator. Collectively, they significantly stabilize the RL training, improve data efficiency, and boost performance. Our work demonstrates the feasibility of building strong reasoning models in a particular domain at low cost. We open-source our model and code at https://github.com/netease-youdao/Confucius3-Math.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.18330</guid>
|
|
<pubDate>Mon, 23 Jun 2025 06:23:53 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Adaptive Domain Modeling with Language Models: A Multi-Agent Approach to Task Planning</title>
|
|
<link>https://arxiv.org/abs/2506.19592</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.19592.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Harisankar Babu, Philipp Schillinger, Tamim Asfour</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> We introduce TAPAS (Task-based Adaptation and Planning using AgentS), a multi-agent framework that integrates Large Language Models (LLMs) with symbolic planning to solve complex tasks without the need for manually defined environment models. TAPAS employs specialized LLM-based agents that collaboratively generate and adapt domain models, initial states, and goal specifications as needed using structured tool-calling mechanisms. Through this tool-based interaction, downstream agents can request modifications from upstream agents, enabling adaptation to novel attributes and constraints without manual domain redefinition. A ReAct (Reason+Act)-style execution agent, coupled with natural language plan translation, bridges the gap between dynamically generated plans and real-world robot capabilities. TAPAS demonstrates strong performance in benchmark planning domains and in the VirtualHome simulated real-world environment.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.19592</guid>
|
|
<pubDate>Tue, 24 Jun 2025 13:02:06 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Noise Consistency Training: A Native Approach for One-Step Generator in Learning Additional Controls</title>
|
|
<link>https://arxiv.org/abs/2506.19741</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.19741.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yihong Luo, Shuchen Xue, Tianyang Hu, Jing Tang</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> The pursuit of efficient and controllable high-quality content generation remains a central challenge in artificial intelligence-generated content (AIGC). While one-step generators, enabled by diffusion distillation techniques, offer excellent generation quality and computational efficiency, adapting them to new control conditions--such as structural constraints, semantic guidelines, or external inputs--poses a significant challenge. Conventional approaches often necessitate computationally expensive modifications to the base model and subsequent diffusion distillation. This paper introduces Noise Consistency Training (NCT), a novel and lightweight approach to directly integrate new control signals into pre-trained one-step generators without requiring access to original training images or retraining the base diffusion model. NCT operates by introducing an adapter module and employs a noise consistency loss in the noise space of the generator. This loss aligns the adapted model's generation behavior across noises that are conditionally dependent to varying degrees, implicitly guiding it to adhere to the new control. Theoretically, this training objective can be understood as minimizing the distributional distance between the adapted generator and the conditional distribution induced by the new conditions. NCT is modular, data-efficient, and easily deployable, relying only on the pre-trained one-step generator and a control signal model. Extensive experiments demonstrate that NCT achieves state-of-the-art controllable generation in a single forward pass, surpassing existing multi-step and distillation-based methods in both generation quality and computational efficiency. Code is available at https://github.com/Luo-Yihong/NCT</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.19741</guid>
|
|
<pubDate>Tue, 24 Jun 2025 15:58:55 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation</title>
|
|
<link>https://arxiv.org/abs/2506.19852</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.19852.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, Song Han</p><p><b>Upvotes:</b> 34</p><p><b>Summary:</b> Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporal Energy Decay in video diffusion models: post-softmax attention scores diminish as spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, we propose Radial Attention, a scalable sparse attention mechanism with O(n log n) complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard O(n^2) dense attention and more expressive than linear attention. Specifically, Radial Attention employs a simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient LoRA-based fine-tuning. Extensive experiments show that Radial Attention maintains video quality across Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9times speedup over the original dense attention. With minimal tuning, it enables video generation up to 4times longer while reducing training costs by up to 4.4times compared to direct fine-tuning and accelerating inference by up to 3.7times compared to dense attention inference.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.19852</guid>
|
|
<pubDate>Tue, 24 Jun 2025 17:59:59 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Ark: An Open-source Python-based Framework for Robot Learning</title>
|
|
<link>https://arxiv.org/abs/2506.21628</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21628.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Magnus Dierking, Christopher E. Mower, Sarthak Das, Huang Helong, Jiacheng Qiu, Cody Reading, Wei Chen, Huidong Liang, Huang Guowei, Jan Peters, Quan Xingyue, Jun Wang, Haitham Bou-Ammar</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Robotics has made remarkable hardware strides-from DARPA's Urban and Robotics Challenges to the first humanoid-robot kickboxing tournament-yet commercial autonomy still lags behind progress in machine learning. A major bottleneck is software: current robot stacks demand steep learning curves, low-level C/C++ expertise, fragmented tooling, and intricate hardware integration, in stark contrast to the Python-centric, well-documented ecosystems that propelled modern AI. We introduce ARK, an open-source, Python-first robotics framework designed to close that gap. ARK presents a Gym-style environment interface that allows users to collect data, preprocess it, and train policies using state-of-the-art imitation-learning algorithms (e.g., ACT, Diffusion Policy) while seamlessly toggling between high-fidelity simulation and physical robots. A lightweight client-server architecture provides networked publisher-subscriber communication, and optional C/C++ bindings ensure real-time performance when needed. ARK ships with reusable modules for control, SLAM, motion planning, system identification, and visualization, along with native ROS interoperability. Comprehensive documentation and case studies-from manipulation to mobile navigation-demonstrate rapid prototyping, effortless hardware swapping, and end-to-end pipelines that rival the convenience of mainstream machine-learning workflows. By unifying robotics and AI practices under a common Python umbrella, ARK lowers entry barriers and accelerates research and commercial deployment of autonomous robots.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21628</guid>
|
|
<pubDate>Tue, 24 Jun 2025 20:23:39 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarios</title>
|
|
<link>https://arxiv.org/abs/2506.20279</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.20279.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Changliang Xia, Chengyou Jia, Zhuohang Dang, Minnan Luo</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Dense prediction tasks hold significant importance of computer vision, aiming to learn pixel-wise annotated label for an input image. Despite advances in this field, existing methods primarily focus on idealized conditions, with limited generalization to real-world scenarios and facing the challenging scarcity of real-world data. To systematically study this problem, we first introduce DenseWorld, a benchmark spanning a broad set of 25 dense prediction tasks that correspond to urgent real-world applications, featuring unified evaluation across tasks. Then, we propose DenseDiT, which maximally exploits generative models' visual priors to perform diverse real-world dense prediction tasks through a unified strategy. DenseDiT combines a parameter-reuse mechanism and two lightweight branches that adaptively integrate multi-scale context, working with less than 0.1% additional parameters. Evaluations on DenseWorld reveal significant performance drops in existing general and specialized baselines, highlighting their limited real-world generalization. In contrast, DenseDiT achieves superior results using less than 0.01% training data of baselines, underscoring its practical value for real-world deployment. Our data, and checkpoints and codes are available at https://xcltql666.github.io/DenseDiTProj</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.20279</guid>
|
|
<pubDate>Wed, 25 Jun 2025 09:40:50 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation</title>
|
|
<link>https://arxiv.org/abs/2506.20639</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.20639.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, Yizhe Zhang</p><p><b>Upvotes:</b> 25</p><p><b>Summary:</b> Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models because their denoising models operate over the entire sequence. The global planning and iterative refinement features of dLLMs are particularly useful for code generation. However, current training and inference mechanisms for dLLMs in coding are still under-explored. To demystify the decoding behavior of dLLMs and unlock their potential for coding, we systematically investigate their denoising processes and reinforcement learning (RL) methods. We train a 7B dLLM, DiffuCoder, on 130B tokens of code. Using this model as a testbed, we analyze its decoding behavior, revealing how it differs from that of AR models: (1) dLLMs can decide how causal their generation should be without relying on semi-AR decoding, and (2) increasing the sampling temperature diversifies not only token choices but also their generation order. This diversity creates a rich search space for RL rollouts. For RL training, to reduce the variance of token log-likelihood estimates and maintain training efficiency, we propose coupled-GRPO, a novel sampling scheme that constructs complementary mask noise for completions used in training. In our experiments, coupled-GRPO significantly improves DiffuCoder's performance on code generation benchmarks (+4.4\% on EvalPlus) and reduces reliance on AR causal during decoding. Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework. https://github.com/apple/ml-diffucoder.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.20639</guid>
|
|
<pubDate>Wed, 25 Jun 2025 17:35:47 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context</title>
|
|
<link>https://arxiv.org/abs/2506.21277</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21277.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qize Yang, Shimin Yao, Weixuan Chen, Shenghao Fu, Detao Bai, Jiaxing Zhao, Boyuan Sun, Bowen Yin, Xihan Wei, Jingren Zhou</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> With the rapid evolution of multimodal large language models, the capacity to deeply understand and interpret human intentions has emerged as a critical capability, which demands detailed and thoughtful reasoning. In recent studies, Reinforcement Learning (RL) has demonstrated potential in enhancing the reasoning capabilities of Large Language Models (LLMs). Nonetheless, the challenges associated with adapting RL to multimodal data and formats remain largely unaddressed. In this paper, we identify two issues in existing multimodal reasoning models: insufficient global context understanding and shortcut problems. Insufficient context understanding can happen when a model misinterprets multimodal context, resulting in incorrect answers. The shortcut problem occurs when the model overlooks crucial clues in multimodal inputs, directly addressing the query without considering the multimodal information. To tackle these issues, we emphasize the necessity for the model to reason with a clear understanding of the global context within multimodal inputs. This global context understanding can effectively prevent the model from overlooking key multimodal cues and ensure a thorough reasoning process. To ensure the accurate interpretation of multimodal context information, we implement a context reward judged by a large language model, alongside format and accuracy rewards. Additionally, to improve complex reasoning capability, we employ the LLM to assess the logical reward, determining whether the reasoning process successfully integrates multimodal information with logical methods. We also introduce a reasoning omni-modal benchmark, IntentBench, aimed at evaluating models in understanding complex human intentions and emotions. Our proposed method demonstrates advanced performance across multiple omni-modal benchmarks compared to other open-source omni-modal models.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21277</guid>
|
|
<pubDate>Thu, 26 Jun 2025 14:01:03 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning</title>
|
|
<link>https://arxiv.org/abs/2506.21355</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21355.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Melanie Rieff, Maya Varma, Ossian Rabow, Subathra Adithan, Julie Kim, Ken Chang, Hannah Lee, Nidhi Rohatgi, Christian Bluethgen, Mohamed S. Muneer, Jean-Benoit Delbrouck, Michael Moor</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Multimodal in-context learning (ICL) remains underexplored despite significant potential for domains such as medicine. Clinicians routinely encounter diverse, specialized tasks requiring adaptation from limited examples, such as drawing insights from a few relevant prior cases or considering a constrained set of differential diagnoses. While multimodal large language models (MLLMs) have shown advances in medical visual question answering (VQA), their ability to learn multimodal tasks from context is largely unknown. We introduce SMMILE, the first expert-driven multimodal ICL benchmark for medical tasks. Eleven medical experts curated problems, each including a multimodal query and multimodal in-context examples as task demonstrations. SMMILE encompasses 111 problems (517 question-image-answer triplets) covering 6 medical specialties and 13 imaging modalities. We further introduce SMMILE++, an augmented variant with 1038 permuted problems. A comprehensive evaluation of 15 MLLMs demonstrates that most models exhibit moderate to poor multimodal ICL ability in medical tasks. In open-ended evaluations, ICL contributes only 8% average improvement over zero-shot on SMMILE and 9.4% on SMMILE++. We observe a susceptibility for irrelevant in-context examples: even a single noisy or irrelevant example can degrade performance by up to 9.5%. Moreover, example ordering exhibits a recency bias, i.e., placing the most relevant example last can lead to substantial performance improvements by up to 71%. Our findings highlight critical limitations and biases in current MLLMs when learning multimodal medical tasks from context.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21355</guid>
|
|
<pubDate>Thu, 26 Jun 2025 15:08:18 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models</title>
|
|
<link>https://arxiv.org/abs/2506.21356</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21356.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongbo Liu, Jingwen He, Yi Jin, Dian Zheng, Yuhao Dong, Fan Zhang, Ziqi Huang, Yinan He, Yangguang Li, Weichao Chen, Yu Qiao, Wanli Ouyang, Shengjie Zhao, Ziwei Liu</p><p><b>Upvotes:</b> 21</p><p><b>Summary:</b> Cinematography, the fundamental visual language of film, is essential for conveying narrative, emotion, and aesthetic quality. While recent Vision-Language Models (VLMs) demonstrate strong general visual understanding, their proficiency in comprehending the nuanced cinematic grammar embedded within individual shots remains largely unexplored and lacks robust evaluation. This critical gap limits both fine-grained visual comprehension and the precision of AI-assisted video generation. To address this, we introduce ShotBench, a comprehensive benchmark specifically designed for cinematic language understanding. It features over 3.5k expert-annotated QA pairs from images and video clips, meticulously curated from over 200 acclaimed (predominantly Oscar-nominated) films and spanning eight key cinematography dimensions. Our evaluation of 24 leading VLMs on ShotBench reveals their substantial limitations: even the top-performing model achieves less than 60% average accuracy, particularly struggling with fine-grained visual cues and complex spatial reasoning. To catalyze advancement in this domain, we construct ShotQA, a large-scale multimodal dataset comprising approximately 70k cinematic QA pairs. Leveraging ShotQA, we develop ShotVL through supervised fine-tuning and Group Relative Policy Optimization. ShotVL significantly outperforms all existing open-source and proprietary models on ShotBench, establishing new state-of-the-art performance. We open-source our models, data, and code to foster rapid progress in this crucial area of AI-driven cinematic understanding and generation.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21356</guid>
|
|
<pubDate>Thu, 26 Jun 2025 15:09:21 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation</title>
|
|
<link>https://arxiv.org/abs/2506.21416</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21416.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bowen Chen, Mengyi Zhao, Haomiao Sun, Li Chen, Xu Wang, Kang Du, Xinglong Wu</p><p><b>Upvotes:</b> 27</p><p><b>Summary:</b> Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21416</guid>
|
|
<pubDate>Thu, 26 Jun 2025 16:04:16 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing</title>
|
|
<link>https://arxiv.org/abs/2506.21448</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21448.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Huadai Liu, Jialei Wang, Kaicheng Luo, Wen Wang, Qian Chen, Zhou Zhao, Wei Xue</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> While end-to-end video-to-audio generation has greatly improved, producing high-fidelity audio that authentically captures the nuances of visual content remains challenging. Like professionals in the creative industries, such generation requires sophisticated reasoning about items such as visual dynamics, acoustic environments, and temporal relationships. We present ThinkSound, a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos. Our approach decomposes the process into three complementary stages: foundational foley generation that creates semantically coherent soundscapes, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions. At each stage, a multimodal large language model generates contextually aligned CoT reasoning that guides a unified audio foundation model. Furthermore, we introduce AudioCoT, a comprehensive dataset with structured reasoning annotations that establishes connections between visual content, textual descriptions, and sound synthesis. Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics and excels in out-of-distribution Movie Gen Audio benchmark. The demo page is available at https://ThinkSound-Project.github.io.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21448</guid>
|
|
<pubDate>Thu, 26 Jun 2025 16:32:06 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Spatial Mental Modeling from Limited Views</title>
|
|
<link>https://arxiv.org/abs/2506.21458</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21458.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Manling Li, Jiajun Wu, Li Fei-Fei</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Can Vision Language Models (VLMs) imagine the full scene from just a few views, like humans do? Humans form spatial mental models, internal representations of unseen space, to reason about layout, perspective, and motion. Our new MindCube benchmark with 21,154 questions across 3,268 images exposes this critical gap, where existing VLMs exhibit near-random performance. Using MindCube, we systematically evaluate how well VLMs build robust spatial mental models through representing positions (cognitive mapping), orientations (perspective-taking), and dynamics (mental simulation for "what-if" movements). We then explore three approaches to help VLMs approximate spatial mental models, including unseen intermediate views, natural language reasoning chains, and cognitive maps. The significant improvement comes from a synergistic approach, "map-then-reason", that jointly trains the model to first generate a cognitive map and then reason upon it. By training models to reason over these internal maps, we boosted accuracy from 37.8% to 60.8% (+23.0%). Adding reinforcement learning pushed performance even further to 70.7% (+32.9%). Our key insight is that such scaffolding of spatial mental models, actively constructing and utilizing internal structured spatial representations with flexible reasoning processes, significantly improves understanding of unobservable space.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21458</guid>
|
|
<pubDate>Thu, 26 Jun 2025 16:38:19 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Global and Local Entailment Learning for Natural World Imagery</title>
|
|
<link>https://arxiv.org/abs/2506.21476</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21476.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Srikumar Sastry, Aayush Dhakal, Eric Xing, Subash Khanal, Nathan Jacobs</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Learning the hierarchical structure of data in vision-language models is a significant challenge. Previous works have attempted to address this challenge by employing entailment learning. However, these approaches fail to model the transitive nature of entailment explicitly, which establishes the relationship between order and semantics within a representation space. In this work, we introduce Radial Cross-Modal Embeddings (RCME), a framework that enables the explicit modeling of transitivity-enforced entailment. Our proposed framework optimizes for the partial order of concepts within vision-language models. By leveraging our framework, we develop a hierarchical vision-language foundation model capable of representing the hierarchy in the Tree of Life. Our experiments on hierarchical species classification and hierarchical retrieval tasks demonstrate the enhanced performance of our models compared to the existing state-of-the-art models. Our code and models are open-sourced at https://vishu26.github.io/RCME/index.html.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21476</guid>
|
|
<pubDate>Thu, 26 Jun 2025 17:05:06 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Data Efficacy for Language Model Training</title>
|
|
<link>https://arxiv.org/abs/2506.21545</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21545.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yalun Dai, Yangyu Huang, Xin Zhang, Wenshan Wu, Chong Li, Wenhui Lu, Shijie Cao, Li Dong, Scarlett Li</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Data is fundamental to the training of language models (LM). Recent research has been dedicated to data efficiency, which aims to maximize performance by selecting a minimal or optimal subset of training data. Techniques such as data filtering, sampling, and selection play a crucial role in this area. To complement it, we define Data Efficacy, which focuses on maximizing performance by optimizing the organization of training data and remains relatively underexplored. This work introduces a general paradigm, DELT, for considering data efficacy in LM training, which highlights the significance of training data organization. DELT comprises three components: Data Scoring, Data Selection, and Data Ordering. Among these components, we design Learnability-Quality Scoring (LQS), as a new instance of Data Scoring, which considers both the learnability and quality of each data sample from the gradient consistency perspective. We also devise Folding Ordering (FO), as a novel instance of Data Ordering, which addresses issues such as model forgetting and data distribution bias. Comprehensive experiments validate the data efficacy in LM training, which demonstrates the following: Firstly, various instances of the proposed DELT enhance LM performance to varying degrees without increasing the data scale and model size. Secondly, among these instances, the combination of our proposed LQS for data scoring and Folding for data ordering achieves the most significant improvement. Lastly, data efficacy can be achieved together with data efficiency by applying data selection. Therefore, we believe that data efficacy is a promising foundational area in LM training.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21545</guid>
|
|
<pubDate>Thu, 26 Jun 2025 17:59:07 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>HalluSegBench: Counterfactual Visual Reasoning for Segmentation Hallucination Evaluation</title>
|
|
<link>https://arxiv.org/abs/2506.21546</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21546.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xinzhuo Li, Adheesh Juvekar, Xingyou Liu, Muntasir Wahed, Kiet A. Nguyen, Ismini Lourentzou</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Recent progress in vision-language segmentation has significantly advanced grounded visual understanding. However, these models often exhibit hallucinations by producing segmentation masks for objects not grounded in the image content or by incorrectly labeling irrelevant regions. Existing evaluation protocols for segmentation hallucination primarily focus on label or textual hallucinations without manipulating the visual context, limiting their capacity to diagnose critical failures. In response, we introduce HalluSegBench, the first benchmark specifically designed to evaluate hallucinations in visual grounding through the lens of counterfactual visual reasoning. Our benchmark consists of a novel dataset of 1340 counterfactual instance pairs spanning 281 unique object classes, and a set of newly introduced metrics that quantify hallucination sensitivity under visually coherent scene edits. Experiments on HalluSegBench with state-of-the-art vision-language segmentation models reveal that vision-driven hallucinations are significantly more prevalent than label-driven ones, with models often persisting in false segmentation, highlighting the need for counterfactual reasoning to diagnose grounding fidelity.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21546</guid>
|
|
<pubDate>Thu, 26 Jun 2025 17:59:12 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs</title>
|
|
<link>https://arxiv.org/abs/2506.21656</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21656.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifan Shen, Yuanzhe Liu, Jingyuan Zhu, Xu Cao, Xiaofeng Zhang, Yixiao He, Wenming Ye, James Matthew Rehg, Ismini Lourentzou</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Current Vision-Language Models (VLMs) struggle with fine-grained spatial reasoning, particularly when multi-step logic and precise spatial alignment are required. In this work, we introduce SpatialReasoner-R1, a vision-language reasoning model designed to address these limitations. To construct high-quality supervision for spatial reasoning, we design a Multi-Model Monte Carlo Tree Search (M3CTS) method that generates diverse, logically consistent Long Chain-of-Thought (LongCoT) reasoning trajectories. In addition, we propose fine-grained Direct Preference Optimization (fDPO), which introduces segment-specific preference granularity for descriptive grounding and logical reasoning, guided by a spatial reward mechanism that evaluates candidate responses based on visual consistency, spatial grounding, and logical coherence. Experimental results demonstrate that fDPO achieves an average improvement of 4.1% over standard DPO across spatial quality tasks, and a 9.0% gain in spatial quantity tasks. SpatialReasoner-R1, trained with fDPO, sets a new SoTA on SPATIALRGPT-Bench, outperforming the strongest baseline by 9.8% in average accuracy, while maintaining competitive performance on general vision-language tasks.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21656</guid>
|
|
<pubDate>Thu, 26 Jun 2025 18:00:00 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Performance Prediction for Large Systems via Text-to-Text Regression</title>
|
|
<link>https://arxiv.org/abs/2506.21718</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21718.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yash Akhauri, Bryan Lewandowski, Cheng-Hsi Lin, Adrian N. Reyes, Grant C. Forbes, Arissa Wongpanich, Bangding Yang, Mohamed S. Abdelfattah, Sagi Perel, Xingyou Song</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> In many industries, predicting metric outcomes of large systems is a fundamental problem, driven largely by traditional tabular regression. However, such methods struggle on complex systems data in the wild such as configuration files or system logs, where feature engineering is often infeasible. We propose text-to-text regression as a general, scalable alternative. For predicting resource efficiency on Borg, Google's massive compute cluster scheduling system, a 60M parameter encoder-decoder, trained from random initialization, achieves up to a near perfect 0.99 (0.9 average) rank correlation across the entire fleet, and 100x lower MSE than tabular approaches. The model also easily adapts to new tasks in only 500 few-shot examples and captures the densities of complex outcome distributions. Ablation studies highlight the importance of using encoders, increasing sequence length, and the model's inherent uncertainty quantification. These findings pave the way for universal simulators of real-world outcomes.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21718</guid>
|
|
<pubDate>Thu, 26 Jun 2025 19:10:08 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs</title>
|
|
<link>https://arxiv.org/abs/2506.21862</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21862.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Boyuan Sun, Jiaxing Zhao, Xihan Wei, Qibin Hou</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> In this paper, we present LLaVA-Scissor, a training-free token compression strategy designed for video multimodal large language models. Previous methods mostly attempt to compress tokens based on attention scores, but fail to effectively capture all semantic regions and often lead to token redundancy. Differently, we propose to leverage the Semantic Connected Components (SCC) approach that assigns tokens to distinct semantic regions within the token set, ensuring comprehensive semantic coverage. The outcome is a two-step spatio-temporal token compression strategy that utilizes SCC in both spatial and temporal domains. This strategy can effectively compress tokens by representing the entire video with a set of non-overlapping semantic tokens. We conduct extensive evaluations of the token compression capabilities of LLaVA-Scissor across diverse video understanding benchmarks, including video question answering, long video understanding, and comprehensive multi-choices benchmarks. Experimental results show that the proposed LLaVA-Scissor outperforms other token compression methods, achieving superior performance in various video understanding benchmarks, particularly at low token retention ratios. Project page: https://github.com/HumanMLLM/LLaVA-Scissor.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21862</guid>
|
|
<pubDate>Fri, 27 Jun 2025 02:29:58 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation</title>
|
|
<link>https://arxiv.org/abs/2506.21876</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21876.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen, David Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu</p><p><b>Upvotes:</b> 25</p><p><b>Summary:</b> Internal world models (WMs) enable agents to understand the world's state and predict transitions, serving as the basis for advanced deliberative reasoning. Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs. While the latest studies have evaluated and shown limitations in specific capabilities such as visual understanding, a systematic evaluation of VLMs' fundamental WM abilities remains absent. Drawing on comparative psychology and cognitive science, we propose a two-stage framework that assesses Perception (visual, spatial, temporal, quantitative, and motion) and Prediction (mechanistic simulation, transitive inference, compositional inference) to provide an atomic evaluation of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations. Through 660 experiments on 15 latest commercial and open-source VLMs, we find that these models exhibit striking limitations in basic world modeling abilities. For instance, almost all models perform at near-random accuracy when distinguishing motion trajectories. Additionally, they lack disentangled understanding -- e.g., some models tend to believe blue objects move faster than green ones. More rich results and analyses reveal significant gaps between VLMs and human-level world modeling.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.21876</guid>
|
|
<pubDate>Fri, 27 Jun 2025 03:24:29 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling</title>
|
|
<link>https://arxiv.org/abs/2506.22049</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22049.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tianhao Chen, Xin Xu, Zijing Liu, Pengxiang Li, Xinyuan Song, Ajay Kumar Jaiswal, Fan Zhang, Jishan Hu, Yang Wang, Hao Chen, Shizhe Diao, Shiwei Liu, Yu Li, Yin Lu, Can Yang</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series, predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While being stable during pretraining and scalable to large model sizes, Pre-LN suffers from an exponential growth in activation variance across layers, causing the residual path to dominate over sub-layer outputs and limiting the learning capacity of deeper layers. To mitigate this issue, we propose Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be used in combination with existing approaches. GPAS works by scaling down the intermediate activations while keeping their gradients unchanged. This leaves information in the activations intact, and avoids the gradient vanishing problem associated with gradient downscaling. Extensive experiments across various model sizes from 71M to 1B show that GPAS achieves consistent performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows promise in improving alternative architectures such as Sandwich-LN and DeepNorm, demonstrating its versatility and potential for improving training dynamics in a wide range of settings.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22049</guid>
|
|
<pubDate>Fri, 27 Jun 2025 09:45:15 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>RetFiner: A Vision-Language Refinement Scheme for Retinal Foundation Models</title>
|
|
<link>https://arxiv.org/abs/2506.22149</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22149.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ronald Fecso, José Morano, Ursula Schmidt-Erfurth, Hrvoje Bogunović</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> The rise of imaging techniques such as optical coherence tomography (OCT) and advances in deep learning (DL) have enabled clinicians and researchers to streamline retinal disease staging. A popular DL approach is self-supervised learning (SSL), where models learn from vast amounts of unlabeled data, avoiding costly annotation. SSL has allowed the development of foundation models (FMs), large models that can be used for a variety of downstream tasks. However, existing FMs for OCT, trained solely on image data, lack a comprehensive and robust semantic understanding of images, as evidenced by their downstream performance (especially for complex tasks), and thus require supervised fine-tuning (which may be unfeasible) to better adapt to specific applications and populations. To address this, we propose RetFiner, an SSL vision-language refinement scheme that improves the representations of existing FMs and enables their efficient and direct adaptation to specific populations for improved downstream performance. Our method uses a diverse set of training objectives which take advantage of the rich supervisory signal found in textual data. We tested RetFiner on the retinal FMs RETFound, UrFound, and VisionFM, showing significant improvements in linear probing performance on seven highly diverse OCT classification tasks, with an average increase of 5.8, 3.9, and 2.1 percentage points over their baselines, respectively. Our code and model weights are publicly available at https://github.com/ronnief1/RetFiner.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22149</guid>
|
|
<pubDate>Fri, 27 Jun 2025 11:53:54 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements</title>
|
|
<link>https://arxiv.org/abs/2506.22419</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22419.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bingchen Zhao, Despoina Magka, Minqi Jiang, Xian Li, Roberta Raileanu, Tatiana Shavrina, Jean-Christophe Gagnon-Audet, Kelvin Niu, Shagun Sodhani, Michael Shvartsman, Andrei Lupu, Alisia Lupidi, Edan Toledo, Karen Hambardzumyan, Martin Josifoski, Thomas Foster, Lucia Cipolina-Kun, Abhishek Charnalia, Derek Dunfield, Alexander H. Miller, Oisin Mac Aodha, Jakob Foerster, Yoram Bachrach</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Rapid advancements in large language models (LLMs) have the potential to assist in scientific progress. A critical capability toward this endeavor is the ability to reproduce existing work. To evaluate the ability of AI agents to reproduce results in an active research area, we introduce the Automated LLM Speedrunning Benchmark, leveraging the research community contributions on the NanoGPT speedrun, a competition to train a GPT-2 model in the shortest time. Each of the 19 speedrun tasks provides the agent with the previous records training script, optionally paired with one of three hint formats, ranging from pseudocode to paper-like descriptions of the new records improvements. Records execute quickly by design and speedrun improvements encompass diverse code-level changes, ranging from high-level algorithmic advancements to hardware-aware optimizations. These features make the benchmark both accessible and realistic for the frontier problem of improving LLM training. We find that recent reasoning LLMs combined with SoTA scaffolds struggle to reimplement already-known innovations in our benchmark, even when given detailed hints. Our benchmark thus provides a simple, non-saturated measure of an LLMs ability to automate scientific reproduction, a necessary (but not sufficient) skill for an autonomous research agent.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22419</guid>
|
|
<pubDate>Fri, 27 Jun 2025 17:44:32 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy</title>
|
|
<link>https://arxiv.org/abs/2506.22432</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22432.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuhao Liu, Tengfei Wang, Fang Liu, Zhenwei Wang, Rynson W. H. Lau</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Recent advances in deep generative modeling have unlocked unprecedented opportunities for video synthesis. In real-world applications, however, users often seek tools to faithfully realize their creative editing intentions with precise and consistent control. Despite the progress achieved by existing methods, ensuring fine-grained alignment with user intentions remains an open and challenging problem. In this work, we present Shape-for-Motion, a novel framework that incorporates a 3D proxy for precise and consistent video editing. Shape-for-Motion achieves this by converting the target object in the input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be performed directly on the proxy and then inferred back to the video frames. To simplify the editing process, we design a novel Dual-Propagation Strategy that allows users to perform edits on the 3D mesh of a single frame, and the edits are then automatically propagated to the 3D meshes of the other frames. The 3D meshes for different frames are further projected onto the 2D space to produce the edited geometry and texture renderings, which serve as inputs to a decoupled video diffusion model for generating edited results. Our framework supports various precise and physically-consistent manipulations across the video frames, including pose editing, rotation, scaling, translation, texture modification, and object composition. Our approach marks a key step toward high-quality, controllable video editing workflows. Extensive experiments demonstrate the superiority and effectiveness of our approach. Project page: https://shapeformotion.github.io/</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22432</guid>
|
|
<pubDate>Fri, 27 Jun 2025 17:59:01 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>MiCo: Multi-image Contrast for Reinforcement Visual Reasoning</title>
|
|
<link>https://arxiv.org/abs/2506.22434</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22434.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xi Chen, Mingkang Zhu, Shaoteng Liu, Xiaoyang Wu, Xiaogang Xu, Yu Liu, Xiang Bai, Hengshuang Zhao</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> This work explores enabling Chain-of-Thought (CoT) reasoning to link visual cues across multiple images. A straightforward solution is to adapt rule-based reinforcement learning for Vision-Language Models (VLMs). However, such methods typically rely on manually curated question-answer pairs, which can be particularly challenging when dealing with fine grained visual details and complex logic across images. Inspired by self-supervised visual representation learning, we observe that images contain inherent constraints that can serve as supervision. Based on this insight, we construct image triplets comprising two augmented views of the same image and a third, similar but distinct image. During training, the model is prompted to generate a reasoning process to compare these images (i.e., determine same or different). Then we optimize the model with rule-based reinforcement learning. Due to the high visual similarity and the presence of augmentations, the model must attend to subtle visual changes and perform logical reasoning to succeed. Experiments show that, although trained solely on visual comparison tasks, the learned reasoning ability generalizes effectively to a wide range of questions. Without relying on any human-annotated question-answer pairs, our method achieves significant improvements on multi-image reasoning benchmarks and shows strong performance on general vision tasks.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22434</guid>
|
|
<pubDate>Fri, 27 Jun 2025 17:59:27 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>RExBench: Can coding agents autonomously implement AI research extensions?</title>
|
|
<link>https://arxiv.org/abs/2506.22598</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22598.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nicholas Edwards, Yukyung Lee, Yujun, Mao, Yulu Qin, Sebastian Schuster, Najoung Kim</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Agents based on Large Language Models (LLMs) have shown promise for performing sophisticated software engineering tasks autonomously. In addition, there has been progress towards developing agents that can perform parts of the research pipeline in machine learning and the natural sciences. We argue that research extension and its implementation is a critical capability for such systems, and introduce RExBench to support the evaluation of this capability. RExBench is a benchmark consisting of 12 realistic research experiment implementation tasks that aim to investigate research hypotheses that have not previously been implemented. Each task is set up as an extension to an existing research paper and codebase, accompanied by domain expert-written instructions. RExBench is robust to data contamination, and supports an automatic evaluation infrastructure that executes agent outputs to determine whether the success criteria are met. We use this benchmark to evaluate nine LLM agents implemented using three different frameworks: aider, Claude Code, and OpenHands. We find that all agents evaluated fail to autonomously implement the majority of the extensions. Although the success rate improves with additional human-written hints, the best performance under this setting remains below 40%. This indicates that current agents are still short of being able to handle realistic research extension tasks without substantial human guidance.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22598</guid>
|
|
<pubDate>Fri, 27 Jun 2025 19:41:41 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs</title>
|
|
<link>https://arxiv.org/abs/2506.22694</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22694.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani, Junyoung Park, Yifan Zao, He Zhang, Tian Liu, Yiping Yang, Xin Yuan, Jiuyan Lu, Chris Lott, Mingu Lee</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> In this paper, we introduce a simple training-free technique to improve the performance of drafter-based speculative decoding (SpD) methods that incorporates language modeling head (LM head) during drafting process. A drafter-based speculative decoding leverages one or more smaller language models, a.k.a. drafters or draft models, to sample a draft sequence or tree consisting of multiple tokens, followed by verification by a base LLM, a target model, accepting a subset as its valid generation. As it is usually considered that the speculative decoding requires one-to-one mapping between vocabularies of the target model and the draft model, it has been natural to share the vocabulary between them, or even share the LM head as in EAGLE or Medusa. We first identify that this draft token sampling scheme inherently contains an unnecessary inference overhead in drafting, especially for some target LLMs with very large vocabularies. Then, we propose a simple technique, VocabTrim, to mitigate the drafting overhead to improve the generation speed in memory-bound environment. VocabTrim reconstructs the drafter LM head to contain only a limited set of tokens, selected by the most frequently sampled from the vocabulary of the target model. While limiting the vocabulary in drafting slightly degrades the acceptance rate, it significantly reduces the drafting latency in memory-bound process which is often the case on edge devices, resulting in higher memory-bound speed up (MBSU). We show that our method can boost the memory-bound speed-up for Llama-3 models on Spec-Bench, specifically by 16% for Llama-3.2-3B-Instruct.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22694</guid>
|
|
<pubDate>Sat, 28 Jun 2025 00:26:40 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography</title>
|
|
<link>https://arxiv.org/abs/2506.22753</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22753.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jianing Zhang, Jiayi Zhu, Feiyu Ji, Xiaokang Yang, Xiaoyun Yuan</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Metalenses offer significant potential for ultra-compact computational imaging but face challenges from complex optical degradation and computational restoration difficulties. Existing methods typically rely on precise optical calibration or massive paired datasets, which are non-trivial for real-world imaging systems. Furthermore, a lack of control over the inference process often results in undesirable hallucinated artifacts. We introduce Degradation-Modeled Multipath Diffusion for tunable metalens photography, leveraging powerful natural image priors from pretrained models instead of large datasets. Our framework uses positive, neutral, and negative-prompt paths to balance high-frequency detail generation, structural fidelity, and suppression of metalens-specific degradation, alongside pseudo data augmentation. A tunable decoder enables controlled trade-offs between fidelity and perceptual quality. Additionally, a spatially varying degradation-aware attention (SVDA) module adaptively models complex optical and sensor-induced degradation. Finally, we design and build a millimeter-scale MetaCamera for real-world validation. Extensive results show that our approach outperforms state-of-the-art methods, achieving high-fidelity and sharp image reconstruction. More materials: https://dmdiff.github.io/.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22753</guid>
|
|
<pubDate>Sat, 28 Jun 2025 04:48:37 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Jan-nano Technical Report</title>
|
|
<link>https://arxiv.org/abs/2506.22760</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22760.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Alan Dao, Dinh Bach Vu</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Most language models face a fundamental tradeoff where powerful capabilities require substantial computational resources. We shatter this constraint with Jan-nano, a 4B parameter language model that redefines efficiency through radical specialization: instead of trying to know everything, it masters the art of finding anything instantly. Fine-tuned from Qwen3-4B using our novel multi-stage RLVR system that completely eliminates reliance on next token prediction training (SFT), Jan-nano achieves 83.2% on SimpleQA benchmark with MCP integration while running on consumer hardware. With 128K context length, Jan-nano proves that intelligence isn't about scale, it's about strategy.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22760</guid>
|
|
<pubDate>Sat, 28 Jun 2025 05:44:57 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Models</title>
|
|
<link>https://arxiv.org/abs/2506.22813</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22813.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhuojun Ding, Wei Wei, Chenghao Fan</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Supervised fine-tuning (SFT) is widely used to align large language models (LLMs) with information extraction (IE) tasks, such as named entity recognition (NER). However, annotating such fine-grained labels and training domain-specific models is costly. Existing works typically train a unified model across multiple domains, but such approaches lack adaptation and scalability since not all training data benefits target domains and scaling trained models remains challenging. We propose the SaM framework, which dynamically Selects and Merges expert models at inference time. Specifically, for a target domain, we select domain-specific experts pre-trained on existing domains based on (i) domain similarity to the target domain and (ii) performance on sampled instances, respectively. The experts are then merged to create task-specific models optimized for the target domain. By dynamically merging experts beneficial to target domains, we improve generalization across various domains without extra training. Additionally, experts can be added or removed conveniently, leading to great scalability. Extensive experiments on multiple benchmarks demonstrate our framework's effectiveness, which outperforms the unified model by an average of 10%. We further provide insights into potential improvements, practical experience, and extensions of our framework.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22813</guid>
|
|
<pubDate>Sat, 28 Jun 2025 08:28:52 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Listener-Rewarded Thinking in VLMs for Image Preferences</title>
|
|
<link>https://arxiv.org/abs/2506.22832</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22832.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Alexander Gambashidze, Li Pengyi, Matvey Skripkin, Andrey Galichin, Anton Gusarov, Konstantin Sobolev, Andrey Kuznetsov, Ivan Oseledets</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> Training robust and generalizable reward models for human visual preferences is essential for aligning text-to-image and text-to-video generative models with human intent. However, current reward models often fail to generalize, and supervised fine-tuning leads to memorization, demanding complex annotation pipelines. While reinforcement learning (RL), specifically Group Relative Policy Optimization (GRPO), improves generalization, we uncover a key failure mode: a significant drop in reasoning accuracy occurs when a model's reasoning trace contradicts that of an independent, frozen vision-language model ("listener") evaluating the same output. To address this, we introduce a listener-augmented GRPO framework. Here, the listener re-evaluates the reasoner's chain-of-thought to provide a dense, calibrated confidence score, shaping the RL reward signal. This encourages the reasoner not only to answer correctly, but to produce explanations that are persuasive to an independent model. Our listener-shaped reward scheme achieves best accuracy on the ImageReward benchmark (67.4%), significantly improves out-of-distribution (OOD) performance on a large-scale human preference dataset (1.2M votes, up to +6% over naive reasoner), and reduces reasoning contradictions compared to strong GRPO and SFT baselines. These results demonstrate that listener-based rewards provide a scalable, data-efficient path to aligning vision-language models with nuanced human preferences. We will release our reasoning model here: https://huggingface.co/alexgambashidze/qwen2.5vl_image_preference_reasoner.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22832</guid>
|
|
<pubDate>Sat, 28 Jun 2025 09:53:17 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>STR-Match: Matching SpatioTemporal Relevance Score for Training-Free Video Editing</title>
|
|
<link>https://arxiv.org/abs/2506.22868</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22868.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junsung Lee, Junoh Kang, Bohyung Han</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Previous text-guided video editing methods often suffer from temporal inconsistency, motion distortion, and-most notably-limited domain transformation. We attribute these limitations to insufficient modeling of spatiotemporal pixel relevance during the editing process. To address this, we propose STR-Match, a training-free video editing algorithm that produces visually appealing and spatiotemporally coherent videos through latent optimization guided by our novel STR score. The score captures spatiotemporal pixel relevance across adjacent frames by leveraging 2D spatial attention and 1D temporal modules in text-to-video (T2V) diffusion models, without the overhead of computationally expensive 3D attention mechanisms. Integrated into a latent optimization framework with a latent mask, STR-Match generates temporally consistent and visually faithful videos, maintaining strong performance even under significant domain transformations while preserving key visual attributes of the source. Extensive experiments demonstrate that STR-Match consistently outperforms existing methods in both visual quality and spatiotemporal consistency.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22868</guid>
|
|
<pubDate>Sat, 28 Jun 2025 12:36:19 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Peccavi: Visual Paraphrase Attack Safe and Distortion Free Image Watermarking Technique for AI-Generated Images</title>
|
|
<link>https://arxiv.org/abs/2506.22960</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22960.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shreyas Dixit, Ashhar Aziz, Shashwat Bajpai, Vasu Sharma, Aman Chadha, Vinija Jain, Amitava Das</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> A report by the European Union Law Enforcement Agency predicts that by 2026, up to 90 percent of online content could be synthetically generated, raising concerns among policymakers, who cautioned that "Generative AI could act as a force multiplier for political disinformation. The combined effect of generative text, images, videos, and audio may surpass the influence of any single modality." In response, California's Bill AB 3211 mandates the watermarking of AI-generated images, videos, and audio. However, concerns remain regarding the vulnerability of invisible watermarking techniques to tampering and the potential for malicious actors to bypass them entirely. Generative AI-powered de-watermarking attacks, especially the newly introduced visual paraphrase attack, have shown an ability to fully remove watermarks, resulting in a paraphrase of the original image. This paper introduces PECCAVI, the first visual paraphrase attack-safe and distortion-free image watermarking technique. In visual paraphrase attacks, an image is altered while preserving its core semantic regions, termed Non-Melting Points (NMPs). PECCAVI strategically embeds watermarks within these NMPs and employs multi-channel frequency domain watermarking. It also incorporates noisy burnishing to counter reverse-engineering efforts aimed at locating NMPs to disrupt the embedded watermark, thereby enhancing durability. PECCAVI is model-agnostic. All relevant resources and codes will be open-sourced.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22960</guid>
|
|
<pubDate>Sat, 28 Jun 2025 17:34:08 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Confident Splatting: Confidence-Based Compression of 3D Gaussian Splatting via Learnable Beta Distributions</title>
|
|
<link>https://arxiv.org/abs/2506.22973</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22973.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> AmirHossein Naghi Razlighi, Elaheh Badali Golezani, Shohreh Kasaei</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> 3D Gaussian Splatting enables high-quality real-time rendering but often produces millions of splats, resulting in excessive storage and computational overhead. We propose a novel lossy compression method based on learnable confidence scores modeled as Beta distributions. Each splat's confidence is optimized through reconstruction-aware losses, enabling pruning of low-confidence splats while preserving visual fidelity. The proposed approach is architecture-agnostic and can be applied to any Gaussian Splatting variant. In addition, the average confidence values serve as a new metric to assess the quality of the scene. Extensive experiments demonstrate favorable trade-offs between compression and fidelity compared to prior work. Our code and data are publicly available at https://github.com/amirhossein-razlighi/Confident-Splatting</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22973</guid>
|
|
<pubDate>Sat, 28 Jun 2025 18:11:30 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning</title>
|
|
<link>https://arxiv.org/abs/2506.22992</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22992.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yulun Jiang, Yekun Chai, Maria Brbić, Michael Moor</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> The ability to process information from multiple modalities and to reason through it step-by-step remains a critical challenge in advancing artificial intelligence. However, existing reasoning benchmarks focus on text-only reasoning, or employ multimodal questions that can be answered by directly retrieving information from a non-text modality. Thus, complex reasoning remains poorly understood in multimodal domains. Here, we present MARBLE, a challenging multimodal reasoning benchmark that is designed to scrutinize multimodal language models (MLLMs) in their ability to carefully reason step-by-step through complex multimodal problems and environments. MARBLE is composed of two highly challenging tasks, M-Portal and M-Cube, that require the crafting and understanding of multistep plans under spatial, visual, and physical constraints. We find that current MLLMs perform poorly on MARBLE -- all the 12 advanced models obtain near-random performance on M-Portal and 0% accuracy on M-Cube. Only in simplified subtasks some models outperform the random baseline, indicating that complex reasoning is still a challenge for existing MLLMs. Moreover, we show that perception remains a bottleneck, where MLLMs occasionally fail to extract information from the visual inputs. By shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the development of the next generation of models with the ability to reason and plan across many, multimodal reasoning steps.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.22992</guid>
|
|
<pubDate>Sat, 28 Jun 2025 19:44:32 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models</title>
|
|
<link>https://arxiv.org/abs/2506.23009</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23009.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Ruiyi Zhang, Changyou Chen</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) have achieved remarkable visual reasoning abilities in natural images, text-rich documents, and graphic designs. However, their ability to interpret music sheets remains underexplored. To bridge this gap, we introduce MusiXQA, the first comprehensive dataset for evaluating and advancing MLLMs in music sheet understanding. MusiXQA features high-quality synthetic music sheets generated via MusiXTeX, with structured annotations covering note pitch and duration, chords, clefs, key/time signatures, and text, enabling diverse visual QA tasks. Through extensive evaluations, we reveal significant limitations of current state-of-the-art MLLMs in this domain. Beyond benchmarking, we developed Phi-3-MusiX, an MLLM fine-tuned on our dataset, achieving significant performance gains over GPT-based methods. The proposed dataset and model establish a foundation for future advances in MLLMs for music sheet understanding. Code, data, and model will be released upon acceptance.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23009</guid>
|
|
<pubDate>Sat, 28 Jun 2025 20:46:47 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Ovis-U1 Technical Report</title>
|
|
<link>https://arxiv.org/abs/2506.23044</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23044.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, Yang Li, Qing-Guo Chen</p><p><b>Upvotes:</b> 58</p><p><b>Summary:</b> In this report, we introduce Ovis-U1, a 3-billion-parameter unified model that integrates multimodal understanding, text-to-image generation, and image editing capabilities. Building on the foundation of the Ovis series, Ovis-U1 incorporates a diffusion-based visual decoder paired with a bidirectional token refiner, enabling image generation tasks comparable to leading models like GPT-4o. Unlike some previous models that use a frozen MLLM for generation tasks, Ovis-U1 utilizes a new unified training approach starting from a language model. Compared to training solely on understanding or generation tasks, unified training yields better performance, demonstrating the enhancement achieved by integrating these two tasks. Ovis-U1 achieves a score of 69.6 on the OpenCompass Multi-modal Academic Benchmark, surpassing recent state-of-the-art models such as Ristretto-3B and SAIL-VL-1.5-2B. In text-to-image generation, it excels with scores of 83.72 and 0.89 on the DPG-Bench and GenEval benchmarks, respectively. For image editing, it achieves 4.00 and 6.42 on the ImgEdit-Bench and GEdit-Bench-EN, respectively. As the initial version of the Ovis unified model series, Ovis-U1 pushes the boundaries of multimodal understanding, generation, and editing.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23044</guid>
|
|
<pubDate>Sun, 29 Jun 2025 00:40:17 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings</title>
|
|
<link>https://arxiv.org/abs/2506.23115</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23115.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haonan Chen, Hong Liu, Yuping Luo, Liang Wang, Nan Yang, Furu Wei, Zhicheng Dou</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> Multimodal embedding models, built upon causal Vision Language Models (VLMs), have shown promise in various tasks. However, current approaches face three key limitations: the use of causal attention in VLM backbones is suboptimal for embedding tasks; scalability issues due to reliance on high-quality labeled paired data for contrastive learning; and limited diversity in training objectives and data. To address these issues, we propose MoCa, a two-stage framework for transforming pre-trained VLMs into effective bidirectional multimodal embedding models. The first stage, Modality-aware Continual Pre-training, introduces a joint reconstruction objective that simultaneously denoises interleaved text and image inputs, enhancing bidirectional context-aware reasoning. The second stage, Heterogeneous Contrastive Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple image-caption pairs to enhance generalization and alignment. Our method addresses the stated limitations by introducing bidirectional attention through continual pre-training, scaling effectively with massive unlabeled datasets via joint reconstruction objectives, and utilizing diverse multimodal data for enhanced representation robustness. Experiments demonstrate that MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results, and exhibits strong scalability with both model size and training data on MMEB.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23115</guid>
|
|
<pubDate>Sun, 29 Jun 2025 06:41:00 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentation</title>
|
|
<link>https://arxiv.org/abs/2506.23121</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23121.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xinlei Yu, Chanmiao Wang, Hui Jin, Ahmed Elazab, Gangyong Jia, Xiang Wan, Changqing Zou, Ruiquan Ge</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Multi-organ medical segmentation is a crucial component of medical image processing, essential for doctors to make accurate diagnoses and develop effective treatment plans. Despite significant progress in this field, current multi-organ segmentation models often suffer from inaccurate details, dependence on geometric prompts and loss of spatial information. Addressing these challenges, we introduce a novel model named CRISP-SAM2 with CRoss-modal Interaction and Semantic Prompting based on SAM2. This model represents a promising approach to multi-organ medical segmentation guided by textual descriptions of organs. Our method begins by converting visual and textual inputs into cross-modal contextualized semantics using a progressive cross-attention interaction mechanism. These semantics are then injected into the image encoder to enhance the detailed understanding of visual information. To eliminate reliance on geometric prompts, we use a semantic prompting strategy, replacing the original prompt encoder to sharpen the perception of challenging targets. In addition, a similarity-sorting self-updating strategy for memory and a mask-refining process is applied to further adapt to medical imaging and enhance localized details. Comparative experiments conducted on seven public datasets indicate that CRISP-SAM2 outperforms existing models. Extensive analysis also demonstrates the effectiveness of our method, thereby confirming its superior performance, especially in addressing the limitations mentioned earlier. Our code is available at: https://github.com/YU-deep/CRISP\_SAM2.git.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23121</guid>
|
|
<pubDate>Sun, 29 Jun 2025 07:05:27 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>RoboScape: Physics-informed Embodied World Model</title>
|
|
<link>https://arxiv.org/abs/2506.23135</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23135.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yu Shang, Xin Zhang, Yinzhou Tang, Lei Jin, Chen Gao, Wei Wu, Yong Li</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> World models have become indispensable tools for embodied intelligence, serving as powerful simulators capable of generating realistic robotic videos while addressing critical data scarcity challenges. However, current embodied world models exhibit limited physical awareness, particularly in modeling 3D geometry and motion dynamics, resulting in unrealistic video generation for contact-rich robotic scenarios. In this paper, we present RoboScape, a unified physics-informed world model that jointly learns RGB video generation and physics knowledge within an integrated framework. We introduce two key physics-informed joint training tasks: temporal depth prediction that enhances 3D geometric consistency in video rendering, and keypoint dynamics learning that implicitly encodes physical properties (e.g., object shape and material characteristics) while improving complex motion modeling. Extensive experiments demonstrate that RoboScape generates videos with superior visual fidelity and physical plausibility across diverse robotic scenarios. We further validate its practical utility through downstream applications including robotic policy training with generated data and policy evaluation. Our work provides new insights for building efficient physics-informed world models to advance embodied intelligence research. The code is available at: https://github.com/tsinghua-fib-lab/RoboScape.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23135</guid>
|
|
<pubDate>Sun, 29 Jun 2025 08:19:45 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame Optical Flow Estimation</title>
|
|
<link>https://arxiv.org/abs/2506.23151</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23151.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Vladislav Bargatin, Egor Chistov, Alexander Yakovenko, Dmitriy Vatolin</p><p><b>Upvotes:</b> 23</p><p><b>Summary:</b> Recent advances in optical flow estimation have prioritized accuracy at the cost of growing GPU memory consumption, particularly for high-resolution (FullHD) inputs. We introduce MEMFOF, a memory-efficient multi-frame optical flow method that identifies a favorable trade-off between multi-frame estimation and GPU memory usage. Notably, MEMFOF requires only 2.09 GB of GPU memory at runtime for 1080p inputs, and 28.5 GB during training, which uniquely positions our method to be trained at native 1080p without the need for cropping or downsampling. We systematically revisit design choices from RAFT-like architectures, integrating reduced correlation volumes and high-resolution training protocols alongside multi-frame estimation, to achieve state-of-the-art performance across multiple benchmarks while substantially reducing memory overhead. Our method outperforms more resource-intensive alternatives in both accuracy and runtime efficiency, validating its robustness for flow estimation at high resolutions. At the time of submission, our method ranks first on the Spring benchmark with a 1-pixel (1px) outlier rate of 3.289, leads Sintel (clean) with an endpoint error (EPE) of 0.963, and achieves the best Fl-all error on KITTI-2015 at 2.94%. The code is available at https://github.com/msu-video-group/memfof.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23151</guid>
|
|
<pubDate>Sun, 29 Jun 2025 09:01:42 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding</title>
|
|
<link>https://arxiv.org/abs/2506.23219</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23219.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jie Feng, Shengyuan Wang, Tianhui Liu, Yanxin Xi, Yong Li</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Urban research involves a wide range of scenarios and tasks that require the understanding of multi-modal data. Current methods often focus on specific data types and lack a unified framework in urban field for processing them comprehensively. The recent success of multi-modal large language models (MLLMs) presents a promising opportunity to overcome this limitation. In this paper, we introduce UrbanLLaVA, a multi-modal large language model designed to process these four types of data simultaneously and achieve strong performance across diverse urban tasks compared with general MLLMs. In UrbanLLaVA, we first curate a diverse urban instruction dataset encompassing both single-modal and cross-modal urban data, spanning from location view to global view of urban environment. Additionally, we propose a multi-stage training framework that decouples spatial reasoning enhancement from domain knowledge learning, thereby improving the compatibility and downstream performance of UrbanLLaVA across diverse urban tasks. Finally, we also extend existing benchmark for urban research to assess the performance of MLLMs across a wide range of urban tasks. Experimental results from three cities demonstrate that UrbanLLaVA outperforms open-source and proprietary MLLMs in both single-modal tasks and complex cross-modal tasks and shows robust generalization abilities across cities. Source codes and data are openly accessible to the research community via https://github.com/tsinghua-fib-lab/UrbanLLaVA.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23219</guid>
|
|
<pubDate>Sun, 29 Jun 2025 13:04:27 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering</title>
|
|
<link>https://arxiv.org/abs/2506.23329</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23329.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Parker Liu, Chenxin Li, Zhengxin Li, Yipeng Wu, Wuyang Li, Zhiqin Yang, Zhenyuan Zhang, Yunlong Lin, Sirui Han, Brandon Y. Feng</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Vision-language models (VLMs) excel at descriptive tasks, but whether they truly understand scenes from visual observations remains uncertain. We introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding through active creation rather than passive recognition. Grounded in the analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs) with actively using programming and rendering tools to recreate the underlying 3D structure of an input image, achieving agentic inverse rendering through tool use. This "understanding-by-creating" approach probes the tool-using generative capacity of VLAs, moving beyond the descriptive or conversational capacity measured by traditional scene understanding benchmarks. We provide a comprehensive suite of metrics to evaluate geometric accuracy, spatial relations, appearance attributes, and overall plausibility. Initial experiments on agentic inverse rendering powered by various state-of-the-art VLMs highlight current limitations, particularly in visual precision rather than basic tool usage. IR3D-Bench, including data and evaluation protocols, is released to facilitate systematic study and development of tool-using VLAs towards genuine scene understanding by creating.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23329</guid>
|
|
<pubDate>Sun, 29 Jun 2025 17:02:57 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Teaching a Language Model to Speak the Language of Tools</title>
|
|
<link>https://arxiv.org/abs/2506.23394</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23394.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Simeon Emanuilov</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> External tool integration through function-calling is essential for practical language model applications, yet most multilingual models lack reliable tool-use capabilities in non-English languages. Even state-of-the-art multilingual models struggle with determining when to use tools and generating the structured outputs required for function calls, often exhibiting language confusion when prompted in lower-resource languages. This work presents a methodology for adapting existing language models to enable robust tool use in any target language, using Bulgarian as a case study. The approach involves continued training of the BgGPT model series (2.6B, 9B, 27B parameters) on a novel bilingual dataset of 10,035 function-calling examples designed to support standardized protocols like MCP (Model Context Protocol). The research introduces TUCAN (Tool-Using Capable Assistant Navigator), which achieves up to 28.75% improvement in function-calling accuracy over base models while preserving core language understanding, as verified on established Bulgarian benchmarks. Beyond accuracy gains, TUCAN models demonstrate production-ready response formatting with clean, parsable function calls, contrasting with the verbose and inconsistent outputs of base models. The models, evaluation framework, and dataset are released to enable replication for other languages. This work demonstrates a practical approach for extending tool-augmented capabilities beyond English-centric systems.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23394</guid>
|
|
<pubDate>Sun, 29 Jun 2025 20:47:27 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Consistent Time-of-Flight Depth Denoising via Graph-Informed Geometric Attention</title>
|
|
<link>https://arxiv.org/abs/2506.23542</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23542.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Weida Wang, Changyong He, Jin Zeng, Di Qiu</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Depth images captured by Time-of-Flight (ToF) sensors are prone to noise, requiring denoising for reliable downstream applications. Previous works either focus on single-frame processing, or perform multi-frame processing without considering depth variations at corresponding pixels across frames, leading to undesirable temporal inconsistency and spatial ambiguity. In this paper, we propose a novel ToF depth denoising network leveraging motion-invariant graph fusion to simultaneously enhance temporal stability and spatial sharpness. Specifically, despite depth shifts across frames, graph structures exhibit temporal self-similarity, enabling cross-frame geometric attention for graph fusion. Then, by incorporating an image smoothness prior on the fused graph and data fidelity term derived from ToF noise distribution, we formulate a maximum a posterior problem for ToF denoising. Finally, the solution is unrolled into iterative filters whose weights are adaptively learned from the graph-informed geometric attention, producing a high-performance yet interpretable network. Experimental results demonstrate that the proposed scheme achieves state-of-the-art performance in terms of accuracy and consistency on synthetic DVToF dataset and exhibits robust generalization on the real Kinectv2 dataset. Source code will be released at https://github.com/davidweidawang/GIGA-ToF{https://github.com/davidweidawang/GIGA-ToF}.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23542</guid>
|
|
<pubDate>Mon, 30 Jun 2025 06:29:24 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching</title>
|
|
<link>https://arxiv.org/abs/2506.23552</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23552.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mingi Kwon, Joonghyuk Shin, Jaeseok Jung, Jaesik Park, Youngjung Uh</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> The intrinsic link between facial motion and speech is often overlooked in generative modeling, where talking head synthesis and text-to-speech (TTS) are typically addressed as separate tasks. This paper introduces JAM-Flow, a unified framework to simultaneously synthesize and condition on both facial motion and speech. Our approach leverages flow matching and a novel Multi-Modal Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT and Audio-DiT modules. These are coupled via selective joint attention layers and incorporate key architectural choices, such as temporally aligned positional embeddings and localized joint attention masking, to enable effective cross-modal interaction while preserving modality-specific strengths. Trained with an inpainting-style objective, JAM-Flow supports a wide array of conditioning inputs-including text, reference audio, and reference motion-facilitating tasks such as synchronized talking head generation from text, audio-driven animation, and much more, within a single, coherent model. JAM-Flow significantly advances multi-modal generative modeling by providing a practical solution for holistic audio-visual synthesis. project page: https://joonghyuk.com/jamflow-web</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23552</guid>
|
|
<pubDate>Mon, 30 Jun 2025 06:51:40 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>VMoBA: Mixture-of-Block Attention for Video Diffusion Models</title>
|
|
<link>https://arxiv.org/abs/2506.23858</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23858.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jianzong Wu, Liang Hou, Haotian Yang, Xin Tao, Ye Tian, Pengfei Wan, Di Zhang, Yunhai Tong</p><p><b>Upvotes:</b> 31</p><p><b>Summary:</b> The quadratic complexity of full attention mechanisms poses a significant bottleneck for Video Diffusion Models (VDMs) aiming to generate long-duration, high-resolution videos. While various sparse attention methods have been proposed, many are designed as training-free inference accelerators or do not optimally capture the unique spatio-temporal characteristics inherent in video data when trained natively. This paper introduces Video Mixture of Block Attention (VMoBA), a novel sparse attention mechanism specifically adapted for VDMs. Motivated by an in-depth analysis of attention patterns within pre-trained video transformers, which revealed strong spatio-temporal locality, varying query importance, and head-specific concentration levels, VMoBA enhances the original MoBA framework with three key modifications: (1) a layer-wise recurrent block partition scheme (1D-2D-3D) to dynamically adapt to diverse spatio-temporal attention patterns and improve efficiency; (2) global block selection to prioritize the most salient query-key block interactions across an entire attention head; and (3) threshold-based block selection to dynamically determine the number of attended blocks based on their cumulative similarity. Extensive experiments demonstrate that VMoBA significantly accelerates the training of VDMs on longer sequences, achieving 2.92x FLOPs and 1.48x latency speedup, while attaining comparable or even superior generation quality to full attention. Furthermore, VMoBA exhibits competitive performance in training-free inference, offering 2.40x FLOPs and 1.35x latency speedup for high-res video generation.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23858</guid>
|
|
<pubDate>Mon, 30 Jun 2025 13:52:31 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers</title>
|
|
<link>https://arxiv.org/abs/2506.23918</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23918.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, Linjie Li, Yu Cheng, Heng Ji, Junxian He, Yi R. Fung</p><p><b>Upvotes:</b> 55</p><p><b>Summary:</b> Recent progress in multimodal reasoning has been significantly advanced by textual Chain-of-Thought (CoT), a paradigm where models conduct reasoning within language. This text-centric approach, however, treats vision as a static, initial context, creating a fundamental "semantic gap" between rich perceptual data and discrete symbolic thought. Human cognition often transcends language, utilizing vision as a dynamic mental sketchpad. A similar evolution is now unfolding in AI, marking a fundamental paradigm shift from models that merely think about images to those that can truly think with images. This emerging paradigm is characterized by models leveraging visual information as intermediate steps in their thought process, transforming vision from a passive input into a dynamic, manipulable cognitive workspace. In this survey, we chart this evolution of intelligence along a trajectory of increasing cognitive autonomy, which unfolds across three key stages: from external tool exploration, through programmatic manipulation, to intrinsic imagination. To structure this rapidly evolving field, our survey makes four key contributions. (1) We establish the foundational principles of the think with image paradigm and its three-stage framework. (2) We provide a comprehensive review of the core methods that characterize each stage of this roadmap. (3) We analyze the critical landscape of evaluation benchmarks and transformative applications. (4) We identify significant challenges and outline promising future directions. By providing this structured overview, we aim to offer a clear roadmap for future research towards more powerful and human-aligned multimodal AI.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.23918</guid>
|
|
<pubDate>Mon, 30 Jun 2025 14:48:35 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Ella: Embodied Social Agents with Lifelong Memory</title>
|
|
<link>https://arxiv.org/abs/2506.24019</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.24019.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongxin Zhang, Zheyuan Zhang, Zeyuan Wang, Zunzhe Zhang, Lixing Fang, Qinhong Zhou, Chuang Gan</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> We introduce Ella, an embodied social agent capable of lifelong learning within a community in a 3D open world, where agents accumulate experiences and acquire knowledge through everyday visual observations and social interactions. At the core of Ella's capabilities is a structured, long-term multimodal memory system that stores, updates, and retrieves information effectively. It consists of a name-centric semantic memory for organizing acquired knowledge and a spatiotemporal episodic memory for capturing multimodal experiences. By integrating this lifelong memory system with foundation models, Ella retrieves relevant information for decision-making, plans daily activities, builds social relationships, and evolves autonomously while coexisting with other intelligent beings in the open world. We conduct capability-oriented evaluations in a dynamic 3D open world where 15 agents engage in social activities for days and are assessed with a suite of unseen controlled evaluations. Experimental results show that Ella can influence, lead, and cooperate with other agents well to achieve goals, showcasing its ability to learn effectively through observation and social interaction. Our findings highlight the transformative potential of combining structured memory systems with foundation models for advancing embodied intelligence. More videos can be found at https://umass-embodied-agi.github.io/Ella/.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.24019</guid>
|
|
<pubDate>Mon, 30 Jun 2025 16:22:51 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning</title>
|
|
<link>https://arxiv.org/abs/2506.24119</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.24119.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, Natasha Jaques</p><p><b>Upvotes:</b> 42</p><p><b>Summary:</b> Recent advances in reinforcement learning have shown that language models can develop sophisticated reasoning through training on tasks with verifiable rewards, but these approaches depend on human-curated problem-answer pairs and domain-specific reward engineering. We introduce SPIRAL, a self-play framework where models learn by playing multi-turn, zero-sum games against continuously improving versions of themselves, eliminating the need for human supervision. Through self-play, SPIRAL generates an infinite curriculum of progressively challenging problems as models must constantly adapt to stronger opponents. To enable this self-play training at scale, We implement a fully online, multi-turn, multi-agent reinforcement learning system for LLMs and propose role-conditioned advantage estimation (RAE) to stabilize multi-agent training. Using SPIRAL, self-play on zero-sum games produces reasoning capabilities that transfer broadly. Training Qwen3-4B-Base on Kuhn Poker alone achieves 8.6% improvement on math and 8.4% on general reasoning, outperforming SFT on 25,000 expert game trajectories. Analysis reveals that this transfer occurs through three cognitive patterns: systematic decomposition, expected value calculation, and case-by-case analysis. Multi-game training (TicTacToe, Kuhn Poker, Simple Negotiation) further enhances performance as each game develops distinct reasoning strengths. Applying SPIRAL to a strong reasoning model (DeepSeek-R1-Distill-Qwen-7B) can still lead to 2.0% average improvement. These results demonstrate that zero-sum games naturally develop transferable reasoning capabilities, highlighting a promising direction for autonomous reasoning development.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.24119</guid>
|
|
<pubDate>Mon, 30 Jun 2025 17:58:13 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Calligrapher: Freestyle Text Image Customization</title>
|
|
<link>https://arxiv.org/abs/2506.24123</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.24123.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yue Ma, Qingyan Bai, Hao Ouyang, Ka Leong Cheng, Qiuyu Wang, Hongyu Liu, Zichen Liu, Haofan Wang, Jingye Chen, Yujun Shen, Qifeng Chen</p><p><b>Upvotes:</b> 31</p><p><b>Summary:</b> We introduce Calligrapher, a novel diffusion-based framework that innovatively integrates advanced text customization with artistic typography for digital calligraphy and design applications. Addressing the challenges of precise style control and data dependency in typographic customization, our framework incorporates three key technical contributions. First, we develop a self-distillation mechanism that leverages the pre-trained text-to-image generative model itself alongside the large language model to automatically construct a style-centric typography benchmark. Second, we introduce a localized style injection framework via a trainable style encoder, which comprises both Qformer and linear layers, to extract robust style features from reference images. An in-context generation mechanism is also employed to directly embed reference images into the denoising process, further enhancing the refined alignment of target styles. Extensive quantitative and qualitative evaluations across diverse fonts and design contexts confirm Calligrapher's accurate reproduction of intricate stylistic details and precise glyph positioning. By automating high-quality, visually consistent typography, Calligrapher surpasses traditional models, empowering creative practitioners in digital art, branding, and contextual typographic design.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2506.24123</guid>
|
|
<pubDate>Mon, 30 Jun 2025 17:59:06 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion</title>
|
|
<link>https://arxiv.org/abs/2507.00162</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00162.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yu Lu, Yi Yang</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Recent advances in video generation models have enabled high-quality short video generation from text prompts. However, extending these models to longer videos remains a significant challenge, primarily due to degraded temporal consistency and visual fidelity. Our preliminary observations show that naively applying short-video generation models to longer sequences leads to noticeable quality degradation. Further analysis identifies a systematic trend where high-frequency components become increasingly distorted as video length grows, an issue we term high-frequency distortion. To address this, we propose FreeLong, a training-free framework designed to balance the frequency distribution of long video features during the denoising process. FreeLong achieves this by blending global low-frequency features, which capture holistic semantics across the full video, with local high-frequency features extracted from short temporal windows to preserve fine details. Building on this, FreeLong++ extends FreeLong dual-branch design into a multi-branch architecture with multiple attention branches, each operating at a distinct temporal scale. By arranging multiple window sizes from global to local, FreeLong++ enables multi-band frequency fusion from low to high frequencies, ensuring both semantic continuity and fine-grained motion dynamics across longer video sequences. Without any additional training, FreeLong++ can be plugged into existing video generation models (e.g. Wan2.1 and LTX-Video) to produce longer videos with substantially improved temporal consistency and visual fidelity. We demonstrate that our approach outperforms previous methods on longer video generation tasks (e.g. 4x and 8x of native length). It also supports coherent multi-prompt video generation with smooth scene transitions and enables controllable video generation using long depth or pose sequences.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00162</guid>
|
|
<pubDate>Mon, 30 Jun 2025 18:11:21 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>μ^2Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation</title>
|
|
<link>https://arxiv.org/abs/2507.00316</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00316.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siyou Li, Pengyao Qin, Huanan Wu, Dong Nie, Arun J. Thirunavukarasu, Juntao Yu, Le Zhang</p><p><b>Upvotes:</b> 14</p><p><b>Summary:</b> Automated radiology report generation (RRG) aims to produce detailed textual reports from clinical imaging, such as computed tomography (CT) scans, to improve the accuracy and efficiency of diagnosis and provision of management advice. RRG is complicated by two key challenges: (1) inherent complexity in extracting relevant information from imaging data under resource constraints, and (2) difficulty in objectively evaluating discrepancies between model-generated and expert-written reports. To address these challenges, we propose mu^2LLM, a textbf{mu}ltiscale textbf{mu}ltimodal large language models for RRG tasks. The novel {mu}^2Tokenizer, as an intermediate layer, integrates multi-modal features from the multiscale visual tokenizer and the text tokenizer, then enhances report generation quality through direct preference optimization (DPO), guided by GREEN-RedLlama. Experimental results on four large CT image-report medical datasetdemonstrate that our method outperforms existing approaches, highlighting the potential of our fine-tuned mu^2LLMs on limited data for RRG tasks.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00316</guid>
|
|
<pubDate>Mon, 30 Jun 2025 23:14:49 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Training for X-Ray Vision: Amodal Segmentation, Amodal Content Completion, and View-Invariant Object Representation from Multi-Camera Video</title>
|
|
<link>https://arxiv.org/abs/2507.00339</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00339.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Alexander Moore, Amar Saini, Kylie Cancilla, Doug Poland, Carmen Carrano</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Amodal segmentation and amodal content completion require using object priors to estimate occluded masks and features of objects in complex scenes. Until now, no data has provided an additional dimension for object context: the possibility of multiple cameras sharing a view of a scene. We introduce MOVi-MC-AC: Multiple Object Video with Multi-Cameras and Amodal Content, the largest amodal segmentation and first amodal content dataset to date. Cluttered scenes of generic household objects are simulated in multi-camera video. MOVi-MC-AC contributes to the growing literature of object detection, tracking, and segmentation by including two new contributions to the deep learning for computer vision world. Multiple Camera (MC) settings where objects can be identified and tracked between various unique camera perspectives are rare in both synthetic and real-world video. We introduce a new complexity to synthetic video by providing consistent object ids for detections and segmentations between both frames and multiple cameras each with unique features and motion patterns on a single scene. Amodal Content (AC) is a reconstructive task in which models predict the appearance of target objects through occlusions. In the amodal segmentation literature, some datasets have been released with amodal detection, tracking, and segmentation labels. While other methods rely on slow cut-and-paste schemes to generate amodal content pseudo-labels, they do not account for natural occlusions present in the modal masks. MOVi-MC-AC provides labels for ~5.8 million object instances, setting a new maximum in the amodal dataset literature, along with being the first to provide ground-truth amodal content. The full dataset is available at https://huggingface.co/datasets/Amar-S/MOVi-MC-AC ,</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00339</guid>
|
|
<pubDate>Tue, 01 Jul 2025 00:36:56 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning</title>
|
|
<link>https://arxiv.org/abs/2507.00432</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00432.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, Xiang Yue</p><p><b>Upvotes:</b> 53</p><p><b>Summary:</b> Math reasoning has become the poster child of progress in large language models (LLMs), with new models rapidly surpassing human-level performance on benchmarks like MATH and AIME. But as math leaderboards improve week by week, it is worth asking: do these gains reflect broader problem-solving ability or just narrow overfitting? To answer this question, we evaluate over 20 open-weight reasoning-tuned models across a broad suite of tasks, including math, scientific QA, agent planning, coding, and standard instruction-following. We surprisingly find that most models that succeed in math fail to transfer their gains to other domains. To rigorously study this phenomenon, we conduct controlled experiments on Qwen3-14B models using math-only data but different tuning methods. We find that reinforcement learning (RL)-tuned models generalize well across domains, while supervised fine-tuning (SFT)-tuned models often forget general capabilities. Latent-space representation and token-space distribution shift analyses reveal that SFT induces substantial representation and output drift, while RL preserves general-domain structure. Our results suggest a need to rethink standard post-training recipes, particularly the reliance on SFT-distilled data for advancing reasoning models.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00432</guid>
|
|
<pubDate>Tue, 01 Jul 2025 05:23:05 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>ARIG: Autoregressive Interactive Head Generation for Real-time Conversations</title>
|
|
<link>https://arxiv.org/abs/2507.00472</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00472.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ying Guo, Xi Liu, Cheng Zhen, Pengfei Yan, Xiaoming Wei</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Face-to-face communication, as a common human activity, motivates the research on interactive head generation. A virtual agent can generate motion responses with both listening and speaking capabilities based on the audio or motion signals of the other user and itself. However, previous clip-wise generation paradigm or explicit listener/speaker generator-switching methods have limitations in future signal acquisition, contextual behavioral understanding, and switching smoothness, making it challenging to be real-time and realistic. In this paper, we propose an autoregressive (AR) based frame-wise framework called ARIG to realize the real-time generation with better interaction realism. To achieve real-time generation, we model motion prediction as a non-vector-quantized AR process. Unlike discrete codebook-index prediction, we represent motion distribution using diffusion procedure, achieving more accurate predictions in continuous space. To improve interaction realism, we emphasize interactive behavior understanding (IBU) and detailed conversational state understanding (CSU). In IBU, based on dual-track dual-modal signals, we summarize short-range behaviors through bidirectional-integrated learning and perform contextual understanding over long ranges. In CSU, we use voice activity signals and context features of IBU to understand the various states (interruption, feedback, pause, etc.) that exist in actual conversations. These serve as conditions for the final progressive motion prediction. Extensive experiments have verified the effectiveness of our model.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00472</guid>
|
|
<pubDate>Tue, 01 Jul 2025 06:38:14 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>FreNBRDF: A Frequency-Rectified Neural Material Representation</title>
|
|
<link>https://arxiv.org/abs/2507.00476</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00476.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chenliang Zhou, Zheyuan Hu, Cengiz Oztireli</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Accurate material modeling is crucial for achieving photorealistic rendering, bridging the gap between computer-generated imagery and real-world photographs. While traditional approaches rely on tabulated BRDF data, recent work has shifted towards implicit neural representations, which offer compact and flexible frameworks for a range of tasks. However, their behavior in the frequency domain remains poorly understood. To address this, we introduce FreNBRDF, a frequency-rectified neural material representation. By leveraging spherical harmonics, we integrate frequency-domain considerations into neural BRDF modeling. We propose a novel frequency-rectified loss, derived from a frequency analysis of neural materials, and incorporate it into a generalizable and adaptive reconstruction and editing pipeline. This framework enhances fidelity, adaptability, and efficiency. Extensive experiments demonstrate that \ours improves the accuracy and robustness of material appearance reconstruction and editing compared to state-of-the-art baselines, enabling more structured and interpretable downstream tasks and applications.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00476</guid>
|
|
<pubDate>Tue, 01 Jul 2025 06:48:50 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Mixture of Reasonings: Teach Large Language Models to Reason with Adaptive Strategies</title>
|
|
<link>https://arxiv.org/abs/2507.00606</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00606.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tao Xiong, Xavier Hu, Wenyan Fan, Shengyu Zhang</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Large language models (LLMs) excel in complex tasks through advanced prompting techniques like Chain-of-Thought (CoT) and Tree-of-Thought (ToT), but their reliance on manually crafted, task-specific prompts limits adaptability and efficiency. We introduce Mixture of Reasoning (MoR), a training framework that embeds diverse reasoning strategies into LLMs for autonomous, task-adaptive reasoning without external prompt engineering. MoR has two phases: Thought Generation, creating reasoning chain templates with models like GPT-4o, and SFT Dataset Construction, pairing templates with benchmark datasets for supervised fine-tuning.Our experiments show that MoR significantly enhances performance, with MoR150 achieving 0.730 (2.2% improvement) using CoT prompting and 0.734 (13.5% improvement) compared to baselines. MoR eliminates the need for task-specific prompts, offering a generalizable solution for robust reasoning across diverse tasks.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00606</guid>
|
|
<pubDate>Tue, 01 Jul 2025 09:39:04 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impact</title>
|
|
<link>https://arxiv.org/abs/2507.00951</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00951.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Rizwan Qureshi, Ranjan Sapkota, Abbas Shah, Amgad Muneer, Anas Zafar, Ashmal Vayani, Maged Shoman, Abdelrahman B. M. Eldaly, Kai Zhang, Ferhat Sadak, Shaina Raza, Xinqi Fan, Ravid Shwartz-Ziv, Hong Yan, Vinjia Jain, Aman Chadha, Manoj Karkee, Jia Wu, Philip Torr, Seyedali Mirjalili</p><p><b>Upvotes:</b> 20</p><p><b>Summary:</b> Can machines truly think, reason and act in domains like humans? This enduring question continues to shape the pursuit of Artificial General Intelligence (AGI). Despite the growing capabilities of models such as GPT-4.5, DeepSeek, Claude 3.5 Sonnet, Phi-4, and Grok 3, which exhibit multimodal fluency and partial reasoning, these systems remain fundamentally limited by their reliance on token-level prediction and lack of grounded agency. This paper offers a cross-disciplinary synthesis of AGI development, spanning artificial intelligence, cognitive neuroscience, psychology, generative models, and agent-based systems. We analyze the architectural and cognitive foundations of general intelligence, highlighting the role of modular reasoning, persistent memory, and multi-agent coordination. In particular, we emphasize the rise of Agentic RAG frameworks that combine retrieval, planning, and dynamic tool use to enable more adaptive behavior. We discuss generalization strategies, including information compression, test-time adaptation, and training-free methods, as critical pathways toward flexible, domain-agnostic intelligence. Vision-Language Models (VLMs) are reexamined not just as perception modules but as evolving interfaces for embodied understanding and collaborative task completion. We also argue that true intelligence arises not from scale alone but from the integration of memory and reasoning: an orchestration of modular, interactive, and self-improving components where compression enables adaptive behavior. Drawing on advances in neurosymbolic systems, reinforcement learning, and cognitive scaffolding, we explore how recent architectures begin to bridge the gap between statistical learning and goal-directed cognition. Finally, we identify key scientific, technical, and ethical challenges on the path to AGI.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.00951</guid>
|
|
<pubDate>Tue, 01 Jul 2025 16:52:25 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>SciArena: An Open Evaluation Platform for Foundation Models in Scientific Literature Tasks</title>
|
|
<link>https://arxiv.org/abs/2507.01001</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01001.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Charles McGrady, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, Arman Cohan</p><p><b>Upvotes:</b> 41</p><p><b>Summary:</b> We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature tasks. Unlike traditional benchmarks for scientific literature understanding and synthesis, SciArena engages the research community directly, following the Chatbot Arena evaluation approach of community voting on model comparisons. By leveraging collective intelligence, SciArena offers a community-driven evaluation of model performance on open-ended scientific tasks that demand literature-grounded, long-form responses. The platform currently supports 23 open-source and proprietary foundation models and has collected over 13,000 votes from trusted researchers across diverse scientific domains. We analyze the data collected so far and confirm that the submitted questions are diverse, aligned with real-world literature needs, and that participating researchers demonstrate strong self-consistency and inter-annotator agreement in their evaluations. We discuss the results and insights based on the model ranking leaderboard. To further promote research in building model-based automated evaluation systems for literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based on our collected preference data. The benchmark measures the accuracy of models in judging answer quality by comparing their pairwise assessments with human votes. Our experiments highlight the benchmark's challenges and emphasize the need for more reliable automated evaluation methods.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01001</guid>
|
|
<pubDate>Tue, 01 Jul 2025 17:51:59 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention</title>
|
|
<link>https://arxiv.org/abs/2507.01004</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01004.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuhong Chou, Zehao Liu, Ruijie Zhu, Xinyi Wan, Tianjian Li, Congying Chu, Qian Liu, Jibin Wu, Zejun Ma</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Linear attention mechanisms deliver significant advantages for Large Language Models (LLMs) by providing linear computational complexity, enabling efficient processing of ultra-long sequences (e.g., 1M context). However, existing Sequence Parallelism (SP) methods, essential for distributing these workloads across devices, become the primary bottleneck due to substantial communication overhead. In this paper, we introduce ZeCO (Zero Communication Overhead) sequence parallelism for linear attention models, a new SP method designed to overcome these limitations and achieve end-to-end near-linear scalability for long sequence training. For example, training a model with a 1M sequence length across 64 devices using ZeCO takes roughly the same time as training with an 16k sequence on a single device. At the heart of ZeCO lies All-Scan, a new collective communication primitive. All-Scan provides each SP rank with precisely the initial operator state it requires while maintaining a minimal communication footprint, effectively eliminating communication overhead. Theoretically, we prove the optimaity of ZeCO, showing that it introduces only negligible time and space overhead. Empirically, we compare the communication costs of different sequence parallelism strategies and demonstrate that All-Scan achieves the fastest communication in SP scenarios. Specifically, on 256 GPUs with an 8M sequence length, ZeCO achieves a 60\% speedup compared to the current state-of-the-art (SOTA) SP method. We believe ZeCO establishes a clear path toward efficiently training next-generation LLMs on previously intractable sequence lengths.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01004</guid>
|
|
<pubDate>Tue, 01 Jul 2025 17:54:53 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning</title>
|
|
<link>https://arxiv.org/abs/2507.01006</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01006.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Leyi Pan, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianle Gong, Wenkai Li, Wei Jia, Xin Lyu, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuxuan Zhang, Zhanxiao Du, Zhenyu Hou, Zhao Xue, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, Jie Tang</p><p><b>Upvotes:</b> 166</p><p><b>Summary:</b> We present GLM-4.1V-Thinking, a vision-language model (VLM) designed to advance general-purpose multimodal reasoning. In this report, we share our key findings in the development of the reasoning-centric training framework. We first develop a capable vision foundation model with significant potential through large-scale pre-training, which arguably sets the upper bound for the final performance. Reinforcement Learning with Curriculum Sampling (RLCS) then unlocks the full potential of the model, leading to comprehensive capability enhancement across a diverse range of tasks, including STEM problem solving, video understanding, content recognition, coding, grounding, GUI-based agents, and long document understanding, among others. To facilitate research in this field, we open-source GLM-4.1V-9B-Thinking, which achieves state-of-the-art performance among models of comparable size. In a comprehensive evaluation across 28 public benchmarks, our model outperforms Qwen2.5-VL-7B on nearly all tasks and achieves comparable or even superior performance on 18 benchmarks relative to the significantly larger Qwen2.5-VL-72B. Notably, GLM-4.1V-9B-Thinking also demonstrates competitive or superior performance compared to closed-source models such as GPT-4o on challenging tasks including long document understanding and STEM reasoning, further underscoring its strong capabilities. Code, models and more information are released at https://github.com/THUDM/GLM-4.1V-Thinking.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01006</guid>
|
|
<pubDate>Tue, 01 Jul 2025 17:55:04 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy</title>
|
|
<link>https://arxiv.org/abs/2507.01352</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01352.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, Yahui Zhou</p><p><b>Upvotes:</b> 38</p><p><b>Summary:</b> Despite the critical role of reward models (RMs) in reinforcement learning from human feedback (RLHF), current state-of-the-art open RMs perform poorly on most existing evaluation benchmarks, failing to capture the spectrum of nuanced and sophisticated human preferences. Even approaches that incorporate advanced training techniques have not yielded meaningful performance improvements. We hypothesize that this brittleness stems primarily from limitations in preference datasets, which are often narrowly scoped, synthetically labeled, or lack rigorous quality control. To address these challenges, we present a large-scale preference dataset comprising 40 million preference pairs, named SynPref-40M. To enable data curation at scale, we design a human-AI synergistic two-stage pipeline that leverages the complementary strengths of human annotation quality and AI scalability. In this pipeline, humans provide verified annotations, while large language models perform automatic curation based on human guidance. Training on this preference mixture, we introduce Skywork-Reward-V2, a suite of eight reward models ranging from 0.6B to 8B parameters, trained on a carefully curated subset of 26 million preference pairs from SynPref-40M. We demonstrate that Skywork-Reward-V2 is versatile across a wide range of capabilities, including alignment with human preferences, objective correctness, safety, resistance to stylistic biases, and best-of-N scaling, achieving state-of-the-art performance across seven major reward model benchmarks. Ablation studies confirm that the effectiveness of our approach stems not only from data scale but also from high-quality curation. The Skywork-Reward-V2 series represents substantial progress in open reward models, highlighting the untapped potential of existing preference datasets and demonstrating how human-AI curation synergy can unlock significantly higher data quality.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01352</guid>
|
|
<pubDate>Wed, 02 Jul 2025 04:40:29 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>MARVIS: Modality Adaptive Reasoning over VISualizations</title>
|
|
<link>https://arxiv.org/abs/2507.01544</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01544.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Benjamin Feuer, Lennart Purucker, Oussama Elachqar, Chinmay Hegde</p><p><b>Upvotes:</b> 10</p><p><b>Summary:</b> Scientific applications of machine learning often rely on small, specialized models tuned to particular domains. Such models often achieve excellent performance, but lack flexibility. Foundation models offer versatility, but typically underperform specialized approaches, especially on non-traditional modalities and long-tail domains. We propose MARVIS (Modality Adaptive Reasoning over VISualizations), a training-free method that enables even small vision-language models to predict any data modality with high accuracy. MARVIS transforms latent embedding spaces into visual representations and then leverages the spatial and fine-grained reasoning skills of VLMs to successfully interpret and utilize them. MARVIS achieves competitive performance on vision, audio, biological, and tabular domains using a single 3B parameter model, achieving results that beat Gemini by 16\% on average and approach specialized methods, without exposing personally identifiable information (P.I.I.) or requiring any domain-specific training. We open source our code and datasets at https://github.com/penfever/marvis</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01544</guid>
|
|
<pubDate>Wed, 02 Jul 2025 09:56:24 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Depth Anything at Any Condition</title>
|
|
<link>https://arxiv.org/abs/2507.01634</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01634.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Boyuan Sun, Modi Jin, Bowen Yin, Qibin Hou</p><p><b>Upvotes:</b> 44</p><p><b>Summary:</b> We present Depth Anything at Any Condition (DepthAnything-AC), a foundation monocular depth estimation (MDE) model capable of handling diverse environmental conditions. Previous foundation MDE models achieve impressive performance across general scenes but not perform well in complex open-world environments that involve challenging conditions, such as illumination variations, adverse weather, and sensor-induced distortions. To overcome the challenges of data scarcity and the inability of generating high-quality pseudo-labels from corrupted images, we propose an unsupervised consistency regularization finetuning paradigm that requires only a relatively small amount of unlabeled data. Furthermore, we propose the Spatial Distance Constraint to explicitly enforce the model to learn patch-level relative relationships, resulting in clearer semantic boundaries and more accurate details. Experimental results demonstrate the zero-shot capabilities of DepthAnything-AC across diverse benchmarks, including real-world adverse weather benchmarks, synthetic corruption benchmarks, and general benchmarks. Project Page: https://ghost233lism.github.io/depthanything-AC-page Code: https://github.com/HVision-NKU/DepthAnythingAC</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01634</guid>
|
|
<pubDate>Wed, 02 Jul 2025 12:05:57 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training</title>
|
|
<link>https://arxiv.org/abs/2507.01663</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01663.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, Jianping Wu</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs). Traditional task-colocated RL frameworks suffer from significant scalability bottlenecks, while task-separated RL frameworks face challenges in complex dataflows and the corresponding resource idling and workload imbalance. Moreover, most existing frameworks are tightly coupled with LLM training or inference engines, making it difficult to support custom-designed engines. To address these challenges, we propose AsyncFlow, an asynchronous streaming RL framework for efficient post-training. Specifically, we introduce a distributed data storage and transfer module that provides a unified data management and fine-grained scheduling capability in a fully streamed manner. This architecture inherently facilitates automated pipeline overlapping among RL tasks and dynamic load balancing. Moreover, we propose a producer-consumer-based asynchronous workflow engineered to minimize computational idleness by strategically deferring parameter update process within staleness thresholds. Finally, the core capability of AsynFlow is architecturally decoupled from underlying training and inference engines and encapsulated by service-oriented user interfaces, offering a modular and customizable user experience. Extensive experiments demonstrate an average of 1.59 throughput improvement compared with state-of-the-art baseline. The presented architecture in this work provides actionable insights for next-generation RL training system designs.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01663</guid>
|
|
<pubDate>Wed, 02 Jul 2025 12:45:34 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>IntFold: A Controllable Foundation Model for General and Specialized Biomolecular Structure Prediction</title>
|
|
<link>https://arxiv.org/abs/2507.02025</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02025.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> The IntFold Team, Leon Qiao, Wayne Bai, He Yan, Gary Liu, Nova Xi, Xiang Zhang</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> We introduce IntFold, a controllable foundation model for both general and specialized biomolecular structure prediction. IntFold demonstrates predictive accuracy comparable to the state-of-the-art AlphaFold3, while utilizing a superior customized attention kernel. Beyond standard structure prediction, IntFold can be adapted to predict allosteric states, constrained structures, and binding affinity through the use of individual adapters. Furthermore, we introduce a novel confidence head to estimate docking quality, offering a more nuanced assessment for challenging targets such as antibody-antigen complexes. Finally, we share insights gained during the training process of this computationally intensive model.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02025</guid>
|
|
<pubDate>Wed, 02 Jul 2025 16:09:47 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>A Survey on Vision-Language-Action Models: An Action Tokenization Perspective</title>
|
|
<link>https://arxiv.org/abs/2507.01925</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01925.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, Zhiquan Qi, Yitao Liang, Yuanpei Chen, Yaodong Yang</p><p><b>Upvotes:</b> 29</p><p><b>Summary:</b> The remarkable advancements of vision and language foundation models in multimodal understanding, reasoning, and generation has sparked growing efforts to extend such intelligence to the physical world, fueling the flourishing of vision-language-action (VLA) models. Despite seemingly diverse approaches, we observe that current VLA models can be unified under a single framework: vision and language inputs are processed by a series of VLA modules, producing a chain of action tokens that progressively encode more grounded and actionable information, ultimately generating executable actions. We further determine that the primary design choice distinguishing VLA models lies in how action tokens are formulated, which can be categorized into language description, code, affordance, trajectory, goal state, latent representation, raw action, and reasoning. However, there remains a lack of comprehensive understanding regarding action tokens, significantly impeding effective VLA development and obscuring future directions. Therefore, this survey aims to categorize and interpret existing VLA research through the lens of action tokenization, distill the strengths and limitations of each token type, and identify areas for improvement. Through this systematic review and analysis, we offer a synthesized outlook on the broader evolution of VLA models, highlight underexplored yet promising directions, and contribute guidance for future research, hoping to bring the field closer to general-purpose intelligence.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01925</guid>
|
|
<pubDate>Wed, 02 Jul 2025 17:34:52 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>LongAnimation: Long Animation Generation with Dynamic Global-Local Memory</title>
|
|
<link>https://arxiv.org/abs/2507.01945</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01945.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nan Chen, Mengqi Huang, Yihao Meng, Zhendong Mao</p><p><b>Upvotes:</b> 68</p><p><b>Summary:</b> Animation colorization is a crucial part of real animation industry production. Long animation colorization has high labor costs. Therefore, automated long animation colorization based on the video generation model has significant research value. Existing studies are limited to short-term colorization. These studies adopt a local paradigm, fusing overlapping features to achieve smooth transitions between local segments. However, the local paradigm neglects global information, failing to maintain long-term color consistency. In this study, we argue that ideal long-term color consistency can be achieved through a dynamic global-local paradigm, i.e., dynamically extracting global color-consistent features relevant to the current generation. Specifically, we propose LongAnimation, a novel framework, which mainly includes a SketchDiT, a Dynamic Global-Local Memory (DGLM), and a Color Consistency Reward. The SketchDiT captures hybrid reference features to support the DGLM module. The DGLM module employs a long video understanding model to dynamically compress global historical features and adaptively fuse them with the current generation features. To refine the color consistency, we introduce a Color Consistency Reward. During inference, we propose a color consistency fusion to smooth the video segment transition. Extensive experiments on both short-term (14 frames) and long-term (average 500 frames) animations show the effectiveness of LongAnimation in maintaining short-term and long-term color consistency for open-domain animation colorization task. The code can be found at https://cn-makers.github.io/long_animation_web/.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01945</guid>
|
|
<pubDate>Wed, 02 Jul 2025 17:55:50 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Kwai Keye-VL Technical Report</title>
|
|
<link>https://arxiv.org/abs/2507.01949</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01949.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kwai Keye Team, Biao Yang, Bin Wen, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, Fan Yang, Guorui Zhou, Hao Peng, Haojie Ding, Jiaming Huang, Jiangxia Cao, Jiankang Chen, Jingyun Hua, Jin Ouyang, Kaibing Chen, Kaiyu Jiang, Kaiyu Tang, Kun Gai, Shengnan Zhang, Siyang Mao, Sui Huang, Tianke Zhang, Tingting Gao, Wei Chen, Wei Yuan, Xiangyu Wu, Xiao Hu, Xingyu Lu, Yang Zhou, Yi-Fan Zhang, Yiping Yang, Yulong Chen, Zhenhua Wu, Zhenyu Li, Zhixin Ling, Ziming Li, Dehua Ma, Di Xu, Haixuan Gao, Hang Li, Jiawei Guo, Jing Wang, Lejian Ren, Muhao Wei, Qianqian Wang, Qigen Hu, Shiyao Wang, Tao Yu, Xinchen Luo, Yan Li, Yiming Liang, Yuhang Hu, Zeyi Lu, Zhuoran Yang, Zixing Zhang</p><p><b>Upvotes:</b> 109</p><p><b>Summary:</b> While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities on static images, they often fall short in comprehending dynamic, information-dense short-form videos, a dominant medium in today's digital landscape. To bridge this gap, we introduce Kwai Keye-VL, an 8-billion-parameter multimodal foundation model engineered for leading-edge performance in short-video understanding while maintaining robust general-purpose vision-language abilities. The development of Keye-VL rests on two core pillars: a massive, high-quality dataset exceeding 600 billion tokens with a strong emphasis on video, and an innovative training recipe. This recipe features a four-stage pre-training process for solid vision-language alignment, followed by a meticulous two-phase post-training process. The first post-training stage enhances foundational capabilities like instruction following, while the second phase focuses on stimulating advanced reasoning. In this second phase, a key innovation is our five-mode ``cold-start'' data mixture, which includes ``thinking'', ``non-thinking'', ``auto-think'', ``think with image'', and high-quality video data. This mixture teaches the model to decide when and how to reason. Subsequent reinforcement learning (RL) and alignment steps further enhance these reasoning capabilities and correct abnormal model behaviors, such as repetitive outputs. To validate our approach, we conduct extensive evaluations, showing that Keye-VL achieves state-of-the-art results on public video benchmarks and remains highly competitive on general image-based tasks (Figure 1). Furthermore, we develop and release the KC-MMBench, a new benchmark tailored for real-world short-video scenarios, where Keye-VL shows a significant advantage.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01949</guid>
|
|
<pubDate>Wed, 02 Jul 2025 17:57:28 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>FreeMorph: Tuning-Free Generalized Image Morphing with Diffusion Model</title>
|
|
<link>https://arxiv.org/abs/2507.01953</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01953.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yukang Cao, Chenyang Si, Jinghao Wang, Ziwei Liu</p><p><b>Upvotes:</b> 19</p><p><b>Summary:</b> We present FreeMorph, the first tuning-free method for image morphing that accommodates inputs with different semantics or layouts. Unlike existing methods that rely on finetuning pre-trained diffusion models and are limited by time constraints and semantic/layout discrepancies, FreeMorph delivers high-fidelity image morphing without requiring per-instance training. Despite their efficiency and potential, tuning-free methods face challenges in maintaining high-quality results due to the non-linear nature of the multi-step denoising process and biases inherited from the pre-trained diffusion model. In this paper, we introduce FreeMorph to address these challenges by integrating two key innovations. 1) We first propose a guidance-aware spherical interpolation design that incorporates explicit guidance from the input images by modifying the self-attention modules, thereby addressing identity loss and ensuring directional transitions throughout the generated sequence. 2) We further introduce a step-oriented variation trend that blends self-attention modules derived from each input image to achieve controlled and consistent transitions that respect both inputs. Our extensive evaluations demonstrate that FreeMorph outperforms existing methods, being 10x ~ 50x faster and establishing a new state-of-the-art for image morphing.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01953</guid>
|
|
<pubDate>Wed, 02 Jul 2025 17:58:20 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation</title>
|
|
<link>https://arxiv.org/abs/2507.01957</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01957.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhuoyang Zhang, Luke J. Huang, Chengyue Wu, Shang Yang, Kelly Peng, Yao Lu, Song Han</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> We present Locality-aware Parallel Decoding (LPD) to accelerate autoregressive image generation. Traditional autoregressive image generation relies on next-patch prediction, a memory-bound process that leads to high latency. Existing works have tried to parallelize next-patch prediction by shifting to multi-patch prediction to accelerate the process, but only achieved limited parallelization. To achieve high parallelization while maintaining generation quality, we introduce two key techniques: (1) Flexible Parallelized Autoregressive Modeling, a novel architecture that enables arbitrary generation ordering and degrees of parallelization. It uses learnable position query tokens to guide generation at target positions while ensuring mutual visibility among concurrently generated tokens for consistent parallel decoding. (2) Locality-aware Generation Ordering, a novel schedule that forms groups to minimize intra-group dependencies and maximize contextual support, enhancing generation quality. With these designs, we reduce the generation steps from 256 to 20 (256times256 res.) and 1024 to 48 (512times512 res.) without compromising quality on the ImageNet class-conditional generation, and achieving at least 3.4times lower latency than previous parallelized autoregressive models.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.01957</guid>
|
|
<pubDate>Wed, 02 Jul 2025 17:59:23 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Energy-Based Transformers are Scalable Learners and Thinkers</title>
|
|
<link>https://arxiv.org/abs/2507.02092</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02092.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal</p><p><b>Upvotes:</b> 23</p><p><b>Summary:</b> Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limitations: they are modality-specific (e.g., working only in text), problem-specific (e.g., verifiable domains like math and coding), or require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards). In this paper, we ask the question "Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?" Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier. Specifically, we train Energy-Based Transformers (EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs improve performance with System 2 Thinking by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using fewer forward passes. Further, we find that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, suggesting that EBTs generalize better than existing approaches. Consequently, EBTs are a promising new paradigm for scaling both the learning and thinking capabilities of models.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02092</guid>
|
|
<pubDate>Wed, 02 Jul 2025 19:17:29 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback</title>
|
|
<link>https://arxiv.org/abs/2507.02321</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02321.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nina Konovalova, Maxim Nikolaev, Andrey Kuznetsov, Aibek Alanov</p><p><b>Upvotes:</b> 38</p><p><b>Summary:</b> Despite significant progress in text-to-image diffusion models, achieving precise spatial control over generated outputs remains challenging. ControlNet addresses this by introducing an auxiliary conditioning module, while ControlNet++ further refines alignment through a cycle consistency loss applied only to the final denoising steps. However, this approach neglects intermediate generation stages, limiting its effectiveness. We propose InnerControl, a training strategy that enforces spatial consistency across all diffusion steps. Our method trains lightweight convolutional probes to reconstruct input control signals (e.g., edges, depth) from intermediate UNet features at every denoising step. These probes efficiently extract signals even from highly noisy latents, enabling pseudo ground truth controls for training. By minimizing the discrepancy between predicted and target conditions throughout the entire diffusion process, our alignment loss improves both control fidelity and generation quality. Combined with established techniques like ControlNet++, InnerControl achieves state-of-the-art performance across diverse conditioning methods (e.g., edges, depth).</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02321</guid>
|
|
<pubDate>Thu, 03 Jul 2025 05:25:53 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>WebSailor: Navigating Super-human Reasoning for Web Agent</title>
|
|
<link>https://arxiv.org/abs/2507.02592</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02592.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, Jingren Zhou</p><p><b>Upvotes:</b> 77</p><p><b>Summary:</b> Transcending human cognitive limitations represents a critical frontier in LLM training. Proprietary agentic systems like DeepResearch have demonstrated superhuman capabilities on extremely complex information-seeking benchmarks such as BrowseComp, a feat previously unattainable. We posit that their success hinges on a sophisticated reasoning pattern absent in open-source models: the ability to systematically reduce extreme uncertainty when navigating vast information landscapes. Based on this insight, we introduce WebSailor, a complete post-training methodology designed to instill this crucial capability. Our approach involves generating novel, high-uncertainty tasks through structured sampling and information obfuscation, RFT cold start, and an efficient agentic RL training algorithm, Duplicating Sampling Policy Optimization (DUPO). With this integrated pipeline, WebSailor significantly outperforms all opensource agents in complex information-seeking tasks, matching proprietary agents' performance and closing the capability gap.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02592</guid>
|
|
<pubDate>Thu, 03 Jul 2025 12:59:07 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Decoupled Planning and Execution: A Hierarchical Reasoning Framework for Deep Search</title>
|
|
<link>https://arxiv.org/abs/2507.02652</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02652.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yang Zhao, Hongjin Qian, Zhicheng Dou</p><p><b>Upvotes:</b> 18</p><p><b>Summary:</b> Complex information needs in real-world search scenarios demand deep reasoning and knowledge synthesis across diverse sources, which traditional retrieval-augmented generation (RAG) pipelines struggle to address effectively. Current reasoning-based approaches suffer from a fundamental limitation: they use a single model to handle both high-level planning and detailed execution, leading to inefficient reasoning and limited scalability. In this paper, we introduce HiRA, a hierarchical framework that separates strategic planning from specialized execution. Our approach decomposes complex search tasks into focused subtasks, assigns each subtask to domain-specific agents equipped with external tools and reasoning capabilities, and coordinates the results through a structured integration mechanism. This separation prevents execution details from disrupting high-level reasoning while enabling the system to leverage specialized expertise for different types of information processing. Experiments on four complex, cross-modal deep search benchmarks demonstrate that HiRA significantly outperforms state-of-the-art RAG and agent-based systems. Our results show improvements in both answer quality and system efficiency, highlighting the effectiveness of decoupled planning and execution for multi-step information seeking tasks. Our code is available at https://github.com/ignorejjj/HiRA.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02652</guid>
|
|
<pubDate>Thu, 03 Jul 2025 14:18:08 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers</title>
|
|
<link>https://arxiv.org/abs/2507.02694</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02694.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhijian Xu, Yilun Zhao, Manasi Patwardhan, Lovekesh Vig, Arman Cohan</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Peer review is fundamental to scientific research, but the growing volume of publications has intensified the challenges of this expertise-intensive process. While LLMs show promise in various scientific tasks, their potential to assist with peer review, particularly in identifying paper limitations, remains understudied. We first present a comprehensive taxonomy of limitation types in scientific research, with a focus on AI. Guided by this taxonomy, for studying limitations, we present LimitGen, the first comprehensive benchmark for evaluating LLMs' capability to support early-stage feedback and complement human peer review. Our benchmark consists of two subsets: LimitGen-Syn, a synthetic dataset carefully created through controlled perturbations of high-quality papers, and LimitGen-Human, a collection of real human-written limitations. To improve the ability of LLM systems to identify limitations, we augment them with literature retrieval, which is essential for grounding identifying limitations in prior scientific findings. Our approach enhances the capabilities of LLM systems to generate limitations in research papers, enabling them to provide more concrete and constructive feedback.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02694</guid>
|
|
<pubDate>Thu, 03 Jul 2025 15:04:38 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Bourbaki: Self-Generated and Goal-Conditioned MDPs for Theorem Proving</title>
|
|
<link>https://arxiv.org/abs/2507.02726</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02726.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Matthieu Zimmer, Xiaotong Ji, Rasul Tutunov, Anthony Bordg, Jun Wang, Haitham Bou Ammar</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Reasoning remains a challenging task for large language models (LLMs), especially within the logically constrained environment of automated theorem proving (ATP), due to sparse rewards and the vast scale of proofs. These challenges are amplified in benchmarks like PutnamBench, which contains university-level problems requiring complex, multi-step reasoning. To address this, we introduce self-generated goal-conditioned MDPs (sG-MDPs), a new framework in which agents generate and pursue their subgoals based on the evolving proof state. Given this more structured generation of goals, the resulting problem becomes more amenable to search. We then apply Monte Carlo Tree Search (MCTS)-like algorithms to solve the sG-MDP, instantiating our approach in Bourbaki (7B), a modular system that can ensemble multiple 7B LLMs for subgoal generation and tactic synthesis. On PutnamBench, Bourbaki (7B) solves 26 problems, achieving new state-of-the-art results with models at this scale.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02726</guid>
|
|
<pubDate>Thu, 03 Jul 2025 15:41:38 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Fast and Simplex: 2-Simplicial Attention in Triton</title>
|
|
<link>https://arxiv.org/abs/2507.02754</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02754.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aurko Roy, Timothy Chou, Sai Surya Duvvuri, Sijia Chen, Jiecao Yu, Xiaodong Wang, Manzil Zaheer, Rohan Anil</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Recent work has shown that training loss scales as a power law with both model size and the number of tokens, and that achieving compute-optimal models requires scaling model size and token count together. However, these scaling laws assume an infinite supply of data and apply primarily in compute-bound settings. As modern large language models increasingly rely on massive internet-scale datasets, the assumption that they are compute-bound is becoming less valid. This shift highlights the need for architectures that prioritize token efficiency. In this work, we investigate the use of the 2-simplicial Transformer, an architecture that generalizes standard dot-product attention to trilinear functions through an efficient Triton kernel implementation. We demonstrate that the 2-simplicial Transformer achieves better token efficiency than standard Transformers: for a fixed token budget, similarly sized models outperform their dot-product counterparts on tasks involving mathematics, coding, reasoning, and logic. We quantify these gains by demonstrating that 2-simplicial attention changes the exponent in the scaling laws for knowledge and reasoning tasks compared to dot product attention.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02754</guid>
|
|
<pubDate>Thu, 03 Jul 2025 16:16:34 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Self-Correction Bench: Revealing and Addressing the Self-Correction Blind Spot in LLMs</title>
|
|
<link>https://arxiv.org/abs/2507.02778</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02778.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ken Tsui</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> Although large language models (LLMs) have become transformative, they still make mistakes and can explore unproductive reasoning paths. Self-correction is an important capability for a trustworthy LLM, particularly an autoregressive LLM. While LLMs can identify error in user input, they exhibit a systematic 'Self-Correction Blind Spot' - failing to correct identical error in their own outputs. To systematically study this phenomenon, we introduce Self-Correction Bench, a systematic framework to measure this phenomenon through controlled error injection at three complexity levels. Testing 14 models, we find an average 64.5% blind spot rate. We find multiple evidences that this limitation relates to training data composition: human training demonstrations predominantly show error-free responses rather than error-correction sequences, unlike RL-trained models that learn error correction through outcome feedback. Remarkably, simply appending "Wait" reduces blind spots by 89.3%, suggesting that the capability exists but requires activation. Our work highlights a critical limitation in current LLMs and offers potential avenues for improving their reliability and trustworthiness.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02778</guid>
|
|
<pubDate>Thu, 03 Jul 2025 16:41:30 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion</title>
|
|
<link>https://arxiv.org/abs/2507.02813</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02813.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fangfu Liu, Hao Li, Jiawei Chi, Hanyang Wang, Minghui Yang, Fudong Wang, Yueqi Duan</p><p><b>Upvotes:</b> 49</p><p><b>Summary:</b> Recovering 3D structures with open-vocabulary scene understanding from 2D images is a fundamental but daunting task. Recent developments have achieved this by performing per-scene optimization with embedded language information. However, they heavily rely on the calibrated dense-view reconstruction paradigm, thereby suffering from severe rendering artifacts and implausible semantic synthesis when limited views are available. In this paper, we introduce a novel generative framework, coined LangScene-X, to unify and generate 3D consistent multi-modality information for reconstruction and understanding. Powered by the generative capability of creating more consistent novel observations, we can build generalizable 3D language-embedded scenes from only sparse views. Specifically, we first train a TriMap video diffusion model that can generate appearance (RGBs), geometry (normals), and semantics (segmentation maps) from sparse inputs through progressive knowledge integration. Furthermore, we propose a Language Quantized Compressor (LQC), trained on large-scale image datasets, to efficiently encode language embeddings, enabling cross-scene generalization without per-scene retraining. Finally, we reconstruct the language surface fields by aligning language information onto the surface of 3D scenes, enabling open-ended language queries. Extensive experiments on real-world data demonstrate the superiority of our LangScene-X over state-of-the-art methods in terms of quality and generalizability. Project Page: https://liuff19.github.io/LangScene-X.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02813</guid>
|
|
<pubDate>Thu, 03 Jul 2025 17:21:23 +0000</pubDate>
|
|
</item>
|
|
<item>
|
|
<title>Answer Matching Outperforms Multiple Choice for Language Model Evaluation</title>
|
|
<link>https://arxiv.org/abs/2507.02856</link>
|
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02856.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, Jonas Geiping</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Multiple choice benchmarks have long been the workhorse of language model evaluation because grading multiple choice is objective and easy to automate. However, we show multiple choice questions from popular benchmarks can often be answered without even seeing the question. These shortcuts arise from a fundamental limitation of discriminative evaluation not shared by evaluations of the model's free-form, generative answers. Until recently, there appeared to be no viable, scalable alternative to multiple choice--but, we show that this has changed. We consider generative evaluation via what we call answer matching: Give the candidate model the question without the options, have it generate a free-form response, then use a modern language model with the reference answer to determine if the response matches the reference. To compare the validity of different evaluation strategies, we annotate MMLU-Pro and GPQA-Diamond to obtain human grading data, and measure the agreement of each evaluation approach. We find answer matching using recent models--even small ones--achieves near-perfect agreement, in the range of inter-annotator agreement. In contrast, both multiple choice evaluation and using LLM-as-a-judge without reference answers aligns poorly with human grading. Improving evaluations via answer matching is not merely a conceptual concern: the rankings of several models change significantly when evaluating their free-form responses with answer matching. In light of these findings, we discuss how to move the evaluation ecosystem from multiple choice to answer matching.</p></description>
|
|
<guid isPermaLink="false">https://arxiv.org/abs/2507.02856</guid>
|
|
<pubDate>Thu, 03 Jul 2025 17:59:02 +0000</pubDate>
|
|
</item>
|
|
</channel>
|
|
</rss>
|