bot: update monthly RSS feed
This commit is contained in:
github-actions[bot]
266
feed_monthly.xml
266
feed_monthly.xml
@@ -7,7 +7,7 @@
|
|||||||
<docs>http://www.rssboard.org/rss-specification</docs>
|
<docs>http://www.rssboard.org/rss-specification</docs>
|
||||||
<generator>python-feedgen</generator>
|
<generator>python-feedgen</generator>
|
||||||
<language>en</language>
|
<language>en</language>
|
||||||
<lastBuildDate>Fri, 25 Apr 2025 10:19:50 +0000</lastBuildDate>
|
<lastBuildDate>Thu, 01 May 2025 00:13:04 +0000</lastBuildDate>
|
||||||
<item>
|
<item>
|
||||||
<title>TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization</title>
|
<title>TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization</title>
|
||||||
<link>https://arxiv.org/abs/2503.19901</link>
|
<link>https://arxiv.org/abs/2503.19901</link>
|
||||||
@@ -18,7 +18,7 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Understanding R1-Zero-Like Training: A Critical Perspective</title>
|
<title>Understanding R1-Zero-Like Training: A Critical Perspective</title>
|
||||||
<link>https://arxiv.org/abs/2503.20783</link>
|
<link>https://arxiv.org/abs/2503.20783</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.20783.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin</p><p><b>Upvotes:</b> 45</p><p><b>Summary:</b> DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.20783.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin</p><p><b>Upvotes:</b> 47</p><p><b>Summary:</b> DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2503.20783</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2503.20783</guid>
|
||||||
<pubDate>Wed, 26 Mar 2025 17:59:14 +0000</pubDate>
|
<pubDate>Wed, 26 Mar 2025 17:59:14 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -46,7 +46,7 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>MoCha: Towards Movie-Grade Talking Character Synthesis</title>
|
<title>MoCha: Towards Movie-Grade Talking Character Synthesis</title>
|
||||||
<link>https://arxiv.org/abs/2503.23307</link>
|
<link>https://arxiv.org/abs/2503.23307</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.23307.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Cong Wei, Bo Sun, Haoyu Ma, Ji Hou, Felix Juefei-Xu, Zecheng He, Xiaoliang Dai, Luxin Zhang, Kunpeng Li, Tingbo Hou, Animesh Sinha, Peter Vajda, Wenhu Chen</p><p><b>Upvotes:</b> 130</p><p><b>Summary:</b> Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.23307.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Cong Wei, Bo Sun, Haoyu Ma, Ji Hou, Felix Juefei-Xu, Zecheng He, Xiaoliang Dai, Luxin Zhang, Kunpeng Li, Tingbo Hou, Animesh Sinha, Peter Vajda, Wenhu Chen</p><p><b>Upvotes:</b> 131</p><p><b>Summary:</b> Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2503.23307</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2503.23307</guid>
|
||||||
<pubDate>Sun, 30 Mar 2025 04:22:09 +0000</pubDate>
|
<pubDate>Sun, 30 Mar 2025 04:22:09 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -60,14 +60,14 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization</title>
|
<title>JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization</title>
|
||||||
<link>https://arxiv.org/abs/2503.23377</link>
|
<link>https://arxiv.org/abs/2503.23377</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.23377.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, Tat-Seng Chua</p><p><b>Upvotes:</b> 52</p><p><b>Summary:</b> This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion Transformer designed for synchronized audio-video generation (JAVG). Built upon the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to generate high-quality audio and video content simultaneously from open-ended user prompts. To ensure optimal synchronization, we introduce a fine-grained spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and fine-grained spatio-temporal priors, guiding the synchronization between the visual and auditory components. Furthermore, we propose a new benchmark, JavisBench, consisting of 10,140 high-quality text-captioned sounding videos spanning diverse scenes and complex real-world scenarios. Further, we specifically devise a robust metric for evaluating the synchronization between generated audio-video pairs in real-world complex content. Experimental results demonstrate that JavisDiT significantly outperforms existing methods by ensuring both high-quality generation and precise synchronization, setting a new standard for JAVG tasks. Our code, model, and dataset will be made publicly available at https://javisdit.github.io/.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.23377.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, Tat-Seng Chua</p><p><b>Upvotes:</b> 54</p><p><b>Summary:</b> This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion Transformer designed for synchronized audio-video generation (JAVG). Built upon the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to generate high-quality audio and video content simultaneously from open-ended user prompts. To ensure optimal synchronization, we introduce a fine-grained spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and fine-grained spatio-temporal priors, guiding the synchronization between the visual and auditory components. Furthermore, we propose a new benchmark, JavisBench, consisting of 10,140 high-quality text-captioned sounding videos spanning diverse scenes and complex real-world scenarios. Further, we specifically devise a robust metric for evaluating the synchronization between generated audio-video pairs in real-world complex content. Experimental results demonstrate that JavisDiT significantly outperforms existing methods by ensuring both high-quality generation and precise synchronization, setting a new standard for JAVG tasks. Our code, model, and dataset will be made publicly available at https://javisdit.github.io/.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2503.23377</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2503.23377</guid>
|
||||||
<pubDate>Sun, 30 Mar 2025 09:40:42 +0000</pubDate>
|
<pubDate>Sun, 30 Mar 2025 09:40:42 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>TextCrafter: Accurately Rendering Multiple Texts in Complex Visual Scenes</title>
|
<title>TextCrafter: Accurately Rendering Multiple Texts in Complex Visual Scenes</title>
|
||||||
<link>https://arxiv.org/abs/2503.23461</link>
|
<link>https://arxiv.org/abs/2503.23461</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.23461.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nikai Du, Zhennan Chen, Zhizhou Chen, Shan Gao, Xi Chen, Zhengkai Jiang, Jian Yang, Ying Tai</p><p><b>Upvotes:</b> 94</p><p><b>Summary:</b> This paper explores the task of Complex Visual Text Generation (CVTG), which centers on generating intricate textual content distributed across diverse regions within visual images. In CVTG, image generation models often rendering distorted and blurred visual text or missing some visual text. To tackle these challenges, we propose TextCrafter, a novel multi-visual text rendering method. TextCrafter employs a progressive strategy to decompose complex visual text into distinct components while ensuring robust alignment between textual content and its visual carrier. Additionally, it incorporates a token focus enhancement mechanism to amplify the prominence of visual text during the generation process. TextCrafter effectively addresses key challenges in CVTG tasks, such as text confusion, omissions, and blurriness. Moreover, we present a new benchmark dataset, CVTG-2K, tailored to rigorously evaluate the performance of generative models on CVTG tasks. Extensive experiments demonstrate that our method surpasses state-of-the-art approaches.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.23461.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nikai Du, Zhennan Chen, Zhizhou Chen, Shan Gao, Xi Chen, Zhengkai Jiang, Jian Yang, Ying Tai</p><p><b>Upvotes:</b> 95</p><p><b>Summary:</b> This paper explores the task of Complex Visual Text Generation (CVTG), which centers on generating intricate textual content distributed across diverse regions within visual images. In CVTG, image generation models often rendering distorted and blurred visual text or missing some visual text. To tackle these challenges, we propose TextCrafter, a novel multi-visual text rendering method. TextCrafter employs a progressive strategy to decompose complex visual text into distinct components while ensuring robust alignment between textual content and its visual carrier. Additionally, it incorporates a token focus enhancement mechanism to amplify the prominence of visual text during the generation process. TextCrafter effectively addresses key challenges in CVTG tasks, such as text confusion, omissions, and blurriness. Moreover, we present a new benchmark dataset, CVTG-2K, tailored to rigorously evaluate the performance of generative models on CVTG tasks. Extensive experiments demonstrate that our method surpasses state-of-the-art approaches.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2503.23461</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2503.23461</guid>
|
||||||
<pubDate>Sun, 30 Mar 2025 14:36:55 +0000</pubDate>
|
<pubDate>Sun, 30 Mar 2025 14:36:55 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -88,7 +88,7 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model</title>
|
<title>Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model</title>
|
||||||
<link>https://arxiv.org/abs/2503.24290</link>
|
<link>https://arxiv.org/abs/2503.24290</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.24290.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum</p><p><b>Upvotes:</b> 62</p><p><b>Summary:</b> We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training focusing on scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE (lambda=1, gamma=1) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both response length and benchmark performance, similar to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating remarkable efficiency -- requiring only a tenth of the training steps, compared to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our source code, parameter settings, training data, and model weights across various sizes.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.24290.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum</p><p><b>Upvotes:</b> 63</p><p><b>Summary:</b> We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training focusing on scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE (lambda=1, gamma=1) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both response length and benchmark performance, similar to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating remarkable efficiency -- requiring only a tenth of the training steps, compared to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our source code, parameter settings, training data, and model weights across various sizes.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2503.24290</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2503.24290</guid>
|
||||||
<pubDate>Mon, 31 Mar 2025 16:36:05 +0000</pubDate>
|
<pubDate>Mon, 31 Mar 2025 16:36:05 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -102,21 +102,21 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation</title>
|
<title>Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation</title>
|
||||||
<link>https://arxiv.org/abs/2503.24379</link>
|
<link>https://arxiv.org/abs/2503.24379</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.24379.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shengqiong Wu, Weicai Ye, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Shuicheng Yan, Hao Fei, Tat-Seng Chua</p><p><b>Upvotes:</b> 75</p><p><b>Summary:</b> To address the bottleneck of accurate user intent interpretation within the current video generation community, we present Any2Caption, a novel framework for controllable video generation under any condition. The key idea is to decouple various condition interpretation steps from the video synthesis step. By leveraging modern multimodal large language models (MLLMs), Any2Caption interprets diverse inputs--text, images, videos, and specialized cues such as region, motion, and camera poses--into dense, structured captions that offer backbone video generators with better guidance. We also introduce Any2CapIns, a large-scale dataset with 337K instances and 407K conditions for any-condition-to-caption instruction tuning. Comprehensive evaluations demonstrate significant improvements of our system in controllability and video quality across various aspects of existing video generation models. Project Page: https://sqwu.top/Any2Cap/</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2503.24379.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shengqiong Wu, Weicai Ye, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Shuicheng Yan, Hao Fei, Tat-Seng Chua</p><p><b>Upvotes:</b> 76</p><p><b>Summary:</b> To address the bottleneck of accurate user intent interpretation within the current video generation community, we present Any2Caption, a novel framework for controllable video generation under any condition. The key idea is to decouple various condition interpretation steps from the video synthesis step. By leveraging modern multimodal large language models (MLLMs), Any2Caption interprets diverse inputs--text, images, videos, and specialized cues such as region, motion, and camera poses--into dense, structured captions that offer backbone video generators with better guidance. We also introduce Any2CapIns, a large-scale dataset with 337K instances and 407K conditions for any-condition-to-caption instruction tuning. Comprehensive evaluations demonstrate significant improvements of our system in controllability and video quality across various aspects of existing video generation models. Project Page: https://sqwu.top/Any2Cap/</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2503.24379</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2503.24379</guid>
|
||||||
<pubDate>Mon, 31 Mar 2025 17:59:01 +0000</pubDate>
|
<pubDate>Mon, 31 Mar 2025 17:59:01 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems</title>
|
<title>Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems</title>
|
||||||
<link>https://arxiv.org/abs/2504.01990</link>
|
<link>https://arxiv.org/abs/2504.01990</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01990.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, Dekun Wu, Fengwei Teng, Xiaojun Jia, Jiawei Xu, Jinyu Xiang, Yizhang Lin, Tianming Liu, Tongliang Liu, Yu Su, Huan Sun, Glen Berseth, Jianyun Nie, Ian Foster, Logan Ward, Qingyun Wu, Yu Gu, Mingchen Zhuge, Xiangru Tang, Haohan Wang, Jiaxuan You, Chi Wang, Jian Pei, Qiang Yang, Xiaoliang Qi, Chenglin Wu</p><p><b>Upvotes:</b> 258</p><p><b>Summary:</b> The advent of large language models (LLMs) has catalyzed a transformative shift in artificial intelligence, paving the way for advanced intelligent agents capable of sophisticated reasoning, robust perception, and versatile action across diverse domains. As these agents increasingly drive AI research and practical applications, their design, evaluation, and continuous improvement present intricate, multifaceted challenges. This survey provides a comprehensive overview, framing intelligent agents within a modular, brain-inspired architecture that integrates principles from cognitive science, neuroscience, and computational research. We structure our exploration into four interconnected parts. First, we delve into the modular foundation of intelligent agents, systematically mapping their cognitive, perceptual, and operational modules onto analogous human brain functionalities, and elucidating core components such as memory, world modeling, reward processing, and emotion-like systems. Second, we discuss self-enhancement and adaptive evolution mechanisms, exploring how agents autonomously refine their capabilities, adapt to dynamic environments, and achieve continual learning through automated optimization paradigms, including emerging AutoML and LLM-driven optimization strategies. Third, we examine collaborative and evolutionary multi-agent systems, investigating the collective intelligence emerging from agent interactions, cooperation, and societal structures, highlighting parallels to human social dynamics. Finally, we address the critical imperative of building safe, secure, and beneficial AI systems, emphasizing intrinsic and extrinsic security threats, ethical alignment, robustness, and practical mitigation strategies necessary for trustworthy real-world deployment.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01990.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, Dekun Wu, Fengwei Teng, Xiaojun Jia, Jiawei Xu, Jinyu Xiang, Yizhang Lin, Tianming Liu, Tongliang Liu, Yu Su, Huan Sun, Glen Berseth, Jianyun Nie, Ian Foster, Logan Ward, Qingyun Wu, Yu Gu, Mingchen Zhuge, Xiangru Tang, Haohan Wang, Jiaxuan You, Chi Wang, Jian Pei, Qiang Yang, Xiaoliang Qi, Chenglin Wu</p><p><b>Upvotes:</b> 271</p><p><b>Summary:</b> The advent of large language models (LLMs) has catalyzed a transformative shift in artificial intelligence, paving the way for advanced intelligent agents capable of sophisticated reasoning, robust perception, and versatile action across diverse domains. As these agents increasingly drive AI research and practical applications, their design, evaluation, and continuous improvement present intricate, multifaceted challenges. This survey provides a comprehensive overview, framing intelligent agents within a modular, brain-inspired architecture that integrates principles from cognitive science, neuroscience, and computational research. We structure our exploration into four interconnected parts. First, we delve into the modular foundation of intelligent agents, systematically mapping their cognitive, perceptual, and operational modules onto analogous human brain functionalities, and elucidating core components such as memory, world modeling, reward processing, and emotion-like systems. Second, we discuss self-enhancement and adaptive evolution mechanisms, exploring how agents autonomously refine their capabilities, adapt to dynamic environments, and achieve continual learning through automated optimization paradigms, including emerging AutoML and LLM-driven optimization strategies. Third, we examine collaborative and evolutionary multi-agent systems, investigating the collective intelligence emerging from agent interactions, cooperation, and societal structures, highlighting parallels to human social dynamics. Finally, we address the critical imperative of building safe, secure, and beneficial AI systems, emphasizing intrinsic and extrinsic security threats, ethical alignment, robustness, and practical mitigation strategies necessary for trustworthy real-world deployment.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.01990</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.01990</guid>
|
||||||
<pubDate>Mon, 31 Mar 2025 18:00:29 +0000</pubDate>
|
<pubDate>Mon, 31 Mar 2025 18:00:29 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources</title>
|
<title>Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources</title>
|
||||||
<link>https://arxiv.org/abs/2504.00595</link>
|
<link>https://arxiv.org/abs/2504.00595</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.00595.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Weizhi Wang, Yu Tian, Linjie Yang, Heng Wang, Xifeng Yan</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> The reproduction of state-of-the-art multimodal LLM pre-training faces barriers at every stage of the pipeline, including high-quality data filtering, multimodal data mixture strategies, sequence packing techniques, and training frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic image resolution and multimodal sequence packing to significantly enhance pre-training efficiency. The training dataset was carefully curated using both MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based filtering methods, substantially improving data quality and training efficiency. The Open-Qwen2VL pre-training is conducted on academic level 8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista, indicating the remarkable training efficiency of Open-Qwen2VL. We open-source all aspects of our work, including compute-efficient and data-efficient training details, data filtering methods, sequence packing scripts, pre-training data in WebDataset format, FSDP-based training codebase, and both base and instruction-tuned model checkpoints. We redefine "fully open" for multimodal LLMs as the complete release of: 1) the training codebase, 2) detailed data filtering techniques, and 3) all pre-training and supervised fine-tuning data used to develop the model.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.00595.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Weizhi Wang, Yu Tian, Linjie Yang, Heng Wang, Xifeng Yan</p><p><b>Upvotes:</b> 36</p><p><b>Summary:</b> The reproduction of state-of-the-art multimodal LLM pre-training faces barriers at every stage of the pipeline, including high-quality data filtering, multimodal data mixture strategies, sequence packing techniques, and training frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic image resolution and multimodal sequence packing to significantly enhance pre-training efficiency. The training dataset was carefully curated using both MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based filtering methods, substantially improving data quality and training efficiency. The Open-Qwen2VL pre-training is conducted on academic level 8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista, indicating the remarkable training efficiency of Open-Qwen2VL. We open-source all aspects of our work, including compute-efficient and data-efficient training details, data filtering methods, sequence packing scripts, pre-training data in WebDataset format, FSDP-based training codebase, and both base and instruction-tuned model checkpoints. We redefine "fully open" for multimodal LLMs as the complete release of: 1) the training codebase, 2) detailed data filtering techniques, and 3) all pre-training and supervised fine-tuning data used to develop the model.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.00595</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.00595</guid>
|
||||||
<pubDate>Tue, 01 Apr 2025 09:54:00 +0000</pubDate>
|
<pubDate>Tue, 01 Apr 2025 09:54:00 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -137,7 +137,7 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Multi-Token Attention</title>
|
<title>Multi-Token Attention</title>
|
||||||
<link>https://arxiv.org/abs/2504.00927</link>
|
<link>https://arxiv.org/abs/2504.00927</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.00927.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar</p><p><b>Upvotes:</b> 46</p><p><b>Summary:</b> Soft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.00927.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar</p><p><b>Upvotes:</b> 49</p><p><b>Summary:</b> Soft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.00927</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.00927</guid>
|
||||||
<pubDate>Tue, 01 Apr 2025 15:59:32 +0000</pubDate>
|
<pubDate>Tue, 01 Apr 2025 15:59:32 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -151,42 +151,28 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization</title>
|
<title>MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization</title>
|
||||||
<link>https://arxiv.org/abs/2504.00999</link>
|
<link>https://arxiv.org/abs/2504.00999</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.00999.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siyuan Li, Luyuan Zhang, Zedong Wang, Juanxi Tian, Cheng Tan, Zicheng Liu, Chang Yu, Qingsong Xie, Haonan Lu, Haoqian Wang, Zhen Lei</p><p><b>Upvotes:</b> 83</p><p><b>Summary:</b> Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation. However, most existing methods struggle to address the trade-off in shared latent space for generation quality vs. representation learning and efficiency. To push the limits of this paradigm, we propose MergeVQ, which incorporates token merging techniques into VQ-based generative models to bridge the gap between image generation and visual representation learning in a unified architecture. During pre-training, MergeVQ decouples top-k semantics from latent space with the token merge module after self-attention blocks in the encoder for subsequent Look-up Free Quantization (LFQ) and global alignment and recovers their fine-grained details through cross-attention in the decoder for reconstruction. As for the second-stage generation, we introduce MergeAR, which performs KV Cache compression for efficient raster-order prediction. Extensive experiments on ImageNet verify that MergeVQ as an AR generative model achieves competitive performance in both visual representation learning and image generation tasks while maintaining favorable token efficiency and inference speed. The code and model will be available at https://apexgen-x.github.io/MergeVQ.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.00999.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siyuan Li, Luyuan Zhang, Zedong Wang, Juanxi Tian, Cheng Tan, Zicheng Liu, Chang Yu, Qingsong Xie, Haonan Lu, Haoqian Wang, Zhen Lei</p><p><b>Upvotes:</b> 87</p><p><b>Summary:</b> Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation. However, most existing methods struggle to address the trade-off in shared latent space for generation quality vs. representation learning and efficiency. To push the limits of this paradigm, we propose MergeVQ, which incorporates token merging techniques into VQ-based generative models to bridge the gap between image generation and visual representation learning in a unified architecture. During pre-training, MergeVQ decouples top-k semantics from latent space with the token merge module after self-attention blocks in the encoder for subsequent Look-up Free Quantization (LFQ) and global alignment and recovers their fine-grained details through cross-attention in the decoder for reconstruction. As for the second-stage generation, we introduce MergeAR, which performs KV Cache compression for efficient raster-order prediction. Extensive experiments on ImageNet verify that MergeVQ as an AR generative model achieves competitive performance in both visual representation learning and image generation tasks while maintaining favorable token efficiency and inference speed. The code and model will be available at https://apexgen-x.github.io/MergeVQ.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.00999</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.00999</guid>
|
||||||
<pubDate>Tue, 01 Apr 2025 17:39:19 +0000</pubDate>
|
<pubDate>Tue, 01 Apr 2025 17:39:19 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction</title>
|
<title>AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction</title>
|
||||||
<link>https://arxiv.org/abs/2504.01014</link>
|
<link>https://arxiv.org/abs/2504.01014</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01014.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junhao Cheng, Yuying Ge, Yixiao Ge, Jing Liao, Ying Shan</p><p><b>Upvotes:</b> 64</p><p><b>Summary:</b> Recent advancements in image and video synthesis have opened up new promise in generative games. One particularly intriguing application is transforming characters from anime films into interactive, playable entities. This allows players to immerse themselves in the dynamic anime world as their favorite characters for life simulation through language instructions. Such games are defined as infinite game since they eliminate predetermined boundaries and fixed gameplay rules, where players can interact with the game world through open-ended language and experience ever-evolving storylines and environments. Recently, a pioneering approach for infinite anime life simulation employs large language models (LLMs) to translate multi-turn text dialogues into language instructions for image generation. However, it neglects historical visual context, leading to inconsistent gameplay. Furthermore, it only generates static images, failing to incorporate the dynamics necessary for an engaging gaming experience. In this work, we propose AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states, as illustrated in Figure 1. We introduce novel action-aware multimodal representations to represent animation shots, which can be decoded into high-quality video clips using a video diffusion model. By taking historical animation shot representations as context and predicting subsequent representations, AnimeGamer can generate games with contextual consistency and satisfactory dynamics. Extensive evaluations using both automated metrics and human evaluations demonstrate that AnimeGamer outperforms existing methods in various aspects of the gaming experience. Codes and checkpoints are available at https://github.com/TencentARC/AnimeGamer.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01014.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junhao Cheng, Yuying Ge, Yixiao Ge, Jing Liao, Ying Shan</p><p><b>Upvotes:</b> 66</p><p><b>Summary:</b> Recent advancements in image and video synthesis have opened up new promise in generative games. One particularly intriguing application is transforming characters from anime films into interactive, playable entities. This allows players to immerse themselves in the dynamic anime world as their favorite characters for life simulation through language instructions. Such games are defined as infinite game since they eliminate predetermined boundaries and fixed gameplay rules, where players can interact with the game world through open-ended language and experience ever-evolving storylines and environments. Recently, a pioneering approach for infinite anime life simulation employs large language models (LLMs) to translate multi-turn text dialogues into language instructions for image generation. However, it neglects historical visual context, leading to inconsistent gameplay. Furthermore, it only generates static images, failing to incorporate the dynamics necessary for an engaging gaming experience. In this work, we propose AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states, as illustrated in Figure 1. We introduce novel action-aware multimodal representations to represent animation shots, which can be decoded into high-quality video clips using a video diffusion model. By taking historical animation shot representations as context and predicting subsequent representations, AnimeGamer can generate games with contextual consistency and satisfactory dynamics. Extensive evaluations using both automated metrics and human evaluations demonstrate that AnimeGamer outperforms existing methods in various aspects of the gaming experience. Codes and checkpoints are available at https://github.com/TencentARC/AnimeGamer.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.01014</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.01014</guid>
|
||||||
<pubDate>Tue, 01 Apr 2025 17:57:18 +0000</pubDate>
|
<pubDate>Tue, 01 Apr 2025 17:57:18 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.01016</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01016.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tian-Xing Xu, Xiangjun Gao, Wenbo Hu, Xiaoyu Li, Song-Hai Zhang, Ying Shan</p><p><b>Upvotes:</b> 29</p><p><b>Summary:</b> Despite remarkable advancements in video depth estimation, existing methods exhibit inherent limitations in achieving geometric fidelity through the affine-invariant predictions, limiting their applicability in reconstruction and other metrically grounded downstream tasks. We propose GeometryCrafter, a novel framework that recovers high-fidelity point map sequences with temporal coherence from open-world videos, enabling accurate 3D/4D reconstruction, camera parameter estimation, and other depth-based applications. At the core of our approach lies a point map Variational Autoencoder (VAE) that learns a latent space agnostic to video latent distributions for effective point map encoding and decoding. Leveraging the VAE, we train a video diffusion model to model the distribution of point map sequences conditioned on the input videos. Extensive evaluations on diverse datasets demonstrate that GeometryCrafter achieves state-of-the-art 3D accuracy, temporal consistency, and generalization capability.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.01016</guid>
|
|
||||||
<pubDate>Tue, 01 Apr 2025 17:58:03 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Scaling Language-Free Visual Representation Learning</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.01017</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01017.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, Saining Xie</p><p><b>Upvotes:</b> 29</p><p><b>Summary:</b> Visual Self-Supervised Learning (SSL) currently underperforms Contrastive Language-Image Pretraining (CLIP) in multimodal settings such as Visual Question Answering (VQA). This multimodal gap is often attributed to the semantics introduced by language supervision, even though visual SSL and CLIP models are often trained on different data. In this work, we ask the question: "Do visual self-supervised approaches lag behind CLIP due to the lack of language supervision, or differences in the training data?" We study this question by training both visual SSL and CLIP models on the same MetaCLIP data, and leveraging VQA as a diverse testbed for vision encoders. In this controlled setup, visual SSL models scale better than CLIP models in terms of data and model capacity, and visual SSL performance does not saturate even after scaling up to 7B parameters. Consequently, we observe visual SSL methods achieve CLIP-level performance on a wide range of VQA and classic vision benchmarks. These findings demonstrate that pure visual SSL can match language-supervised visual pretraining at scale, opening new opportunities for vision-centric representation learning.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.01017</guid>
|
|
||||||
<pubDate>Tue, 01 Apr 2025 17:59:15 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>DeepSeek-R1 Thoughtology: Let's <think> about LLM Reasoning</title>
|
<title>DeepSeek-R1 Thoughtology: Let's <think> about LLM Reasoning</title>
|
||||||
<link>https://arxiv.org/abs/2504.07128</link>
|
<link>https://arxiv.org/abs/2504.07128</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.07128.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lù, Nicholas Meade, Dongchan Shin, Amirhossein Kazemnejad, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Siva Reddy</p><p><b>Upvotes:</b> 82</p><p><b>Summary:</b> Large Reasoning Models like DeepSeek-R1 mark a fundamental shift in how LLMs approach complex problems. Instead of directly producing an answer for a given input, DeepSeek-R1 creates detailed multi-step reasoning chains, seemingly "thinking" about a problem before providing an answer. This reasoning process is publicly available to the user, creating endless opportunities for studying the reasoning behaviour of the model and opening up the field of Thoughtology. Starting from a taxonomy of DeepSeek-R1's basic building blocks of reasoning, our analyses on DeepSeek-R1 investigate the impact and controllability of thought length, management of long or confusing contexts, cultural and safety concerns, and the status of DeepSeek-R1 vis-\`a-vis cognitive phenomena, such as human-like language processing and world modelling. Our findings paint a nuanced picture. Notably, we show DeepSeek-R1 has a 'sweet spot' of reasoning, where extra inference time can impair model performance. Furthermore, we find a tendency for DeepSeek-R1 to persistently ruminate on previously explored problem formulations, obstructing further exploration. We also note strong safety vulnerabilities of DeepSeek-R1 compared to its non-reasoning counterpart, which can also compromise safety-aligned LLMs.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.07128.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lù, Nicholas Meade, Dongchan Shin, Amirhossein Kazemnejad, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Siva Reddy</p><p><b>Upvotes:</b> 83</p><p><b>Summary:</b> Large Reasoning Models like DeepSeek-R1 mark a fundamental shift in how LLMs approach complex problems. Instead of directly producing an answer for a given input, DeepSeek-R1 creates detailed multi-step reasoning chains, seemingly "thinking" about a problem before providing an answer. This reasoning process is publicly available to the user, creating endless opportunities for studying the reasoning behaviour of the model and opening up the field of Thoughtology. Starting from a taxonomy of DeepSeek-R1's basic building blocks of reasoning, our analyses on DeepSeek-R1 investigate the impact and controllability of thought length, management of long or confusing contexts, cultural and safety concerns, and the status of DeepSeek-R1 vis-\`a-vis cognitive phenomena, such as human-like language processing and world modelling. Our findings paint a nuanced picture. Notably, we show DeepSeek-R1 has a 'sweet spot' of reasoning, where extra inference time can impair model performance. Furthermore, we find a tendency for DeepSeek-R1 to persistently ruminate on previously explored problem formulations, obstructing further exploration. We also note strong safety vulnerabilities of DeepSeek-R1 compared to its non-reasoning counterpart, which can also compromise safety-aligned LLMs.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.07128</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.07128</guid>
|
||||||
<pubDate>Wed, 02 Apr 2025 00:36:08 +0000</pubDate>
|
<pubDate>Wed, 02 Apr 2025 00:36:08 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidance</title>
|
<title>DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidance</title>
|
||||||
<link>https://arxiv.org/abs/2504.01724</link>
|
<link>https://arxiv.org/abs/2504.01724</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01724.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuxuan Luo, Zhengkun Rong, Lizhen Wang, Longhao Zhang, Tianshu Hu, Yongming Zhu</p><p><b>Upvotes:</b> 64</p><p><b>Summary:</b> While recent image-based human animation methods achieve realistic body and facial motion synthesis, critical gaps remain in fine-grained holistic controllability, multi-scale adaptability, and long-term temporal coherence, which leads to their lower expressiveness and robustness. We propose a diffusion transformer (DiT) based framework, DreamActor-M1, with hybrid guidance to overcome these limitations. For motion guidance, our hybrid control signals that integrate implicit facial representations, 3D head spheres, and 3D body skeletons achieve robust control of facial expressions and body movements, while producing expressive and identity-preserving animations. For scale adaptation, to handle various body poses and image scales ranging from portraits to full-body views, we employ a progressive training strategy using data with varying resolutions and scales. For appearance guidance, we integrate motion patterns from sequential frames with complementary visual references, ensuring long-term temporal coherence for unseen regions during complex movements. Experiments demonstrate that our method outperforms the state-of-the-art works, delivering expressive results for portraits, upper-body, and full-body generation with robust long-term consistency. Project Page: https://grisoon.github.io/DreamActor-M1/.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.01724.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuxuan Luo, Zhengkun Rong, Lizhen Wang, Longhao Zhang, Tianshu Hu, Yongming Zhu</p><p><b>Upvotes:</b> 65</p><p><b>Summary:</b> While recent image-based human animation methods achieve realistic body and facial motion synthesis, critical gaps remain in fine-grained holistic controllability, multi-scale adaptability, and long-term temporal coherence, which leads to their lower expressiveness and robustness. We propose a diffusion transformer (DiT) based framework, DreamActor-M1, with hybrid guidance to overcome these limitations. For motion guidance, our hybrid control signals that integrate implicit facial representations, 3D head spheres, and 3D body skeletons achieve robust control of facial expressions and body movements, while producing expressive and identity-preserving animations. For scale adaptation, to handle various body poses and image scales ranging from portraits to full-body views, we employ a progressive training strategy using data with varying resolutions and scales. For appearance guidance, we integrate motion patterns from sequential frames with complementary visual references, ensuring long-term temporal coherence for unseen regions during complex movements. Experiments demonstrate that our method outperforms the state-of-the-art works, delivering expressive results for portraits, upper-body, and full-body generation with robust long-term consistency. Project Page: https://grisoon.github.io/DreamActor-M1/.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.01724</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.01724</guid>
|
||||||
<pubDate>Wed, 02 Apr 2025 13:30:32 +0000</pubDate>
|
<pubDate>Wed, 02 Apr 2025 13:30:32 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -214,42 +200,35 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>SkyReels-A2: Compose Anything in Video Diffusion Transformers</title>
|
<title>SkyReels-A2: Compose Anything in Video Diffusion Transformers</title>
|
||||||
<link>https://arxiv.org/abs/2504.02436</link>
|
<link>https://arxiv.org/abs/2504.02436</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02436.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, Yahui Zhou</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> This paper presents SkyReels-A2, a controllable video generation framework capable of assembling arbitrary visual elements (e.g., characters, objects, backgrounds) into synthesized videos based on textual prompts while maintaining strict consistency with reference images for each element. We term this task elements-to-video (E2V), whose primary challenges lie in preserving the fidelity of each reference element, ensuring coherent composition of the scene, and achieving natural outputs. To address these, we first design a comprehensive data pipeline to construct prompt-reference-video triplets for model training. Next, we propose a novel image-text joint embedding model to inject multi-element representations into the generative process, balancing element-specific consistency with global coherence and text alignment. We also optimize the inference pipeline for both speed and output stability. Moreover, we introduce a carefully curated benchmark for systematic evaluation, i.e, A2 Bench. Experiments demonstrate that our framework can generate diverse, high-quality videos with precise element control. SkyReels-A2 is the first open-source commercial grade model for the generation of E2V, performing favorably against advanced closed-source commercial models. We anticipate SkyReels-A2 will advance creative applications such as drama and virtual e-commerce, pushing the boundaries of controllable video generation.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02436.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, Yahui Zhou</p><p><b>Upvotes:</b> 36</p><p><b>Summary:</b> This paper presents SkyReels-A2, a controllable video generation framework capable of assembling arbitrary visual elements (e.g., characters, objects, backgrounds) into synthesized videos based on textual prompts while maintaining strict consistency with reference images for each element. We term this task elements-to-video (E2V), whose primary challenges lie in preserving the fidelity of each reference element, ensuring coherent composition of the scene, and achieving natural outputs. To address these, we first design a comprehensive data pipeline to construct prompt-reference-video triplets for model training. Next, we propose a novel image-text joint embedding model to inject multi-element representations into the generative process, balancing element-specific consistency with global coherence and text alignment. We also optimize the inference pipeline for both speed and output stability. Moreover, we introduce a carefully curated benchmark for systematic evaluation, i.e, A2 Bench. Experiments demonstrate that our framework can generate diverse, high-quality videos with precise element control. SkyReels-A2 is the first open-source commercial grade model for the generation of E2V, performing favorably against advanced closed-source commercial models. We anticipate SkyReels-A2 will advance creative applications such as drama and virtual e-commerce, pushing the boundaries of controllable video generation.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02436</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.02436</guid>
|
||||||
<pubDate>Thu, 03 Apr 2025 09:50:50 +0000</pubDate>
|
<pubDate>Thu, 03 Apr 2025 09:50:50 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Inference-Time Scaling for Generalist Reward Modeling</title>
|
<title>Inference-Time Scaling for Generalist Reward Modeling</title>
|
||||||
<link>https://arxiv.org/abs/2504.02495</link>
|
<link>https://arxiv.org/abs/2504.02495</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02495.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu</p><p><b>Upvotes:</b> 53</p><p><b>Summary:</b> Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that proper learning methods could enable effective inference-time scalability. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models will be released and open-sourced.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02495.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu</p><p><b>Upvotes:</b> 54</p><p><b>Summary:</b> Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that proper learning methods could enable effective inference-time scalability. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models will be released and open-sourced.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02495</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.02495</guid>
|
||||||
<pubDate>Thu, 03 Apr 2025 11:19:49 +0000</pubDate>
|
<pubDate>Thu, 03 Apr 2025 11:19:49 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>ZClip: Adaptive Spike Mitigation for LLM Pre-Training</title>
|
<title>ZClip: Adaptive Spike Mitigation for LLM Pre-Training</title>
|
||||||
<link>https://arxiv.org/abs/2504.02507</link>
|
<link>https://arxiv.org/abs/2504.02507</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02507.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Abhay Kumar, Louis Owen, Nilabhra Roy Chowdhury, Fabian Güra</p><p><b>Upvotes:</b> 76</p><p><b>Summary:</b> Training large language models (LLMs) presents numerous challenges, including gradient instability and loss spikes. These phenomena can lead to catastrophic divergence, requiring costly checkpoint restoration and data batch skipping. Traditional gradient clipping techniques, such as constant or norm-based methods, fail to address these issues effectively due to their reliance on fixed thresholds or heuristics, leading to inefficient learning and requiring frequent manual intervention. In this work, we propose ZClip, an adaptive gradient clipping algorithm that dynamically adjusts the clipping threshold based on statistical properties of gradient norms over time. Unlike prior reactive strategies, ZClip proactively adapts to training dynamics without making any prior assumptions on the scale and the temporal evolution of gradient norms. At its core, it leverages z-score-based anomaly detection to identify and mitigate large gradient spikes, preventing malignant loss spikes while not interfering with convergence otherwise. Our code is available at: https://github.com/bluorion-com/ZClip.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02507.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Abhay Kumar, Louis Owen, Nilabhra Roy Chowdhury, Fabian Güra</p><p><b>Upvotes:</b> 77</p><p><b>Summary:</b> Training large language models (LLMs) presents numerous challenges, including gradient instability and loss spikes. These phenomena can lead to catastrophic divergence, requiring costly checkpoint restoration and data batch skipping. Traditional gradient clipping techniques, such as constant or norm-based methods, fail to address these issues effectively due to their reliance on fixed thresholds or heuristics, leading to inefficient learning and requiring frequent manual intervention. In this work, we propose ZClip, an adaptive gradient clipping algorithm that dynamically adjusts the clipping threshold based on statistical properties of gradient norms over time. Unlike prior reactive strategies, ZClip proactively adapts to training dynamics without making any prior assumptions on the scale and the temporal evolution of gradient norms. At its core, it leverages z-score-based anomaly detection to identify and mitigate large gradient spikes, preventing malignant loss spikes while not interfering with convergence otherwise. Our code is available at: https://github.com/bluorion-com/ZClip.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02507</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.02507</guid>
|
||||||
<pubDate>Thu, 03 Apr 2025 11:41:55 +0000</pubDate>
|
<pubDate>Thu, 03 Apr 2025 11:41:55 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation</title>
|
<title>Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation</title>
|
||||||
<link>https://arxiv.org/abs/2504.02542</link>
|
<link>https://arxiv.org/abs/2504.02542</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02542.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fa-Ting Hong, Zunnan Xu, Zixiang Zhou, Jun Zhou, Xiu Li, Qin Lin, Qinglin Lu, Dan Xu</p><p><b>Upvotes:</b> 41</p><p><b>Summary:</b> Talking head synthesis is vital for virtual avatars and human-computer interaction. However, most existing methods are typically limited to accepting control from a single primary modality, restricting their practical utility. To this end, we introduce ACTalker, an end-to-end video diffusion framework that supports both multi-signals control and single-signal control for talking head video generation. For multiple control, we design a parallel mamba structure with multiple branches, each utilizing a separate driving signal to control specific facial regions. A gate mechanism is applied across all branches, providing flexible control over video generation. To ensure natural coordination of the controlled video both temporally and spatially, we employ the mamba structure, which enables driving signals to manipulate feature tokens across both dimensions in each branch. Additionally, we introduce a mask-drop strategy that allows each driving signal to independently control its corresponding facial region within the mamba structure, preventing control conflicts. Experimental results demonstrate that our method produces natural-looking facial videos driven by diverse signals and that the mamba layer seamlessly integrates multiple driving modalities without conflict.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02542.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fa-Ting Hong, Zunnan Xu, Zixiang Zhou, Jun Zhou, Xiu Li, Qin Lin, Qinglin Lu, Dan Xu</p><p><b>Upvotes:</b> 44</p><p><b>Summary:</b> Talking head synthesis is vital for virtual avatars and human-computer interaction. However, most existing methods are typically limited to accepting control from a single primary modality, restricting their practical utility. To this end, we introduce ACTalker, an end-to-end video diffusion framework that supports both multi-signals control and single-signal control for talking head video generation. For multiple control, we design a parallel mamba structure with multiple branches, each utilizing a separate driving signal to control specific facial regions. A gate mechanism is applied across all branches, providing flexible control over video generation. To ensure natural coordination of the controlled video both temporally and spatially, we employ the mamba structure, which enables driving signals to manipulate feature tokens across both dimensions in each branch. Additionally, we introduce a mask-drop strategy that allows each driving signal to independently control its corresponding facial region within the mamba structure, preventing control conflicts. Experimental results demonstrate that our method produces natural-looking facial videos driven by diverse signals and that the mamba layer seamlessly integrates multiple driving modalities without conflict.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02542</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.02542</guid>
|
||||||
<pubDate>Thu, 03 Apr 2025 12:44:41 +0000</pubDate>
|
<pubDate>Thu, 03 Apr 2025 12:44:41 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.02587</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02587.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, Pengfei Liu</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Reinforcement learning (RL) has recently shown strong potential in improving the reasoning capabilities of large language models and is now being actively extended to vision-language models (VLMs). However, existing RL applications in VLMs often rely on heavily engineered frameworks that hinder reproducibility and accessibility, while lacking standardized evaluation protocols, making it difficult to compare results or interpret training dynamics. This work introduces a transparent, from-scratch framework for RL in VLMs, offering a minimal yet functional four-step pipeline validated across multiple models and datasets. In addition, a standardized evaluation scheme is proposed to assess training dynamics and reflective behaviors. Extensive experiments on visual reasoning tasks uncover key empirical findings: response length is sensitive to random seeds, reflection correlates with output length, and RL consistently outperforms supervised fine-tuning (SFT) in generalization, even with high-quality data. These findings, together with the proposed framework, aim to establish a reproducible baseline and support broader engagement in RL-based VLM research.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02587</guid>
|
|
||||||
<pubDate>Thu, 03 Apr 2025 13:53:28 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving</title>
|
<title>Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving</title>
|
||||||
<link>https://arxiv.org/abs/2504.02605</link>
|
<link>https://arxiv.org/abs/2504.02605</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02605.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, Liang Xiang</p><p><b>Upvotes:</b> 44</p><p><b>Summary:</b> The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue. However, existing benchmarks, such as SWE-bench, focus almost exclusively on Python, making them insufficient for evaluating Large Language Models (LLMs) across diverse software ecosystems. To address this, we introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a total of 1,632 high-quality instances, which were carefully annotated from 2,456 candidates by 68 expert annotators, ensuring that the benchmark can provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we evaluate a series of state-of-the-art models using three representative methods (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with key empirical insights. In addition, we launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning (RL) training datasets for issue-resolving tasks. As an initial contribution, we release a set of 4,723 well-structured instances spanning seven programming languages, laying a solid foundation for RL research in this domain. More importantly, we open-source our entire data production pipeline, along with detailed tutorials, encouraging the open-source community to continuously contribute and expand the dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL community as catalysts for advancing RL toward its full potential, bringing us one step closer to the dawn of AGI.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02605.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, Liang Xiang</p><p><b>Upvotes:</b> 45</p><p><b>Summary:</b> The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue. However, existing benchmarks, such as SWE-bench, focus almost exclusively on Python, making them insufficient for evaluating Large Language Models (LLMs) across diverse software ecosystems. To address this, we introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a total of 1,632 high-quality instances, which were carefully annotated from 2,456 candidates by 68 expert annotators, ensuring that the benchmark can provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we evaluate a series of state-of-the-art models using three representative methods (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with key empirical insights. In addition, we launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning (RL) training datasets for issue-resolving tasks. As an initial contribution, we release a set of 4,723 well-structured instances spanning seven programming languages, laying a solid foundation for RL research in this domain. More importantly, we open-source our entire data production pipeline, along with detailed tutorials, encouraging the open-source community to continuously contribute and expand the dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL community as catalysts for advancing RL toward its full potential, bringing us one step closer to the dawn of AGI.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02605</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.02605</guid>
|
||||||
<pubDate>Thu, 03 Apr 2025 14:06:17 +0000</pubDate>
|
<pubDate>Thu, 03 Apr 2025 14:06:17 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -260,13 +239,6 @@
|
|||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02782</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.02782</guid>
|
||||||
<pubDate>Thu, 03 Apr 2025 17:23:16 +0000</pubDate>
|
<pubDate>Thu, 03 Apr 2025 17:23:16 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>MegaMath: Pushing the Limits of Open Math Corpora</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.02807</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.02807.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, Eric P. Xing</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Mathematical reasoning is a cornerstone of human intelligence and a key benchmark for advanced capabilities in large language models (LLMs). However, the research community still lacks an open, large-scale, high-quality corpus tailored to the demands of math-centric LLM pre-training. We present MegaMath, an open dataset curated from diverse, math-focused sources through following practices: (1) Revisiting web data: We re-extracted mathematical documents from Common Crawl with math-oriented HTML optimizations, fasttext-based filtering and deduplication, all for acquiring higher-quality data on the Internet. (2) Recalling Math-related code data: We identified high quality math-related code from large code training corpus, Stack-V2, further enhancing data diversity. (3) Exploring Synthetic data: We synthesized QA-style text, math-related code, and interleaved text-code blocks from web data or code data. By integrating these strategies and validating their effectiveness through extensive ablations, MegaMath delivers 371B tokens with the largest quantity and top quality among existing open math pre-training datasets.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02807</guid>
|
|
||||||
<pubDate>Thu, 03 Apr 2025 17:52:07 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing</title>
|
<title>Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing</title>
|
||||||
<link>https://arxiv.org/abs/2504.02826</link>
|
<link>https://arxiv.org/abs/2504.02826</link>
|
||||||
@@ -274,13 +246,6 @@
|
|||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.02826</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.02826</guid>
|
||||||
<pubDate>Thu, 03 Apr 2025 17:59:56 +0000</pubDate>
|
<pubDate>Thu, 03 Apr 2025 17:59:56 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>Agentic Knowledgeable Self-awareness</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.03553</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.03553.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shuofei Qiao, Zhisong Qiu, Baochang Ren, Xiaobin Wang, Xiangyuan Ru, Ningyu Zhang, Xiang Chen, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen</p><p><b>Upvotes:</b> 28</p><p><b>Summary:</b> Large Language Models (LLMs) have achieved considerable performance across various agentic planning tasks. However, traditional agent planning approaches adopt a "flood irrigation" methodology that indiscriminately injects gold trajectories, external feedback, and domain knowledge into agent models. This practice overlooks the fundamental human cognitive principle of situational self-awareness during decision-making-the ability to dynamically assess situational demands and strategically employ resources during decision-making. We propose agentic knowledgeable self-awareness to address this gap, a novel paradigm enabling LLM-based agents to autonomously regulate knowledge utilization. Specifically, we propose KnowSelf, a data-centric approach that applies agents with knowledgeable self-awareness like humans. Concretely, we devise a heuristic situation judgement criterion to mark special tokens on the agent's self-explored trajectories for collecting training data. Through a two-stage training process, the agent model can switch between different situations by generating specific special tokens, achieving optimal planning effects with minimal costs. Our experiments demonstrate that KnowSelf can outperform various strong baselines on different tasks and models with minimal use of external knowledge. Code is available at https://github.com/zjunlp/KnowSelf.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.03553</guid>
|
|
||||||
<pubDate>Fri, 04 Apr 2025 16:03:38 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>Rethinking Reflection in Pre-Training</title>
|
<title>Rethinking Reflection in Pre-Training</title>
|
||||||
<link>https://arxiv.org/abs/2504.04022</link>
|
<link>https://arxiv.org/abs/2504.04022</link>
|
||||||
@@ -295,38 +260,31 @@
|
|||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.04718</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.04718</guid>
|
||||||
<pubDate>Mon, 07 Apr 2025 04:01:17 +0000</pubDate>
|
<pubDate>Mon, 07 Apr 2025 04:01:17 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.04823</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.04823.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, Lu Hou</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Recent advancements in reasoning language models have demonstrated remarkable performance in complex tasks, but their extended chain-of-thought reasoning process increases inference overhead. While quantization has been widely adopted to reduce the inference cost of large language models, its impact on reasoning models remains understudied. In this study, we conduct the first systematic study on quantized reasoning models, evaluating the open-sourced DeepSeek-R1-Distilled Qwen and LLaMA families ranging from 1.5B to 70B parameters, and QwQ-32B. Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths, with extensive evaluation across mathematical (AIME, MATH-500), scientific (GPQA), and programming (LiveCodeBench) reasoning benchmarks. Our findings reveal that while lossless quantization can be achieved with W8A8 or W4A16 quantization, lower bit-widths introduce significant accuracy risks. We further identify model size, model origin, and task difficulty as critical determinants of performance. Contrary to expectations, quantized models do not exhibit increased output lengths. In addition, strategically scaling the model sizes or reasoning steps can effectively enhance the performance. All quantized models and codes will be open-sourced in https://github.com/ruikangliu/Quantized-Reasoning-Models.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.04823</guid>
|
|
||||||
<pubDate>Mon, 07 Apr 2025 08:22:45 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis</title>
|
<title>FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis</title>
|
||||||
<link>https://arxiv.org/abs/2504.04842</link>
|
<link>https://arxiv.org/abs/2504.04842</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.04842.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, Mu Xu</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> Creating a realistic animatable avatar from a single static portrait remains challenging. Existing approaches often struggle to capture subtle facial expressions, the associated global body movements, and the dynamic background. To address these limitations, we propose a novel framework that leverages a pretrained video diffusion transformer model to generate high-fidelity, coherent talking portraits with controllable motion dynamics. At the core of our work is a dual-stage audio-visual alignment strategy. In the first stage, we employ a clip-level training scheme to establish coherent global motion by aligning audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals. To preserve identity without compromising motion flexibility, we replace the commonly used reference network with a facial-focused cross-attention module that effectively maintains facial consistency throughout the video. Furthermore, we integrate a motion intensity modulation module that explicitly controls expression and body motion intensity, enabling controllable manipulation of portrait movements beyond mere lip motion. Extensive experimental results show that our proposed approach achieves higher quality with better realism, coherence, motion intensity, and identity preservation. Ours project page: https://fantasy-amap.github.io/fantasy-talking/.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.04842.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, Mu Xu</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> Creating a realistic animatable avatar from a single static portrait remains challenging. Existing approaches often struggle to capture subtle facial expressions, the associated global body movements, and the dynamic background. To address these limitations, we propose a novel framework that leverages a pretrained video diffusion transformer model to generate high-fidelity, coherent talking portraits with controllable motion dynamics. At the core of our work is a dual-stage audio-visual alignment strategy. In the first stage, we employ a clip-level training scheme to establish coherent global motion by aligning audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals. To preserve identity without compromising motion flexibility, we replace the commonly used reference network with a facial-focused cross-attention module that effectively maintains facial consistency throughout the video. Furthermore, we integrate a motion intensity modulation module that explicitly controls expression and body motion intensity, enabling controllable manipulation of portrait movements beyond mere lip motion. Extensive experimental results show that our proposed approach achieves higher quality with better realism, coherence, motion intensity, and identity preservation. Ours project page: https://fantasy-amap.github.io/fantasy-talking/.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.04842</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.04842</guid>
|
||||||
<pubDate>Mon, 07 Apr 2025 08:56:01 +0000</pubDate>
|
<pubDate>Mon, 07 Apr 2025 08:56:01 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday Home Clusters</title>
|
<title>PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday Home Clusters</title>
|
||||||
<link>https://arxiv.org/abs/2504.08791</link>
|
<link>https://arxiv.org/abs/2504.08791</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08791.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zonghang Li, Tao Li, Wenjiao Feng, Mohsen Guizani, Hongfang Yu</p><p><b>Upvotes:</b> 123</p><p><b>Summary:</b> Emergency of DeepSeek R1 and QwQ 32B have broken through performance barriers for running frontier large language models (LLMs) on home devices. While consumer hardware is getting stronger and model quantization is improving, existing end-side solutions still demand GPU clusters, large RAM/VRAM, and high bandwidth, far beyond what a common home cluster can handle. This paper introduces prima.cpp, a distributed inference system that runs 70B-scale models on everyday home devices using a mix of CPU/GPU, low RAM/VRAM, Wi-Fi, and cross-platform support. It uses mmap to manage model weights and introduces piped-ring parallelism with prefetching to hide disk loading. By modeling heterogeneity in computation, communication, disk, memory (and its management behavior), and OS, it optimally assigns model layers to each device's CPU and GPU, further reducing token latency. An elegant algorithm named Halda is proposed to solve this NP-hard assignment problem. We evaluate prima.cpp on a common four-node home cluster. It outperforms llama.cpp, exo, and dllama on 30B+ models while keeping memory pressure below 6%. This brings frontier 30B-70B models, such as Llama 3, DeepSeek R1, Qwen 2.5, and QwQ to home assistants, making advanced AI truly accessible to individuals. The code is open source and available at https://github.com/Lizonghang/prima.cpp.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08791.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zonghang Li, Tao Li, Wenjiao Feng, Mohsen Guizani, Hongfang Yu</p><p><b>Upvotes:</b> 127</p><p><b>Summary:</b> Emergency of DeepSeek R1 and QwQ 32B have broken through performance barriers for running frontier large language models (LLMs) on home devices. While consumer hardware is getting stronger and model quantization is improving, existing end-side solutions still demand GPU clusters, large RAM/VRAM, and high bandwidth, far beyond what a common home cluster can handle. This paper introduces prima.cpp, a distributed inference system that runs 70B-scale models on everyday home devices using a mix of CPU/GPU, low RAM/VRAM, Wi-Fi, and cross-platform support. It uses mmap to manage model weights and introduces piped-ring parallelism with prefetching to hide disk loading. By modeling heterogeneity in computation, communication, disk, memory (and its management behavior), and OS, it optimally assigns model layers to each device's CPU and GPU, further reducing token latency. An elegant algorithm named Halda is proposed to solve this NP-hard assignment problem. We evaluate prima.cpp on a common four-node home cluster. It outperforms llama.cpp, exo, and dllama on 30B+ models while keeping memory pressure below 6%. This brings frontier 30B-70B models, such as Llama 3, DeepSeek R1, Qwen 2.5, and QwQ to home assistants, making advanced AI truly accessible to individuals. The code is open source and available at https://github.com/Lizonghang/prima.cpp.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.08791</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.08791</guid>
|
||||||
<pubDate>Mon, 07 Apr 2025 13:46:21 +0000</pubDate>
|
<pubDate>Mon, 07 Apr 2025 13:46:21 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>One-Minute Video Generation with Test-Time Training</title>
|
<title>One-Minute Video Generation with Test-Time Training</title>
|
||||||
<link>https://arxiv.org/abs/2504.05298</link>
|
<link>https://arxiv.org/abs/2504.05298</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.05298.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, Tatsunori Hashimoto, Sanmi Koyejo, Yejin Choi, Yu Sun, Xiaolong Wang</p><p><b>Upvotes:</b> 98</p><p><b>Summary:</b> Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.05298.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, Tatsunori Hashimoto, Sanmi Koyejo, Yejin Choi, Yu Sun, Xiaolong Wang</p><p><b>Upvotes:</b> 100</p><p><b>Summary:</b> Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.05298</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.05298</guid>
|
||||||
<pubDate>Mon, 07 Apr 2025 17:56:31 +0000</pubDate>
|
<pubDate>Mon, 07 Apr 2025 17:56:31 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>SmolVLM: Redefining small and efficient multimodal models</title>
|
<title>SmolVLM: Redefining small and efficient multimodal models</title>
|
||||||
<link>https://arxiv.org/abs/2504.05299</link>
|
<link>https://arxiv.org/abs/2504.05299</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.05299.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, Thomas Wolf</p><p><b>Upvotes:</b> 171</p><p><b>Summary:</b> Large Vision-Language Models (VLMs) deliver exceptional performance but require significant computational resources, limiting their deployment on mobile and edge devices. Smaller VLMs typically mirror design choices of larger models, such as extensive image tokenization, leading to inefficient GPU memory usage and constrained practicality for on-device applications. We introduce SmolVLM, a series of compact multimodal models specifically engineered for resource-efficient inference. We systematically explore architectural configurations, tokenization strategies, and data curation optimized for low computational overhead. Through this, we identify key design choices that yield substantial performance gains on image and video tasks with minimal memory footprints. Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during inference and outperforms the 300-times larger Idefics-80B model, despite an 18-month development gap. Our largest model, at 2.2B parameters, rivals state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend beyond static images, demonstrating robust video comprehension capabilities. Our results emphasize that strategic architectural optimizations, aggressive yet efficient tokenization, and carefully curated training data significantly enhance multimodal performance, facilitating practical, energy-efficient deployments at significantly smaller scales.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.05299.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, Thomas Wolf</p><p><b>Upvotes:</b> 177</p><p><b>Summary:</b> Large Vision-Language Models (VLMs) deliver exceptional performance but require significant computational resources, limiting their deployment on mobile and edge devices. Smaller VLMs typically mirror design choices of larger models, such as extensive image tokenization, leading to inefficient GPU memory usage and constrained practicality for on-device applications. We introduce SmolVLM, a series of compact multimodal models specifically engineered for resource-efficient inference. We systematically explore architectural configurations, tokenization strategies, and data curation optimized for low computational overhead. Through this, we identify key design choices that yield substantial performance gains on image and video tasks with minimal memory footprints. Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during inference and outperforms the 300-times larger Idefics-80B model, despite an 18-month development gap. Our largest model, at 2.2B parameters, rivals state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend beyond static images, demonstrating robust video comprehension capabilities. Our results emphasize that strategic architectural optimizations, aggressive yet efficient tokenization, and carefully curated training data significantly enhance multimodal performance, facilitating practical, energy-efficient deployments at significantly smaller scales.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.05299</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.05299</guid>
|
||||||
<pubDate>Mon, 07 Apr 2025 17:58:57 +0000</pubDate>
|
<pubDate>Mon, 07 Apr 2025 17:58:57 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -368,14 +326,14 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Hogwild! Inference: Parallel LLM Generation via Concurrent Attention</title>
|
<title>Hogwild! Inference: Parallel LLM Generation via Concurrent Attention</title>
|
||||||
<link>https://arxiv.org/abs/2504.06261</link>
|
<link>https://arxiv.org/abs/2504.06261</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.06261.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Gleb Rodionov, Roman Garipov, Alina Shutova, George Yakushev, Vage Egiazarian, Anton Sinitsin, Denis Kuznedelev, Dan Alistarh</p><p><b>Upvotes:</b> 104</p><p><b>Summary:</b> Large Language Models (LLMs) have demonstrated the ability to tackle increasingly complex tasks through advanced reasoning, long-form content generation, and tool use. Solving these tasks often involves long inference-time computations. In human problem solving, a common strategy to expedite work is collaboration: by dividing the problem into sub-tasks, exploring different strategies concurrently, etc. Recent research has shown that LLMs can also operate in parallel by implementing explicit cooperation frameworks, such as voting mechanisms or the explicit creation of independent sub-tasks that can be executed in parallel. However, each of these frameworks may not be suitable for all types of tasks, which can hinder their applicability. In this work, we propose a different design approach: we run LLM "workers" in parallel , allowing them to synchronize via a concurrently-updated attention cache and prompt these workers to decide how best to collaborate. Our approach allows the instances to come up with their own collaboration strategy for the problem at hand, all the while "seeing" each other's partial progress in the concurrent cache. We implement this approach via Hogwild! Inference: a parallel LLM inference engine where multiple instances of the same LLM run in parallel with the same attention cache, with "instant" access to each other's generated tokens. Hogwild! inference takes advantage of Rotary Position Embeddings (RoPE) to avoid recomputation while improving parallel hardware utilization. We find that modern reasoning-capable LLMs can perform inference with shared Key-Value cache out of the box, without additional fine-tuning.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.06261.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Gleb Rodionov, Roman Garipov, Alina Shutova, George Yakushev, Vage Egiazarian, Anton Sinitsin, Denis Kuznedelev, Dan Alistarh</p><p><b>Upvotes:</b> 107</p><p><b>Summary:</b> Large Language Models (LLMs) have demonstrated the ability to tackle increasingly complex tasks through advanced reasoning, long-form content generation, and tool use. Solving these tasks often involves long inference-time computations. In human problem solving, a common strategy to expedite work is collaboration: by dividing the problem into sub-tasks, exploring different strategies concurrently, etc. Recent research has shown that LLMs can also operate in parallel by implementing explicit cooperation frameworks, such as voting mechanisms or the explicit creation of independent sub-tasks that can be executed in parallel. However, each of these frameworks may not be suitable for all types of tasks, which can hinder their applicability. In this work, we propose a different design approach: we run LLM "workers" in parallel , allowing them to synchronize via a concurrently-updated attention cache and prompt these workers to decide how best to collaborate. Our approach allows the instances to come up with their own collaboration strategy for the problem at hand, all the while "seeing" each other's partial progress in the concurrent cache. We implement this approach via Hogwild! Inference: a parallel LLM inference engine where multiple instances of the same LLM run in parallel with the same attention cache, with "instant" access to each other's generated tokens. Hogwild! inference takes advantage of Rotary Position Embeddings (RoPE) to avoid recomputation while improving parallel hardware utilization. We find that modern reasoning-capable LLMs can perform inference with shared Key-Value cache out of the box, without additional fine-tuning.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.06261</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.06261</guid>
|
||||||
<pubDate>Tue, 08 Apr 2025 17:59:41 +0000</pubDate>
|
<pubDate>Tue, 08 Apr 2025 17:59:41 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>OmniSVG: A Unified Scalable Vector Graphics Generation Model</title>
|
<title>OmniSVG: A Unified Scalable Vector Graphics Generation Model</title>
|
||||||
<link>https://arxiv.org/abs/2504.06263</link>
|
<link>https://arxiv.org/abs/2504.06263</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.06263.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yiying Yang, Wei Cheng, Sijin Chen, Xianfang Zeng, Jiaxu Zhang, Liao Wang, Gang Yu, Xingjun Ma, Yu-Gang Jiang</p><p><b>Upvotes:</b> 151</p><p><b>Summary:</b> Scalable Vector Graphics (SVG) is an important image format widely adopted in graphic design because of their resolution independence and editability. The study of generating high-quality SVG has continuously drawn attention from both designers and researchers in the AIGC community. However, existing methods either produces unstructured outputs with huge computational cost or is limited to generating monochrome icons of over-simplified structures. To produce high-quality and complex SVG, we propose OmniSVG, a unified framework that leverages pre-trained Vision-Language Models (VLMs) for end-to-end multimodal SVG generation. By parameterizing SVG commands and coordinates into discrete tokens, OmniSVG decouples structural logic from low-level geometry for efficient training while maintaining the expressiveness of complex SVG structure. To further advance the development of SVG synthesis, we introduce MMSVG-2M, a multimodal dataset with two million richly annotated SVG assets, along with a standardized evaluation protocol for conditional SVG generation tasks. Extensive experiments show that OmniSVG outperforms existing methods and demonstrates its potential for integration into professional SVG design workflows.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.06263.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yiying Yang, Wei Cheng, Sijin Chen, Xianfang Zeng, Jiaxu Zhang, Liao Wang, Gang Yu, Xingjun Ma, Yu-Gang Jiang</p><p><b>Upvotes:</b> 156</p><p><b>Summary:</b> Scalable Vector Graphics (SVG) is an important image format widely adopted in graphic design because of their resolution independence and editability. The study of generating high-quality SVG has continuously drawn attention from both designers and researchers in the AIGC community. However, existing methods either produces unstructured outputs with huge computational cost or is limited to generating monochrome icons of over-simplified structures. To produce high-quality and complex SVG, we propose OmniSVG, a unified framework that leverages pre-trained Vision-Language Models (VLMs) for end-to-end multimodal SVG generation. By parameterizing SVG commands and coordinates into discrete tokens, OmniSVG decouples structural logic from low-level geometry for efficient training while maintaining the expressiveness of complex SVG structure. To further advance the development of SVG synthesis, we introduce MMSVG-2M, a multimodal dataset with two million richly annotated SVG assets, along with a standardized evaluation protocol for conditional SVG generation tasks. Extensive experiments show that OmniSVG outperforms existing methods and demonstrates its potential for integration into professional SVG design workflows.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.06263</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.06263</guid>
|
||||||
<pubDate>Tue, 08 Apr 2025 17:59:49 +0000</pubDate>
|
<pubDate>Tue, 08 Apr 2025 17:59:49 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -389,17 +347,10 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability</title>
|
<title>Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability</title>
|
||||||
<link>https://arxiv.org/abs/2504.08003</link>
|
<link>https://arxiv.org/abs/2504.08003</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08003.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ning Li, Jingran Zhang, Justin Cui</p><p><b>Upvotes:</b> 47</p><p><b>Summary:</b> OpenAI's multimodal GPT-4o has demonstrated remarkable capabilities in image generation and editing, yet its ability to achieve world knowledge-informed semantic synthesis--seamlessly integrating domain knowledge, contextual reasoning, and instruction adherence--remains unproven. In this study, we systematically evaluate these capabilities across three critical dimensions: (1) Global Instruction Adherence, (2) Fine-Grained Editing Precision, and (3) Post-Generation Reasoning. While existing benchmarks highlight GPT-4o's strong capabilities in image generation and editing, our evaluation reveals GPT-4o's persistent limitations: the model frequently defaults to literal interpretations of instructions, inconsistently applies knowledge constraints, and struggles with conditional reasoning tasks. These findings challenge prevailing assumptions about GPT-4o's unified understanding and generation capabilities, exposing significant gaps in its dynamic knowledge integration. Our study calls for the development of more robust benchmarks and training strategies that go beyond surface-level alignment, emphasizing context-aware and reasoning-grounded multimodal generation.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08003.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ning Li, Jingran Zhang, Justin Cui</p><p><b>Upvotes:</b> 49</p><p><b>Summary:</b> OpenAI's multimodal GPT-4o has demonstrated remarkable capabilities in image generation and editing, yet its ability to achieve world knowledge-informed semantic synthesis--seamlessly integrating domain knowledge, contextual reasoning, and instruction adherence--remains unproven. In this study, we systematically evaluate these capabilities across three critical dimensions: (1) Global Instruction Adherence, (2) Fine-Grained Editing Precision, and (3) Post-Generation Reasoning. While existing benchmarks highlight GPT-4o's strong capabilities in image generation and editing, our evaluation reveals GPT-4o's persistent limitations: the model frequently defaults to literal interpretations of instructions, inconsistently applies knowledge constraints, and struggles with conditional reasoning tasks. These findings challenge prevailing assumptions about GPT-4o's unified understanding and generation capabilities, exposing significant gaps in its dynamic knowledge integration. Our study calls for the development of more robust benchmarks and training strategies that go beyond surface-level alignment, emphasizing context-aware and reasoning-grounded multimodal generation.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.08003</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.08003</guid>
|
||||||
<pubDate>Wed, 09 Apr 2025 16:10:15 +0000</pubDate>
|
<pubDate>Wed, 09 Apr 2025 16:10:15 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>A Unified Agentic Framework for Evaluating Conditional Image Generation</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.07046</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.07046.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jifang Wang, Xue Yang, Longyue Wang, Zhenran Xu, Yiyu Wang, Yaowei Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, Min Zhang</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Conditional image generation has gained significant attention for its ability to personalize content. However, the field faces challenges in developing task-agnostic, reliable, and explainable evaluation metrics. This paper introduces CIGEval, a unified agentic framework for comprehensive evaluation of conditional image generation tasks. CIGEval utilizes large multimodal models (LMMs) as its core, integrating a multi-functional toolbox and establishing a fine-grained evaluation framework. Additionally, we synthesize evaluation trajectories for fine-tuning, empowering smaller LMMs to autonomously select appropriate tools and conduct nuanced analyses based on tool outputs. Experiments across seven prominent conditional image generation tasks demonstrate that CIGEval (GPT-4o version) achieves a high correlation of 0.4625 with human assessments, closely matching the inter-annotator correlation of 0.47. Moreover, when implemented with 7B open-source LMMs using only 2.3K training trajectories, CIGEval surpasses the previous GPT-4o-based state-of-the-art method. Case studies on GPT-4o image generation highlight CIGEval's capability in identifying subtle issues related to subject consistency and adherence to control guidance, indicating its great potential for automating evaluation of image generation tasks with human-level reliability.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.07046</guid>
|
|
||||||
<pubDate>Wed, 09 Apr 2025 17:04:14 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens</title>
|
<title>OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens</title>
|
||||||
<link>https://arxiv.org/abs/2504.07096</link>
|
<link>https://arxiv.org/abs/2504.07096</link>
|
||||||
@@ -410,17 +361,10 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Kimi-VL Technical Report</title>
|
<title>Kimi-VL Technical Report</title>
|
||||||
<link>https://arxiv.org/abs/2504.07491</link>
|
<link>https://arxiv.org/abs/2504.07491</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.07491.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen</p><p><b>Upvotes:</b> 121</p><p><b>Summary:</b> We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2.8B parameters in its language decoder (Kimi-VL-A3B). Kimi-VL demonstrates strong performance across challenging domains: as a general-purpose VLM, Kimi-VL excels in multi-turn agent tasks (e.g., OSWorld), matching flagship models. Furthermore, it exhibits remarkable capabilities across diverse challenging vision language tasks, including college-level image and video comprehension, OCR, mathematical reasoning, and multi-image understanding. In comparative evaluations, it effectively competes with cutting-edge efficient VLMs such as GPT-4o-mini, Qwen2.5-VL-7B, and Gemma-3-12B-IT, while surpassing GPT-4o in several key domains. Kimi-VL also advances in processing long contexts and perceiving clearly. With a 128K extended context window, Kimi-VL can process diverse long inputs, achieving impressive scores of 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc. Its native-resolution vision encoder, MoonViT, further allows it to see and understand ultra-high-resolution visual inputs, achieving 83.2 on InfoVQA and 34.5 on ScreenSpot-Pro, while maintaining lower computational cost for common tasks. Building upon Kimi-VL, we introduce an advanced long-thinking variant: Kimi-VL-Thinking. Developed through long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL), this model exhibits strong long-horizon reasoning capabilities. It achieves scores of 61.7 on MMMU, 36.8 on MathVision, and 71.3 on MathVista while maintaining the compact 2.8B activated LLM parameters, setting a new standard for efficient multimodal thinking models. Code and models are publicly accessible at https://github.com/MoonshotAI/Kimi-VL.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.07491.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen</p><p><b>Upvotes:</b> 122</p><p><b>Summary:</b> We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2.8B parameters in its language decoder (Kimi-VL-A3B). Kimi-VL demonstrates strong performance across challenging domains: as a general-purpose VLM, Kimi-VL excels in multi-turn agent tasks (e.g., OSWorld), matching flagship models. Furthermore, it exhibits remarkable capabilities across diverse challenging vision language tasks, including college-level image and video comprehension, OCR, mathematical reasoning, and multi-image understanding. In comparative evaluations, it effectively competes with cutting-edge efficient VLMs such as GPT-4o-mini, Qwen2.5-VL-7B, and Gemma-3-12B-IT, while surpassing GPT-4o in several key domains. Kimi-VL also advances in processing long contexts and perceiving clearly. With a 128K extended context window, Kimi-VL can process diverse long inputs, achieving impressive scores of 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc. Its native-resolution vision encoder, MoonViT, further allows it to see and understand ultra-high-resolution visual inputs, achieving 83.2 on InfoVQA and 34.5 on ScreenSpot-Pro, while maintaining lower computational cost for common tasks. Building upon Kimi-VL, we introduce an advanced long-thinking variant: Kimi-VL-Thinking. Developed through long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL), this model exhibits strong long-horizon reasoning capabilities. It achieves scores of 61.7 on MMMU, 36.8 on MathVision, and 71.3 on MathVista while maintaining the compact 2.8B activated LLM parameters, setting a new standard for efficient multimodal thinking models. Code and models are publicly accessible at https://github.com/MoonshotAI/Kimi-VL.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.07491</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.07491</guid>
|
||||||
<pubDate>Thu, 10 Apr 2025 06:48:26 +0000</pubDate>
|
<pubDate>Thu, 10 Apr 2025 06:48:26 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.07615</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.07615.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, Tiancheng Zhao</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Recently DeepSeek R1 has shown that reinforcement learning (RL) can substantially improve the reasoning capabilities of Large Language Models (LLMs) through a simple yet effective design. The core of R1 lies in its rule-based reward formulation, which leverages tasks with deterministic ground-truth answers to enable precise and stable reward computation. In the visual domain, we similarly observe that a wide range of visual understanding tasks are inherently equipped with well-defined ground-truth annotations. This property makes them naturally compatible with rule-based reward mechanisms. Motivated by this observation, we investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs), aiming to enhance their visual reasoning capabilities. To this end, we develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks. Using this framework, we further explore the feasibility of applying RL to visual domain. Experimental results indicate that the RL-based model not only delivers competitive performance on visual understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in generalization ability. Furthermore, we conduct comprehensive ablation studies that uncover a series of noteworthy insights, including the presence of reward hacking in object detection, the emergence of the "OD aha moment", the impact of training data quality, and the scaling behavior of RL across different model sizes. Through these analyses, we aim to deepen the understanding of how reinforcement learning enhances the capabilities of vision-language models, and we hope our findings and open-source contributions will support continued progress in the vision-language RL community. Our code and model are available at https://github.com/om-ai-lab/VLM-R1</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.07615</guid>
|
|
||||||
<pubDate>Thu, 10 Apr 2025 10:05:15 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness</title>
|
<title>ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness</title>
|
||||||
<link>https://arxiv.org/abs/2504.10514</link>
|
<link>https://arxiv.org/abs/2504.10514</link>
|
||||||
@@ -435,13 +379,6 @@
|
|||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.08837</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.08837</guid>
|
||||||
<pubDate>Thu, 10 Apr 2025 17:41:56 +0000</pubDate>
|
<pubDate>Thu, 10 Apr 2025 17:41:56 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>HoloPart: Generative 3D Part Amodal Segmentation</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.07943</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.07943.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yunhan Yang, Yuan-Chen Guo, Yukun Huang, Zi-Xin Zou, Zhipeng Yu, Yangguang Li, Yan-Pei Cao, Xihui Liu</p><p><b>Upvotes:</b> 28</p><p><b>Summary:</b> 3D part amodal segmentation--decomposing a 3D shape into complete, semantically meaningful parts, even when occluded--is a challenging but crucial task for 3D content creation and understanding. Existing 3D part segmentation methods only identify visible surface patches, limiting their utility. Inspired by 2D amodal segmentation, we introduce this novel task to the 3D domain and propose a practical, two-stage approach, addressing the key challenges of inferring occluded 3D geometry, maintaining global shape consistency, and handling diverse shapes with limited training data. First, we leverage existing 3D part segmentation to obtain initial, incomplete part segments. Second, we introduce HoloPart, a novel diffusion-based model, to complete these segments into full 3D parts. HoloPart utilizes a specialized architecture with local attention to capture fine-grained part geometry and global shape context attention to ensure overall shape consistency. We introduce new benchmarks based on the ABO and PartObjaverse-Tiny datasets and demonstrate that HoloPart significantly outperforms state-of-the-art shape completion methods. By incorporating HoloPart with existing segmentation techniques, we achieve promising results on 3D part amodal segmentation, opening new avenues for applications in geometry editing, animation, and material assignment.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.07943</guid>
|
|
||||||
<pubDate>Thu, 10 Apr 2025 17:53:31 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning</title>
|
<title>VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning</title>
|
||||||
<link>https://arxiv.org/abs/2504.07956</link>
|
<link>https://arxiv.org/abs/2504.07956</link>
|
||||||
@@ -480,14 +417,14 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning</title>
|
<title>Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning</title>
|
||||||
<link>https://arxiv.org/abs/2504.08672</link>
|
<link>https://arxiv.org/abs/2504.08672</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08672.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, Zhiyong Wu</p><p><b>Upvotes:</b> 53</p><p><b>Summary:</b> Advancing LLM reasoning skills has captivated wide interest. However, current post-training techniques rely heavily on supervisory signals, such as outcome supervision or auxiliary reward models, which face the problem of scalability and high annotation costs. This motivates us to enhance LLM reasoning without the need for external supervision. We introduce a generalizable and purely unsupervised self-training framework, named Genius. Without external auxiliary, Genius requires to seek the optimal response sequence in a stepwise manner and optimize the LLM. To explore the potential steps and exploit the optimal ones, Genius introduces a stepwise foresight re-sampling strategy to sample and estimate the step value by simulating future outcomes. Further, we recognize that the unsupervised setting inevitably induces the intrinsic noise and uncertainty. To provide a robust optimization, we propose an advantage-calibrated optimization (ACO) loss function to mitigate estimation inconsistencies. Combining these techniques together, Genius provides an advanced initial step towards self-improve LLM reasoning with general queries and without supervision, revolutionizing reasoning scaling laws given the vast availability of general queries. The code will be released at https://github.com/xufangzhi/Genius.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08672.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, Zhiyong Wu</p><p><b>Upvotes:</b> 54</p><p><b>Summary:</b> Advancing LLM reasoning skills has captivated wide interest. However, current post-training techniques rely heavily on supervisory signals, such as outcome supervision or auxiliary reward models, which face the problem of scalability and high annotation costs. This motivates us to enhance LLM reasoning without the need for external supervision. We introduce a generalizable and purely unsupervised self-training framework, named Genius. Without external auxiliary, Genius requires to seek the optimal response sequence in a stepwise manner and optimize the LLM. To explore the potential steps and exploit the optimal ones, Genius introduces a stepwise foresight re-sampling strategy to sample and estimate the step value by simulating future outcomes. Further, we recognize that the unsupervised setting inevitably induces the intrinsic noise and uncertainty. To provide a robust optimization, we propose an advantage-calibrated optimization (ACO) loss function to mitigate estimation inconsistencies. Combining these techniques together, Genius provides an advanced initial step towards self-improve LLM reasoning with general queries and without supervision, revolutionizing reasoning scaling laws given the vast availability of general queries. The code will be released at https://github.com/xufangzhi/Genius.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.08672</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.08672</guid>
|
||||||
<pubDate>Fri, 11 Apr 2025 16:26:23 +0000</pubDate>
|
<pubDate>Fri, 11 Apr 2025 16:26:23 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model</title>
|
<title>Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model</title>
|
||||||
<link>https://arxiv.org/abs/2504.08685</link>
|
<link>https://arxiv.org/abs/2504.08685</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08685.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, Feng Cheng, Feilong Zuo Xuejiao Zeng, Ziyan Yang, Fangyuan Kong, Zhiwu Qing, Fei Xiao, Meng Wei, Tuyen Hoang, Siyu Zhang, Peihao Zhu, Qi Zhao, Jiangqiao Yan, Liangke Gui, Sheng Bi, Jiashi Li, Yuxi Ren, Rui Wang, Huixia Li, Xuefeng Xiao, Shu Liu, Feng Ling, Heng Zhang, Houmin Wei, Huafeng Kuang, Jerry Duncan, Junda Zhang, Junru Zheng, Li Sun, Manlin Zhang, Renfei Sun, Xiaobin Zhuang, Xiaojie Li, Xin Xia, Xuyan Chi, Yanghua Peng, Yuping Wang, Yuxuan Wang, Zhongkai Zhao, Zhuo Chen, Zuquan Song, Zhenheng Yang, Jiashi Feng, Jianchao Yang, Lu Jiang</p><p><b>Upvotes:</b> 121</p><p><b>Summary:</b> This technical report presents a cost-efficient strategy for training a video generation foundation model. We present a mid-sized research model with approximately 7 billion parameters (7B) called Seaweed-7B trained from scratch using 665,000 H100 GPU hours. Despite being trained with moderate computational resources, Seaweed-7B demonstrates highly competitive performance compared to contemporary video generation models of much larger size. Design choices are especially crucial in a resource-constrained setting. This technical report highlights the key design decisions that enhance the performance of the medium-sized diffusion model. Empirically, we make two observations: (1) Seaweed-7B achieves performance comparable to, or even surpasses, larger models trained on substantially greater GPU resources, and (2) our model, which exhibits strong generalization ability, can be effectively adapted across a wide range of downstream applications either by lightweight fine-tuning or continue training. See the project page at https://seaweed.video/</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.08685.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, Feng Cheng, Feilong Zuo Xuejiao Zeng, Ziyan Yang, Fangyuan Kong, Zhiwu Qing, Fei Xiao, Meng Wei, Tuyen Hoang, Siyu Zhang, Peihao Zhu, Qi Zhao, Jiangqiao Yan, Liangke Gui, Sheng Bi, Jiashi Li, Yuxi Ren, Rui Wang, Huixia Li, Xuefeng Xiao, Shu Liu, Feng Ling, Heng Zhang, Houmin Wei, Huafeng Kuang, Jerry Duncan, Junda Zhang, Junru Zheng, Li Sun, Manlin Zhang, Renfei Sun, Xiaobin Zhuang, Xiaojie Li, Xin Xia, Xuyan Chi, Yanghua Peng, Yuping Wang, Yuxuan Wang, Zhongkai Zhao, Zhuo Chen, Zuquan Song, Zhenheng Yang, Jiashi Feng, Jianchao Yang, Lu Jiang</p><p><b>Upvotes:</b> 122</p><p><b>Summary:</b> This technical report presents a cost-efficient strategy for training a video generation foundation model. We present a mid-sized research model with approximately 7 billion parameters (7B) called Seaweed-7B trained from scratch using 665,000 H100 GPU hours. Despite being trained with moderate computational resources, Seaweed-7B demonstrates highly competitive performance compared to contemporary video generation models of much larger size. Design choices are especially crucial in a resource-constrained setting. This technical report highlights the key design decisions that enhance the performance of the medium-sized diffusion model. Empirically, we make two observations: (1) Seaweed-7B achieves performance comparable to, or even surpasses, larger models trained on substantially greater GPU resources, and (2) our model, which exhibits strong generalization ability, can be effectively adapted across a wide range of downstream applications either by lightweight fine-tuning or continue training. See the project page at https://seaweed.video/</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.08685</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.08685</guid>
|
||||||
<pubDate>Fri, 11 Apr 2025 16:46:20 +0000</pubDate>
|
<pubDate>Fri, 11 Apr 2025 16:46:20 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -512,13 +449,6 @@
|
|||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.09925</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.09925</guid>
|
||||||
<pubDate>Mon, 14 Apr 2025 06:33:29 +0000</pubDate>
|
<pubDate>Mon, 14 Apr 2025 06:33:29 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>Mavors: Multi-granularity Video Representation for Multimodal Large Language Model</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.10068</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.10068.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, Bohan Zeng, Wentao Zhang, Fuzheng Zhang, Wenjing Yang, Di Zhang</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose Mavors, a novel framework that introduces Multi-granularity video representation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.10068</guid>
|
|
||||||
<pubDate>Mon, 14 Apr 2025 10:14:44 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>Heimdall: test-time scaling on the generative verification</title>
|
<title>Heimdall: test-time scaling on the generative verification</title>
|
||||||
<link>https://arxiv.org/abs/2504.10337</link>
|
<link>https://arxiv.org/abs/2504.10337</link>
|
||||||
@@ -526,17 +456,10 @@
|
|||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.10337</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.10337</guid>
|
||||||
<pubDate>Mon, 14 Apr 2025 15:46:33 +0000</pubDate>
|
<pubDate>Mon, 14 Apr 2025 15:46:33 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
|
||||||
<title>Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.10465</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.10465.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tao Zhang, Xiangtai Li, Zilong Huang, Yanwei Li, Weixian Lei, Xueqing Deng, Shihao Chen, Shunping Ji, Jiashi Feng</p><p><b>Upvotes:</b> 28</p><p><b>Summary:</b> Multimodal Large Language Models (MLLMs) achieve remarkable performance for fine-grained pixel-level understanding tasks. However, all the works rely heavily on extra components, such as vision encoder (CLIP), segmentation experts, leading to high system complexity and limiting model scaling. In this work, our goal is to explore a highly simplified MLLM without introducing extra components. Our work is motivated by the recent works on Single trAnsformer as a unified vIsion-Language Model (SAIL) design, where these works jointly learn vision tokens and text tokens in transformers. We present Pixel-SAIL, a single transformer for pixel-wise MLLM tasks. In particular, we present three technical improvements on the plain baseline. First, we design a learnable upsampling module to refine visual token features. Secondly, we propose a novel visual prompt injection strategy to enable the single transformer to understand visual prompt inputs and benefit from the early fusion of visual prompt embeddings and vision tokens. Thirdly, we introduce a vision expert distillation strategy to efficiently enhance the single transformer's fine-grained feature extraction capability. In addition, we have collected a comprehensive pixel understanding benchmark (PerBench), using a manual check. It includes three tasks: detailed object description, visual prompt-based question answering, and visual-text referring segmentation. Extensive experiments on four referring segmentation benchmarks, one visual prompt benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or even better results with a much simpler pipeline. Code and model will be released at https://github.com/magic-research/Sa2VA.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.10465</guid>
|
|
||||||
<pubDate>Mon, 14 Apr 2025 17:52:22 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
<item>
|
||||||
<title>InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models</title>
|
<title>InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models</title>
|
||||||
<link>https://arxiv.org/abs/2504.10479</link>
|
<link>https://arxiv.org/abs/2504.10479</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.10479.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Yue Cao, Yangzhou Liu, Weiye Xu, Hao Li, Jiahao Wang, Han Lv, Dengnian Chen, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang</p><p><b>Upvotes:</b> 239</p><p><b>Summary:</b> We introduce InternVL3, a significant advancement in the InternVL series featuring a native multimodal pre-training paradigm. Rather than adapting a text-only large language model (LLM) into a multimodal large language model (MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and linguistic capabilities from both diverse multimodal data and pure-text corpora during a single pre-training stage. This unified training paradigm effectively addresses the complexities and alignment challenges commonly encountered in conventional post-hoc training pipelines for MLLMs. To further improve performance and scalability, InternVL3 incorporates variable visual position encoding (V2PE) to support extended multimodal contexts, employs advanced post-training techniques such as supervised fine-tuning (SFT) and mixed preference optimization (MPO), and adopts test-time scaling strategies alongside an optimized training infrastructure. Extensive empirical evaluations demonstrate that InternVL3 delivers superior performance across a wide range of multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its capabilities remain highly competitive with leading proprietary models, including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also maintaining strong pure-language proficiency. In pursuit of open-science principles, we will publicly release both the training data and model weights to foster further research and development in next-generation MLLMs.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.10479.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Yue Cao, Yangzhou Liu, Weiye Xu, Hao Li, Jiahao Wang, Han Lv, Dengnian Chen, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang</p><p><b>Upvotes:</b> 250</p><p><b>Summary:</b> We introduce InternVL3, a significant advancement in the InternVL series featuring a native multimodal pre-training paradigm. Rather than adapting a text-only large language model (LLM) into a multimodal large language model (MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and linguistic capabilities from both diverse multimodal data and pure-text corpora during a single pre-training stage. This unified training paradigm effectively addresses the complexities and alignment challenges commonly encountered in conventional post-hoc training pipelines for MLLMs. To further improve performance and scalability, InternVL3 incorporates variable visual position encoding (V2PE) to support extended multimodal contexts, employs advanced post-training techniques such as supervised fine-tuning (SFT) and mixed preference optimization (MPO), and adopts test-time scaling strategies alongside an optimized training infrastructure. Extensive empirical evaluations demonstrate that InternVL3 delivers superior performance across a wide range of multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its capabilities remain highly competitive with leading proprietary models, including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also maintaining strong pure-language proficiency. In pursuit of open-science principles, we will publicly release both the training data and model weights to foster further research and development in next-generation MLLMs.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.10479</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.10479</guid>
|
||||||
<pubDate>Mon, 14 Apr 2025 17:59:25 +0000</pubDate>
|
<pubDate>Mon, 14 Apr 2025 17:59:25 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -550,42 +473,42 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients</title>
|
<title>How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients</title>
|
||||||
<link>https://arxiv.org/abs/2504.10766</link>
|
<link>https://arxiv.org/abs/2504.10766</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.10766.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ming Li, Yanhong Li, Ziyue Li, Tianyi Zhou</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> As the post-training of large language models (LLMs) advances from instruction-following to complex reasoning tasks, understanding how different data affect finetuning dynamics remains largely unexplored. In this paper, we present a spectral analysis of layer-wise gradients induced by low/high-quality instruction and reasoning data for LLM post-training. Our analysis reveals that widely-studied metrics for data evaluation, e.g., IFD, InsTag, Difficulty, and Reward, can be explained and unified by spectral properties computed from gradients' singular value decomposition (SVD). Specifically, higher-quality data are usually associated with lower nuclear norms and higher effective ranks. Notably, effective rank exhibits better robustness and resolution than nuclear norm in capturing subtle quality differences. For example, reasoning data achieves substantially higher effective ranks than instruction data, implying richer gradient structures on more complex tasks. Our experiments also highlight that models within the same family share similar gradient patterns regardless of their sizes, whereas different model families diverge significantly. Providing a unified view on the effects of data quality across instruction and reasoning data, this work illuminates the interplay between data quality and training stability, shedding novel insights into developing better data exploration strategies for post-training.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.10766.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ming Li, Yanhong Li, Ziyue Li, Tianyi Zhou</p><p><b>Upvotes:</b> 40</p><p><b>Summary:</b> As the post-training of large language models (LLMs) advances from instruction-following to complex reasoning tasks, understanding how different data affect finetuning dynamics remains largely unexplored. In this paper, we present a spectral analysis of layer-wise gradients induced by low/high-quality instruction and reasoning data for LLM post-training. Our analysis reveals that widely-studied metrics for data evaluation, e.g., IFD, InsTag, Difficulty, and Reward, can be explained and unified by spectral properties computed from gradients' singular value decomposition (SVD). Specifically, higher-quality data are usually associated with lower nuclear norms and higher effective ranks. Notably, effective rank exhibits better robustness and resolution than nuclear norm in capturing subtle quality differences. For example, reasoning data achieves substantially higher effective ranks than instruction data, implying richer gradient structures on more complex tasks. Our experiments also highlight that models within the same family share similar gradient patterns regardless of their sizes, whereas different model families diverge significantly. Providing a unified view on the effects of data quality across instruction and reasoning data, this work illuminates the interplay between data quality and training stability, shedding novel insights into developing better data exploration strategies for post-training.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.10766</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.10766</guid>
|
||||||
<pubDate>Mon, 14 Apr 2025 23:53:47 +0000</pubDate>
|
<pubDate>Mon, 14 Apr 2025 23:53:47 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents</title>
|
<title>X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents</title>
|
||||||
<link>https://arxiv.org/abs/2504.13203</link>
|
<link>https://arxiv.org/abs/2504.13203</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13203.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, Saadia Gabriel</p><p><b>Upvotes:</b> 29</p><p><b>Summary:</b> Multi-turn interactions with language models (LMs) pose critical safety risks, as harmful intent can be strategically spread across exchanges. Yet, the vast majority of prior work has focused on single-turn safety, while adaptability and diversity remain among the key challenges of multi-turn red-teaming. To address these challenges, we present X-Teaming, a scalable framework that systematically explores how seemingly harmless interactions escalate into harmful outcomes and generates corresponding attack scenarios. X-Teaming employs collaborative agents for planning, attack optimization, and verification, achieving state-of-the-art multi-turn jailbreak effectiveness and diversity with success rates up to 98.1% across representative leading open-weight and closed-source models. In particular, X-Teaming achieves a 96.2% attack success rate against the latest Claude 3.7 Sonnet model, which has been considered nearly immune to single-turn attacks. Building on X-Teaming, we introduce XGuard-Train, an open-source multi-turn safety training dataset that is 20x larger than the previous best resource, comprising 30K interactive jailbreaks, designed to enable robust multi-turn safety alignment for LMs. Our work offers essential tools and insights for mitigating sophisticated conversational attacks, advancing the multi-turn safety of LMs.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13203.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, Saadia Gabriel</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> Multi-turn interactions with language models (LMs) pose critical safety risks, as harmful intent can be strategically spread across exchanges. Yet, the vast majority of prior work has focused on single-turn safety, while adaptability and diversity remain among the key challenges of multi-turn red-teaming. To address these challenges, we present X-Teaming, a scalable framework that systematically explores how seemingly harmless interactions escalate into harmful outcomes and generates corresponding attack scenarios. X-Teaming employs collaborative agents for planning, attack optimization, and verification, achieving state-of-the-art multi-turn jailbreak effectiveness and diversity with success rates up to 98.1% across representative leading open-weight and closed-source models. In particular, X-Teaming achieves a 96.2% attack success rate against the latest Claude 3.7 Sonnet model, which has been considered nearly immune to single-turn attacks. Building on X-Teaming, we introduce XGuard-Train, an open-source multi-turn safety training dataset that is 20x larger than the previous best resource, comprising 30K interactive jailbreaks, designed to enable robust multi-turn safety alignment for LMs. Our work offers essential tools and insights for mitigating sophisticated conversational attacks, advancing the multi-turn safety of LMs.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.13203</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.13203</guid>
|
||||||
<pubDate>Tue, 15 Apr 2025 16:11:28 +0000</pubDate>
|
<pubDate>Tue, 15 Apr 2025 16:11:28 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Seedream 3.0 Technical Report</title>
|
<title>Seedream 3.0 Technical Report</title>
|
||||||
<link>https://arxiv.org/abs/2504.11346</link>
|
<link>https://arxiv.org/abs/2504.11346</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.11346.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, Wei Liu, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Rui Wang, Xuanda Wang, Xun Wang, Ye Wang, Guofeng Wu, Jie Wu, Xin Xia, Xuefeng Xiao, Zhonghua Zhai, Xinyu Zhang, Qi Zhang, Yuwei Zhang, Shijia Zhao, Jianchao Yang, Weilin Huang</p><p><b>Upvotes:</b> 48</p><p><b>Summary:</b> We present Seedream 3.0, a high-performance Chinese-English bilingual image generation foundation model. We develop several technical improvements to address existing challenges in Seedream 2.0, including alignment with complicated prompts, fine-grained typography generation, suboptimal visual aesthetics and fidelity, and limited image resolutions. Specifically, the advancements of Seedream 3.0 stem from improvements across the entire pipeline, from data construction to model deployment. At the data stratum, we double the dataset using a defect-aware training paradigm and a dual-axis collaborative data-sampling framework. Furthermore, we adopt several effective techniques such as mixed-resolution training, cross-modality RoPE, representation alignment loss, and resolution-aware timestep sampling in the pre-training phase. During the post-training stage, we utilize diversified aesthetic captions in SFT, and a VLM-based reward model with scaling, thereby achieving outputs that well align with human preferences. Furthermore, Seedream 3.0 pioneers a novel acceleration paradigm. By employing consistent noise expectation and importance-aware timestep sampling, we achieve a 4 to 8 times speedup while maintaining image quality. Seedream 3.0 demonstrates significant improvements over Seedream 2.0: it enhances overall capabilities, in particular for text-rendering in complicated Chinese characters which is important to professional typography generation. In addition, it provides native high-resolution output (up to 2K), allowing it to generate images with high visual quality.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.11346.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, Wei Liu, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Rui Wang, Xuanda Wang, Xun Wang, Ye Wang, Guofeng Wu, Jie Wu, Xin Xia, Xuefeng Xiao, Zhonghua Zhai, Xinyu Zhang, Qi Zhang, Yuwei Zhang, Shijia Zhao, Jianchao Yang, Weilin Huang</p><p><b>Upvotes:</b> 52</p><p><b>Summary:</b> We present Seedream 3.0, a high-performance Chinese-English bilingual image generation foundation model. We develop several technical improvements to address existing challenges in Seedream 2.0, including alignment with complicated prompts, fine-grained typography generation, suboptimal visual aesthetics and fidelity, and limited image resolutions. Specifically, the advancements of Seedream 3.0 stem from improvements across the entire pipeline, from data construction to model deployment. At the data stratum, we double the dataset using a defect-aware training paradigm and a dual-axis collaborative data-sampling framework. Furthermore, we adopt several effective techniques such as mixed-resolution training, cross-modality RoPE, representation alignment loss, and resolution-aware timestep sampling in the pre-training phase. During the post-training stage, we utilize diversified aesthetic captions in SFT, and a VLM-based reward model with scaling, thereby achieving outputs that well align with human preferences. Furthermore, Seedream 3.0 pioneers a novel acceleration paradigm. By employing consistent noise expectation and importance-aware timestep sampling, we achieve a 4 to 8 times speedup while maintaining image quality. Seedream 3.0 demonstrates significant improvements over Seedream 2.0: it enhances overall capabilities, in particular for text-rendering in complicated Chinese characters which is important to professional typography generation. In addition, it provides native high-resolution output (up to 2K), allowing it to generate images with high visual quality.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.11346</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.11346</guid>
|
||||||
<pubDate>Tue, 15 Apr 2025 16:19:07 +0000</pubDate>
|
<pubDate>Tue, 15 Apr 2025 16:19:07 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>ReTool: Reinforcement Learning for Strategic Tool Use in LLMs</title>
|
<title>ReTool: Reinforcement Learning for Strategic Tool Use in LLMs</title>
|
||||||
<link>https://arxiv.org/abs/2504.11536</link>
|
<link>https://arxiv.org/abs/2504.11536</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.11536.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, Wanjun Zhong</p><p><b>Upvotes:</b> 58</p><p><b>Summary:</b> While reasoning models (e.g., DeepSeek R1) trained with reinforcement learning (RL), excel in textual reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning, concise computation, or complex equation solving-areas where computational tools like code interpreters (CI) demonstrate distinct advantages. To bridge this gap, we propose ReTool, which enhances long-form reasoning with tool-integrated learning, including two key features: (1) dynamic interleaving of real-time code execution within natural language reasoning processes, and (2) an automated RL paradigm that allows policy rollouts with multi-turn real-time code execution and teaches the model in learning when and how to invoke tools based on outcome feedback. ReTool employs a systematic training framework, beginning with synthetic cold-start data generation to produce code-augmented long-form reasoning traces for fine-tuning base models. Subsequent RL training leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns without human priors. Experiments on the challenging MATH Olympiad benchmark AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with 400 training steps, outperforming text-based RL baseline (40% accuracy, 1080 steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5% accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further analysis reveals emergent behaviors such as code self-correction, signaling an ''aha moment'' in which the model autonomously masters adaptive tool use. These findings highlight the promise of outcome-driven tool integration for advancing complex mathematical reasoning and offer new insights into hybrid neuro-symbolic systems.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.11536.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, Wanjun Zhong</p><p><b>Upvotes:</b> 60</p><p><b>Summary:</b> While reasoning models (e.g., DeepSeek R1) trained with reinforcement learning (RL), excel in textual reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning, concise computation, or complex equation solving-areas where computational tools like code interpreters (CI) demonstrate distinct advantages. To bridge this gap, we propose ReTool, which enhances long-form reasoning with tool-integrated learning, including two key features: (1) dynamic interleaving of real-time code execution within natural language reasoning processes, and (2) an automated RL paradigm that allows policy rollouts with multi-turn real-time code execution and teaches the model in learning when and how to invoke tools based on outcome feedback. ReTool employs a systematic training framework, beginning with synthetic cold-start data generation to produce code-augmented long-form reasoning traces for fine-tuning base models. Subsequent RL training leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns without human priors. Experiments on the challenging MATH Olympiad benchmark AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with 400 training steps, outperforming text-based RL baseline (40% accuracy, 1080 steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5% accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further analysis reveals emergent behaviors such as code self-correction, signaling an ''aha moment'' in which the model autonomously masters adaptive tool use. These findings highlight the promise of outcome-driven tool integration for advancing complex mathematical reasoning and offer new insights into hybrid neuro-symbolic systems.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.11536</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.11536</guid>
|
||||||
<pubDate>Tue, 15 Apr 2025 18:10:22 +0000</pubDate>
|
<pubDate>Tue, 15 Apr 2025 18:10:22 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>NodeRAG: Structuring Graph-based RAG with Heterogeneous Nodes</title>
|
<title>NodeRAG: Structuring Graph-based RAG with Heterogeneous Nodes</title>
|
||||||
<link>https://arxiv.org/abs/2504.11544</link>
|
<link>https://arxiv.org/abs/2504.11544</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.11544.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tianyang Xu, Haojie Zheng, Chengze Li, Haoxiang Chen, Yixin Liu, Ruoxi Chen, Lichao Sun</p><p><b>Upvotes:</b> 38</p><p><b>Summary:</b> Retrieval-augmented generation (RAG) empowers large language models to access external and private corpus, enabling factually consistent responses in specific domains. By exploiting the inherent structure of the corpus, graph-based RAG methods further enrich this process by building a knowledge graph index and leveraging the structural nature of graphs. However, current graph-based RAG approaches seldom prioritize the design of graph structures. Inadequately designed graph not only impede the seamless integration of diverse graph algorithms but also result in workflow inconsistencies and degraded performance. To further unleash the potential of graph for RAG, we propose NodeRAG, a graph-centric framework introducing heterogeneous graph structures that enable the seamless and holistic integration of graph-based methodologies into the RAG workflow. By aligning closely with the capabilities of LLMs, this framework ensures a fully cohesive and efficient end-to-end process. Through extensive experiments, we demonstrate that NodeRAG exhibits performance advantages over previous methods, including GraphRAG and LightRAG, not only in indexing time, query time, and storage efficiency but also in delivering superior question-answering performance on multi-hop benchmarks and open-ended head-to-head evaluations with minimal retrieval tokens. Our GitHub repository could be seen at https://github.com/Terry-Xu-666/NodeRAG.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.11544.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tianyang Xu, Haojie Zheng, Chengze Li, Haoxiang Chen, Yixin Liu, Ruoxi Chen, Lichao Sun</p><p><b>Upvotes:</b> 41</p><p><b>Summary:</b> Retrieval-augmented generation (RAG) empowers large language models to access external and private corpus, enabling factually consistent responses in specific domains. By exploiting the inherent structure of the corpus, graph-based RAG methods further enrich this process by building a knowledge graph index and leveraging the structural nature of graphs. However, current graph-based RAG approaches seldom prioritize the design of graph structures. Inadequately designed graph not only impede the seamless integration of diverse graph algorithms but also result in workflow inconsistencies and degraded performance. To further unleash the potential of graph for RAG, we propose NodeRAG, a graph-centric framework introducing heterogeneous graph structures that enable the seamless and holistic integration of graph-based methodologies into the RAG workflow. By aligning closely with the capabilities of LLMs, this framework ensures a fully cohesive and efficient end-to-end process. Through extensive experiments, we demonstrate that NodeRAG exhibits performance advantages over previous methods, including GraphRAG and LightRAG, not only in indexing time, query time, and storage efficiency but also in delivering superior question-answering performance on multi-hop benchmarks and open-ended head-to-head evaluations with minimal retrieval tokens. Our GitHub repository could be seen at https://github.com/Terry-Xu-666/NodeRAG.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.11544</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.11544</guid>
|
||||||
<pubDate>Tue, 15 Apr 2025 18:24:00 +0000</pubDate>
|
<pubDate>Tue, 15 Apr 2025 18:24:00 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>BitNet b1.58 2B4T Technical Report</title>
|
<title>BitNet b1.58 2B4T Technical Report</title>
|
||||||
<link>https://arxiv.org/abs/2504.12285</link>
|
<link>https://arxiv.org/abs/2504.12285</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.12285.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, Furu Wei</p><p><b>Upvotes:</b> 66</p><p><b>Summary:</b> We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.12285.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, Furu Wei</p><p><b>Upvotes:</b> 68</p><p><b>Summary:</b> We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.12285</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.12285</guid>
|
||||||
<pubDate>Wed, 16 Apr 2025 17:51:43 +0000</pubDate>
|
<pubDate>Wed, 16 Apr 2025 17:51:43 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -599,14 +522,14 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>ToolRL: Reward is All Tool Learning Needs</title>
|
<title>ToolRL: Reward is All Tool Learning Needs</title>
|
||||||
<link>https://arxiv.org/abs/2504.13958</link>
|
<link>https://arxiv.org/abs/2504.13958</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13958.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji</p><p><b>Upvotes:</b> 38</p><p><b>Summary:</b> Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning. In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using Group Relative Policy Optimization (GRPO). Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17% improvement over base models and a 15% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13958.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji</p><p><b>Upvotes:</b> 42</p><p><b>Summary:</b> Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning. In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using Group Relative Policy Optimization (GRPO). Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17% improvement over base models and a 15% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.13958</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.13958</guid>
|
||||||
<pubDate>Wed, 16 Apr 2025 21:45:32 +0000</pubDate>
|
<pubDate>Wed, 16 Apr 2025 21:45:32 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Packing Input Frame Context in Next-Frame Prediction Models for Video Generation</title>
|
<title>Packing Input Frame Context in Next-Frame Prediction Models for Video Generation</title>
|
||||||
<link>https://arxiv.org/abs/2504.12626</link>
|
<link>https://arxiv.org/abs/2504.12626</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.12626.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lvmin Zhang, Maneesh Agrawala</p><p><b>Upvotes:</b> 47</p><p><b>Summary:</b> We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. The FramePack compresses input frames to make the transformer context length a fixed number regardless of the video length. As a result, we are able to process a large number of frames using video diffusion with computation bottleneck similar to image diffusion. This also makes the training video batch sizes significantly higher (batch sizes become comparable to image diffusion training). We also propose an anti-drifting sampling method that generates frames in inverted temporal order with early-established endpoints to avoid exposure bias (error accumulation over iterations). Finally, we show that existing video diffusion models can be finetuned with FramePack, and their visual quality may be improved because the next-frame prediction supports more balanced diffusion schedulers with less extreme flow shift timesteps.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.12626.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lvmin Zhang, Maneesh Agrawala</p><p><b>Upvotes:</b> 48</p><p><b>Summary:</b> We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. The FramePack compresses input frames to make the transformer context length a fixed number regardless of the video length. As a result, we are able to process a large number of frames using video diffusion with computation bottleneck similar to image diffusion. This also makes the training video batch sizes significantly higher (batch sizes become comparable to image diffusion training). We also propose an anti-drifting sampling method that generates frames in inverted temporal order with early-established endpoints to avoid exposure bias (error accumulation over iterations). Finally, we show that existing video diffusion models can be finetuned with FramePack, and their visual quality may be improved because the next-frame prediction supports more balanced diffusion schedulers with less extreme flow shift timesteps.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.12626</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.12626</guid>
|
||||||
<pubDate>Thu, 17 Apr 2025 04:02:31 +0000</pubDate>
|
<pubDate>Thu, 17 Apr 2025 04:02:31 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
@@ -620,128 +543,205 @@
|
|||||||
<item>
|
<item>
|
||||||
<title>CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training</title>
|
<title>CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training</title>
|
||||||
<link>https://arxiv.org/abs/2504.13161</link>
|
<link>https://arxiv.org/abs/2504.13161</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13161.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan, Lin, Jan Kautz, Pavlo Molchanov</p><p><b>Upvotes:</b> 86</p><p><b>Summary:</b> Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting. Specifically, CLIMB embeds and clusters large-scale datasets in a semantic space and then iteratively searches for optimal mixtures using a smaller proxy model and a predictor. When continuously trained on 400B tokens with this mixture, our 1B model exceeds the state-of-the-art Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific domain (e.g., Social Sciences) yields a 5% improvement over random sampling. Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and ClimbMix, a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. We analyze the final data mixture, elucidating the characteristics of an optimal data mixture. Our data is available at: https://research.nvidia.com/labs/lpr/climb/</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13161.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan, Lin, Jan Kautz, Pavlo Molchanov</p><p><b>Upvotes:</b> 87</p><p><b>Summary:</b> Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting. Specifically, CLIMB embeds and clusters large-scale datasets in a semantic space and then iteratively searches for optimal mixtures using a smaller proxy model and a predictor. When continuously trained on 400B tokens with this mixture, our 1B model exceeds the state-of-the-art Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific domain (e.g., Social Sciences) yields a 5% improvement over random sampling. Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and ClimbMix, a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. We analyze the final data mixture, elucidating the characteristics of an optimal data mixture. Our data is available at: https://research.nvidia.com/labs/lpr/climb/</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.13161</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.13161</guid>
|
||||||
<pubDate>Thu, 17 Apr 2025 17:58:13 +0000</pubDate>
|
<pubDate>Thu, 17 Apr 2025 17:58:13 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling</title>
|
<title>Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling</title>
|
||||||
<link>https://arxiv.org/abs/2504.13169</link>
|
<link>https://arxiv.org/abs/2504.13169</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13169.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E. Gonzalez, Trevor Darrell, David M. Chan</p><p><b>Upvotes:</b> 38</p><p><b>Summary:</b> Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations, where they generate descriptions of nonexistent objects, actions, or concepts, posing significant risks in safety-critical applications. Existing hallucination mitigation methods typically follow one of two paradigms: generation adjustment, which modifies decoding behavior to align text with visual inputs, and post-hoc verification, where external models assess and correct outputs. While effective, generation adjustment methods often rely on heuristics and lack correction mechanisms, while post-hoc verification is complicated, typically requiring multiple models and tending to reject outputs rather than refine them. In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification. By leveraging a new hallucination-verification dataset containing over 1.3M semi-synthetic samples, along with a novel inference-time retrospective resampling technique, our approach enables VLMs to both detect hallucinations during generation and dynamically revise those hallucinations. Our evaluations show that REVERSE achieves state-of-the-art hallucination reduction, outperforming the best existing methods by up to 12% on CHAIR-MSCOCO and 28% on HaloQuest. Our dataset, model, and code are available at: https://reverse-vlm.github.io.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13169.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E. Gonzalez, Trevor Darrell, David M. Chan</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations, where they generate descriptions of nonexistent objects, actions, or concepts, posing significant risks in safety-critical applications. Existing hallucination mitigation methods typically follow one of two paradigms: generation adjustment, which modifies decoding behavior to align text with visual inputs, and post-hoc verification, where external models assess and correct outputs. While effective, generation adjustment methods often rely on heuristics and lack correction mechanisms, while post-hoc verification is complicated, typically requiring multiple models and tending to reject outputs rather than refine them. In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification. By leveraging a new hallucination-verification dataset containing over 1.3M semi-synthetic samples, along with a novel inference-time retrospective resampling technique, our approach enables VLMs to both detect hallucinations during generation and dynamically revise those hallucinations. Our evaluations show that REVERSE achieves state-of-the-art hallucination reduction, outperforming the best existing methods by up to 12% on CHAIR-MSCOCO and 28% on HaloQuest. Our dataset, model, and code are available at: https://reverse-vlm.github.io.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.13169</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.13169</guid>
|
||||||
<pubDate>Thu, 17 Apr 2025 17:59:22 +0000</pubDate>
|
<pubDate>Thu, 17 Apr 2025 17:59:22 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>Perception Encoder: The best visual embeddings are not at the output of the network</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.13181</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13181.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, Christoph Feichtenhofer</p><p><b>Upvotes:</b> 31</p><p><b>Summary:</b> We introduce Perception Encoder (PE), a state-of-the-art encoder for image and video understanding trained via simple vision-language learning. Traditionally, vision encoders have relied on a variety of pretraining objectives, each tailored to specific downstream tasks such as classification, captioning, or localization. Surprisingly, after scaling our carefully tuned image pretraining recipe and refining with our robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods, language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together with the core contrastive checkpoint, our PE family of models achieves state-of-the-art performance on a wide variety of tasks, including zero-shot image and video classification and retrieval; document, image, and video Q&A; and spatial tasks such as detection, depth estimation, and tracking. To foster further research, we are releasing our models, code, and a novel dataset of synthetically and human-annotated videos.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.13181</guid>
|
||||||
|
<pubDate>Thu, 17 Apr 2025 17:59:57 +0000</pubDate>
|
||||||
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space</title>
|
<title>MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space</title>
|
||||||
<link>https://arxiv.org/abs/2504.13835</link>
|
<link>https://arxiv.org/abs/2504.13835</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13835.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> Data quality and diversity are key to the construction of effective instruction-tuning datasets. % With the increasing availability of open-source instruction-tuning datasets, it is advantageous to automatically select high-quality and diverse subsets from a vast amount of data. % Existing methods typically prioritize instance quality and use heuristic rules to maintain diversity. % However, this absence of a comprehensive view of the entire collection often leads to suboptimal results. % Moreover, heuristic rules generally focus on distance or clustering within the embedding space, which fails to accurately capture the intent of complex instructions in the semantic space. % To bridge this gap, we propose a unified method for quantifying the information content of datasets. This method models the semantic space by constructing a label graph and quantifies diversity based on the distribution of information within the graph. % Based on such a measurement, we further introduce an efficient sampling method that selects data samples iteratively to Maximize the Information Gain (MIG) in semantic space. % Experiments on various datasets and base models demonstrate that MIG consistently outperforms state-of-the-art methods. % Notably, the model fine-tuned with 5\% Tulu3 data sampled by MIG achieves comparable performance to the official SFT model trained on the full dataset, with improvements of +5.73\% on AlpacaEval and +6.89\% on Wildbench.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13835.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen</p><p><b>Upvotes:</b> 36</p><p><b>Summary:</b> Data quality and diversity are key to the construction of effective instruction-tuning datasets. % With the increasing availability of open-source instruction-tuning datasets, it is advantageous to automatically select high-quality and diverse subsets from a vast amount of data. % Existing methods typically prioritize instance quality and use heuristic rules to maintain diversity. % However, this absence of a comprehensive view of the entire collection often leads to suboptimal results. % Moreover, heuristic rules generally focus on distance or clustering within the embedding space, which fails to accurately capture the intent of complex instructions in the semantic space. % To bridge this gap, we propose a unified method for quantifying the information content of datasets. This method models the semantic space by constructing a label graph and quantifies diversity based on the distribution of information within the graph. % Based on such a measurement, we further introduce an efficient sampling method that selects data samples iteratively to Maximize the Information Gain (MIG) in semantic space. % Experiments on various datasets and base models demonstrate that MIG consistently outperforms state-of-the-art methods. % Notably, the model fine-tuned with 5\% Tulu3 data sampled by MIG achieves comparable performance to the official SFT model trained on the full dataset, with improvements of +5.73\% on AlpacaEval and +6.89\% on Wildbench.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.13835</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.13835</guid>
|
||||||
<pubDate>Fri, 18 Apr 2025 17:59:46 +0000</pubDate>
|
<pubDate>Fri, 18 Apr 2025 17:59:46 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?</title>
|
<title>Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?</title>
|
||||||
<link>https://arxiv.org/abs/2504.13837</link>
|
<link>https://arxiv.org/abs/2504.13837</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13837.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang</p><p><b>Upvotes:</b> 96</p><p><b>Summary:</b> Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning capabilities of LLMs, particularly in mathematics and programming tasks. It is widely believed that RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning abilities that exceed corresponding base models' capacity. In this study, however, we critically re-examines this assumption by measuring the pass@k metric with large values of k to explore the reasoning capability boundary of the models across a wide range of model families and benchmarks. Surprisingly, the RL does not, in fact, elicit fundamentally new reasoning patterns. While RL-trained models outperform their base models at smaller values of k (\eg, k=1), base models can achieve a comparable or even higher pass@k score compared to their RL counterparts at large k values. The reasoning paths generated by RL-trained models are already included in the base models' sampling distribution, suggesting that most reasoning abilities manifested in RL-trained models are already obtained by base models. Further analysis shows that RL training boosts the performance by biasing the model's output distribution toward paths that are more likely to yield rewards, therefore sampling correct responses more efficiently. But this also results in a narrower reasoning capability boundary compared to base models. Similar results are observed in visual reasoning tasks trained with RLVR. Moreover, we find that distillation can genuinely introduce new knowledge into the model, different from RLVR. These findings underscore a critical limitation of RLVR in advancing LLM reasoning abilities which requires us to fundamentally rethink the impact of RL training in reasoning LLMs and the need of a better paradigm. Project Page: https://limit-of-RLVR.github.io</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.13837.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang</p><p><b>Upvotes:</b> 116</p><p><b>Summary:</b> Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning capabilities of LLMs, particularly in mathematics and programming tasks. It is widely believed that RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning abilities that exceed corresponding base models' capacity. In this study, however, we critically re-examines this assumption by measuring the pass@k metric with large values of k to explore the reasoning capability boundary of the models across a wide range of model families and benchmarks. Surprisingly, the RL does not, in fact, elicit fundamentally new reasoning patterns. While RL-trained models outperform their base models at smaller values of k (\eg, k=1), base models can achieve a comparable or even higher pass@k score compared to their RL counterparts at large k values. The reasoning paths generated by RL-trained models are already included in the base models' sampling distribution, suggesting that most reasoning abilities manifested in RL-trained models are already obtained by base models. Further analysis shows that RL training boosts the performance by biasing the model's output distribution toward paths that are more likely to yield rewards, therefore sampling correct responses more efficiently. But this also results in a narrower reasoning capability boundary compared to base models. Similar results are observed in visual reasoning tasks trained with RLVR. Moreover, we find that distillation can genuinely introduce new knowledge into the model, different from RLVR. These findings underscore a critical limitation of RLVR in advancing LLM reasoning abilities which requires us to fundamentally rethink the impact of RL training in reasoning LLMs and the need of a better paradigm. Project Page: https://limit-of-RLVR.github.io</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.13837</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.13837</guid>
|
||||||
<pubDate>Fri, 18 Apr 2025 17:59:56 +0000</pubDate>
|
<pubDate>Fri, 18 Apr 2025 17:59:56 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>DreamID: High-Fidelity and Fast diffusion-based Face Swapping via Triplet ID Group Learning</title>
|
<title>DreamID: High-Fidelity and Fast diffusion-based Face Swapping via Triplet ID Group Learning</title>
|
||||||
<link>https://arxiv.org/abs/2504.14509</link>
|
<link>https://arxiv.org/abs/2504.14509</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.14509.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fulong Ye, Miao Hua, Pengze Zhang, Xinghui Li, Qichao Sun, Songtao Zhao, Qian He, Xinglong Wu</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> In this paper, we introduce DreamID, a diffusion-based face swapping model that achieves high levels of ID similarity, attribute preservation, image fidelity, and fast inference speed. Unlike the typical face swapping training process, which often relies on implicit supervision and struggles to achieve satisfactory results. DreamID establishes explicit supervision for face swapping by constructing Triplet ID Group data, significantly enhancing identity similarity and attribute preservation. The iterative nature of diffusion models poses challenges for utilizing efficient image-space loss functions, as performing time-consuming multi-step sampling to obtain the generated image during training is impractical. To address this issue, we leverage the accelerated diffusion model SD Turbo, reducing the inference steps to a single iteration, enabling efficient pixel-level end-to-end training with explicit Triplet ID Group supervision. Additionally, we propose an improved diffusion-based model architecture comprising SwapNet, FaceNet, and ID Adapter. This robust architecture fully unlocks the power of the Triplet ID Group explicit supervision. Finally, to further extend our method, we explicitly modify the Triplet ID Group data during training to fine-tune and preserve specific attributes, such as glasses and face shape. Extensive experiments demonstrate that DreamID outperforms state-of-the-art methods in terms of identity similarity, pose and expression preservation, and image fidelity. Overall, DreamID achieves high-quality face swapping results at 512*512 resolution in just 0.6 seconds and performs exceptionally well in challenging scenarios such as complex lighting, large angles, and occlusions.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.14509.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fulong Ye, Miao Hua, Pengze Zhang, Xinghui Li, Qichao Sun, Songtao Zhao, Qian He, Xinglong Wu</p><p><b>Upvotes:</b> 48</p><p><b>Summary:</b> In this paper, we introduce DreamID, a diffusion-based face swapping model that achieves high levels of ID similarity, attribute preservation, image fidelity, and fast inference speed. Unlike the typical face swapping training process, which often relies on implicit supervision and struggles to achieve satisfactory results. DreamID establishes explicit supervision for face swapping by constructing Triplet ID Group data, significantly enhancing identity similarity and attribute preservation. The iterative nature of diffusion models poses challenges for utilizing efficient image-space loss functions, as performing time-consuming multi-step sampling to obtain the generated image during training is impractical. To address this issue, we leverage the accelerated diffusion model SD Turbo, reducing the inference steps to a single iteration, enabling efficient pixel-level end-to-end training with explicit Triplet ID Group supervision. Additionally, we propose an improved diffusion-based model architecture comprising SwapNet, FaceNet, and ID Adapter. This robust architecture fully unlocks the power of the Triplet ID Group explicit supervision. Finally, to further extend our method, we explicitly modify the Triplet ID Group data during training to fine-tune and preserve specific attributes, such as glasses and face shape. Extensive experiments demonstrate that DreamID outperforms state-of-the-art methods in terms of identity similarity, pose and expression preservation, and image fidelity. Overall, DreamID achieves high-quality face swapping results at 512*512 resolution in just 0.6 seconds and performs exceptionally well in challenging scenarios such as complex lighting, large angles, and occlusions.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.14509</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.14509</guid>
|
||||||
<pubDate>Sun, 20 Apr 2025 06:53:00 +0000</pubDate>
|
<pubDate>Sun, 20 Apr 2025 06:53:00 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Learning to Reason under Off-Policy Guidance</title>
|
<title>Learning to Reason under Off-Policy Guidance</title>
|
||||||
<link>https://arxiv.org/abs/2504.14945</link>
|
<link>https://arxiv.org/abs/2504.14945</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.14945.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, Yue Zhang</p><p><b>Upvotes:</b> 70</p><p><b>Summary:</b> Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning (RL) with simple rule-based rewards. However, existing zero-RL approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY guidance), a framework that augments zero-RL with off-policy reasoning traces. LUFFY dynamically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. Notably, we propose policy shaping via regularized importance sampling to avoid superficial and rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an over +7.0 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks. It also substantially surpasses imitation-based supervised fine-tuning (SFT), particularly in generalization. Analysis shows LUFFY not only imitates effectively but also explores beyond demonstrations, offering a scalable path to train generalizable reasoning models with off-policy guidance.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.14945.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, Yue Zhang</p><p><b>Upvotes:</b> 77</p><p><b>Summary:</b> Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning (RL) with simple rule-based rewards. However, existing zero-RL approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY guidance), a framework that augments zero-RL with off-policy reasoning traces. LUFFY dynamically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. Notably, we propose policy shaping via regularized importance sampling to avoid superficial and rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an over +7.0 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks. It also substantially surpasses imitation-based supervised fine-tuning (SFT), particularly in generalization. Analysis shows LUFFY not only imitates effectively but also explores beyond demonstrations, offering a scalable path to train generalizable reasoning models with off-policy guidance.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.14945</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.14945</guid>
|
||||||
<pubDate>Mon, 21 Apr 2025 08:09:13 +0000</pubDate>
|
<pubDate>Mon, 21 Apr 2025 08:09:13 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Kuwain 1.5B: An Arabic SLM via Language Injection</title>
|
<title>Kuwain 1.5B: An Arabic SLM via Language Injection</title>
|
||||||
<link>https://arxiv.org/abs/2504.15120</link>
|
<link>https://arxiv.org/abs/2504.15120</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15120.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Khalil Hennara, Sara Chrouf, Mohamed Motaism Hamed, Zeina Aldallal, Omar Hadid, Safwan AlModhayan</p><p><b>Upvotes:</b> 106</p><p><b>Summary:</b> Enhancing existing models with new knowledge is a crucial aspect of AI development. This paper introduces a novel method for integrating a new language into a large language model (LLM). Our approach successfully incorporates a previously unseen target language into an existing LLM without compromising its prior knowledge. We trained a tiny model with 1.5 billion parameters named Kuwain by injecting the Arabic language into a small open-source model mainly trained in English. Our method demonstrates significant improvements in Arabic language performance, with an average 8% improvement across various benchmarks, while retaining the model's existing knowledge with a minimum amount of the original model's data. This offers a cost-effective alternative to training a comprehensive model in both English and Arabic. The results highlight the potential for efficient, targeted language model expansion without extensive retraining or resource-intensive processes.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15120.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Khalil Hennara, Sara Chrouf, Mohamed Motaism Hamed, Zeina Aldallal, Omar Hadid, Safwan AlModhayan</p><p><b>Upvotes:</b> 113</p><p><b>Summary:</b> Enhancing existing models with new knowledge is a crucial aspect of AI development. This paper introduces a novel method for integrating a new language into a large language model (LLM). Our approach successfully incorporates a previously unseen target language into an existing LLM without compromising its prior knowledge. We trained a tiny model with 1.5 billion parameters named Kuwain by injecting the Arabic language into a small open-source model mainly trained in English. Our method demonstrates significant improvements in Arabic language performance, with an average 8% improvement across various benchmarks, while retaining the model's existing knowledge with a minimum amount of the original model's data. This offers a cost-effective alternative to training a comprehensive model in both English and Arabic. The results highlight the potential for efficient, targeted language model expansion without extensive retraining or resource-intensive processes.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15120</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15120</guid>
|
||||||
<pubDate>Mon, 21 Apr 2025 14:17:25 +0000</pubDate>
|
<pubDate>Mon, 21 Apr 2025 14:17:25 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>FlowReasoner: Reinforcing Query-Level Meta-Agents</title>
|
<title>FlowReasoner: Reinforcing Query-Level Meta-Agents</title>
|
||||||
<link>https://arxiv.org/abs/2504.15257</link>
|
<link>https://arxiv.org/abs/2504.15257</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15257.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, Tianyu Pang</p><p><b>Upvotes:</b> 42</p><p><b>Summary:</b> This paper proposes a query-level meta-agent named FlowReasoner to automate the design of query-level multi-agent systems, i.e., one system per user query. Our core idea is to incentivize a reasoning-based meta-agent via external execution feedback. Concretely, by distilling DeepSeek R1, we first endow the basic reasoning ability regarding the generation of multi-agent systems to FlowReasoner. Then, we further enhance it via reinforcement learning (RL) with external execution feedback. A multi-purpose reward is designed to guide the RL training from aspects of performance, complexity, and efficiency. In this manner, FlowReasoner is enabled to generate a personalized multi-agent system for each user query via deliberative reasoning. Experiments on both engineering and competition code benchmarks demonstrate the superiority of FlowReasoner. Remarkably, it surpasses o1-mini by 10.52% accuracy across three benchmarks. The code is available at https://github.com/sail-sg/FlowReasoner.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15257.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, Tianyu Pang</p><p><b>Upvotes:</b> 46</p><p><b>Summary:</b> This paper proposes a query-level meta-agent named FlowReasoner to automate the design of query-level multi-agent systems, i.e., one system per user query. Our core idea is to incentivize a reasoning-based meta-agent via external execution feedback. Concretely, by distilling DeepSeek R1, we first endow the basic reasoning ability regarding the generation of multi-agent systems to FlowReasoner. Then, we further enhance it via reinforcement learning (RL) with external execution feedback. A multi-purpose reward is designed to guide the RL training from aspects of performance, complexity, and efficiency. In this manner, FlowReasoner is enabled to generate a personalized multi-agent system for each user query via deliberative reasoning. Experiments on both engineering and competition code benchmarks demonstrate the superiority of FlowReasoner. Remarkably, it surpasses o1-mini by 10.52% accuracy across three benchmarks. The code is available at https://github.com/sail-sg/FlowReasoner.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15257</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15257</guid>
|
||||||
<pubDate>Mon, 21 Apr 2025 17:35:42 +0000</pubDate>
|
<pubDate>Mon, 21 Apr 2025 17:35:42 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models</title>
|
<title>Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models</title>
|
||||||
<link>https://arxiv.org/abs/2504.15271</link>
|
<link>https://arxiv.org/abs/2504.15271</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15271.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, Tyler Poon, Max Ehrlich, Tuomas Rintamaki, Tyler Poon, Tong Lu, Limin Wang, Bryan Catanzaro, Jan Kautz, Andrew Tao, Zhiding Yu, Guilin Liu</p><p><b>Upvotes:</b> 61</p><p><b>Summary:</b> We introduce Eagle 2.5, a family of frontier vision-language models (VLMs) for long-context multimodal learning. Our work addresses the challenges in long video comprehension and high-resolution image understanding, introducing a generalist framework for both tasks. The proposed training framework incorporates Automatic Degrade Sampling and Image Area Preservation, two techniques that preserve contextual integrity and visual details. The framework also includes numerous efficiency optimizations in the pipeline for long-context data training. Finally, we propose Eagle-Video-110K, a novel dataset that integrates both story-level and clip-level annotations, facilitating long-video understanding. Eagle 2.5 demonstrates substantial improvements on long-context multimodal benchmarks, providing a robust solution to the limitations of existing VLMs. Notably, our best model Eagle 2.5-8B achieves 72.4% on Video-MME with 512 input frames, matching the results of top-tier commercial model such as GPT-4o and large-scale open-source models like Qwen2.5-VL-72B and InternVL2.5-78B.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15271.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, Tyler Poon, Max Ehrlich, Tuomas Rintamaki, Tyler Poon, Tong Lu, Limin Wang, Bryan Catanzaro, Jan Kautz, Andrew Tao, Zhiding Yu, Guilin Liu</p><p><b>Upvotes:</b> 65</p><p><b>Summary:</b> We introduce Eagle 2.5, a family of frontier vision-language models (VLMs) for long-context multimodal learning. Our work addresses the challenges in long video comprehension and high-resolution image understanding, introducing a generalist framework for both tasks. The proposed training framework incorporates Automatic Degrade Sampling and Image Area Preservation, two techniques that preserve contextual integrity and visual details. The framework also includes numerous efficiency optimizations in the pipeline for long-context data training. Finally, we propose Eagle-Video-110K, a novel dataset that integrates both story-level and clip-level annotations, facilitating long-video understanding. Eagle 2.5 demonstrates substantial improvements on long-context multimodal benchmarks, providing a robust solution to the limitations of existing VLMs. Notably, our best model Eagle 2.5-8B achieves 72.4% on Video-MME with 512 input frames, matching the results of top-tier commercial model such as GPT-4o and large-scale open-source models like Qwen2.5-VL-72B and InternVL2.5-78B.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15271</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15271</guid>
|
||||||
<pubDate>Mon, 21 Apr 2025 17:57:28 +0000</pubDate>
|
<pubDate>Mon, 21 Apr 2025 17:57:28 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models</title>
|
<title>VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models</title>
|
||||||
<link>https://arxiv.org/abs/2504.15279</link>
|
<link>https://arxiv.org/abs/2504.15279</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15279.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, Jinguo Zhu</p><p><b>Upvotes:</b> 59</p><p><b>Summary:</b> Visual reasoning is a core component of human intelligence and a critical capability for advanced multimodal models. Yet current reasoning evaluations of multimodal large language models (MLLMs) often rely on text descriptions and allow language-based reasoning shortcuts, failing to measure genuine vision-centric reasoning. To address this, we introduce VisuLogic: a benchmark of 1,000 human-verified problems across six categories (e.g., quantitative shifts, spatial relations, attribute comparisons). These various types of questions can be evaluated to assess the visual reasoning capabilities of MLLMs from multiple perspectives. We evaluate leading MLLMs on this benchmark and analyze their results to identify common failure modes. Most models score below 30% accuracy-only slightly above the 25% random baseline and far below the 51.4% achieved by humans-revealing significant gaps in visual reasoning. Furthermore, we provide a supplementary training dataset and a reinforcement-learning baseline to support further progress.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15279.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, Jinguo Zhu</p><p><b>Upvotes:</b> 71</p><p><b>Summary:</b> Visual reasoning is a core component of human intelligence and a critical capability for advanced multimodal models. Yet current reasoning evaluations of multimodal large language models (MLLMs) often rely on text descriptions and allow language-based reasoning shortcuts, failing to measure genuine vision-centric reasoning. To address this, we introduce VisuLogic: a benchmark of 1,000 human-verified problems across six categories (e.g., quantitative shifts, spatial relations, attribute comparisons). These various types of questions can be evaluated to assess the visual reasoning capabilities of MLLMs from multiple perspectives. We evaluate leading MLLMs on this benchmark and analyze their results to identify common failure modes. Most models score below 30% accuracy-only slightly above the 25% random baseline and far below the 51.4% achieved by humans-revealing significant gaps in visual reasoning. Furthermore, we provide a supplementary training dataset and a reinforcement-learning baseline to support further progress.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15279</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15279</guid>
|
||||||
<pubDate>Mon, 21 Apr 2025 17:59:53 +0000</pubDate>
|
<pubDate>Mon, 21 Apr 2025 17:59:53 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>Towards Understanding Camera Motions in Any Video</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.15376</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15376.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, Rushikesh Zawar, Xue Bai, Yilun Du, Chuang Gan, Deva Ramanan</p><p><b>Upvotes:</b> 145</p><p><b>Summary:</b> We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding. CameraBench consists of ~3,000 diverse internet videos, annotated by experts through a rigorous multi-stage quality control process. One of our contributions is a taxonomy of camera motion primitives, designed in collaboration with cinematographers. We find, for example, that some motions like "follow" (or tracking) require understanding scene content like moving subjects. We conduct a large-scale human study to quantify human annotation performance, revealing that domain expertise and tutorial-based training can significantly enhance accuracy. For example, a novice may confuse zoom-in (a change of intrinsics) with translating forward (a change of extrinsics), but can be trained to differentiate the two. Using CameraBench, we evaluate Structure-from-Motion (SfM) and Video-Language Models (VLMs), finding that SfM models struggle to capture semantic primitives that depend on scene content, while VLMs struggle to capture geometric primitives that require precise estimation of trajectories. We then fine-tune a generative VLM on CameraBench to achieve the best of both worlds and showcase its applications, including motion-augmented captioning, video question answering, and video-text retrieval. We hope our taxonomy, benchmark, and tutorials will drive future efforts towards the ultimate goal of understanding camera motions in any video.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15376</guid>
|
||||||
|
<pubDate>Mon, 21 Apr 2025 18:34:57 +0000</pubDate>
|
||||||
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Learning Adaptive Parallel Reasoning with Language Models</title>
|
<title>Learning Adaptive Parallel Reasoning with Language Models</title>
|
||||||
<link>https://arxiv.org/abs/2504.15466</link>
|
<link>https://arxiv.org/abs/2504.15466</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15466.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, Alane Suhr</p><p><b>Upvotes:</b> 38</p><p><b>Summary:</b> Scaling inference-time computation has substantially improved the reasoning capabilities of language models. However, existing methods have significant limitations: serialized chain-of-thought approaches generate overly long outputs, leading to increased latency and exhausted context windows, while parallel methods such as self-consistency suffer from insufficient coordination, resulting in redundant computations and limited performance gains. To address these shortcomings, we propose Adaptive Parallel Reasoning (APR), a novel reasoning framework that enables language models to orchestrate both serialized and parallel computations end-to-end. APR generalizes existing reasoning methods by enabling adaptive multi-threaded inference using spawn() and join() operations. A key innovation is our end-to-end reinforcement learning strategy, optimizing both parent and child inference threads to enhance task success rate without requiring predefined reasoning structures. Experiments on the Countdown reasoning task demonstrate significant benefits of APR: (1) higher performance within the same context window (83.4% vs. 60.0% at 4k context); (2) superior scalability with increased computation (80.1% vs. 66.6% at 20k total tokens); (3) improved accuracy at equivalent latency (75.2% vs. 57.3% at approximately 5,000ms). APR represents a step towards enabling language models to autonomously optimize their reasoning processes through adaptive allocation of computation.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15466.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, Alane Suhr</p><p><b>Upvotes:</b> 42</p><p><b>Summary:</b> Scaling inference-time computation has substantially improved the reasoning capabilities of language models. However, existing methods have significant limitations: serialized chain-of-thought approaches generate overly long outputs, leading to increased latency and exhausted context windows, while parallel methods such as self-consistency suffer from insufficient coordination, resulting in redundant computations and limited performance gains. To address these shortcomings, we propose Adaptive Parallel Reasoning (APR), a novel reasoning framework that enables language models to orchestrate both serialized and parallel computations end-to-end. APR generalizes existing reasoning methods by enabling adaptive multi-threaded inference using spawn() and join() operations. A key innovation is our end-to-end reinforcement learning strategy, optimizing both parent and child inference threads to enhance task success rate without requiring predefined reasoning structures. Experiments on the Countdown reasoning task demonstrate significant benefits of APR: (1) higher performance within the same context window (83.4% vs. 60.0% at 4k context); (2) superior scalability with increased computation (80.1% vs. 66.6% at 20k total tokens); (3) improved accuracy at equivalent latency (75.2% vs. 57.3% at approximately 5,000ms). APR represents a step towards enabling language models to autonomously optimize their reasoning processes through adaptive allocation of computation.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15466</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15466</guid>
|
||||||
<pubDate>Mon, 21 Apr 2025 22:29:02 +0000</pubDate>
|
<pubDate>Mon, 21 Apr 2025 22:29:02 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks</title>
|
<title>The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks</title>
|
||||||
<link>https://arxiv.org/abs/2504.15521</link>
|
<link>https://arxiv.org/abs/2504.15521</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15521.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang</p><p><b>Upvotes:</b> 57</p><p><b>Summary:</b> As large language models (LLMs) continue to advance in linguistic capabilities, robust multilingual evaluation has become essential for promoting equitable technological progress. This position paper examines over 2,000 multilingual (non-English) benchmarks from 148 countries, published between 2021 and 2024, to evaluate past, present, and future practices in multilingual benchmarking. Our findings reveal that, despite significant investments amounting to tens of millions of dollars, English remains significantly overrepresented in these benchmarks. Additionally, most benchmarks rely on original language content rather than translations, with the majority sourced from high-resource countries such as China, India, Germany, the UK, and the USA. Furthermore, a comparison of benchmark performance with human judgments highlights notable disparities. STEM-related tasks exhibit strong correlations with human evaluations (0.70 to 0.85), while traditional NLP tasks like question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30). Moreover, translating English benchmarks into other languages proves insufficient, as localized benchmarks demonstrate significantly higher alignment with local human judgments (0.68) than their translated counterparts (0.47). This underscores the importance of creating culturally and linguistically tailored benchmarks rather than relying solely on translations. Through this comprehensive analysis, we highlight six key limitations in current multilingual evaluation practices, propose the guiding principles accordingly for effective multilingual benchmarking, and outline five critical research directions to drive progress in the field. Finally, we call for a global collaborative effort to develop human-aligned benchmarks that prioritize real-world applications.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15521.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang</p><p><b>Upvotes:</b> 61</p><p><b>Summary:</b> As large language models (LLMs) continue to advance in linguistic capabilities, robust multilingual evaluation has become essential for promoting equitable technological progress. This position paper examines over 2,000 multilingual (non-English) benchmarks from 148 countries, published between 2021 and 2024, to evaluate past, present, and future practices in multilingual benchmarking. Our findings reveal that, despite significant investments amounting to tens of millions of dollars, English remains significantly overrepresented in these benchmarks. Additionally, most benchmarks rely on original language content rather than translations, with the majority sourced from high-resource countries such as China, India, Germany, the UK, and the USA. Furthermore, a comparison of benchmark performance with human judgments highlights notable disparities. STEM-related tasks exhibit strong correlations with human evaluations (0.70 to 0.85), while traditional NLP tasks like question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30). Moreover, translating English benchmarks into other languages proves insufficient, as localized benchmarks demonstrate significantly higher alignment with local human judgments (0.68) than their translated counterparts (0.47). This underscores the importance of creating culturally and linguistically tailored benchmarks rather than relying solely on translations. Through this comprehensive analysis, we highlight six key limitations in current multilingual evaluation practices, propose the guiding principles accordingly for effective multilingual benchmarking, and outline five critical research directions to drive progress in the field. Finally, we call for a global collaborative effort to develop human-aligned benchmarks that prioritize real-world applications.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15521</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15521</guid>
|
||||||
<pubDate>Tue, 22 Apr 2025 01:47:37 +0000</pubDate>
|
<pubDate>Tue, 22 Apr 2025 01:47:37 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Tina: Tiny Reasoning Models via LoRA</title>
|
<title>Tina: Tiny Reasoning Models via LoRA</title>
|
||||||
<link>https://arxiv.org/abs/2504.15777</link>
|
<link>https://arxiv.org/abs/2504.15777</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15777.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, Willie Neiswanger</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> How cost-effectively can strong reasoning abilities be achieved in language models? Driven by this fundamental question, we present Tina, a family of tiny reasoning models achieved with high cost-efficiency. Notably, Tina demonstrates that substantial reasoning performance can be developed using only minimal resources, by applying parameter-efficient updates during reinforcement learning (RL), using low-rank adaptation (LoRA), to an already tiny 1.5B parameter base model. This minimalist approach produces models that achieve reasoning performance which is competitive with, and sometimes surpasses, SOTA RL reasoning models built upon the same base model. Crucially, this is achieved at a tiny fraction of the computational post-training cost employed by existing SOTA models. In fact, the best Tina model achieves a >20\% reasoning performance increase and 43.33\% Pass@1 accuracy on AIME24, at only \$9 USD post-training and evaluation cost (i.e., an estimated 260x cost reduction). Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA. We validate this across multiple open-source reasoning datasets and various ablation settings starting with a single, fixed set of hyperparameters. Furthermore, we hypothesize that this effectiveness and efficiency stem from LoRA rapidly adapting the model to the structural format of reasoning rewarded by RL, while largely preserving the base model's underlying knowledge. In service of accessibility and open research, we fully open-source all code, training logs, and model weights \& checkpoints.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15777.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, Willie Neiswanger</p><p><b>Upvotes:</b> 50</p><p><b>Summary:</b> How cost-effectively can strong reasoning abilities be achieved in language models? Driven by this fundamental question, we present Tina, a family of tiny reasoning models achieved with high cost-efficiency. Notably, Tina demonstrates that substantial reasoning performance can be developed using only minimal resources, by applying parameter-efficient updates during reinforcement learning (RL), using low-rank adaptation (LoRA), to an already tiny 1.5B parameter base model. This minimalist approach produces models that achieve reasoning performance which is competitive with, and sometimes surpasses, SOTA RL reasoning models built upon the same base model. Crucially, this is achieved at a tiny fraction of the computational post-training cost employed by existing SOTA models. In fact, the best Tina model achieves a >20\% reasoning performance increase and 43.33\% Pass@1 accuracy on AIME24, at only \$9 USD post-training and evaluation cost (i.e., an estimated 260x cost reduction). Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA. We validate this across multiple open-source reasoning datasets and various ablation settings starting with a single, fixed set of hyperparameters. Furthermore, we hypothesize that this effectiveness and efficiency stem from LoRA rapidly adapting the model to the structural format of reasoning rewarded by RL, while largely preserving the base model's underlying knowledge. In service of accessibility and open research, we fully open-source all code, training logs, and model weights \& checkpoints.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15777</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15777</guid>
|
||||||
<pubDate>Tue, 22 Apr 2025 10:38:00 +0000</pubDate>
|
<pubDate>Tue, 22 Apr 2025 10:38:00 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.16030</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16030.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, Mike Zheng Shou</p><p><b>Upvotes:</b> 32</p><p><b>Summary:</b> Recent video large language models (Video LLMs) often depend on costly human annotations or proprietary model APIs (e.g., GPT-4o) to produce training data, which limits their training at scale. In this paper, we explore large-scale training for Video LLM with cheap automatic speech recognition (ASR) transcripts. Specifically, we propose a novel streaming training approach that densely interleaves the ASR words and video frames according to their timestamps. Compared to previous studies in vision-language representation with ASR, our method naturally fits the streaming characteristics of ASR, thus enabling the model to learn temporally-aligned, fine-grained vision-language modeling. To support the training algorithm, we introduce a data production pipeline to process YouTube videos and their closed captions (CC, same as ASR), resulting in Live-CC-5M dataset for pre-training and Live-WhisperX-526K dataset for high-quality supervised fine-tuning (SFT). Remarkably, even without SFT, the ASR-only pre-trained LiveCC-7B-Base model demonstrates competitive general video QA performance and exhibits a new capability in real-time video commentary. To evaluate this, we carefully design a new LiveSports-3K benchmark, using LLM-as-a-judge to measure the free-form commentary. Experiments show our final LiveCC-7B-Instruct model can surpass advanced 72B models (Qwen2.5-VL-72B-Instruct, LLaVA-Video-72B) in commentary quality even working in a real-time mode. Meanwhile, it achieves state-of-the-art results at the 7B/8B scale on popular video QA benchmarks such as VideoMME and OVOBench, demonstrating the broad generalizability of our approach. All resources of this paper have been released at https://showlab.github.io/livecc.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.16030</guid>
|
||||||
|
<pubDate>Tue, 22 Apr 2025 16:52:09 +0000</pubDate>
|
||||||
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Describe Anything: Detailed Localized Image and Video Captioning</title>
|
<title>Describe Anything: Detailed Localized Image and Video Captioning</title>
|
||||||
<link>https://arxiv.org/abs/2504.16072</link>
|
<link>https://arxiv.org/abs/2504.16072</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16072.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui</p><p><b>Upvotes:</b> 48</p><p><b>Summary:</b> Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16072.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui</p><p><b>Upvotes:</b> 58</p><p><b>Summary:</b> Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.16072</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.16072</guid>
|
||||||
<pubDate>Tue, 22 Apr 2025 17:51:41 +0000</pubDate>
|
<pubDate>Tue, 22 Apr 2025 17:51:41 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.16074</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16074.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Yutong Ren, Zizhuo Fu, Weike Wang, Xudong Tian, Anqi Lv, Laifu Man, Jianxiang Li, Feiyu Tao, Qihua Sun, Zhou Liang, Yushu Mu, Zhongxuan Li, Jing-Jun Zhang, Shutao Zhang, Xiaotian Li, Xingqi Xia, Jiawei Lin, Zheyu Shen, Jiahang Chen, Qiuhao Xiong, Binran Wang, Fengyuan Wang, Ziyang Ni, Bohan Zhang, Fan Cui, Changkun Shao, Qing-Hong Cao, Ming-xing Luo, Muhan Zhang, Hua Xing Zhu</p><p><b>Upvotes:</b> 34</p><p><b>Summary:</b> We introduce PHYBench, a novel, high-quality benchmark designed for evaluating reasoning capabilities of large language models (LLMs) in physical contexts. PHYBench consists of 500 meticulously curated physics problems based on real-world physical scenarios, designed to assess the ability of models to understand and reason about realistic physical processes. Covering mechanics, electromagnetism, thermodynamics, optics, modern physics, and advanced physics, the benchmark spans difficulty levels from high school exercises to undergraduate problems and Physics Olympiad challenges. Additionally, we propose the Expression Edit Distance (EED) Score, a novel evaluation metric based on the edit distance between mathematical expressions, which effectively captures differences in model reasoning processes and results beyond traditional binary scoring methods. We evaluate various LLMs on PHYBench and compare their performance with human experts. Our results reveal that even state-of-the-art reasoning models significantly lag behind human experts, highlighting their limitations and the need for improvement in complex physical reasoning scenarios. Our benchmark results and dataset are publicly available at https://phybench-official.github.io/phybench-demo/.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.16074</guid>
|
||||||
|
<pubDate>Tue, 22 Apr 2025 17:53:29 +0000</pubDate>
|
||||||
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>TTRL: Test-Time Reinforcement Learning</title>
|
<title>TTRL: Test-Time Reinforcement Learning</title>
|
||||||
<link>https://arxiv.org/abs/2504.16084</link>
|
<link>https://arxiv.org/abs/2504.16084</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16084.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, Bowen Zhou</p><p><b>Upvotes:</b> 76</p><p><b>Summary:</b> This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning (TTRL), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 159% on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks, and highlight TTRL's potential for broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16084.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, Bowen Zhou</p><p><b>Upvotes:</b> 96</p><p><b>Summary:</b> This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning (TTRL), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 159% on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks, and highlight TTRL's potential for broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.16084</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.16084</guid>
|
||||||
<pubDate>Tue, 22 Apr 2025 17:59:56 +0000</pubDate>
|
<pubDate>Tue, 22 Apr 2025 17:59:56 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.16656</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16656.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chris, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, Yang Liu, Yahui Zhou</p><p><b>Upvotes:</b> 50</p><p><b>Summary:</b> We present Skywork R1V2, a next-generation multimodal reasoning model and a major leap forward from its predecessor, Skywork R1V. At its core, R1V2 introduces a hybrid reinforcement learning paradigm that harmonizes reward-model guidance with rule-based strategies, thereby addressing the long-standing challenge of balancing sophisticated reasoning capabilities with broad generalization. To further enhance training efficiency, we propose the Selective Sample Buffer (SSB) mechanism, which effectively counters the ``Vanishing Advantages'' dilemma inherent in Group Relative Policy Optimization (GRPO) by prioritizing high-value samples throughout the optimization process. Notably, we observe that excessive reinforcement signals can induce visual hallucinations--a phenomenon we systematically monitor and mitigate through calibrated reward thresholds throughout the training process. Empirical results affirm the exceptional capability of R1V2, with benchmark-leading performances such as 62.6 on OlympiadBench, 79.0 on AIME2024, 63.6 on LiveCodeBench, and 74.0 on MMMU. These results underscore R1V2's superiority over existing open-source models and demonstrate significant progress in closing the performance gap with premier proprietary systems, including Gemini 2.5 and OpenAI o4-mini. The Skywork R1V2 model weights have been publicly released to promote openness and reproducibility https://huggingface.co/Skywork/Skywork-R1V2-38B.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.16656</guid>
|
||||||
|
<pubDate>Wed, 23 Apr 2025 12:24:10 +0000</pubDate>
|
||||||
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning</title>
|
<title>Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning</title>
|
||||||
<link>https://arxiv.org/abs/2504.17192</link>
|
<link>https://arxiv.org/abs/2504.17192</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17192.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Minju Seo, Jinheon Baek, Seongyun Lee, Sung Ju Hwang</p><p><b>Upvotes:</b> 31</p><p><b>Summary:</b> Despite the rapid growth of machine learning research, corresponding code implementations are often unavailable, making it slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent Large Language Models (LLMs) excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, a multi-agent LLM framework that transforms machine learning papers into functional code repositories. PaperCoder operates in three stages: planning, where it constructs a high-level roadmap, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files; analysis, which focuses on interpreting implementation-specific details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through a set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, specifically from the original paper authors, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful implementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17192.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Minju Seo, Jinheon Baek, Seongyun Lee, Sung Ju Hwang</p><p><b>Upvotes:</b> 95</p><p><b>Summary:</b> Despite the rapid growth of machine learning research, corresponding code implementations are often unavailable, making it slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent Large Language Models (LLMs) excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, a multi-agent LLM framework that transforms machine learning papers into functional code repositories. PaperCoder operates in three stages: planning, where it constructs a high-level roadmap, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files; analysis, which focuses on interpreting implementation-specific details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through a set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, specifically from the original paper authors, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful implementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17192</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17192</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 01:57:01 +0000</pubDate>
|
<pubDate>Thu, 24 Apr 2025 01:57:01 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.17432</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17432.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, Jiankang Deng</p><p><b>Upvotes:</b> 37</p><p><b>Summary:</b> The Contrastive Language-Image Pre-training (CLIP) framework has become a widely used approach for multimodal representation learning, particularly in image-text retrieval and clustering. However, its efficacy is constrained by three key limitations: (1) text token truncation, (2) isolated image-text encoding, and (3) deficient compositionality due to bag-of-words behavior. While recent Multimodal Large Language Models (MLLMs) have demonstrated significant advances in generalized vision-language understanding, their potential for learning transferable multimodal representations remains underexplored.In this work, we present UniME (Universal Multimodal Embedding), a novel two-stage framework that leverages MLLMs to learn discriminative representations for diverse downstream tasks. In the first stage, we perform textual discriminative knowledge distillation from a powerful LLM-based teacher model to enhance the embedding capability of the MLLM\'s language component. In the second stage, we introduce hard negative enhanced instruction tuning to further advance discriminative representation learning. Specifically, we initially mitigate false negative contamination and then sample multiple hard negatives per instance within each batch, forcing the model to focus on challenging samples. This approach not only improves discriminative power but also enhances instruction-following ability in downstream tasks. We conduct extensive experiments on the MMEB benchmark and multiple retrieval tasks, including short and long caption retrieval and compositional retrieval. Results demonstrate that UniME achieves consistent performance improvement across all tasks, exhibiting superior discriminative and compositional capabilities.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17432</guid>
|
||||||
|
<pubDate>Thu, 24 Apr 2025 10:51:52 +0000</pubDate>
|
||||||
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generation</title>
|
<title>RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generation</title>
|
||||||
<link>https://arxiv.org/abs/2504.17502</link>
|
<link>https://arxiv.org/abs/2504.17502</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17502.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon, Michal Sokolik, Nitzan Bitton Guetta, Almog Gueta, Royi Rassin, Itay Laish, Dani Lischinski, Idan Szpektor</p><p><b>Upvotes:</b> 36</p><p><b>Summary:</b> Subject-driven text-to-image (T2I) generation aims to produce images that align with a given textual description, while preserving the visual identity from a referenced subject image. Despite its broad downstream applicability -- ranging from enhanced personalization in image generation to consistent character representation in video rendering -- progress in this field is limited by the lack of reliable automatic evaluation. Existing methods either assess only one aspect of the task (i.e., textual alignment or subject preservation), misalign with human judgments, or rely on costly API-based evaluation. To address this, we introduce RefVNLI, a cost-effective metric that evaluates both textual alignment and subject preservation in a single prediction. Trained on a large-scale dataset derived from video-reasoning benchmarks and image perturbations, RefVNLI outperforms or matches existing baselines across multiple benchmarks and subject categories (e.g., Animal, Object), achieving up to 6.4-point gains in textual alignment and 8.5-point gains in subject consistency. It also excels with lesser-known concepts, aligning with human preferences at over 87\% accuracy.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17502.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon, Michal Sokolik, Nitzan Bitton Guetta, Almog Gueta, Royi Rassin, Itay Laish, Dani Lischinski, Idan Szpektor</p><p><b>Upvotes:</b> 54</p><p><b>Summary:</b> Subject-driven text-to-image (T2I) generation aims to produce images that align with a given textual description, while preserving the visual identity from a referenced subject image. Despite its broad downstream applicability -- ranging from enhanced personalization in image generation to consistent character representation in video rendering -- progress in this field is limited by the lack of reliable automatic evaluation. Existing methods either assess only one aspect of the task (i.e., textual alignment or subject preservation), misalign with human judgments, or rely on costly API-based evaluation. To address this, we introduce RefVNLI, a cost-effective metric that evaluates both textual alignment and subject preservation in a single prediction. Trained on a large-scale dataset derived from video-reasoning benchmarks and image perturbations, RefVNLI outperforms or matches existing baselines across multiple benchmarks and subject categories (e.g., Animal, Object), achieving up to 6.4-point gains in textual alignment and 8.5-point gains in subject consistency. It also excels with lesser-known concepts, aligning with human preferences at over 87\% accuracy.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17502</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17502</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 12:44:51 +0000</pubDate>
|
<pubDate>Thu, 24 Apr 2025 12:44:51 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Step1X-Edit: A Practical Framework for General Image Editing</title>
|
<title>Step1X-Edit: A Practical Framework for General Image Editing</title>
|
||||||
<link>https://arxiv.org/abs/2504.17761</link>
|
<link>https://arxiv.org/abs/2504.17761</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17761.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, Guopeng Li, Yuang Peng, Quan Sun, Jingwei Wu, Yan Cai, Zheng Ge, Ranchen Ming, Lei Xia, Xianfang Zeng, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Gang Yu, Daxin Jiang</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> In recent years, image editing models have witnessed remarkable and rapid development. The recent unveiling of cutting-edge multimodal models such as GPT-4o and Gemini2 Flash has introduced highly promising image editing capabilities. These models demonstrate an impressive aptitude for fulfilling a vast majority of user-driven editing requirements, marking a significant advancement in the field of image manipulation. However, there is still a large gap between the open-source algorithm with these closed-source models. Thus, in this paper, we aim to release a state-of-the-art image editing model, called Step1X-Edit, which can provide comparable performance against the closed-source models like GPT-4o and Gemini2 Flash. More specifically, we adopt the Multimodal LLM to process the reference image and the user's editing instruction. A latent embedding has been extracted and integrated with a diffusion image decoder to obtain the target image. To train the model, we build a data generation pipeline to produce a high-quality dataset. For evaluation, we develop the GEdit-Bench, a novel benchmark rooted in real-world user instructions. Experimental results on GEdit-Bench demonstrate that Step1X-Edit outperforms existing open-source baselines by a substantial margin and approaches the performance of leading proprietary models, thereby making significant contributions to the field of image editing.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17761.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, Guopeng Li, Yuang Peng, Quan Sun, Jingwei Wu, Yan Cai, Zheng Ge, Ranchen Ming, Lei Xia, Xianfang Zeng, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Gang Yu, Daxin Jiang</p><p><b>Upvotes:</b> 83</p><p><b>Summary:</b> In recent years, image editing models have witnessed remarkable and rapid development. The recent unveiling of cutting-edge multimodal models such as GPT-4o and Gemini2 Flash has introduced highly promising image editing capabilities. These models demonstrate an impressive aptitude for fulfilling a vast majority of user-driven editing requirements, marking a significant advancement in the field of image manipulation. However, there is still a large gap between the open-source algorithm with these closed-source models. Thus, in this paper, we aim to release a state-of-the-art image editing model, called Step1X-Edit, which can provide comparable performance against the closed-source models like GPT-4o and Gemini2 Flash. More specifically, we adopt the Multimodal LLM to process the reference image and the user's editing instruction. A latent embedding has been extracted and integrated with a diffusion image decoder to obtain the target image. To train the model, we build a data generation pipeline to produce a high-quality dataset. For evaluation, we develop the GEdit-Bench, a novel benchmark rooted in real-world user instructions. Experimental results on GEdit-Bench demonstrate that Step1X-Edit outperforms existing open-source baselines by a substantial margin and approaches the performance of leading proprietary models, thereby making significant contributions to the field of image editing.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17761</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17761</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 17:25:12 +0000</pubDate>
|
<pubDate>Thu, 24 Apr 2025 17:25:12 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.18415</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.18415.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongyu Wang, Shuming Ma, Furu Wei</p><p><b>Upvotes:</b> 39</p><p><b>Summary:</b> Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.18415</guid>
|
||||||
|
<pubDate>Fri, 25 Apr 2025 15:17:52 +0000</pubDate>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>Reinforcement Learning for Reasoning in Large Language Models with One Training Example</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.20571</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.20571.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen</p><p><b>Upvotes:</b> 47</p><p><b>Summary:</b> We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0% to 73.6%, and improves the average performance across six common mathematical reasoning benchmarks from 17.6% to 35.7%. This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6%, average: 35.9%), which includes the aforementioned example. Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples (many of which yield approximately 30% or greater improvement on MATH500 when employed as a single training example). In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-domain generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post-saturation generalization. Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., by adding entropy loss with an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe that applying entropy loss alone, without any outcome reward, significantly enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings can inspire future work on RLVR data efficiency and encourage a re-examination of both recent progress and the underlying mechanisms in RLVR. Our code, model, and data are open source at https://github.com/ypwang61/One-Shot-RLVR</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.20571</guid>
|
||||||
|
<pubDate>Tue, 29 Apr 2025 09:24:30 +0000</pubDate>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>ReasonIR: Training Retrievers for Reasoning Tasks</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.20595</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.20595.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, Luke Zettlemoyer</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> We present ReasonIR-8B, the first retriever specifically trained for general reasoning tasks. Existing retrievers have shown limited gains on reasoning tasks, in part because existing training datasets focus on short factual queries tied to documents that straightforwardly answer them. We develop a synthetic data generation pipeline that, for each document, our pipeline creates a challenging and relevant query, along with a plausibly related but ultimately unhelpful hard negative. By training on a mixture of our synthetic data and existing public data, ReasonIR-8B achieves a new state-of-the-art of 29.9 nDCG@10 without reranker and 36.9 nDCG@10 with reranker on BRIGHT, a widely-used reasoning-intensive information retrieval (IR) benchmark. When applied to RAG tasks, ReasonIR-8B improves MMLU and GPQA performance by 6.4% and 22.6% respectively, relative to the closed-book baseline, outperforming other retrievers and search engines. In addition, ReasonIR-8B uses test-time compute more effectively: on BRIGHT, its performance consistently increases with longer and more information-rich rewritten queries; it continues to outperform other retrievers when combined with an LLM reranker. Our training recipe is general and can be easily extended to future LLMs; to this end, we open-source our code, data, and model.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.20595</guid>
|
||||||
|
<pubDate>Tue, 29 Apr 2025 09:49:28 +0000</pubDate>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>UniversalRAG: Retrieval-Augmented Generation over Multiple Corpora with Diverse Modalities and Granularities</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.20734</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.20734.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang</p><p><b>Upvotes:</b> 42</p><p><b>Summary:</b> Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing RAG approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus. In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To address this, we introduce UniversalRAG, a novel RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single combined corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose a modality-aware routing mechanism that dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it. Also, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query. We validate UniversalRAG on 8 benchmarks spanning multiple modalities, showing its superiority over modality-specific and unified baselines.</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.20734</guid>
|
||||||
|
<pubDate>Tue, 29 Apr 2025 13:18:58 +0000</pubDate>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>The Leaderboard Illusion</title>
|
||||||
|
<link>https://arxiv.org/abs/2504.20879</link>
|
||||||
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.20879.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> Measuring progress is fundamental to the advancement of any scientific field. As benchmarks play an increasingly central role, they also grow more susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard for ranking the most capable AI systems. Yet, in this work we identify systematic issues that have resulted in a distorted playing field. We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired. We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results. At an extreme, we identify 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release. We also establish that proprietary closed models are sampled at higher rates (number of battles) and have fewer models removed from the arena than open-weight and open-source alternatives. Both these policies lead to large data access asymmetries over time. Providers like Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the arena, respectively. In contrast, a combined 83 open-weight models have only received an estimated 29.7% of the total data. We show that access to Chatbot Arena data yields substantial benefits; even limited additional data can result in relative performance gains of up to 112% on the arena distribution, based on our conservative estimates. Together, these dynamics result in overfitting to Arena-specific dynamics rather than general model quality. The Arena builds on the substantial efforts of both the organizers and an open community that maintains this valuable evaluation platform. We offer actionable recommendations to reform the Chatbot Arena's evaluation framework and promote fairer, more transparent benchmarking for the field</p></description>
|
||||||
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.20879</guid>
|
||||||
|
<pubDate>Tue, 29 Apr 2025 15:48:49 +0000</pubDate>
|
||||||
|
</item>
|
||||||
</channel>
|
</channel>
|
||||||
</rss>
|
</rss>
|
||||||
|
|||||||
Reference in New Issue
Block a user