diff --git a/feed.xml b/feed.xml index a859ed3..ea9a5f7 100644 --- a/feed.xml +++ b/feed.xml @@ -7,83 +7,125 @@ http://www.rssboard.org/rss-specification python-feedgen en - Fri, 04 Jul 2025 00:03:04 +0000 + Sat, 05 Jul 2025 00:02:26 +0000 - STR-Match: Matching SpatioTemporal Relevance Score for Training-Free Video Editing - https://arxiv.org/abs/2506.22868 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22868.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Junsung Lee, Junoh Kang, Bohyung Han</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Previous text-guided video editing methods often suffer from temporal inconsistency, motion distortion, and-most notably-limited domain transformation. We attribute these limitations to insufficient modeling of spatiotemporal pixel relevance during the editing process. To address this, we propose STR-Match, a training-free video editing algorithm that produces visually appealing and spatiotemporally coherent videos through latent optimization guided by our novel STR score. The score captures spatiotemporal pixel relevance across adjacent frames by leveraging 2D spatial attention and 1D temporal modules in text-to-video (T2V) diffusion models, without the overhead of computationally expensive 3D attention mechanisms. Integrated into a latent optimization framework with a latent mask, STR-Match generates temporally consistent and visually faithful videos, maintaining strong performance even under significant domain transformations while preserving key visual attributes of the source. Extensive experiments demonstrate that STR-Match consistently outperforms existing methods in both visual quality and spatiotemporal consistency.</p> - https://arxiv.org/abs/2506.22868 - Sat, 28 Jun 2025 12:36:19 +0000 + HalluSegBench: Counterfactual Visual Reasoning for Segmentation Hallucination Evaluation + https://arxiv.org/abs/2506.21546 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.21546.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xinzhuo Li, Adheesh Juvekar, Xingyou Liu, Muntasir Wahed, Kiet A. Nguyen, Ismini Lourentzou</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Recent progress in vision-language segmentation has significantly advanced grounded visual understanding. However, these models often exhibit hallucinations by producing segmentation masks for objects not grounded in the image content or by incorrectly labeling irrelevant regions. Existing evaluation protocols for segmentation hallucination primarily focus on label or textual hallucinations without manipulating the visual context, limiting their capacity to diagnose critical failures. In response, we introduce HalluSegBench, the first benchmark specifically designed to evaluate hallucinations in visual grounding through the lens of counterfactual visual reasoning. Our benchmark consists of a novel dataset of 1340 counterfactual instance pairs spanning 281 unique object classes, and a set of newly introduced metrics that quantify hallucination sensitivity under visually coherent scene edits. Experiments on HalluSegBench with state-of-the-art vision-language segmentation models reveal that vision-driven hallucinations are significantly more prevalent than label-driven ones, with models often persisting in false segmentation, highlighting the need for counterfactual reasoning to diagnose grounding fidelity.</p> + https://arxiv.org/abs/2506.21546 + Thu, 26 Jun 2025 17:59:12 +0000 - JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching - https://arxiv.org/abs/2506.23552 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23552.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mingi Kwon, Joonghyuk Shin, Jaeseok Jung, Jaesik Park, Youngjung Uh</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> The intrinsic link between facial motion and speech is often overlooked in generative modeling, where talking head synthesis and text-to-speech (TTS) are typically addressed as separate tasks. This paper introduces JAM-Flow, a unified framework to simultaneously synthesize and condition on both facial motion and speech. Our approach leverages flow matching and a novel Multi-Modal Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT and Audio-DiT modules. These are coupled via selective joint attention layers and incorporate key architectural choices, such as temporally aligned positional embeddings and localized joint attention masking, to enable effective cross-modal interaction while preserving modality-specific strengths. Trained with an inpainting-style objective, JAM-Flow supports a wide array of conditioning inputs-including text, reference audio, and reference motion-facilitating tasks such as synchronized talking head generation from text, audio-driven animation, and much more, within a single, coherent model. JAM-Flow significantly advances multi-modal generative modeling by providing a practical solution for holistic audio-visual synthesis. project page: https://joonghyuk.com/jamflow-web</p> - https://arxiv.org/abs/2506.23552 - Mon, 30 Jun 2025 06:51:40 +0000 + Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Models + https://arxiv.org/abs/2506.22813 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.22813.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhuojun Ding, Wei Wei, Chenghao Fan</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Supervised fine-tuning (SFT) is widely used to align large language models (LLMs) with information extraction (IE) tasks, such as named entity recognition (NER). However, annotating such fine-grained labels and training domain-specific models is costly. Existing works typically train a unified model across multiple domains, but such approaches lack adaptation and scalability since not all training data benefits target domains and scaling trained models remains challenging. We propose the SaM framework, which dynamically Selects and Merges expert models at inference time. Specifically, for a target domain, we select domain-specific experts pre-trained on existing domains based on (i) domain similarity to the target domain and (ii) performance on sampled instances, respectively. The experts are then merged to create task-specific models optimized for the target domain. By dynamically merging experts beneficial to target domains, we improve generalization across various domains without extra training. Additionally, experts can be added or removed conveniently, leading to great scalability. Extensive experiments on multiple benchmarks demonstrate our framework's effectiveness, which outperforms the unified model by an average of 10%. We further provide insights into potential improvements, practical experience, and extensions of our framework.</p> + https://arxiv.org/abs/2506.22813 + Sat, 28 Jun 2025 08:28:52 +0000 - μ^2Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation - https://arxiv.org/abs/2507.00316 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00316.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Siyou Li, Pengyao Qin, Huanan Wu, Dong Nie, Arun J. Thirunavukarasu, Juntao Yu, Le Zhang</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Automated radiology report generation (RRG) aims to produce detailed textual reports from clinical imaging, such as computed tomography (CT) scans, to improve the accuracy and efficiency of diagnosis and provision of management advice. RRG is complicated by two key challenges: (1) inherent complexity in extracting relevant information from imaging data under resource constraints, and (2) difficulty in objectively evaluating discrepancies between model-generated and expert-written reports. To address these challenges, we propose mu^2LLM, a textbf{mu}ltiscale textbf{mu}ltimodal large language models for RRG tasks. The novel {mu}^2Tokenizer, as an intermediate layer, integrates multi-modal features from the multiscale visual tokenizer and the text tokenizer, then enhances report generation quality through direct preference optimization (DPO), guided by GREEN-RedLlama. Experimental results on four large CT image-report medical datasetdemonstrate that our method outperforms existing approaches, highlighting the potential of our fine-tuned mu^2LLMs on limited data for RRG tasks.</p> - https://arxiv.org/abs/2507.00316 - Mon, 30 Jun 2025 23:14:49 +0000 + CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentation + https://arxiv.org/abs/2506.23121 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23121.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xinlei Yu, Chanmiao Wang, Hui Jin, Ahmed Elazab, Gangyong Jia, Xiang Wan, Changqing Zou, Ruiquan Ge</p><p><b>Upvotes:</b> 1</p><p><b>Summary:</b> Multi-organ medical segmentation is a crucial component of medical image processing, essential for doctors to make accurate diagnoses and develop effective treatment plans. Despite significant progress in this field, current multi-organ segmentation models often suffer from inaccurate details, dependence on geometric prompts and loss of spatial information. Addressing these challenges, we introduce a novel model named CRISP-SAM2 with CRoss-modal Interaction and Semantic Prompting based on SAM2. This model represents a promising approach to multi-organ medical segmentation guided by textual descriptions of organs. Our method begins by converting visual and textual inputs into cross-modal contextualized semantics using a progressive cross-attention interaction mechanism. These semantics are then injected into the image encoder to enhance the detailed understanding of visual information. To eliminate reliance on geometric prompts, we use a semantic prompting strategy, replacing the original prompt encoder to sharpen the perception of challenging targets. In addition, a similarity-sorting self-updating strategy for memory and a mask-refining process is applied to further adapt to medical imaging and enhance localized details. Comparative experiments conducted on seven public datasets indicate that CRISP-SAM2 outperforms existing models. Extensive analysis also demonstrates the effectiveness of our method, thereby confirming its superior performance, especially in addressing the limitations mentioned earlier. Our code is available at: https://github.com/YU-deep/CRISP\_SAM2.git.</p> + https://arxiv.org/abs/2506.23121 + Sun, 29 Jun 2025 07:05:27 +0000 - ARIG: Autoregressive Interactive Head Generation for Real-time Conversations - https://arxiv.org/abs/2507.00472 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.00472.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ying Guo, Xi Liu, Cheng Zhen, Pengfei Yan, Xiaoming Wei</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Face-to-face communication, as a common human activity, motivates the research on interactive head generation. A virtual agent can generate motion responses with both listening and speaking capabilities based on the audio or motion signals of the other user and itself. However, previous clip-wise generation paradigm or explicit listener/speaker generator-switching methods have limitations in future signal acquisition, contextual behavioral understanding, and switching smoothness, making it challenging to be real-time and realistic. In this paper, we propose an autoregressive (AR) based frame-wise framework called ARIG to realize the real-time generation with better interaction realism. To achieve real-time generation, we model motion prediction as a non-vector-quantized AR process. Unlike discrete codebook-index prediction, we represent motion distribution using diffusion procedure, achieving more accurate predictions in continuous space. To improve interaction realism, we emphasize interactive behavior understanding (IBU) and detailed conversational state understanding (CSU). In IBU, based on dual-track dual-modal signals, we summarize short-range behaviors through bidirectional-integrated learning and perform contextual understanding over long ranges. In CSU, we use voice activity signals and context features of IBU to understand the various states (interruption, feedback, pause, etc.) that exist in actual conversations. These serve as conditions for the final progressive motion prediction. Extensive experiments have verified the effectiveness of our model.</p> - https://arxiv.org/abs/2507.00472 - Tue, 01 Jul 2025 06:38:14 +0000 + Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers + https://arxiv.org/abs/2506.23918 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.23918.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, Linjie Li, Yu Cheng, Heng Ji, Junxian He, Yi R. Fung</p><p><b>Upvotes:</b> 26</p><p><b>Summary:</b> Recent progress in multimodal reasoning has been significantly advanced by textual Chain-of-Thought (CoT), a paradigm where models conduct reasoning within language. This text-centric approach, however, treats vision as a static, initial context, creating a fundamental "semantic gap" between rich perceptual data and discrete symbolic thought. Human cognition often transcends language, utilizing vision as a dynamic mental sketchpad. A similar evolution is now unfolding in AI, marking a fundamental paradigm shift from models that merely think about images to those that can truly think with images. This emerging paradigm is characterized by models leveraging visual information as intermediate steps in their thought process, transforming vision from a passive input into a dynamic, manipulable cognitive workspace. In this survey, we chart this evolution of intelligence along a trajectory of increasing cognitive autonomy, which unfolds across three key stages: from external tool exploration, through programmatic manipulation, to intrinsic imagination. To structure this rapidly evolving field, our survey makes four key contributions. (1) We establish the foundational principles of the think with image paradigm and its three-stage framework. (2) We provide a comprehensive review of the core methods that characterize each stage of this roadmap. (3) We analyze the critical landscape of evaluation benchmarks and transformative applications. (4) We identify significant challenges and outline promising future directions. By providing this structured overview, we aim to offer a clear roadmap for future research towards more powerful and human-aligned multimodal AI.</p> + https://arxiv.org/abs/2506.23918 + Mon, 30 Jun 2025 14:48:35 +0000 - MARVIS: Modality Adaptive Reasoning over VISualizations - https://arxiv.org/abs/2507.01544 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01544.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Benjamin Feuer, Lennart Purucker, Oussama Elachqar, Chinmay Hegde</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Scientific applications of machine learning often rely on small, specialized models tuned to particular domains. Such models often achieve excellent performance, but lack flexibility. Foundation models offer versatility, but typically underperform specialized approaches, especially on non-traditional modalities and long-tail domains. We propose MARVIS (Modality Adaptive Reasoning over VISualizations), a training-free method that enables even small vision-language models to predict any data modality with high accuracy. MARVIS transforms latent embedding spaces into visual representations and then leverages the spatial and fine-grained reasoning skills of VLMs to successfully interpret and utilize them. MARVIS achieves competitive performance on vision, audio, biological, and tabular domains using a single 3B parameter model, achieving results that beat Gemini by 16\% on average and approach specialized methods, without exposing personally identifiable information (P.I.I.) or requiring any domain-specific training. We open source our code and datasets at https://github.com/penfever/marvis</p> - https://arxiv.org/abs/2507.01544 - Wed, 02 Jul 2025 09:56:24 +0000 + ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention + https://arxiv.org/abs/2507.01004 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01004.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yuhong Chou, Zehao Liu, Ruijie Zhu, Xinyi Wan, Tianjian Li, Congying Chu, Qian Liu, Jibin Wu, Zejun Ma</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Linear attention mechanisms deliver significant advantages for Large Language Models (LLMs) by providing linear computational complexity, enabling efficient processing of ultra-long sequences (e.g., 1M context). However, existing Sequence Parallelism (SP) methods, essential for distributing these workloads across devices, become the primary bottleneck due to substantial communication overhead. In this paper, we introduce ZeCO (Zero Communication Overhead) sequence parallelism for linear attention models, a new SP method designed to overcome these limitations and achieve end-to-end near-linear scalability for long sequence training. For example, training a model with a 1M sequence length across 64 devices using ZeCO takes roughly the same time as training with an 16k sequence on a single device. At the heart of ZeCO lies All-Scan, a new collective communication primitive. All-Scan provides each SP rank with precisely the initial operator state it requires while maintaining a minimal communication footprint, effectively eliminating communication overhead. Theoretically, we prove the optimaity of ZeCO, showing that it introduces only negligible time and space overhead. Empirically, we compare the communication costs of different sequence parallelism strategies and demonstrate that All-Scan achieves the fastest communication in SP scenarios. Specifically, on 256 GPUs with an 8M sequence length, ZeCO achieves a 60\% speedup compared to the current state-of-the-art (SOTA) SP method. We believe ZeCO establishes a clear path toward efficiently training next-generation LLMs on previously intractable sequence lengths.</p> + https://arxiv.org/abs/2507.01004 + Tue, 01 Jul 2025 17:54:53 +0000 - Depth Anything at Any Condition - https://arxiv.org/abs/2507.01634 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01634.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Boyuan Sun, Modi Jin, Bowen Yin, Qibin Hou</p><p><b>Upvotes:</b> 34</p><p><b>Summary:</b> We present Depth Anything at Any Condition (DepthAnything-AC), a foundation monocular depth estimation (MDE) model capable of handling diverse environmental conditions. Previous foundation MDE models achieve impressive performance across general scenes but not perform well in complex open-world environments that involve challenging conditions, such as illumination variations, adverse weather, and sensor-induced distortions. To overcome the challenges of data scarcity and the inability of generating high-quality pseudo-labels from corrupted images, we propose an unsupervised consistency regularization finetuning paradigm that requires only a relatively small amount of unlabeled data. Furthermore, we propose the Spatial Distance Constraint to explicitly enforce the model to learn patch-level relative relationships, resulting in clearer semantic boundaries and more accurate details. Experimental results demonstrate the zero-shot capabilities of DepthAnything-AC across diverse benchmarks, including real-world adverse weather benchmarks, synthetic corruption benchmarks, and general benchmarks. Project Page: https://ghost233lism.github.io/depthanything-AC-page Code: https://github.com/HVision-NKU/DepthAnythingAC</p> - https://arxiv.org/abs/2507.01634 - Wed, 02 Jul 2025 12:05:57 +0000 + Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy + https://arxiv.org/abs/2507.01352 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01352.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, Yahui Zhou</p><p><b>Upvotes:</b> 31</p><p><b>Summary:</b> Despite the critical role of reward models (RMs) in reinforcement learning from human feedback (RLHF), current state-of-the-art open RMs perform poorly on most existing evaluation benchmarks, failing to capture the spectrum of nuanced and sophisticated human preferences. Even approaches that incorporate advanced training techniques have not yielded meaningful performance improvements. We hypothesize that this brittleness stems primarily from limitations in preference datasets, which are often narrowly scoped, synthetically labeled, or lack rigorous quality control. To address these challenges, we present a large-scale preference dataset comprising 40 million preference pairs, named SynPref-40M. To enable data curation at scale, we design a human-AI synergistic two-stage pipeline that leverages the complementary strengths of human annotation quality and AI scalability. In this pipeline, humans provide verified annotations, while large language models perform automatic curation based on human guidance. Training on this preference mixture, we introduce Skywork-Reward-V2, a suite of eight reward models ranging from 0.6B to 8B parameters, trained on a carefully curated subset of 26 million preference pairs from SynPref-40M. We demonstrate that Skywork-Reward-V2 is versatile across a wide range of capabilities, including alignment with human preferences, objective correctness, safety, resistance to stylistic biases, and best-of-N scaling, achieving state-of-the-art performance across seven major reward model benchmarks. Ablation studies confirm that the effectiveness of our approach stems not only from data scale but also from high-quality curation. The Skywork-Reward-V2 series represents substantial progress in open reward models, highlighting the untapped potential of existing preference datasets and demonstrating how human-AI curation synergy can unlock significantly higher data quality.</p> + https://arxiv.org/abs/2507.01352 + Wed, 02 Jul 2025 04:40:29 +0000 - A Survey on Vision-Language-Action Models: An Action Tokenization Perspective - https://arxiv.org/abs/2507.01925 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01925.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, Zhiquan Qi, Yitao Liang, Yuanpei Chen, Yaodong Yang</p><p><b>Upvotes:</b> 19</p><p><b>Summary:</b> The remarkable advancements of vision and language foundation models in multimodal understanding, reasoning, and generation has sparked growing efforts to extend such intelligence to the physical world, fueling the flourishing of vision-language-action (VLA) models. Despite seemingly diverse approaches, we observe that current VLA models can be unified under a single framework: vision and language inputs are processed by a series of VLA modules, producing a chain of action tokens that progressively encode more grounded and actionable information, ultimately generating executable actions. We further determine that the primary design choice distinguishing VLA models lies in how action tokens are formulated, which can be categorized into language description, code, affordance, trajectory, goal state, latent representation, raw action, and reasoning. However, there remains a lack of comprehensive understanding regarding action tokens, significantly impeding effective VLA development and obscuring future directions. Therefore, this survey aims to categorize and interpret existing VLA research through the lens of action tokenization, distill the strengths and limitations of each token type, and identify areas for improvement. Through this systematic review and analysis, we offer a synthesized outlook on the broader evolution of VLA models, highlight underexplored yet promising directions, and contribute guidance for future research, hoping to bring the field closer to general-purpose intelligence.</p> - https://arxiv.org/abs/2507.01925 - Wed, 02 Jul 2025 17:34:52 +0000 + AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training + https://arxiv.org/abs/2507.01663 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01663.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, Jianping Wu</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs). Traditional task-colocated RL frameworks suffer from significant scalability bottlenecks, while task-separated RL frameworks face challenges in complex dataflows and the corresponding resource idling and workload imbalance. Moreover, most existing frameworks are tightly coupled with LLM training or inference engines, making it difficult to support custom-designed engines. To address these challenges, we propose AsyncFlow, an asynchronous streaming RL framework for efficient post-training. Specifically, we introduce a distributed data storage and transfer module that provides a unified data management and fine-grained scheduling capability in a fully streamed manner. This architecture inherently facilitates automated pipeline overlapping among RL tasks and dynamic load balancing. Moreover, we propose a producer-consumer-based asynchronous workflow engineered to minimize computational idleness by strategically deferring parameter update process within staleness thresholds. Finally, the core capability of AsynFlow is architecturally decoupled from underlying training and inference engines and encapsulated by service-oriented user interfaces, offering a modular and customizable user experience. Extensive experiments demonstrate an average of 1.59 throughput improvement compared with state-of-the-art baseline. The presented architecture in this work provides actionable insights for next-generation RL training system designs.</p> + https://arxiv.org/abs/2507.01663 + Wed, 02 Jul 2025 12:45:34 +0000 - LongAnimation: Long Animation Generation with Dynamic Global-Local Memory - https://arxiv.org/abs/2507.01945 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01945.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nan Chen, Mengqi Huang, Yihao Meng, Zhendong Mao</p><p><b>Upvotes:</b> 60</p><p><b>Summary:</b> Animation colorization is a crucial part of real animation industry production. Long animation colorization has high labor costs. Therefore, automated long animation colorization based on the video generation model has significant research value. Existing studies are limited to short-term colorization. These studies adopt a local paradigm, fusing overlapping features to achieve smooth transitions between local segments. However, the local paradigm neglects global information, failing to maintain long-term color consistency. In this study, we argue that ideal long-term color consistency can be achieved through a dynamic global-local paradigm, i.e., dynamically extracting global color-consistent features relevant to the current generation. Specifically, we propose LongAnimation, a novel framework, which mainly includes a SketchDiT, a Dynamic Global-Local Memory (DGLM), and a Color Consistency Reward. The SketchDiT captures hybrid reference features to support the DGLM module. The DGLM module employs a long video understanding model to dynamically compress global historical features and adaptively fuse them with the current generation features. To refine the color consistency, we introduce a Color Consistency Reward. During inference, we propose a color consistency fusion to smooth the video segment transition. Extensive experiments on both short-term (14 frames) and long-term (average 500 frames) animations show the effectiveness of LongAnimation in maintaining short-term and long-term color consistency for open-domain animation colorization task. The code can be found at https://cn-makers.github.io/long_animation_web/.</p> - https://arxiv.org/abs/2507.01945 - Wed, 02 Jul 2025 17:55:50 +0000 + IntFold: A Controllable Foundation Model for General and Specialized Biomolecular Structure Prediction + https://arxiv.org/abs/2507.02025 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02025.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> The IntFold Team, Leon Qiao, Wayne Bai, He Yan, Gary Liu, Nova Xi, Xiang Zhang</p><p><b>Upvotes:</b> 30</p><p><b>Summary:</b> We introduce IntFold, a controllable foundation model for both general and specialized biomolecular structure prediction. IntFold demonstrates predictive accuracy comparable to the state-of-the-art AlphaFold3, while utilizing a superior customized attention kernel. Beyond standard structure prediction, IntFold can be adapted to predict allosteric states, constrained structures, and binding affinity through the use of individual adapters. Furthermore, we introduce a novel confidence head to estimate docking quality, offering a more nuanced assessment for challenging targets such as antibody-antigen complexes. Finally, we share insights gained during the training process of this computationally intensive model.</p> + https://arxiv.org/abs/2507.02025 + Wed, 02 Jul 2025 16:09:47 +0000 - Kwai Keye-VL Technical Report - https://arxiv.org/abs/2507.01949 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01949.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kwai Keye Team, Biao Yang, Bin Wen, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, Fan Yang, Guorui Zhou, Hao Peng, Haojie Ding, Jiaming Huang, Jiangxia Cao, Jiankang Chen, Jingyun Hua, Jin Ouyang, Kaibing Chen, Kaiyu Jiang, Kaiyu Tang, Kun Gai, Shengnan Zhang, Siyang Mao, Sui Huang, Tianke Zhang, Tingting Gao, Wei Chen, Wei Yuan, Xiangyu Wu, Xiao Hu, Xingyu Lu, Yang Zhou, Yi-Fan Zhang, Yiping Yang, Yulong Chen, Zhenhua Wu, Zhenyu Li, Zhixin Ling, Ziming Li, Dehua Ma, Di Xu, Haixuan Gao, Hang Li, Jiawei Guo, Jing Wang, Lejian Ren, Muhao Wei, Qianqian Wang, Qigen Hu, Shiyao Wang, Tao Yu, Xinchen Luo, Yan Li, Yiming Liang, Yuhang Hu, Zeyi Lu, Zhuoran Yang, Zixing Zhang</p><p><b>Upvotes:</b> 94</p><p><b>Summary:</b> While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities on static images, they often fall short in comprehending dynamic, information-dense short-form videos, a dominant medium in today's digital landscape. To bridge this gap, we introduce Kwai Keye-VL, an 8-billion-parameter multimodal foundation model engineered for leading-edge performance in short-video understanding while maintaining robust general-purpose vision-language abilities. The development of Keye-VL rests on two core pillars: a massive, high-quality dataset exceeding 600 billion tokens with a strong emphasis on video, and an innovative training recipe. This recipe features a four-stage pre-training process for solid vision-language alignment, followed by a meticulous two-phase post-training process. The first post-training stage enhances foundational capabilities like instruction following, while the second phase focuses on stimulating advanced reasoning. In this second phase, a key innovation is our five-mode ``cold-start'' data mixture, which includes ``thinking'', ``non-thinking'', ``auto-think'', ``think with image'', and high-quality video data. This mixture teaches the model to decide when and how to reason. Subsequent reinforcement learning (RL) and alignment steps further enhance these reasoning capabilities and correct abnormal model behaviors, such as repetitive outputs. To validate our approach, we conduct extensive evaluations, showing that Keye-VL achieves state-of-the-art results on public video benchmarks and remains highly competitive on general image-based tasks (Figure 1). Furthermore, we develop and release the KC-MMBench, a new benchmark tailored for real-world short-video scenarios, where Keye-VL shows a significant advantage.</p> - https://arxiv.org/abs/2507.01949 - Wed, 02 Jul 2025 17:57:28 +0000 + Energy-Based Transformers are Scalable Learners and Thinkers + https://arxiv.org/abs/2507.02092 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02092.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limitations: they are modality-specific (e.g., working only in text), problem-specific (e.g., verifiable domains like math and coding), or require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards). In this paper, we ask the question "Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?" Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier. Specifically, we train Energy-Based Transformers (EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs improve performance with System 2 Thinking by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using fewer forward passes. Further, we find that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, suggesting that EBTs generalize better than existing approaches. Consequently, EBTs are a promising new paradigm for scaling both the learning and thinking capabilities of models.</p> + https://arxiv.org/abs/2507.02092 + Wed, 02 Jul 2025 19:17:29 +0000 - FreeMorph: Tuning-Free Generalized Image Morphing with Diffusion Model - https://arxiv.org/abs/2507.01953 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01953.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Yukang Cao, Chenyang Si, Jinghao Wang, Ziwei Liu</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> We present FreeMorph, the first tuning-free method for image morphing that accommodates inputs with different semantics or layouts. Unlike existing methods that rely on finetuning pre-trained diffusion models and are limited by time constraints and semantic/layout discrepancies, FreeMorph delivers high-fidelity image morphing without requiring per-instance training. Despite their efficiency and potential, tuning-free methods face challenges in maintaining high-quality results due to the non-linear nature of the multi-step denoising process and biases inherited from the pre-trained diffusion model. In this paper, we introduce FreeMorph to address these challenges by integrating two key innovations. 1) We first propose a guidance-aware spherical interpolation design that incorporates explicit guidance from the input images by modifying the self-attention modules, thereby addressing identity loss and ensuring directional transitions throughout the generated sequence. 2) We further introduce a step-oriented variation trend that blends self-attention modules derived from each input image to achieve controlled and consistent transitions that respect both inputs. Our extensive evaluations demonstrate that FreeMorph outperforms existing methods, being 10x ~ 50x faster and establishing a new state-of-the-art for image morphing.</p> - https://arxiv.org/abs/2507.01953 - Wed, 02 Jul 2025 17:58:20 +0000 + Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback + https://arxiv.org/abs/2507.02321 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02321.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Nina Konovalova, Maxim Nikolaev, Andrey Kuznetsov, Aibek Alanov</p><p><b>Upvotes:</b> 33</p><p><b>Summary:</b> Despite significant progress in text-to-image diffusion models, achieving precise spatial control over generated outputs remains challenging. ControlNet addresses this by introducing an auxiliary conditioning module, while ControlNet++ further refines alignment through a cycle consistency loss applied only to the final denoising steps. However, this approach neglects intermediate generation stages, limiting its effectiveness. We propose InnerControl, a training strategy that enforces spatial consistency across all diffusion steps. Our method trains lightweight convolutional probes to reconstruct input control signals (e.g., edges, depth) from intermediate UNet features at every denoising step. These probes efficiently extract signals even from highly noisy latents, enabling pseudo ground truth controls for training. By minimizing the discrepancy between predicted and target conditions throughout the entire diffusion process, our alignment loss improves both control fidelity and generation quality. Combined with established techniques like ControlNet++, InnerControl achieves state-of-the-art performance across diverse conditioning methods (e.g., edges, depth).</p> + https://arxiv.org/abs/2507.02321 + Thu, 03 Jul 2025 05:25:53 +0000 - Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation - https://arxiv.org/abs/2507.01957 - <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.01957.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhuoyang Zhang, Luke J. Huang, Chengyue Wu, Shang Yang, Kelly Peng, Yao Lu, Song Han</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> We present Locality-aware Parallel Decoding (LPD) to accelerate autoregressive image generation. Traditional autoregressive image generation relies on next-patch prediction, a memory-bound process that leads to high latency. Existing works have tried to parallelize next-patch prediction by shifting to multi-patch prediction to accelerate the process, but only achieved limited parallelization. To achieve high parallelization while maintaining generation quality, we introduce two key techniques: (1) Flexible Parallelized Autoregressive Modeling, a novel architecture that enables arbitrary generation ordering and degrees of parallelization. It uses learnable position query tokens to guide generation at target positions while ensuring mutual visibility among concurrently generated tokens for consistent parallel decoding. (2) Locality-aware Generation Ordering, a novel schedule that forms groups to minimize intra-group dependencies and maximize contextual support, enhancing generation quality. With these designs, we reduce the generation steps from 256 to 20 (256times256 res.) and 1024 to 48 (512times512 res.) without compromising quality on the ImageNet class-conditional generation, and achieving at least 3.4times lower latency than previous parallelized autoregressive models.</p> - https://arxiv.org/abs/2507.01957 - Wed, 02 Jul 2025 17:59:23 +0000 + WebSailor: Navigating Super-human Reasoning for Web Agent + https://arxiv.org/abs/2507.02592 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02592.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, Jingren Zhou</p><p><b>Upvotes:</b> 53</p><p><b>Summary:</b> Transcending human cognitive limitations represents a critical frontier in LLM training. Proprietary agentic systems like DeepResearch have demonstrated superhuman capabilities on extremely complex information-seeking benchmarks such as BrowseComp, a feat previously unattainable. We posit that their success hinges on a sophisticated reasoning pattern absent in open-source models: the ability to systematically reduce extreme uncertainty when navigating vast information landscapes. Based on this insight, we introduce WebSailor, a complete post-training methodology designed to instill this crucial capability. Our approach involves generating novel, high-uncertainty tasks through structured sampling and information obfuscation, RFT cold start, and an efficient agentic RL training algorithm, Duplicating Sampling Policy Optimization (DUPO). With this integrated pipeline, WebSailor significantly outperforms all opensource agents in complex information-seeking tasks, matching proprietary agents' performance and closing the capability gap.</p> + https://arxiv.org/abs/2507.02592 + Thu, 03 Jul 2025 12:59:07 +0000 + + + Decoupled Planning and Execution: A Hierarchical Reasoning Framework for Deep Search + https://arxiv.org/abs/2507.02652 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02652.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yang Zhao, Hongjin Qian, Zhicheng Dou</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Complex information needs in real-world search scenarios demand deep reasoning and knowledge synthesis across diverse sources, which traditional retrieval-augmented generation (RAG) pipelines struggle to address effectively. Current reasoning-based approaches suffer from a fundamental limitation: they use a single model to handle both high-level planning and detailed execution, leading to inefficient reasoning and limited scalability. In this paper, we introduce HiRA, a hierarchical framework that separates strategic planning from specialized execution. Our approach decomposes complex search tasks into focused subtasks, assigns each subtask to domain-specific agents equipped with external tools and reasoning capabilities, and coordinates the results through a structured integration mechanism. This separation prevents execution details from disrupting high-level reasoning while enabling the system to leverage specialized expertise for different types of information processing. Experiments on four complex, cross-modal deep search benchmarks demonstrate that HiRA significantly outperforms state-of-the-art RAG and agent-based systems. Our results show improvements in both answer quality and system efficiency, highlighting the effectiveness of decoupled planning and execution for multi-step information seeking tasks. Our code is available at https://github.com/ignorejjj/HiRA.</p> + https://arxiv.org/abs/2507.02652 + Thu, 03 Jul 2025 14:18:08 +0000 + + + Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers + https://arxiv.org/abs/2507.02694 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02694.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhijian Xu, Yilun Zhao, Manasi Patwardhan, Lovekesh Vig, Arman Cohan</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> Peer review is fundamental to scientific research, but the growing volume of publications has intensified the challenges of this expertise-intensive process. While LLMs show promise in various scientific tasks, their potential to assist with peer review, particularly in identifying paper limitations, remains understudied. We first present a comprehensive taxonomy of limitation types in scientific research, with a focus on AI. Guided by this taxonomy, for studying limitations, we present LimitGen, the first comprehensive benchmark for evaluating LLMs' capability to support early-stage feedback and complement human peer review. Our benchmark consists of two subsets: LimitGen-Syn, a synthetic dataset carefully created through controlled perturbations of high-quality papers, and LimitGen-Human, a collection of real human-written limitations. To improve the ability of LLM systems to identify limitations, we augment them with literature retrieval, which is essential for grounding identifying limitations in prior scientific findings. Our approach enhances the capabilities of LLM systems to generate limitations in research papers, enabling them to provide more concrete and constructive feedback.</p> + https://arxiv.org/abs/2507.02694 + Thu, 03 Jul 2025 15:04:38 +0000 + + + Bourbaki: Self-Generated and Goal-Conditioned MDPs for Theorem Proving + https://arxiv.org/abs/2507.02726 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02726.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Matthieu Zimmer, Xiaotong Ji, Rasul Tutunov, Anthony Bordg, Jun Wang, Haitham Bou Ammar</p><p><b>Upvotes:</b> 12</p><p><b>Summary:</b> Reasoning remains a challenging task for large language models (LLMs), especially within the logically constrained environment of automated theorem proving (ATP), due to sparse rewards and the vast scale of proofs. These challenges are amplified in benchmarks like PutnamBench, which contains university-level problems requiring complex, multi-step reasoning. To address this, we introduce self-generated goal-conditioned MDPs (sG-MDPs), a new framework in which agents generate and pursue their subgoals based on the evolving proof state. Given this more structured generation of goals, the resulting problem becomes more amenable to search. We then apply Monte Carlo Tree Search (MCTS)-like algorithms to solve the sG-MDP, instantiating our approach in Bourbaki (7B), a modular system that can ensemble multiple 7B LLMs for subgoal generation and tactic synthesis. On PutnamBench, Bourbaki (7B) solves 26 problems, achieving new state-of-the-art results with models at this scale.</p> + https://arxiv.org/abs/2507.02726 + Thu, 03 Jul 2025 15:41:38 +0000 + + + Fast and Simplex: 2-Simplicial Attention in Triton + https://arxiv.org/abs/2507.02754 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02754.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aurko Roy, Timothy Chou, Sai Surya Duvvuri, Sijia Chen, Jiecao Yu, Xiaodong Wang, Manzil Zaheer, Rohan Anil</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Recent work has shown that training loss scales as a power law with both model size and the number of tokens, and that achieving compute-optimal models requires scaling model size and token count together. However, these scaling laws assume an infinite supply of data and apply primarily in compute-bound settings. As modern large language models increasingly rely on massive internet-scale datasets, the assumption that they are compute-bound is becoming less valid. This shift highlights the need for architectures that prioritize token efficiency. In this work, we investigate the use of the 2-simplicial Transformer, an architecture that generalizes standard dot-product attention to trilinear functions through an efficient Triton kernel implementation. We demonstrate that the 2-simplicial Transformer achieves better token efficiency than standard Transformers: for a fixed token budget, similarly sized models outperform their dot-product counterparts on tasks involving mathematics, coding, reasoning, and logic. We quantify these gains by demonstrating that 2-simplicial attention changes the exponent in the scaling laws for knowledge and reasoning tasks compared to dot product attention.</p> + https://arxiv.org/abs/2507.02754 + Thu, 03 Jul 2025 16:16:34 +0000 + + + Self-Correction Bench: Revealing and Addressing the Self-Correction Blind Spot in LLMs + https://arxiv.org/abs/2507.02778 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02778.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Ken Tsui</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Although large language models (LLMs) have become transformative, they still make mistakes and can explore unproductive reasoning paths. Self-correction is an important capability for a trustworthy LLM, particularly an autoregressive LLM. While LLMs can identify error in user input, they exhibit a systematic 'Self-Correction Blind Spot' - failing to correct identical error in their own outputs. To systematically study this phenomenon, we introduce Self-Correction Bench, a systematic framework to measure this phenomenon through controlled error injection at three complexity levels. Testing 14 models, we find an average 64.5% blind spot rate. We find multiple evidences that this limitation relates to training data composition: human training demonstrations predominantly show error-free responses rather than error-correction sequences, unlike RL-trained models that learn error correction through outcome feedback. Remarkably, simply appending "Wait" reduces blind spots by 89.3%, suggesting that the capability exists but requires activation. Our work highlights a critical limitation in current LLMs and offers potential avenues for improving their reliability and trustworthiness.</p> + https://arxiv.org/abs/2507.02778 + Thu, 03 Jul 2025 16:41:30 +0000 + + + LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion + https://arxiv.org/abs/2507.02813 + <p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2507.02813.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fangfu Liu, Hao Li, Jiawei Chi, Hanyang Wang, Minghui Yang, Fudong Wang, Yueqi Duan</p><p><b>Upvotes:</b> 44</p><p><b>Summary:</b> Recovering 3D structures with open-vocabulary scene understanding from 2D images is a fundamental but daunting task. Recent developments have achieved this by performing per-scene optimization with embedded language information. However, they heavily rely on the calibrated dense-view reconstruction paradigm, thereby suffering from severe rendering artifacts and implausible semantic synthesis when limited views are available. In this paper, we introduce a novel generative framework, coined LangScene-X, to unify and generate 3D consistent multi-modality information for reconstruction and understanding. Powered by the generative capability of creating more consistent novel observations, we can build generalizable 3D language-embedded scenes from only sparse views. Specifically, we first train a TriMap video diffusion model that can generate appearance (RGBs), geometry (normals), and semantics (segmentation maps) from sparse inputs through progressive knowledge integration. Furthermore, we propose a Language Quantized Compressor (LQC), trained on large-scale image datasets, to efficiently encode language embeddings, enabling cross-scene generalization without per-scene retraining. Finally, we reconstruct the language surface fields by aligning language information onto the surface of 3D scenes, enabling open-ended language queries. Extensive experiments on real-world data demonstrate the superiority of our LangScene-X over state-of-the-art methods in terms of quality and generalizability. Project Page: https://liuff19.github.io/LangScene-X.</p> + https://arxiv.org/abs/2507.02813 + Thu, 03 Jul 2025 17:21:23 +0000