bot: update RSS feed
This commit is contained in:
github-actions[bot]
164
feed.xml
164
feed.xml
@@ -7,132 +7,90 @@
|
|||||||
<docs>http://www.rssboard.org/rss-specification</docs>
|
<docs>http://www.rssboard.org/rss-specification</docs>
|
||||||
<generator>python-feedgen</generator>
|
<generator>python-feedgen</generator>
|
||||||
<language>en</language>
|
<language>en</language>
|
||||||
<lastBuildDate>Mon, 28 Apr 2025 00:02:36 +0000</lastBuildDate>
|
<lastBuildDate>Tue, 29 Apr 2025 00:02:34 +0000</lastBuildDate>
|
||||||
<item>
|
<item>
|
||||||
<title>ViSMaP: Unsupervised Hour-long Video Summarisation by Meta-Prompting</title>
|
<title>DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency</title>
|
||||||
<link>https://arxiv.org/abs/2504.15921</link>
|
<link>https://arxiv.org/abs/2504.12080</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15921.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jian Hu, Dimitrios Korkinof, Shaogang Gong, Mariano Beguerisse-Diaz</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> We introduce ViSMap: Unsupervised Video Summarisation by Meta Prompting, a system to summarise hour long videos with no-supervision. Most existing video understanding models work well on short videos of pre-segmented events, yet they struggle to summarise longer videos where relevant events are sparsely distributed and not pre-segmented. Moreover, long-form video understanding often relies on supervised hierarchical training that needs extensive annotations which are costly, slow and prone to inconsistency. With ViSMaP we bridge the gap between short videos (where annotated data is plentiful) and long ones (where it's not). We rely on LLMs to create optimised pseudo-summaries of long videos using segment descriptions from short ones. These pseudo-summaries are used as training data for a model that generates long-form video summaries, bypassing the need for expensive annotations of long videos. Specifically, we adopt a meta-prompting strategy to iteratively generate and refine creating pseudo-summaries of long videos. The strategy leverages short clip descriptions obtained from a supervised short video model to guide the summary. Each iteration uses three LLMs working in sequence: one to generate the pseudo-summary from clip descriptions, another to evaluate it, and a third to optimise the prompt of the generator. This iteration is necessary because the quality of the pseudo-summaries is highly dependent on the generator prompt, and varies widely among videos. We evaluate our summaries extensively on multiple datasets; our results show that ViSMaP achieves performance comparable to fully supervised state-of-the-art models while generalising across domains without sacrificing performance. Code will be released upon publication.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.12080.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Mengshi Qi, Pengfei Zhu, Xiangtai Li, Xiaoyang Bi, Lu Qi, Huadong Ma, Ming-Hsuan Yang</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> Given a single labeled example, in-context segmentation aims to segment corresponding objects. This setting, known as one-shot segmentation in few-shot learning, explores the segmentation model's generalization ability and has been applied to various vision tasks, including scene understanding and image/video editing. While recent Segment Anything Models have achieved state-of-the-art results in interactive segmentation, these approaches are not directly applicable to in-context segmentation. In this work, we propose the Dual Consistency SAM (DC-SAM) method based on prompt-tuning to adapt SAM and SAM2 for in-context segmentation of both images and videos. Our key insights are to enhance the features of the SAM's prompt encoder in segmentation by providing high-quality visual prompts. When generating a mask prior, we fuse the SAM features to better align the prompt encoder. Then, we design a cycle-consistent cross-attention on fused features and initial visual prompts. Next, a dual-branch design is provided by using the discriminative positive and negative prompts in the prompt encoder. Furthermore, we design a simple mask-tube training strategy to adopt our proposed dual consistency method into the mask tube. Although the proposed DC-SAM is primarily designed for images, it can be seamlessly extended to the video domain with the support of SAM2. Given the absence of in-context segmentation in the video domain, we manually curate and construct the first benchmark from existing video segmentation datasets, named In-Context Video Object Segmentation (IC-VOS), to better assess the in-context capability of the model. Extensive experiments demonstrate that our method achieves 55.5 (+1.4) mIoU on COCO-20i, 73.0 (+1.1) mIoU on PASCAL-5i, and a J&F score of 71.52 on the proposed IC-VOS benchmark. Our source code and benchmark are available at https://github.com/zaplm/DC-SAM.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.15921</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.12080</guid>
|
||||||
<pubDate>Tue, 22 Apr 2025 14:06:01 +0000</pubDate>
|
<pubDate>Wed, 16 Apr 2025 13:41:59 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Boosting Generative Image Modeling via Joint Image-Feature Synthesis</title>
|
<title>Towards Understanding Camera Motions in Any Video</title>
|
||||||
<link>https://arxiv.org/abs/2504.16064</link>
|
<link>https://arxiv.org/abs/2504.15376</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16064.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Latent diffusion models (LDMs) dominate high-quality image generation, yet integrating representation learning with generative modeling remains a challenge. We introduce a novel generative image modeling framework that seamlessly bridges this gap by leveraging a diffusion model to jointly model low-level image latents (from a variational autoencoder) and high-level semantic features (from a pretrained self-supervised encoder like DINO). Our latent-semantic diffusion approach learns to generate coherent image-feature pairs from pure noise, significantly enhancing both generative quality and training efficiency, all while requiring only minimal modifications to standard Diffusion Transformer architectures. By eliminating the need for complex distillation objectives, our unified design simplifies training and unlocks a powerful new inference strategy: Representation Guidance, which leverages learned semantics to steer and refine image generation. Evaluated in both conditional and unconditional settings, our method delivers substantial improvements in image quality and training convergence speed, establishing a new direction for representation-aware generative modeling.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15376.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, Rushikesh Zawar, Xue Bai, Yilun Du, Chuang Gan, Deva Ramanan</p><p><b>Upvotes:</b> 120</p><p><b>Summary:</b> We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding. CameraBench consists of ~3,000 diverse internet videos, annotated by experts through a rigorous multi-stage quality control process. One of our contributions is a taxonomy of camera motion primitives, designed in collaboration with cinematographers. We find, for example, that some motions like "follow" (or tracking) require understanding scene content like moving subjects. We conduct a large-scale human study to quantify human annotation performance, revealing that domain expertise and tutorial-based training can significantly enhance accuracy. For example, a novice may confuse zoom-in (a change of intrinsics) with translating forward (a change of extrinsics), but can be trained to differentiate the two. Using CameraBench, we evaluate Structure-from-Motion (SfM) and Video-Language Models (VLMs), finding that SfM models struggle to capture semantic primitives that depend on scene content, while VLMs struggle to capture geometric primitives that require precise estimation of trajectories. We then fine-tune a generative VLM on CameraBench to achieve the best of both worlds and showcase its applications, including motion-augmented captioning, video question answering, and video-text retrieval. We hope our taxonomy, benchmark, and tutorials will drive future efforts towards the ultimate goal of understanding camera motions in any video.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.16064</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15376</guid>
|
||||||
<pubDate>Tue, 22 Apr 2025 17:41:42 +0000</pubDate>
|
<pubDate>Mon, 21 Apr 2025 18:34:57 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining</title>
|
<title>DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models</title>
|
||||||
<link>https://arxiv.org/abs/2504.16511</link>
|
<link>https://arxiv.org/abs/2504.15716</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16511.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Fengze Liu, Weidong Zhou, Binbin Liu, Zhimiao Yu, Yifan Zhang, Haobin Lin, Yifeng Yu, Xiaohuan Zhou, Taifeng Wang, Yong Cao</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> Quality and diversity are two critical metrics for the training data of large language models (LLMs), positively impacting performance. Existing studies often optimize these metrics separately, typically by first applying quality filtering and then adjusting data proportions. However, these approaches overlook the inherent trade-off between quality and diversity, necessitating their joint consideration. Given a fixed training quota, it is essential to evaluate both the quality of each data point and its complementary effect on the overall dataset. In this paper, we introduce a unified data selection framework called QuaDMix, which automatically optimizes the data distribution for LLM pretraining while balancing both quality and diversity. Specifically, we first propose multiple criteria to measure data quality and employ domain classification to distinguish data points, thereby measuring overall diversity. QuaDMix then employs a unified parameterized data sampling function that determines the sampling probability of each data point based on these quality and diversity related labels. To accelerate the search for the optimal parameters involved in the QuaDMix framework, we conduct simulated experiments on smaller models and use LightGBM for parameters searching, inspired by the RegMix method. Our experiments across diverse models and datasets demonstrate that QuaDMix achieves an average performance improvement of 7.2% across multiple benchmarks. These results outperform the independent strategies for quality and diversity, highlighting the necessity and ability to balance data quality and diversity.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.15716.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Jie Zhu, Qian Chen, Huaixia Dou, Junhui Li, Lifan Guo, Feng Chen, Chi Zhang</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> Effective reasoning remains a core challenge for large language models (LLMs) in the financial domain, where tasks often require domain-specific knowledge, precise numerical calculations, and strict adherence to compliance rules. We propose DianJin-R1, a reasoning-enhanced framework designed to address these challenges through reasoning-augmented supervision and reinforcement learning. Central to our approach is DianJin-R1-Data, a high-quality dataset constructed from CFLUE, FinQA, and a proprietary compliance corpus (Chinese Compliance Check, CCC), combining diverse financial reasoning scenarios with verified annotations. Our models, DianJin-R1-7B and DianJin-R1-32B, are fine-tuned from Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct using a structured format that generates both reasoning steps and final answers. To further refine reasoning quality, we apply Group Relative Policy Optimization (GRPO), a reinforcement learning method that incorporates dual reward signals: one encouraging structured outputs and another rewarding answer correctness. We evaluate our models on five benchmarks: three financial datasets (CFLUE, FinQA, and CCC) and two general reasoning benchmarks (MATH-500 and GPQA-Diamond). Experimental results show that DianJin-R1 models consistently outperform their non-reasoning counterparts, especially on complex financial tasks. Moreover, on the real-world CCC dataset, our single-call reasoning models match or even surpass the performance of multi-agent systems that require significantly more computational cost. These findings demonstrate the effectiveness of DianJin-R1 in enhancing financial reasoning through structured supervision and reward-aligned learning, offering a scalable and practical solution for real-world applications.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.16511</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.15716</guid>
|
||||||
<pubDate>Wed, 23 Apr 2025 08:36:50 +0000</pubDate>
|
<pubDate>Tue, 22 Apr 2025 09:01:04 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Process Reward Models That Think</title>
|
<title>Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark</title>
|
||||||
<link>https://arxiv.org/abs/2504.16828</link>
|
<link>https://arxiv.org/abs/2504.16427</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16828.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, Lu Wang</p><p><b>Upvotes:</b> 13</p><p><b>Summary:</b> Step-by-step verifiers -- also known as process reward models (PRMs) -- are a key ingredient for test-time scaling. PRMs require step-level supervision, making them expensive to train. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM, a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers -- using only 1% of the process labels in PRM800K -- across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME '24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation on a subset of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. Our work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training. Our code, data, and models will be released at https://github.com/mukhal/thinkprm.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16427.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hanlei Zhang, Zhuohang Li, Yeshuang Zhu, Hua Xu, Peiwu Wang, Haige Zhu, Jie Zhou, Jinchao Zhang</p><p><b>Upvotes:</b> 11</p><p><b>Summary:</b> Multimodal language analysis is a rapidly evolving field that leverages multiple modalities to enhance the understanding of high-level semantics underlying human conversational utterances. Despite its significance, little research has investigated the capability of multimodal large language models (MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA comprises over 61K multimodal utterances drawn from both staged and real-world scenarios, covering six core dimensions of multimodal semantics: intent, emotion, dialogue act, sentiment, speaking style, and communication behavior. We evaluate eight mainstream branches of LLMs and MLLMs using three methods: zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive experiments reveal that even fine-tuned models achieve only about 60%~70% accuracy, underscoring the limitations of current MLLMs in understanding complex human language. We believe that MMLA will serve as a solid foundation for exploring the potential of large language models in multimodal language analysis and provide valuable resources to advance this field. The datasets and code are open-sourced at https://github.com/thuiar/MMLA.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.16828</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.16427</guid>
|
||||||
<pubDate>Wed, 23 Apr 2025 15:44:54 +0000</pubDate>
|
<pubDate>Wed, 23 Apr 2025 05:25:13 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>IberBench: LLM Evaluation on Iberian Languages</title>
|
<title>Subject-driven Video Generation via Disentangled Identity and Motion</title>
|
||||||
<link>https://arxiv.org/abs/2504.16921</link>
|
<link>https://arxiv.org/abs/2504.17816</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16921.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> José Ángel González, Ian Borrego Obrador, Álvaro Romo Herrero, Areg Mikael Sarvazyan, Mara Chinea-Ríos, Angelo Basile, Marc Franco-Salvador</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Large Language Models (LLMs) remain difficult to evaluate comprehensively, particularly for languages other than English, where high-quality data is often limited. Existing benchmarks and leaderboards are predominantly English-centric, with only a few addressing other languages. These benchmarks fall short in several key areas: they overlook the diversity of language varieties, prioritize fundamental Natural Language Processing (NLP) capabilities over tasks of industrial relevance, and are static. With these aspects in mind, we present IberBench, a comprehensive and extensible benchmark designed to assess LLM performance on both fundamental and industry-relevant NLP tasks, in languages spoken across the Iberian Peninsula and Ibero-America. IberBench integrates 101 datasets from evaluation campaigns and recent benchmarks, covering 22 task categories such as sentiment and emotion analysis, toxicity detection, and summarization. The benchmark addresses key limitations in current evaluation practices, such as the lack of linguistic diversity and static evaluation setups by enabling continual updates and community-driven model and dataset submissions moderated by a committee of experts. We evaluate 23 LLMs ranging from 100 million to 14 billion parameters and provide empirical insights into their strengths and limitations. Our findings indicate that (i) LLMs perform worse on industry-relevant tasks than in fundamental ones, (ii) performance is on average lower for Galician and Basque, (iii) some tasks show results close to random, and (iv) in other tasks LLMs perform above random but below shared task systems. IberBench offers open-source implementations for the entire evaluation pipeline, including dataset normalization and hosting, incremental evaluation of LLMs, and a publicly accessible leaderboard.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17816.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Daneul Kim, Jingxu Zhang, Wonjoon Jin, Sunghyun Cho, Qi Dai, Jaesik Park, Chong Luo</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> We propose to train a subject-driven customized video generation model through decoupling the subject-specific learning from temporal dynamics in zero-shot without additional tuning. A traditional method for video customization that is tuning-free often relies on large, annotated video datasets, which are computationally expensive and require extensive annotation. In contrast to the previous approach, we introduce the use of an image customization dataset directly on training video customization models, factorizing the video customization into two folds: (1) identity injection through image customization dataset and (2) temporal modeling preservation with a small set of unannotated videos through the image-to-video training method. Additionally, we employ random image token dropping with randomized image initialization during image-to-video fine-tuning to mitigate the copy-and-paste issue. To further enhance learning, we introduce stochastic switching during joint optimization of subject-specific and temporal features, mitigating catastrophic forgetting. Our method achieves strong subject consistency and scalability, outperforming existing video customization models in zero-shot settings, demonstrating the effectiveness of our framework.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.16921</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17816</guid>
|
||||||
<pubDate>Wed, 23 Apr 2025 17:48:25 +0000</pubDate>
|
<pubDate>Wed, 23 Apr 2025 06:48:31 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs</title>
|
<title>Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning</title>
|
||||||
<link>https://arxiv.org/abs/2504.17040</link>
|
<link>https://arxiv.org/abs/2504.16656</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17040.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, Ran Xu</p><p><b>Upvotes:</b> 9</p><p><b>Summary:</b> We present DyMU, an efficient, training-free framework that dynamically reduces the computational burden of vision-language models (VLMs) while maintaining high task performance. Our approach comprises two key components. First, Dynamic Token Merging (DToMe) reduces the number of visual token embeddings by merging similar tokens based on image complexity, addressing the inherent inefficiency of fixed-length outputs in vision transformers. Second, Virtual Token Unmerging (VTU) simulates the expected token sequence for large language models (LLMs) by efficiently reconstructing the attention dynamics of a full sequence, thus preserving the downstream performance without additional fine-tuning. Unlike previous approaches, our method dynamically adapts token compression to the content of the image and operates completely training-free, making it readily applicable to most state-of-the-art VLM architectures. Extensive experiments on image and video understanding tasks demonstrate that DyMU can reduce the average visual token count by 32%-85% while achieving comparable performance to full-length models across diverse VLM architectures, including the recently popularized AnyRes-based visual encoders. Furthermore, through qualitative analyses, we demonstrate that DToMe effectively adapts token reduction based on image complexity and, unlike existing systems, provides users more control over computational costs. Project page: https://mikewangwzhl.github.io/dymu/.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.16656.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chris, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, Yang Liu, Yahui Zhou</p><p><b>Upvotes:</b> 43</p><p><b>Summary:</b> We present Skywork R1V2, a next-generation multimodal reasoning model and a major leap forward from its predecessor, Skywork R1V. At its core, R1V2 introduces a hybrid reinforcement learning paradigm that harmonizes reward-model guidance with rule-based strategies, thereby addressing the long-standing challenge of balancing sophisticated reasoning capabilities with broad generalization. To further enhance training efficiency, we propose the Selective Sample Buffer (SSB) mechanism, which effectively counters the ``Vanishing Advantages'' dilemma inherent in Group Relative Policy Optimization (GRPO) by prioritizing high-value samples throughout the optimization process. Notably, we observe that excessive reinforcement signals can induce visual hallucinations--a phenomenon we systematically monitor and mitigate through calibrated reward thresholds throughout the training process. Empirical results affirm the exceptional capability of R1V2, with benchmark-leading performances such as 62.6 on OlympiadBench, 79.0 on AIME2024, 63.6 on LiveCodeBench, and 74.0 on MMMU. These results underscore R1V2's superiority over existing open-source models and demonstrate significant progress in closing the performance gap with premier proprietary systems, including Gemini 2.5 and OpenAI o4-mini. The Skywork R1V2 model weights have been publicly released to promote openness and reproducibility https://huggingface.co/Skywork/Skywork-R1V2-38B.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17040</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.16656</guid>
|
||||||
<pubDate>Wed, 23 Apr 2025 18:38:18 +0000</pubDate>
|
<pubDate>Wed, 23 Apr 2025 12:24:10 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Distilling semantically aware orders for autoregressive image generation</title>
|
<title>VideoVista-CulturalLingo: 360^circ Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension</title>
|
||||||
<link>https://arxiv.org/abs/2504.17069</link>
|
<link>https://arxiv.org/abs/2504.17821</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17069.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Rishav Pramanik, Antoine Poupon, Juan A. Rodriguez, Masih Aminbeidokhti, David Vazquez, Christopher Pal, Zhaozheng Yin, Marco Pedersoli</p><p><b>Upvotes:</b> 4</p><p><b>Summary:</b> Autoregressive patch-based image generation has recently shown competitive results in terms of image quality and scalability. It can also be easily integrated and scaled within Vision-Language models. Nevertheless, autoregressive models require a defined order for patch generation. While a natural order based on the dictation of the words makes sense for text generation, there is no inherent generation order that exists for image generation. Traditionally, a raster-scan order (from top-left to bottom-right) guides autoregressive image generation models. In this paper, we argue that this order is suboptimal, as it fails to respect the causality of the image content: for instance, when conditioned on a visual description of a sunset, an autoregressive model may generate clouds before the sun, even though the color of clouds should depend on the color of the sun and not the inverse. In this work, we show that first by training a model to generate patches in any-given-order, we can infer both the content and the location (order) of each patch during generation. Secondly, we use these extracted orders to finetune the any-given-order model to produce better-quality images. Through our experiments, we show on two datasets that this new generation method produces better images than the traditional raster-scan approach, with similar training costs and no extra annotations.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17821.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xinyu Chen, Yunxin Li, Haoyuan Shi, Baotian Hu, Wenhan Luo, Yaowei Wang, Min Zhang</p><p><b>Upvotes:</b> 19</p><p><b>Summary:</b> Assessing the video comprehension capabilities of multimodal AI systems can effectively measure their understanding and reasoning abilities. Most video evaluation benchmarks are limited to a single language, typically English, and predominantly feature videos rooted in Western cultural contexts. In this paper, we present VideoVista-CulturalLingo, the first video evaluation benchmark designed to bridge cultural, linguistic, and domain divide in video comprehension. Our work differs from existing benchmarks in the following ways: 1) Cultural diversity, incorporating cultures from China, North America, and Europe; 2) Multi-linguistics, with questions presented in Chinese and English-two of the most widely spoken languages; and 3) Broad domain, featuring videos sourced from hundreds of human-created domains. VideoVista-CulturalLingo contains 1,389 videos and 3,134 QA pairs, and we have evaluated 24 recent open-source or proprietary video large models. From the experiment results, we observe that: 1) Existing models perform worse on Chinese-centric questions than Western-centric ones, particularly those related to Chinese history; 2) Current open-source models still exhibit limitations in temporal understanding, especially in the Event Localization task, achieving a maximum score of only 45.2%; 3) Mainstream models demonstrate strong performance in general scientific questions, while open-source models demonstrate weak performance in mathematics.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17069</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17821</guid>
|
||||||
<pubDate>Wed, 23 Apr 2025 19:33:58 +0000</pubDate>
|
<pubDate>Wed, 23 Apr 2025 13:47:30 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning</title>
|
<title>Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation</title>
|
||||||
<link>https://arxiv.org/abs/2504.17192</link>
|
<link>https://arxiv.org/abs/2504.17025</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17192.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Minju Seo, Jinheon Baek, Seongyun Lee, Sung Ju Hwang</p><p><b>Upvotes:</b> 79</p><p><b>Summary:</b> Despite the rapid growth of machine learning research, corresponding code implementations are often unavailable, making it slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent Large Language Models (LLMs) excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, a multi-agent LLM framework that transforms machine learning papers into functional code repositories. PaperCoder operates in three stages: planning, where it constructs a high-level roadmap, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files; analysis, which focuses on interpreting implementation-specific details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through a set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, specifically from the original paper authors, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful implementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17025.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Luca Moroni, Giovanni Puccetti, Pere-Lluis Huguet Cabot, Andrei Stefan Bejgu, Edoardo Barba, Alessio Miaschi, Felice Dell'Orletta, Andrea Esuli, Roberto Navigli</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> The number of pretrained Large Language Models (LLMs) is increasing steadily, though the majority are designed predominantly for the English language. While state-of-the-art LLMs can handle other languages, due to language contamination or some degree of multilingual pretraining data, they are not optimized for non-English languages, leading to inefficient encoding (high token "fertility") and slower inference speed. In this work, we thoroughly compare a variety of vocabulary adaptation techniques for optimizing English LLMs for the Italian language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a novel method that leverages neural mapping for vocabulary substitution. SAVA achieves competitive performance across multiple downstream tasks, enhancing grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and reducing the number of parameters by 1 billion. We show that, following the adaptation of the vocabulary, these models can recover their performance with a relatively limited stage of continual training on the target language. Finally, we test the capabilities of the adapted models on various multi-choice and generative tasks.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17192</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17025</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 01:57:01 +0000</pubDate>
|
<pubDate>Wed, 23 Apr 2025 18:12:27 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation</title>
|
<title>The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs</title>
|
||||||
<link>https://arxiv.org/abs/2504.17207</link>
|
<link>https://arxiv.org/abs/2504.17768</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17207.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Phillip Y. Lee, Jihyeon Je, Chanho Park, Mikaela Angelina Uy, Leonidas Guibas, Minhyuk Sung</p><p><b>Upvotes:</b> 26</p><p><b>Summary:</b> We present a framework for perspective-aware reasoning in vision-language models (VLMs) through mental imagery simulation. Perspective-taking, the ability to perceive an environment or situation from an alternative viewpoint, is a key benchmark for human-level visual understanding, essential for environmental interaction and collaboration with autonomous agents. Despite advancements in spatial reasoning within VLMs, recent research has shown that modern VLMs significantly lack perspective-aware reasoning capabilities and exhibit a strong bias toward egocentric interpretations. To bridge the gap between VLMs and human perception, we focus on the role of mental imagery, where humans perceive the world through abstracted representations that facilitate perspective shifts. Motivated by this, we propose a framework for perspective-aware reasoning, named Abstract Perspective Change (APC), that effectively leverages vision foundation models, such as object detection, segmentation, and orientation estimation, to construct scene abstractions and enable perspective transformations. Our experiments on synthetic and real-image benchmarks, compared with various VLMs, demonstrate significant improvements in perspective-aware reasoning with our framework, further outperforming fine-tuned spatial reasoning models and novel-view-synthesis-based approaches.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17768.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, Edoardo M. Ponti</p><p><b>Upvotes:</b> 8</p><p><b>Summary:</b> Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its viability, its efficiency-accuracy trade-offs, and systematic scaling studies remain unexplored. To address this gap, we perform a careful comparison of training-free sparse attention methods at varying model scales, sequence lengths, and sparsity levels on a diverse collection of long-sequence tasks-including novel ones that rely on natural language while remaining controllable and easy to evaluate. Based on our experiments, we report a series of key findings: 1) an isoFLOPS analysis reveals that for very long sequences, larger and highly sparse models are preferable to smaller and dense ones. 2) The level of sparsity attainable while statistically guaranteeing accuracy preservation is higher during decoding than prefilling, and correlates with model size in the former. 3) There is no clear strategy that performs best across tasks and phases, with different units of sparsification or budget adaptivity needed for different scenarios. Even moderate sparsity levels often result in significant performance degradation on at least one task, highlighting that sparse attention is not a universal solution. 4) We introduce and validate novel scaling laws specifically tailored for sparse attention, providing evidence that our findings are likely to hold true beyond our range of experiments. Through these insights, we demonstrate that sparse attention is a key tool to enhance the capabilities of Transformer LLMs for processing longer sequences, but requires careful evaluation of trade-offs for performance-sensitive applications.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17207</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.17768</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 02:41:34 +0000</pubDate>
|
<pubDate>Thu, 24 Apr 2025 17:39:25 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos</title>
|
<title>Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family</title>
|
||||||
<link>https://arxiv.org/abs/2504.17343</link>
|
<link>https://arxiv.org/abs/2504.18225</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17343.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, Lingpeng Kong, Qi Liu, Yuanxing Zhang, Xu Sun</p><p><b>Upvotes:</b> 6</p><p><b>Summary:</b> The rapid growth of online video platforms, particularly live streaming services, has created an urgent need for real-time video understanding systems. These systems must process continuous video streams and respond to user queries instantaneously, presenting unique challenges for current Video Large Language Models (VideoLLMs). While existing VideoLLMs excel at processing complete videos, they face significant limitations in streaming scenarios due to their inability to handle dense, redundant frames efficiently. We introduce TimeChat-Online, a novel online VideoLLM that revolutionizes real-time video interaction. At its core lies our innovative Differential Token Drop (DTD) module, which addresses the fundamental challenge of visual redundancy in streaming videos. Drawing inspiration from human visual perception's Change Blindness phenomenon, DTD preserves meaningful temporal changes while filtering out static, redundant content between frames. Remarkably, our experiments demonstrate that DTD achieves an 82.8% reduction in video tokens while maintaining 98% performance on StreamingBench, revealing that over 80% of visual content in streaming videos is naturally redundant without requiring language guidance. To enable seamless real-time interaction, we present TimeChat-Online-139K, a comprehensive streaming video dataset featuring diverse interaction patterns including backward-tracing, current-perception, and future-responding scenarios. TimeChat-Online's unique Proactive Response capability, naturally achieved through continuous monitoring of video scene transitions via DTD, sets it apart from conventional approaches. Our extensive evaluation demonstrates TimeChat-Online's superior performance on streaming benchmarks (StreamingBench and OvOBench) and maintaining competitive results on long-form video tasks such as Video-MME and MLVU.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.18225.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Pierre-Carl Langlais, Pavel Chizhov, Mattia Nee, Carlos Rosas Hinostroza, Matthieu Delsart, Irène Girard, Othman Hicheur, Anastasia Stasenko, Ivan P. Yamshchikov</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> We introduce a new generation of small reasoning models for RAG, search, and source summarization. Pleias-RAG-350m and Pleias-RAG-1B are mid-trained on a large synthetic dataset emulating the retrieval of a wide variety of multilingual open sources from the Common Corpus. They provide native support for citation and grounding with literal quotes and reintegrate multiple features associated with RAG workflows, such as query routing, query reformulation, and source reranking. Pleias-RAG-350m and Pleias-RAG-1B outperform SLMs below 4 billion parameters on standardized RAG benchmarks (HotPotQA, 2wiki) and are competitive with popular larger models, including Qwen-2.5-7B, Llama-3.1-8B, and Gemma-3-4B. They are the only SLMs to date maintaining consistent RAG performance across leading European languages and ensuring systematic reference grounding for statements. Due to their size and ease of deployment on constrained infrastructure and higher factuality by design, the models unlock a range of new use cases for generative AI.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17343</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.18225</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 07:59:46 +0000</pubDate>
|
<pubDate>Fri, 25 Apr 2025 10:17:04 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>3DV-TON: Textured 3D-Guided Consistent Video Try-on via Diffusion Models</title>
|
<title>BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs</title>
|
||||||
<link>https://arxiv.org/abs/2504.17414</link>
|
<link>https://arxiv.org/abs/2504.18415</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17414.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Min Wei, Chaohui Yu, Jingkai Zhou, Fan Wang</p><p><b>Upvotes:</b> 5</p><p><b>Summary:</b> Video try-on replaces clothing in videos with target garments. Existing methods struggle to generate high-quality and temporally consistent results when handling complex clothing patterns and diverse body poses. We present 3DV-TON, a novel diffusion-based framework for generating high-fidelity and temporally consistent video try-on results. Our approach employs generated animatable textured 3D meshes as explicit frame-level guidance, alleviating the issue of models over-focusing on appearance fidelity at the expanse of motion coherence. This is achieved by enabling direct reference to consistent garment texture movements throughout video sequences. The proposed method features an adaptive pipeline for generating dynamic 3D guidance: (1) selecting a keyframe for initial 2D image try-on, followed by (2) reconstructing and animating a textured 3D mesh synchronized with original video poses. We further introduce a robust rectangular masking strategy that successfully mitigates artifact propagation caused by leaking clothing information during dynamic human and garment movements. To advance video try-on research, we introduce HR-VVT, a high-resolution benchmark dataset containing 130 videos with diverse clothing types and scenarios. Quantitative and qualitative results demonstrate our superior performance over existing methods. The project page is at this link https://2y7c3.github.io/3DV-TON/</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.18415.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Hongyu Wang, Shuming Ma, Furu Wei</p><p><b>Upvotes:</b> 24</p><p><b>Summary:</b> Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17414</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.18415</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 10:12:40 +0000</pubDate>
|
<pubDate>Fri, 25 Apr 2025 15:17:52 +0000</pubDate>
|
||||||
</item>
|
</item>
|
||||||
<item>
|
<item>
|
||||||
<title>Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs</title>
|
<title>Kimi-Audio Technical Report</title>
|
||||||
<link>https://arxiv.org/abs/2504.17432</link>
|
<link>https://arxiv.org/abs/2504.18425</link>
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17432.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, Jiankang Deng</p><p><b>Upvotes:</b> 35</p><p><b>Summary:</b> The Contrastive Language-Image Pre-training (CLIP) framework has become a widely used approach for multimodal representation learning, particularly in image-text retrieval and clustering. However, its efficacy is constrained by three key limitations: (1) text token truncation, (2) isolated image-text encoding, and (3) deficient compositionality due to bag-of-words behavior. While recent Multimodal Large Language Models (MLLMs) have demonstrated significant advances in generalized vision-language understanding, their potential for learning transferable multimodal representations remains underexplored.In this work, we present UniME (Universal Multimodal Embedding), a novel two-stage framework that leverages MLLMs to learn discriminative representations for diverse downstream tasks. In the first stage, we perform textual discriminative knowledge distillation from a powerful LLM-based teacher model to enhance the embedding capability of the MLLM\'s language component. In the second stage, we introduce hard negative enhanced instruction tuning to further advance discriminative representation learning. Specifically, we initially mitigate false negative contamination and then sample multiple hard negatives per instance within each batch, forcing the model to focus on challenging samples. This approach not only improves discriminative power but also enhances instruction-following ability in downstream tasks. We conduct extensive experiments on the MMEB benchmark and multiple retrieval tasks, including short and long caption retrieval and compositional retrieval. Results demonstrate that UniME achieves consistent performance improvement across all tasks, exhibiting superior discriminative and compositional capabilities.</p></description>
|
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.18425.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, Zaida Zhou</p><p><b>Upvotes:</b> 7</p><p><b>Summary:</b> We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.</p></description>
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17432</guid>
|
<guid isPermaLink="false">https://arxiv.org/abs/2504.18425</guid>
|
||||||
<pubDate>Thu, 24 Apr 2025 10:51:52 +0000</pubDate>
|
<pubDate>Fri, 25 Apr 2025 15:31:46 +0000</pubDate>
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generation</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.17502</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17502.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Aviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon, Michal Sokolik, Nitzan Bitton Guetta, Almog Gueta, Royi Rassin, Itay Laish, Dani Lischinski, Idan Szpektor</p><p><b>Upvotes:</b> 51</p><p><b>Summary:</b> Subject-driven text-to-image (T2I) generation aims to produce images that align with a given textual description, while preserving the visual identity from a referenced subject image. Despite its broad downstream applicability -- ranging from enhanced personalization in image generation to consistent character representation in video rendering -- progress in this field is limited by the lack of reliable automatic evaluation. Existing methods either assess only one aspect of the task (i.e., textual alignment or subject preservation), misalign with human judgments, or rely on costly API-based evaluation. To address this, we introduce RefVNLI, a cost-effective metric that evaluates both textual alignment and subject preservation in a single prediction. Trained on a large-scale dataset derived from video-reasoning benchmarks and image perturbations, RefVNLI outperforms or matches existing baselines across multiple benchmarks and subject categories (e.g., Animal, Object), achieving up to 6.4-point gains in textual alignment and 8.5-point gains in subject consistency. It also excels with lesser-known concepts, aligning with human preferences at over 87\% accuracy.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17502</guid>
|
|
||||||
<pubDate>Thu, 24 Apr 2025 12:44:51 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Interpretable non-linear dimensionality reduction using gaussian weighted linear transformation</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.17601</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17601.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Erik Bergh</p><p><b>Upvotes:</b> 2</p><p><b>Summary:</b> Dimensionality reduction techniques are fundamental for analyzing and visualizing high-dimensional data. With established methods like t-SNE and PCA presenting a trade-off between representational power and interpretability. This paper introduces a novel approach that bridges this gap by combining the interpretability of linear methods with the expressiveness of non-linear transformations. The proposed algorithm constructs a non-linear mapping between high-dimensional and low-dimensional spaces through a combination of linear transformations, each weighted by Gaussian functions. This architecture enables complex non-linear transformations while preserving the interpretability advantages of linear methods, as each transformation can be analyzed independently. The resulting model provides both powerful dimensionality reduction and transparent insights into the transformed space. Techniques for interpreting the learned transformations are presented, including methods for identifying suppressed dimensions and how space is expanded and contracted. These tools enable practitioners to understand how the algorithm preserves and modifies geometric relationships during dimensionality reduction. To ensure the practical utility of this algorithm, the creation of user-friendly software packages is emphasized, facilitating its adoption in both academia and industry.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17601</guid>
|
|
||||||
<pubDate>Thu, 24 Apr 2025 14:26:42 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>DiMeR: Disentangled Mesh Reconstruction Model</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.17670</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17670.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Lutao Jiang, Jiantao Lin, Kanghao Chen, Wenhang Ge, Xin Yang, Yifan Jiang, Yuanhuiyi Lyu, Xu Zheng, Yingcong Chen</p><p><b>Upvotes:</b> 17</p><p><b>Summary:</b> With the advent of large-scale 3D datasets, feed-forward 3D generative models, such as the Large Reconstruction Model (LRM), have gained significant attention and achieved remarkable success. However, we observe that RGB images often lead to conflicting training objectives and lack the necessary clarity for geometry reconstruction. In this paper, we revisit the inductive biases associated with mesh reconstruction and introduce DiMeR, a novel disentangled dual-stream feed-forward model for sparse-view mesh reconstruction. The key idea is to disentangle both the input and framework into geometry and texture parts, thereby reducing the training difficulty for each part according to the Principle of Occam's Razor. Given that normal maps are strictly consistent with geometry and accurately capture surface variations, we utilize normal maps as exclusive input for the geometry branch to reduce the complexity between the network's input and output. Moreover, we improve the mesh extraction algorithm to introduce 3D ground truth supervision. As for texture branch, we use RGB images as input to obtain the textured mesh. Overall, DiMeR demonstrates robust capabilities across various tasks, including sparse-view reconstruction, single-image-to-3D, and text-to-3D. Numerous experiments show that DiMeR significantly outperforms previous methods, achieving over 30% improvement in Chamfer Distance on the GSO and OmniObject3D dataset.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17670</guid>
|
|
||||||
<pubDate>Thu, 24 Apr 2025 15:39:20 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Step1X-Edit: A Practical Framework for General Image Editing</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.17761</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17761.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, Guopeng Li, Yuang Peng, Quan Sun, Jingwei Wu, Yan Cai, Zheng Ge, Ranchen Ming, Lei Xia, Xianfang Zeng, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Gang Yu, Daxin Jiang</p><p><b>Upvotes:</b> 71</p><p><b>Summary:</b> In recent years, image editing models have witnessed remarkable and rapid development. The recent unveiling of cutting-edge multimodal models such as GPT-4o and Gemini2 Flash has introduced highly promising image editing capabilities. These models demonstrate an impressive aptitude for fulfilling a vast majority of user-driven editing requirements, marking a significant advancement in the field of image manipulation. However, there is still a large gap between the open-source algorithm with these closed-source models. Thus, in this paper, we aim to release a state-of-the-art image editing model, called Step1X-Edit, which can provide comparable performance against the closed-source models like GPT-4o and Gemini2 Flash. More specifically, we adopt the Multimodal LLM to process the reference image and the user's editing instruction. A latent embedding has been extracted and integrated with a diffusion image decoder to obtain the target image. To train the model, we build a data generation pipeline to produce a high-quality dataset. For evaluation, we develop the GEdit-Bench, a novel benchmark rooted in real-world user instructions. Experimental results on GEdit-Bench demonstrate that Step1X-Edit outperforms existing open-source baselines by a substantial margin and approaches the performance of leading proprietary models, thereby making significant contributions to the field of image editing.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17761</guid>
|
|
||||||
<pubDate>Thu, 24 Apr 2025 17:25:12 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Dynamic Camera Poses and Where to Find Them</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.17788</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17788.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Chris Rockwell, Joseph Tung, Tsung-Yi Lin, Ming-Yu Liu, David F. Fouhey, Chen-Hsuan Lin</p><p><b>Upvotes:</b> 3</p><p><b>Summary:</b> Annotating camera poses on dynamic Internet videos at scale is critical for advancing fields like realistic video generation and simulation. However, collecting such a dataset is difficult, as most Internet videos are unsuitable for pose estimation. Furthermore, annotating dynamic Internet videos present significant challenges even for state-of-theart methods. In this paper, we introduce DynPose-100K, a large-scale dataset of dynamic Internet videos annotated with camera poses. Our collection pipeline addresses filtering using a carefully combined set of task-specific and generalist models. For pose estimation, we combine the latest techniques of point tracking, dynamic masking, and structure-from-motion to achieve improvements over the state-of-the-art approaches. Our analysis and experiments demonstrate that DynPose-100K is both large-scale and diverse across several key attributes, opening up avenues for advancements in various downstream applications.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17788</guid>
|
|
||||||
<pubDate>Thu, 24 Apr 2025 17:59:56 +0000</pubDate>
|
|
||||||
</item>
|
|
||||||
<item>
|
|
||||||
<title>Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models</title>
|
|
||||||
<link>https://arxiv.org/abs/2504.17789</link>
|
|
||||||
<description><p><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2504.17789.png" alt="Paper thumbnail" style="max-width: 300px; height: auto;" /></p><p><b>Authors:</b> Xu Ma, Peize Sun, Haoyu Ma, Hao Tang, Chih-Yao Ma, Jialiang Wang, Kunpeng Li, Xiaoliang Dai, Yujun Shi, Xuan Ju, Yushi Hu, Artsiom Sanakoyeu, Felix Juefei-Xu, Ji Hou, Junjiao Tian, Tao Xu, Tingbo Hou, Yen-Cheng Liu, Zecheng He, Zijian He, Matt Feiszli, Peizhao Zhang, Peter Vajda, Sam Tsai, Yun Fu</p><p><b>Upvotes:</b> 15</p><p><b>Summary:</b> Autoregressive (AR) models, long dominant in language generation, are increasingly applied to image synthesis but are often considered less competitive than Diffusion-based models. A primary limitation is the substantial number of image tokens required for AR models, which constrains both training and inference efficiency, as well as image resolution. To address this, we present Token-Shuffle, a novel yet simple method that reduces the number of image tokens in Transformer. Our key insight is the dimensional redundancy of visual vocabularies in Multimodal Large Language Models (MLLMs), where low-dimensional visual codes from visual encoder are directly mapped to high-dimensional language vocabularies. Leveraging this, we consider two key operations: token-shuffle, which merges spatially local tokens along channel dimension to decrease the input token number, and token-unshuffle, which untangles the inferred tokens after Transformer blocks to restore the spatial arrangement for output. Jointly training with textual prompts, our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis in a unified next-token prediction way while maintaining efficient training and inference. For the first time, we push the boundary of AR text-to-image generation to a resolution of 2048x2048 with gratifying generation performance. In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15. Exhaustive large-scale human evaluations also demonstrate our prominent image generation ability in terms of text-alignment, visual flaw, and visual appearance. We hope that Token-Shuffle can serve as a foundational design for efficient high-resolution image generation within MLLMs.</p></description>
|
|
||||||
<guid isPermaLink="false">https://arxiv.org/abs/2504.17789</guid>
|
|
||||||
<pubDate>Thu, 24 Apr 2025 17:59:56 +0000</pubDate>
|
|
||||||
</item>
|
</item>
|
||||||
</channel>
|
</channel>
|
||||||
</rss>
|
</rss>
|
||||||
|
|||||||
Reference in New Issue
Block a user