Abstract
Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to 4% improvement in POPE F1 scores and up to 36% reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms for improved multimodal performance.
Original language | English |
---|---|
Title of host publication | 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Proceedings of the Conference |
Editors | Lun-Wei Ku, Andre Martins, Vivek Srikumar |
Publisher | Association for Computational Linguistics (ACL) |
Pages | 6008-6022 |
Number of pages | 15 |
ISBN (Electronic) | 9798891760998 |
State | Published - 2024 |
Externally published | Yes |
Event | Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Hybrid, Bangkok, Thailand Duration: 11 Aug 2024 → 16 Aug 2024 |
Publication series
Name | Proceedings of the Annual Meeting of the Association for Computational Linguistics |
---|---|
ISSN (Print) | 0736-587X |
Conference
Conference | Findings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 |
---|---|
Country/Territory | Thailand |
City | Hybrid, Bangkok |
Period | 11/08/24 → 16/08/24 |
Bibliographical note
Publisher Copyright:© 2024 Association for Computational Linguistics.