תקציר
Coherence is a linguistic term that refers to the relations between small textual units (sentences, propositions), which make the text logically consistent and meaningful to the reader. With the advances of generative foundational models in NLP, there is a pressing need to automatically assess the human-perceived coherence of automatically generated texts. Up until now, little work has been done on explicitly assessing the coherence of generated texts and analyzing the factors contributing to (in)coherence. Previous work on the topic used other tasks, e.g., sentence reordering, as proxies of coherence, rather than approaching coherence detection heads on. In this paper, we introduce COHESENTIA, a novel benchmark of human-perceived coherence of automatically generated texts. Our annotation protocol reflects two perspectives; one is global, assigning a single coherence score, and the other is incremental, scoring sentence by sentence. The incremental method produces an (in)coherence score for each text fragment and also pinpoints reasons for incoherence at that point. Our benchmark contains 500 automatically-generated and human-annotated paragraphs, each annotated in both methods, by multiple raters. Our analysis shows that the inter-annotator agreement in the incremental mode is higher than in the holistic alternative, and our experiments show that standard LMs fine-tuned for coherence detection show varied performance on the different factors contributing to (in)coherence. All in all, these models yield unsatisfactory performance, emphasizing the need for developing more reliable methods for coherence assessment.
שפה מקורית | אנגלית |
---|---|
כותר פרסום המארח | EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings |
עורכים | Houda Bouamor, Juan Pino, Kalika Bali |
מוציא לאור | Association for Computational Linguistics (ACL) |
עמודים | 5328-5343 |
מספר עמודים | 16 |
מסת"ב (אלקטרוני) | 9798891760608 |
סטטוס פרסום | פורסם - 2023 |
פורסם באופן חיצוני | כן |
אירוע | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - Hybrid, Singapore, סינגפור משך הזמן: 6 דצמ׳ 2023 → 10 דצמ׳ 2023 |
סדרות פרסומים
שם | EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings |
---|
כנס
כנס | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 |
---|---|
מדינה/אזור | סינגפור |
עיר | Hybrid, Singapore |
תקופה | 6/12/23 → 10/12/23 |
הערה ביבליוגרפית
Publisher Copyright:© 2023 Association for Computational Linguistics.