In the age of information overload, it is more important than ever to discern fact from fiction. From the internet to traditional media, we are constantly confronted with a deluge of information, much of which comes from politicians and other public figures who wield significant influence. In this paper, we introduce HeTrue: a new, publicly available dataset for evaluating the credibility of statements made by Israeli public figures and politicians. This dataset consists of 1021 statements, manually annotated by Israeli professional journalists, for their credibility status. Using this corpus, we set out to assess whether the credibility of statements can be predicted based on the text alone. To establish a baseline, we compare text-only methods with others using additional data like metadata, context, and evidence. Furthermore, we develop several credibility assessment models, including a feature-based model that utilizes linguistic features, and state-of-the-art transformer-based models with contextualized embeddings from a pre-trained encoder. Empirical results demonstrate improved performance when models integrate statement and context, outperforming those relying on the statement text alone. Our best model, which also integrates evidence, achieves a 48.3 F1 Score, suggesting that HeTrue is a challenging benchmark, calling for further work on this task.
|Title of host publication
|Findings of the Association for Computational Linguistics
|Subtitle of host publication
|Association for Computational Linguistics (ACL)
|Number of pages
|Published - 2023
|2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Singapore, Singapore
Duration: 6 Dec 2023 → 10 Dec 2023
|Findings of the Association for Computational Linguistics: EMNLP 2023
|2023 Findings of the Association for Computational Linguistics: EMNLP 2023
|6/12/23 → 10/12/23
Bibliographical notePublisher Copyright:
© 2023 Association for Computational Linguistics.