Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking

Ronen Tamari, Kyle Richardson, Noam Kahlon, Aviad Sar-Shalom, Nelson F. Liu, Reut Tsarfaty, Dafna Shahaf

פרסום מחקרי: פרק בספר / בדוח / בכנספרסום בספר כנסביקורת עמיתים

תקציר

While neural language models often perform surprisingly well on natural language understanding (NLU) tasks, their strengths and limitations remain poorly understood. Controlled synthetic tasks are thus an increasingly important resource for diagnosing model behavior. In this work we focus on story understanding, a core competency for NLU systems. However, the main synthetic resource for story understanding, the bAbI benchmark, lacks such a systematic mechanism for controllable task generation. We develop Dyna-bAbI, a dynamic framework providing fine-grained control over task generation in bAbI. We demonstrate our ideas by constructing three new tasks requiring compositional generalization, an important evaluation setting absent from the original benchmark. We tested both special-purpose models developed for bAbI as well as state-of-the-art pre-trained methods, and found that while both approaches solve the original tasks (>99% accuracy), neither approach succeeded in the compositional generalization setting, indicating the limitations of the original training data. We explored ways to augment the original data, and found that though diversifying training data was far more useful than simply increasing dataset size, it was still insufficient for driving robust compositional generalization (with <70% accuracy for complex compositions). Our results underscore the importance of highly controllable task generators for creating robust NLU systems through a virtuous cycle of model and data development.

שפה מקוריתאנגלית
כותר פרסום המארח*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference
עורכיםVivi Nastase, Ellie Pavlick, Mohammad Taher Pilehvar, Jose Camacho-Collados, Alessandro Raganato
מוציא לאורAssociation for Computational Linguistics (ACL)
עמודים101-122
מספר עמודים22
מסת"ב (אלקטרוני)9781955917988
סטטוס פרסוםפורסם - 2022
פורסם באופן חיצוניכן
אירוע11th Joint Conference on Lexical and Computational Semantics, *SEM 2022 - Seattle, ארצות הברית
משך הזמן: 14 יולי 202215 יולי 2022

סדרות פרסומים

שם*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference

כנס

כנס11th Joint Conference on Lexical and Computational Semantics, *SEM 2022
מדינה/אזורארצות הברית
עירSeattle
תקופה14/07/2215/07/22

הערה ביבליוגרפית

Publisher Copyright:
© 2022 Association for Computational Linguistics.

טביעת אצבע

להלן מוצגים תחומי המחקר של הפרסום 'Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking'. יחד הם יוצרים טביעת אצבע ייחודית.

פורמט ציטוט ביבליוגרפי