Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking

Ronen Tamari, Kyle Richardson, Noam Kahlon, Aviad Sar-Shalom, Nelson F. Liu, Reut Tsarfaty, Dafna Shahaf

نتاج البحث: فصل من :كتاب / تقرير / مؤتمرمنشور من مؤتمرمراجعة النظراء

ملخص

While neural language models often perform surprisingly well on natural language understanding (NLU) tasks, their strengths and limitations remain poorly understood. Controlled synthetic tasks are thus an increasingly important resource for diagnosing model behavior. In this work we focus on story understanding, a core competency for NLU systems. However, the main synthetic resource for story understanding, the bAbI benchmark, lacks such a systematic mechanism for controllable task generation. We develop Dyna-bAbI, a dynamic framework providing fine-grained control over task generation in bAbI. We demonstrate our ideas by constructing three new tasks requiring compositional generalization, an important evaluation setting absent from the original benchmark. We tested both special-purpose models developed for bAbI as well as state-of-the-art pre-trained methods, and found that while both approaches solve the original tasks (>99% accuracy), neither approach succeeded in the compositional generalization setting, indicating the limitations of the original training data. We explored ways to augment the original data, and found that though diversifying training data was far more useful than simply increasing dataset size, it was still insufficient for driving robust compositional generalization (with <70% accuracy for complex compositions). Our results underscore the importance of highly controllable task generators for creating robust NLU systems through a virtuous cycle of model and data development.

اللغة الأصليةالإنجليزيّة
عنوان منشور المضيف*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference
المحررونVivi Nastase, Ellie Pavlick, Mohammad Taher Pilehvar, Jose Camacho-Collados, Alessandro Raganato
ناشرAssociation for Computational Linguistics (ACL)
الصفحات101-122
عدد الصفحات22
رقم المعيار الدولي للكتب (الإلكتروني)9781955917988
حالة النشرنُشِر - 2022
منشور خارجيًانعم
الحدث11th Joint Conference on Lexical and Computational Semantics, *SEM 2022 - Seattle, الولايات المتّحدة
المدة: ١٤ يوليو ٢٠٢٢١٥ يوليو ٢٠٢٢

سلسلة المنشورات

الاسم*SEM 2022 - 11th Joint Conference on Lexical and Computational Semantics, Proceedings of the Conference

!!Conference

!!Conference11th Joint Conference on Lexical and Computational Semantics, *SEM 2022
الدولة/الإقليمالولايات المتّحدة
المدينةSeattle
المدة١٤/٠٧/٢٢١٥/٠٧/٢٢

ملاحظة ببليوغرافية

Publisher Copyright:
© 2022 Association for Computational Linguistics.

بصمة

أدرس بدقة موضوعات البحث “Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking'. فهما يشكلان معًا بصمة فريدة.

قم بذكر هذا