ZEST: Zero-shot learning from text descriptions using textual similarity and visual summarization

Tzuf Paz-Argaman, Yuval Atzmon, Gal Chechik, Reut Tsarfaty

פרסום מחקרי: פרק בספר / בדוח / בכנספרסום בספר כנסביקורת עמיתים

תקציר

We study the problem of recognizing visual entities from the textual descriptions of their classes. Specifically, given birds’ images with free-text descriptions of their species, we learn to classify images of previously-unseen species based on specie descriptions. This setup has been studied in the vision community under the name zero-shot learning from text, focusing on learning to transfer knowledge about visual aspects of birds from seen classes to previously-unseen ones. Here, we suggest focusing on the textual description and distilling from the description the most relevant information to effectively match visual features to the parts of the text that discuss them. Specifically, (1) we propose to leverage the similarity between species, reflected in the similarity between text descriptions of the species. (2) we derive visual summaries of the texts, i.e., extractive summaries that focus on the visual features that tend to be reflected in images. We propose a simple attention-based model augmented with the similarity and visual summaries components. Our empirical results consistently and significantly outperform the state-of-the-art on the largest benchmarks for text-based zero-shot learning, illustrating the critical importance of texts for zero-shot image-recognition.

שפה מקוריתאנגלית
כותר פרסום המארחFindings of the Association for Computational Linguistics Findings of ACL
כותר משנה של פרסום המארחEMNLP 2020
מוציא לאורAssociation for Computational Linguistics (ACL)
עמודים569-579
מספר עמודים11
מסת"ב (אלקטרוני)9781952148903
סטטוס פרסוםפורסם - 2020
פורסם באופן חיצוניכן
אירועFindings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020 - Virtual, Online
משך הזמן: 16 נוב׳ 202020 נוב׳ 2020

סדרות פרסומים

שםFindings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020

כנס

כנסFindings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020
עירVirtual, Online
תקופה16/11/2020/11/20

הערה ביבליוגרפית

Publisher Copyright:
© 2020 Association for Computational Linguistics

טביעת אצבע

להלן מוצגים תחומי המחקר של הפרסום 'ZEST: Zero-shot learning from text descriptions using textual similarity and visual summarization'. יחד הם יוצרים טביעת אצבע ייחודית.

פורמט ציטוט ביבליוגרפי