GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction

Samrudhdhi B. Rangrej, Kevin J. Liang, Tal Hassner, James J. Clark

نتاج البحث: فصل من :كتاب / تقرير / مؤتمرمنشور من مؤتمرمراجعة النظراء

ملخص

Many online action prediction models observe complete frames to locate and attend to informative subregions in the frames called glimpses and recognize an ongoing action based on global and local information. However, in applications with constrained resources, an agent may not be able to observe the complete frame, yet must still locate useful glimpses to predict an incomplete action based on local information only. In this paper, we develop Glimpse Transformers (GliTr), which observe only narrow glimpses at all times, thus predicting an ongoing action and the following most informative glimpse location based on the partial spatiotemporal information collected so far. In the absence of a ground truth for the optimal glimpse locations for action recognition, we train GliTr using a novel spatiotemporal consistency objective: We require GliTr to attend to the glimpses with features similar to the corresponding complete frames (i.e. spatial consistency) and the resultant class logits at time t equivalent to the ones predicted using whole frames up to t (i.e. temporal consistency). Inclusion of our proposed consistency objective yields ∼ 10% higher accuracy on the Something-Something-v2 (SSv2) dataset than the baseline cross-entropy objective. Overall, despite observing only ∼ 33% of the total area per frame, GliTr achieves 53.02% and 93.91% accuracy on the SSv2 and Jester datasets, respectively.

اللغة الأصليةالإنجليزيّة
عنوان منشور المضيفProceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023
ناشرInstitute of Electrical and Electronics Engineers Inc.
الصفحات3402-3412
عدد الصفحات11
رقم المعيار الدولي للكتب (الإلكتروني)9781665493468
المعرِّفات الرقمية للأشياء
حالة النشرنُشِر - 2023
الحدث23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023 - Waikoloa, الولايات المتّحدة
المدة: ٣ يناير ٢٠٢٣٧ يناير ٢٠٢٣

سلسلة المنشورات

الاسمProceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023

!!Conference

!!Conference23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023
الدولة/الإقليمالولايات المتّحدة
المدينةWaikoloa
المدة٣/٠١/٢٣٧/٠١/٢٣

ملاحظة ببليوغرافية

Publisher Copyright:
© 2023 IEEE.

بصمة

أدرس بدقة موضوعات البحث “GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction'. فهما يشكلان معًا بصمة فريدة.

قم بذكر هذا