A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others

Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, Mark Ibrahim

פרסום מחקרי: פרק בספר / בדוח / בכנספרסום בספר כנסביקורת עמיתים

תקציר

Machine learning models have been found to learn shortcuts - unintended decision rules that are unable to generalize - undermining models' reliability. Previous works address this problem under the tenuous assumption that only a single shortcut exists in the training data. Real-world images are rife with multiple visual cues from background to texture. Key to advancing the reliability of vision systems is understanding whether existing methods can overcome multiple shortcuts or struggle in a Whac-A-Mole game, i.e., where mitigating one shortcut amplifies reliance on others. To address this shortcoming, we propose two benchmarks: 1) UrbanCars, a dataset with precisely controlled spurious cues, and 2) ImageNet-W, an evaluation set based on ImageNet for watermark, a shortcut we discovered affects nearly every modern vision model. Along with texture and background, ImageNet-W allows us to study multiple shortcuts emerging from training on natural images. We find computer vision models, including large foundation models - regardless of training set, architecture, and supervision - struggle when multiple shortcuts are present. Even methods explicitly designed to combat shortcuts struggle in a Whac-A-Mole dilemma. To tackle this challenge, we propose Last Layer Ensemble, a simple-yet-effective method to mitigate multiple shortcuts without Whac-A-Mole behavior. Our results surface multi-shortcut mitigation as an overlooked challenge critical to advancing the reliability of vision systems. The datasets and code are released: https://github.com/facebookresearch/Whac-A-Mole.

שפה מקוריתאנגלית
כותר פרסום המארחProceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
מוציא לאורIEEE Computer Society
עמודים20071-20082
מספר עמודים12
מסת"ב (אלקטרוני)9798350301298
מזהי עצם דיגיטלי (DOIs)
סטטוס פרסוםפורסם - 2023
אירוע2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 - Vancouver, קנדה
משך הזמן: 18 יוני 202322 יוני 2023

סדרות פרסומים

שםProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
כרך2023-June
ISSN (מודפס)1063-6919

כנס

כנס2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
מדינה/אזורקנדה
עירVancouver
תקופה18/06/2322/06/23

הערה ביבליוגרפית

Publisher Copyright:
© 2023 IEEE.

טביעת אצבע

להלן מוצגים תחומי המחקר של הפרסום 'A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others'. יחד הם יוצרים טביעת אצבע ייחודית.

פורמט ציטוט ביבליוגרפי