תקציר
Machine learning models have been found to learn shortcuts - unintended decision rules that are unable to generalize - undermining models' reliability. Previous works address this problem under the tenuous assumption that only a single shortcut exists in the training data. Real-world images are rife with multiple visual cues from background to texture. Key to advancing the reliability of vision systems is understanding whether existing methods can overcome multiple shortcuts or struggle in a Whac-A-Mole game, i.e., where mitigating one shortcut amplifies reliance on others. To address this shortcoming, we propose two benchmarks: 1) UrbanCars, a dataset with precisely controlled spurious cues, and 2) ImageNet-W, an evaluation set based on ImageNet for watermark, a shortcut we discovered affects nearly every modern vision model. Along with texture and background, ImageNet-W allows us to study multiple shortcuts emerging from training on natural images. We find computer vision models, including large foundation models - regardless of training set, architecture, and supervision - struggle when multiple shortcuts are present. Even methods explicitly designed to combat shortcuts struggle in a Whac-A-Mole dilemma. To tackle this challenge, we propose Last Layer Ensemble, a simple-yet-effective method to mitigate multiple shortcuts without Whac-A-Mole behavior. Our results surface multi-shortcut mitigation as an overlooked challenge critical to advancing the reliability of vision systems. The datasets and code are released: https://github.com/facebookresearch/Whac-A-Mole.
שפה מקורית | אנגלית |
---|---|
כותר פרסום המארח | Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 |
מוציא לאור | IEEE Computer Society |
עמודים | 20071-20082 |
מספר עמודים | 12 |
מסת"ב (אלקטרוני) | 9798350301298 |
מזהי עצם דיגיטלי (DOIs) | |
סטטוס פרסום | פורסם - 2023 |
אירוע | 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 - Vancouver, קנדה משך הזמן: 18 יוני 2023 → 22 יוני 2023 |
סדרות פרסומים
שם | Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition |
---|---|
כרך | 2023-June |
ISSN (מודפס) | 1063-6919 |
כנס
כנס | 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 |
---|---|
מדינה/אזור | קנדה |
עיר | Vancouver |
תקופה | 18/06/23 → 22/06/23 |
הערה ביבליוגרפית
Publisher Copyright:© 2023 IEEE.