TY - JOUR
T1 - DeepFake Detection Based on Discrepancies Between Faces and their Context
AU - Nirkin, Yuval
AU - Wolf, Lior
AU - Keller, Yosi
AU - Hassner, Tal
N1 - Publisher Copyright:
IEEE
PY - 2021
Y1 - 2021
N2 - We propose a method for detecting face swapping and other identity manipulations in single images. Face swapping methods, such as DeepFake, manipulate the face region, aiming to adjust the face to the appearance of its context, while leaving the context unchanged. We show that this modus operandi produces discrepancies between the two regions (e.g., Fig. 1). These discrepancies offer exploitable telltale signs of manipulation. Our approach involves two networks: (i) a face identification network that considers the face region bounded by a tight semantic segmentation, and (ii) a context recognition network that considers the face context (e.g., hair, ears, neck). We describe a method which uses the recognition signals from our two networks to detect such discrepancies, providing a complementary detection signal that improves conventional real vs. fake classifiers commonly used for detecting fake images. Our method achieves state of the art results on the FaceForensics++, Celeb-DF-v2, and DFDC benchmarks for face manipulation detection, and even generalizes to detect fakes produced by unseen methods.
AB - We propose a method for detecting face swapping and other identity manipulations in single images. Face swapping methods, such as DeepFake, manipulate the face region, aiming to adjust the face to the appearance of its context, while leaving the context unchanged. We show that this modus operandi produces discrepancies between the two regions (e.g., Fig. 1). These discrepancies offer exploitable telltale signs of manipulation. Our approach involves two networks: (i) a face identification network that considers the face region bounded by a tight semantic segmentation, and (ii) a context recognition network that considers the face context (e.g., hair, ears, neck). We describe a method which uses the recognition signals from our two networks to detect such discrepancies, providing a complementary detection signal that improves conventional real vs. fake classifiers commonly used for detecting fake images. Our method achieves state of the art results on the FaceForensics++, Celeb-DF-v2, and DFDC benchmarks for face manipulation detection, and even generalizes to detect fakes produced by unseen methods.
KW - Benchmark testing
KW - Deep Fake
KW - Deep Learning
KW - Face Swapping
KW - Faces
KW - Fake image Detection
KW - Hair
KW - Image Forensics
KW - Information integrity
KW - Neck
KW - Training
KW - Videos
UR - http://www.scopus.com/inward/record.url?scp=85110837771&partnerID=8YFLogxK
U2 - 10.1109/TPAMI.2021.3093446
DO - 10.1109/TPAMI.2021.3093446
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???
C2 - 34185639
AN - SCOPUS:85110837771
SN - 0162-8828
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
JF - IEEE Transactions on Pattern Analysis and Machine Intelligence
ER -