Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild

Iacopo Masi; Feng Ju Chang; Jongmoo Choi; Shai Harel; Jungyeon Kim; Kanggeon Kim; Jatuporn Leksut; Stephen Rawls; Yue Wu; Tal Hassner; Wael AbdAlmageed; Gerard Medioni; Louis Philippe Morency; Prem Natarajan; Ram Nevatia

doi:10.1109/TPAMI.2018.2792452

Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild

Iacopo Masi, Feng Ju Chang, Jongmoo Choi, Shai Harel, Jungyeon Kim, Kanggeon Kim, Jatuporn Leksut, Stephen Rawls, Yue Wu, Tal Hassner, Wael AbdAlmageed, Gerard Medioni, Louis Philippe Morency, Prem Natarajan, Ram Nevatia

Computer Science discipline

Research output: Contribution to journal › Article › peer-review

Abstract

We propose a method designed to push the frontiers of unconstrained face recognition in the wild with an emphasis on extreme out-of-plane pose variations. Existing methods either expect a single model to learn pose invariance by training on massive amounts of data or else normalize images by aligning faces to a single frontal pose. Contrary to these, our method is designed to explicitly tackle pose variations. Our proposed Pose-Aware Models (PAM) process a face image using several pose-specific, deep convolutional neural networks (CNN). 3D rendering is used to synthesize multiple face poses from input images to both train these models and to provide additional robustness to pose variations at test time. Our paper presents an extensive analysis of the IARPA Janus Benchmark A (IJB-A), evaluating the effects that landmark detection accuracy, CNN layer selection, and pose model selection all have on the performance of the recognition pipeline. It further provides comparative evaluations on IJB-A and the PIPA dataset. These tests show that our approach outperforms existing methods, even surprisingly matching the accuracy of methods that were specifically fine-tuned to the target dataset. Parts of this work previously appeared in [1] and [2].

Original language	English
Article number	8255649
Pages (from-to)	379-393
Number of pages	15
Journal	IEEE Transactions on Pattern Analysis and Machine Intelligence
Volume	41
Issue number	2
DOIs	https://doi.org/10.1109/TPAMI.2018.2792452
State	Published - 1 Feb 2019

Bibliographical note

Funding Information:
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA 2014-14071600010. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purpose notwithstanding any copyright annotation thereon. Moreover, we gratefully acknowledge USC HPC for hyper-computing and the support of NVIDIA Corporation for their donation of an NVIDIA Titan X. I.M. is the lead author; F.C., J.C., S.H., J.K., K.K., J.L., S.R., and Y.W. in alphabetical order; T.H. is the lead senior author; all remaining senior authors appear next in alphabetical order.

Publisher Copyright:
© 1979-2012 IEEE.

Keywords

CNN
Face recognition
pose-aware

Access to Document

10.1109/TPAMI.2018.2792452

Cite this

Masi, I., Chang, F. J., Choi, J., Harel, S., Kim, J., Kim, K., Leksut, J., Rawls, S., Wu, Y., Hassner, T., AbdAlmageed, W., Medioni, G., Morency, L. P., Natarajan, P., & Nevatia, R. (2019). Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 379-393. Article 8255649. https://doi.org/10.1109/TPAMI.2018.2792452

@article{7b52116ce9694217aa1c2aef301aaf35,

title = "Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild",

abstract = "We propose a method designed to push the frontiers of unconstrained face recognition in the wild with an emphasis on extreme out-of-plane pose variations. Existing methods either expect a single model to learn pose invariance by training on massive amounts of data or else normalize images by aligning faces to a single frontal pose. Contrary to these, our method is designed to explicitly tackle pose variations. Our proposed Pose-Aware Models (PAM) process a face image using several pose-specific, deep convolutional neural networks (CNN). 3D rendering is used to synthesize multiple face poses from input images to both train these models and to provide additional robustness to pose variations at test time. Our paper presents an extensive analysis of the IARPA Janus Benchmark A (IJB-A), evaluating the effects that landmark detection accuracy, CNN layer selection, and pose model selection all have on the performance of the recognition pipeline. It further provides comparative evaluations on IJB-A and the PIPA dataset. These tests show that our approach outperforms existing methods, even surprisingly matching the accuracy of methods that were specifically fine-tuned to the target dataset. Parts of this work previously appeared in [1] and [2].",

keywords = "CNN, Face recognition, pose-aware",

author = "Iacopo Masi and Chang, {Feng Ju} and Jongmoo Choi and Shai Harel and Jungyeon Kim and Kanggeon Kim and Jatuporn Leksut and Stephen Rawls and Yue Wu and Tal Hassner and Wael AbdAlmageed and Gerard Medioni and Morency, {Louis Philippe} and Prem Natarajan and Ram Nevatia",

note = "Publisher Copyright: {\textcopyright} 1979-2012 IEEE.",

year = "2019",

month = feb,

day = "1",

doi = "10.1109/TPAMI.2018.2792452",

language = "אנגלית",

volume = "41",

pages = "379--393",

journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence",

issn = "0162-8828",

publisher = "IEEE Computer Society",

number = "2",

}

Masi, I, Chang, FJ, Choi, J, Harel, S, Kim, J, Kim, K, Leksut, J, Rawls, S, Wu, Y, Hassner, T, AbdAlmageed, W, Medioni, G, Morency, LP, Natarajan, P & Nevatia, R 2019, 'Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild', IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, 8255649, pp. 379-393. https://doi.org/10.1109/TPAMI.2018.2792452

TY - JOUR

T1 - Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild

AU - Masi, Iacopo

AU - Chang, Feng Ju

AU - Choi, Jongmoo

AU - Harel, Shai

AU - Kim, Jungyeon

AU - Kim, Kanggeon

AU - Leksut, Jatuporn

AU - Rawls, Stephen

AU - Wu, Yue

AU - Hassner, Tal

AU - AbdAlmageed, Wael

AU - Medioni, Gerard

AU - Morency, Louis Philippe

AU - Natarajan, Prem

AU - Nevatia, Ram

PY - 2019/2/1

Y1 - 2019/2/1

N2 - We propose a method designed to push the frontiers of unconstrained face recognition in the wild with an emphasis on extreme out-of-plane pose variations. Existing methods either expect a single model to learn pose invariance by training on massive amounts of data or else normalize images by aligning faces to a single frontal pose. Contrary to these, our method is designed to explicitly tackle pose variations. Our proposed Pose-Aware Models (PAM) process a face image using several pose-specific, deep convolutional neural networks (CNN). 3D rendering is used to synthesize multiple face poses from input images to both train these models and to provide additional robustness to pose variations at test time. Our paper presents an extensive analysis of the IARPA Janus Benchmark A (IJB-A), evaluating the effects that landmark detection accuracy, CNN layer selection, and pose model selection all have on the performance of the recognition pipeline. It further provides comparative evaluations on IJB-A and the PIPA dataset. These tests show that our approach outperforms existing methods, even surprisingly matching the accuracy of methods that were specifically fine-tuned to the target dataset. Parts of this work previously appeared in [1] and [2].

AB - We propose a method designed to push the frontiers of unconstrained face recognition in the wild with an emphasis on extreme out-of-plane pose variations. Existing methods either expect a single model to learn pose invariance by training on massive amounts of data or else normalize images by aligning faces to a single frontal pose. Contrary to these, our method is designed to explicitly tackle pose variations. Our proposed Pose-Aware Models (PAM) process a face image using several pose-specific, deep convolutional neural networks (CNN). 3D rendering is used to synthesize multiple face poses from input images to both train these models and to provide additional robustness to pose variations at test time. Our paper presents an extensive analysis of the IARPA Janus Benchmark A (IJB-A), evaluating the effects that landmark detection accuracy, CNN layer selection, and pose model selection all have on the performance of the recognition pipeline. It further provides comparative evaluations on IJB-A and the PIPA dataset. These tests show that our approach outperforms existing methods, even surprisingly matching the accuracy of methods that were specifically fine-tuned to the target dataset. Parts of this work previously appeared in [1] and [2].

KW - CNN

KW - Face recognition

KW - pose-aware

UR - http://www.scopus.com/inward/record.url?scp=85041197472&partnerID=8YFLogxK

U2 - 10.1109/TPAMI.2018.2792452

DO - 10.1109/TPAMI.2018.2792452

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

C2 - 29994497

AN - SCOPUS:85041197472

SN - 0162-8828

VL - 41

SP - 379

EP - 393

JO - IEEE Transactions on Pattern Analysis and Machine Intelligence

JF - IEEE Transactions on Pattern Analysis and Machine Intelligence

IS - 2

M1 - 8255649

ER -

Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this