Facial Landmark Detection with Tweaked Convolutional Neural Networks

Yue Wu, Tal Hassner, Kanggeon Kim, Gerard Medioni, Prem Natarajan

Research output: Contribution to journalArticlepeer-review


This paper concerns the problem of facial landmark detection. We provide a unique new analysis of the features produced at intermediate layers of a convolutional neural network (CNN) trained to regress facial landmark coordinates. This analysis shows that while being processed by the CNN, face images can be partitioned in an unsupervised manner into subsets containing faces in similar poses (i.e., 3D views) and facial properties (e.g., presence or absence of eye-wear). Based on this finding, we describe a novel CNN architecture, specialized to regress the facial landmark coordinates of faces in specific poses and appearances. To address the shortage of training data, particularly in extreme profile poses, we additionally present data augmentation techniques designed to provide sufficient training examples for each of these specialized sub-networks. The proposed Tweaked CNN (TCNN) architecture is shown to outperform existing landmark detection methods in an extensive battery of tests on the AFW, ALFW, and 300W benchmarks. Finally, to promote reproducibility of our results, we make code and trained models publicly available through our project webpage.

Original languageEnglish
Article number8239860
Pages (from-to)3067-3074
Number of pages8
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number12
StatePublished - 1 Dec 2018
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 1979-2012 IEEE.


  • Face and gesture recognition
  • Neural nets


Dive into the research topics of 'Facial Landmark Detection with Tweaked Convolutional Neural Networks'. Together they form a unique fingerprint.

Cite this