Center for Identification Technology Research (CITeR): 2020 Year in Review

2020 | The Year in Review – SCIENCE OF BIOMETRICS – NOVEL BIOMETRICS – MULTISPECTRAL BIOMETRICS – BEHAVIORAL BIOMETRICS – TRUST & PRIVACY – VIDEO ANALYTICS – CYBERSECURITY – MOBILE & COMPUTING – FUSION & PERFORMANCE

A NATIONAL SCIENCE FOUNDATION INDUSTRY/UNIVERSITY COOPERATIVE RESEARCH CENTER

A YEAR IN REVIEW

With our interdisciplinary group of faculty, researchers and students, CITeR is the only National Science Foundation (NSF) Industry/University Cooperative Research Center (I/UCRC) focusing on serving its affiliates in the rapidly growing areas of identity and biometrics.

CITeR Affiliates ACV Auctions Army Futures Command — Combat Capabilities Development Command — Armaments Center (CCDC — Armaments) Athena Sciences

38

master’s and PhD students currently engaged in CITeR research

AWARE Inc. Defence Research and Development Canada (DRDC) Defense Forensic Science Center (DFSC) Department of Defense — Defense Forensics and Biometrics Agency (DFBA)

28

undergraduates currently engaged in CITeR research

Department of Homeland Security — Office of Biometric Identity Management (OBIM) Department of Homeland Security — Science and Technology Directorate (S&T) Federal Bureau of Investigation (FBI)

23

faculty engaged in research and teaching biometrics

HID Global IDEMIA InCadence Strategic Solutions Northrop Grumman Corporation Precise Biometrics

16

webinars featuring results from recently completed CITeR projects

c i t e r. c l a r k s o n . e d u

Qualcomm Incorporated SICPA Veridium

DIRECTOR’S MESSAGE

Message From the Director In a year filled with uncertainty, biometric recognition has also had its share of challenges and scrutiny. A continued focus on bias in biometrics, threats due to deepfakes and morphing, and public First video in the animated educational video series. confusion on the diverse applications of biometric recognition technology were some of the top news stories of 2020. CITeR continues to address the challenges in these areas with its internationally recognized researchers and diverse stakeholders from industry and government.

Bias in biometrics as a technologic problem refers to differences in performance for different demographics, such as ethnicity, age and gender.

(Other forms of bias are societal, where solutions need to go beyond the technical.) CITeR projects are looking specifically at metrics to assess differential performance, as well as image based methods to extract skin tone, which can be used for bias mitigation when training with unlabeled datasets.

Deepfake generation technology, with its surprising realism, continues to improve, and the cat-and-mouse game of threats and threat mitigation has begun. Face morphing is a unique biometric threat related to deepfakes where two individuals’ photos are fused to create a morphed photo to match both. CITeR is addressing these through extensive research in both the creation of datasets of Mia and Sofia discuss biometrics in the airport. sophisticated face morphs, as well as the development of methods to detect and reject face-morphed images.

As researchers, our hope is that increased public attention fosters the conversation that leads to developing well-informed laws and policies around biometric technology. Furthering the understanding of biometric recognition is required now more than ever; to that effort, CITeR is developing an animated educational video series that covers the fundamentals of biometric recognition and addresses some of the common misunderstandings of the technology. Over the coming year, we look forward to continuing to address these challenges, as well as many others, through steady, long-term research collaborations and a shared research portfolio developed and executed by the CITeR community Stephanie Schuckers, Director of the Center for Identification Technology Research (CITeR)

Video credits.

C I Te R | 1

Research Highlights - WEST VIRGINIA UNIVERSITY - SUNY AT BUFFALO

Deep Profile-Frontal Face Verification in the Wild Fariborz Taherkhani, Veeru Talreja, Jeremy Dawson, Matthew Valenti, Nasser Nasrabadi (West Virginia University)

Profile GAN Module

i

x PR

Real Profile Image

Profile Discriminator Real or Reconstructed

Profile Generator

i

GPR(z\xiPR)

x PR

3

64

Perceptual Network 128

224x224 224x224

i

z1(x PR)

256

112x112

512

56x56

28x28

Contrastive Loss (Lcont) j

z2(x FR)

Frontal GAN Module

j

x FR

Real Frontal Image

Frontal Discriminator Real or Reconstructed

Frontal Generator

j

x FR

GFR(z\xjFR) 3

64

Perceptual Network

224x224 224x224

128

112x112

256

512

56x56

28x28

Matching profile-to-frontal face images is a challenging problem in video surveillance applications. The features available in both frontal and profile views vary significantly from each other and are, therefore, difficult to match. The performance of current featurebased face recognition (FR) algorithms or subspace-based algorithms degrade significantly when comparing frontal to profile faces under unconstrained conditions. As a result, when a profile face appears in a surveillance camera, the recognition performance of current commercial FR systems is degraded. This research utilized a novel face verification approach based on a coupled deep neural network (DNN) framework, which assumes that the two poses (profile and frontal faces of a person) can be related to each other by a latent feature embedding. The coupled DNN consists of two generative adversarial networks (GANs), each dedicated to a different pose (profile and frontal) but coupled together to produce a common (shared) latent feature vector that represents the hidden relationship between the two poses of the same subject.

We named the proposed network in this project the Profile-Frontal Coupled GAN (PF-CpGAN). The research resulted in a profile-tofrontal matching module that can be applied to any commercial face recognition system, fulfilling the need for matching a profile face probe against a gallery of frontal face images. WVU PhD students Fariborz Taherkhani and Veeru Talreja were awarded Best Student Paper at the IEEE IJCB 2020 for presenting the work associated with this project.

Detecting Deepfakes Using Physiological, Physical and Signal Features Siwei Lyu, David Doermann, Srirangaraj Setlur (SUNY at Buffalo) For this project, researchers at the University at Buffalo developed an innovative approach to expose AI-generated fake videos, including deepfakes, using forensics technology. As these automated, deep learning-based ways to synthesize and c i t e r. c l a r k s o n . e d u

Research Highlights - CLARKSON UNIVERSITY

manipulate videos continue to advance, the resulting videos pose new challenges for existing digital forensic tools. The team’s previous work on media forensics indicates that these deep learning models and processing pipelines that produce deepfakes lack essential physiological signals, have inconsistent Face Detection & Alignment physical characteristics and produce signallevel artifacts. The team’s approach exploits this trace evidence by combining multiple types of cues to detect deepfake videos more effectively. The targeted features improved explainability and increased the robustness to quality degradation and social media laundering.



Video Level Prediction

Behavioral Pattern

CNN CNN CNN

Temporal RNN

Orientation & Lighting

Real Fake Frame Level Prediction

CNN CNN CNN

Real Fake

Artifact Filter

Iris Recognition Performance in Children: A Longitudinal Study Priyanka Das, Stephanie Schuckers (Clarkson University) CITeR graduate student Priyanka Das and research advisors Dr. Stephanie Schuckers, Dr. Daniel Rissacher and Laura Holsopple at Clarkson University and Dr. Michael Schuckers at St. Lawrence University focused on the performance of iris biometrics in children as the time between enrollment and query increases.

Their research is published in IEEE Transactions on Biometrics, Behavior and Identity Science and the International Joint Conference on Biometrics publication.

Their work presents a longitudinal analysis from 209 subjects, aged 4 to 11 years at enrollment and six additional sessions over a period of three years. The influence of time, dilation and enrollment age on iris recognition were analyzed and their statistical importance was evaluated. The impact of age and aging on dilation and the impact of variation in dilation on the iris recognition performance in children was studied. A minor aging effect, which was statistically significant, was noted, but it was practically insignificant and comparatively less important than other variability factors. Practical biometric applications of iris recognition in children are feasible for a time frame of at least three years between samples, for children aged 4 to 11 years, even in the presence of aging, although the researchers noted practical difficulties in enrolling young children with cameras not designed for the purpose.

C I Te R | 3

Research Highlights - MICHIGAN STATE - IDIAP

Homomorphically Encrypted Matching and Search Anil Jain, Joshua Engelsma (Michigan State) Unencrypted Face Recognition System

Enrollment Database ID:01 ID:02 ID:03

Dr. Anil Jain and his team utilized both cryptographic techniques (data encoding schemes) and machine learning   Impersonation?

Reconstruction Feature techniques (dimensionality reduction) to enable the use of     Extraction  Algorithm   Gender?

Age? Race?  Query Face Reconstructed fully homomorphic encryption (FHE) for a practical featureStolen Matcher Face Template based search in the encrypted domain at scale. FHE enables Hacker securely matching templates directly within the encrypted domain, without a need to decrypt or a negative impact Encrypted Database Encrypted Face Recognition System on the biometric recognition accuracy. Existing methods ID:01 ID:02 Encrypted designed specifically for encrypted 1-1 matching were unable ID:03 Template Template   ... to easily leverage FHE for a large-scale search due to its Privacy Feature   HERS      ?  Preserved  Extraction incredible computational complexity. The practicality of the  Query Face Encrypted Matcher  developed system was demonstrated on face and fingerprint (Encrypted Domain) Stolen Template datasets at scale. The search time for a fingerprint query Hacker template in the encrypted domain against a gallery of 1.1 million was reduced from nearly three hours to 40 seconds while only losing 0.1% Rank-1 accuracy. The resulting encrypted matching algorithm allows users a technical edge over competitors by protecting the privacy of users at all times (enrollment into the database and during the matching procedure).

Template

 

 

...

1.0

Fairness in Biometrics: A Figure of Merit to Assess Biometric Verification Systems

FDR(τ)

0.9 0.8

Tiago de Freitas Pereira, Sebastien Marcel (Idiap)

0.7 0.6 10-1 10-2 10-3 DNN COTS

10-4

10-5

τ = FMR10

-x

c i t e r. c l a r k s o n . e d u

10-6

Modern face recognition (FR) systems are based on Deep Convolutional Neural Networks and present skewed recognition scores toward covariates of a test population (i.e., demographic differentials with respect to age, gender and ethnicity). A pressing need for fair FR systems and techniques to reduce demographic differentials led to this research. This project investigated bias mitigation techniques based on regularization and score-normalization techniques. Based on this work, a figure of merit to measure fairness in biometric systems has been proposed in a paper entitled Fairness in Biometrics: A Figure of Merit to Assess Biometric Verification Systems, which is under review but available here as preprint: arxiv.org/pdf/2011.02395.pdf.

STUDENT HIGHLIGHTS | AWARDS

West Virginia University Veeru Talreja, PhD in Electrical Engineering, expected 2021 graduation.

“The first CITeR project I worked on — A Cloud-Based Biometric Service Model for Iris and Ocular Recognition Using a Smartphone — got me interested in biometric recognition. The immense experience I have gained doing CITeR projects has helped me to grow as a researcher. The exposure and knowledge sharing among researchers during CITeR meetings and workshops has helped me stay updated and learn new advancements in biometrics and computer vision. The most recent project I worked on, Generative Adversarial Network (GAN) Model for Profile-to-Frontal Face Recognition in the Wild, was awarded Best Student Paper at the 2020 International Joint Conference on Biometrics.”

Veeru Talreja

Clarkson University David Yambay, PhD in Electrical Engineering, 2019.

“I started my first CITeR project as a researcher in 2010 when I took over the Clarkson side of the LivDet competitions. I grew throughout my years in our organization and continued to expand my CITeR research into the IARPA N2N competition, contactless fingerprints and the longitudinal study on child biometrics. CITeR has been a home for me throughout my education.”

David Yambay

University at Buffalo Shu Hu, PhD in Computer Science, expected June 2023 graduation.

“I am very lucky to be involved in one of the CITeR-funded projects, Detecting Deepfakes Using Physiological, Physical and Signal Features. In this project, my mentor and I developed a method to automatically extract and compare corneal specular highlights from two eyes. We show that deepfake face images can be exposed with the inconsistent corneal specular highlights between two eyes.

Working on this CITeR project enabled me to learn more advanced knowledge of computer vision.”

Shu Hu

Michigan State University Joshua Engelsman, PhD in Computer Science, expected 2021 graduation.

“I started working with CITeR-funded projects in 2020. The first project was to develop a practical solution for fully homomorphically encrypted biometric search at scale to better secure user privacy. The second project was to develop an endto-end solution for mobile finger photo to legacy contact-based fingerprint image matching. Both of these projects gave me the opportunity to develop solutions for challenging, relevant and real-world problems.”

Joshua Engelsman

C I Te R | 5

CITeR PROJECTS

CITeR Fall 2019 Projects (Performance period: 1/1/2020-12/31/2020) – Evaluation of Match Performance of Livescan vs.

Contactless Mobile Phone Fingerprints Jeremy Dawson (WVU) – Deep Cross-Spectral Iris Matching: High-Resolution Visible Iris Against Low-Resolution NIR Iris Jeremy Dawson (WVU) and Nasser Nasrabadi (WVU) – Federated Biometrics: Towards a Trustworthy Solution to Data Privacy Guodong Guo (WVU) – How to Assess Face Image Quality With Deep Learning?

Guodong Guo (WVU) – Age Invariant Face Recognition in Children Stephanie Schuckers (CU), Xin Li (WVU), Chen Liu (CU), Keivan Bahmani (CU) – Multispectral Anti-Spoofing and Liveness Detection Based on the Front-View Camera and the Screen of a Smartphone Jun Xia (UB)

– Adversarial Learning-Based Approach Against Face Morphing Attacks Chen Liu, Stephanie Schuckers, Zander Blasingame (CU) – Detecting Morphed Faces Using Deep Siamese Network Nasser M. Nasrabadi and Jeremy Dawson (WVU) – Face in Motion for Disambiguating Doppelgangers and Twins Arun Ross (MSU) – Biometric Recognition Technology Under Scrutiny: Public Outreach on Technology Fundamentals Stephanie Schuckers, Laura Holsopple, Daqing Hou, Mahesh Banavar (CU) – Face Quality Index Assessment for Sensor and Subject Based Distortions (DHS Special Project) Nasser M. Nasrabadi and Jeremy Dawson (WVU) – FMONET: FAce MOrphing With Adversarial NETworks and Challenge David Doermann (UB AI Institute), Ranga Setlur (CUBS) – Understanding (and Mitigating) the Public Concerns in Biometric Authentication Wenyao Xu, Srirangaraj Setlur, Mark Frank (UB)

– Faster and More Accurate Mobile Touch Dynamics via Feature Selection and Sequence Learning Daqing Hou, Mahesh Banavar, Stephanie Schuckers (CU)

CITeR Spring 2020 Projects

– Deep Fingerprint Matching From Contactless to Contact Fingerprints for Increased Interoperability Jeremy Dawson, Nasser Nasrabadi, Ali Dabouei (WVU)

– Benchmarking Video Face Super-Resolution Algorithms Nasser Nasrabadi (WVU), Jeremy Dawson (WVU), Moktari Mostofa (WVU)

– Abnormality Detection From Automobile Audio Using Spectrography Nishant Sankaran, Deen Dayal Mohan, Srirangaraj Setlur (UB) – Mitigation of Demographic Variability in Face Recognition Using Relative Skin Reflectance Trained With a Direct Measure of Skin Reflectance Stephanie Schuckers (CU), Mahesh Banavar (CU), Keivan Bahmani (CU) – Identical Twins as a Benchmark for Human Facial Recognition Jeremy Dawson & Nasser Nasrabadi (WVU) – Evaluation of the Equitability of Speaker Recognition Algorithms Jeremy Dawson & Nasser Nasrabadi (WVU) c i t e r. c l a r k s o n . e d u

(Performance period: 7/1/2020-6/30/2021)

– Biometric and Biographic Data Cleanup With Deduplication and Quality Scoring Guodong Guo (WVU) and Xin Li (WVU) – Comparative Detection of Facial Image Manipulation Techniques Zander Blasingame (CU), Chen Liu (CU) – Contactless Fingerprint Recognition Using Smartphone Cameras Anil Jain (MSU) – Detecting Deepfake Videos by Analyzing Both Audio and Visual Arun Ross (MSU), Xiaoming Liu (MSU) – Fingerprint Segmentation for Juveniles and Adults Keivan Bahmani (CU), Stephanie Schuckers (CU)

FINANCIALS | OUTCOMES CITeR Income Summary 2016-20 $1,600,000 $1,400,000

– Investigating Cardiac Waveform and Related Physiological Parameters in Personal Identification Jun Xia (UB) – LivDet 2021 Sandip Purnapatra (CU), David Yambay (CU), Stephanie Schuckers (CU), Thirimachos Bourlai (WVU) – Deep Spoof Detection for Text-Independent Speaker Verification Nasser Nasrabadi (WVU), Jeremy Dawson (WVU), Sobhan Soleymani (WVU) – Detecting Deepfakes Using Physiological, Physical and Signal Features Siwei Lyu (UB), David Doermann (UB) and Srirangaraj Setlur (UB)

$800,000 $600,000 $400,000 $200,000 $0

Year

– Determining the Uniqueness of Facial Images in Large Datasets Jeremy Dawson (WVU), Nasser Nasrabadi (WVU) – Digitization of 10-Print Card Fingerprints Using Cellphone Cameras Jeremy Dawson (WVU), Nasser Nasrabadi (WVU) – Homomorphically Encrypted Matching and Search Anil Jain (MSU) – On the Uniqueness of Facial Identity Xiaoming Liu (MSU), Anil Jain (MSU) – Privacy Enhancing Biometric Sensors Arun Ross (MSU)

2016

2017

2018

2019 NSF I/UCRC

2020 Affiliate $

total # of publications (all researchers)

2018

57

2019

53

2020

44

CITeR Citations

Year

total # of citations (all researchers)

2018

3245

2019

3529

2020

3287

Database Requests

– Privacy-Preserving Biometric-Based Authentication Using Secure Multi-Party Computation Marina Blanton (UB) – Revisiting Voice Biometric Entropy With Demographic Impacts Wenyao Xu (UB), Srirangaraj (Ranga) Setlur (UB) – Uniqueness and Permanence of Iris Priyanka Das (CU), Stephanie Schuckers (CU), Joseph Scufka (CU), Natalia Schmid (WVU), Matt Valenti (WVU)

– Wavelet-Based Morphed Artifacts Detection Nasser Nasrabadi (WVU), Jeremy Dawson (WVU)

$1,000,000

CITeR Publications

– Detecting Face Morphing: Dataset Construction and Benchmark Evaluation Jacob Dameron (WVU), Guodong Guo (WVU), and Xin Li (WVU)

– Video Series Expansion — Biometric Recognition Technology Under Scrutiny: Public Outreach on Technology Fundamentals Stephanie Schuckers (CU), Laura Holsopple (CU), Daqing Hou (CU), Mahesh Banavar (CU)

$1,200,000

Year 2018

# fulfilled requests (all universities) 57

2019

88

2020

$50K

70

affiliate membership gains access to the community and a portfolio of in projects

$1.5M

C I Te R | 7

citer.clarkson.edu Contacts Clarkson University Dr. Stephanie Schuckers Director 315-268-6536 sschucke@clarkson.edu

Affiliate Advisory Board Chair Laura Holsopple Managing Director 315-268-2134 lholsopp@clarkson.edu

West Virginia University Dr. Matthew Valenti Site Director 304-293-9139 matthew.valenti@mail.wvu.edu

Dr. Nasser Nasrabadi Site Co-Director 304-293-4815 nasser.nasrabadi@mail.wvu.edu

University at Buffalo Dr. Venu Govindaraju Site Director 716-645-3321 venu@cubs.buffalo.edu

Srirangaraj Setlur Site Co-Director 716-645-1568 setlur@buffalo.edu

Michigan State University Dr. Arun Ross Site Director 517-353-9731 rossarun@cse.msu.edu Idiap Research Institute (international site) Dr. Sébastien Marcel Site Director +41 27 721 77 27 marcel@idiap.ch

Kody West Technical Fellow Northrop Grumman Corporation Affiliate Executive Committee Terry Riopka Director of Research AWARE Inc. Chris Chamberlin Project Manager Office of Biometric Identity Management Department of Homeland Security Arun Vemury Director, Biometric and Identity Technology Center U.S. Department of Homeland Security Brian Green S&T Program Manager, Border Security and Biometrics Defence Research and Development Canada Department of National Defence, Government of Canada