An Introduction to PAI’s Facial Recognition Systems Project
Facial recognition. What do you think of when you hear that term? How do these systems know your name? How accurate are they? And what else can they tell you about someone whose image is in the system?
These questions and others led the Partnership on AI (PAI) to begin the facial recognition systems project. During a series of workshops with our partners, we discovered it was first necessary to grasp how these systems work. The result was PAI’s paper “Understanding Facial Recognition Systems,” which defines the technology used in systems that attempt to verify who someone says they are or identify who someone is.
A productive discussion about the roles of these systems in society starts when we speak the same language, and also understand the importance and meaning of technical terms such as “training the system,” “enrollment database,” and “match thresholds.”
Let’s begin — keeping in mind that the graphics below do not represent any specific system, and are meant only to illustrate how the technology works.
How Facial Recognition Systems Work
Understanding how facial recognition systems work is essential to being able to examine the technical, social & cultural implications of these systems.
Let’s describe how a facial recognition system works. First, the system detects whether an image contains a face. If so, it then tries to recognize the face in one of two ways:
During facial verification: The system attempts to verify the identity of the face. It does so by determining whether the face in the image matches a specific face previously stored in the system.
During facial identification: The system attempts to predict the identity of the face. It does so by determining whether the face in the image potentially matches any of the faces previously stored in the system.
Let’s look at these steps in greater detail
A facial recognition system needs to first be trained, with two main factors influencing how the system performs: firstly, the quality of images (such as the angle, lighting, and resolution) and secondly the diversity of the faces in the dataset used to train the system.
An enrollment databaseconsisting of faces and names is also created. The faces can also be stored in the form of templates.
The first step in using any facial recognition system is when a probe image, derived from either a photo or a video, is submitted to the system. The system then detects the face in the image and creates a template.
There are two paths that can be taken
The template derived from the probe image can be compared to a single template in the enrollment database. This “1:1” process is called facial verification.
Alternatively, the template derived from the probe image can be compared to all templates in the enrollment database. This “1:MANY” process is called facial identification.
The system compares two templates – one from the probe image and one from the enrollment database – in order to find a potential match.
A match score is generated for each pair of templates, indicating how similar the two images are. A match threshold is set by the system’s developer or operator. Any two templates whose match score is above that threshold are considered similar enough to be a potential match.
The match threshold setting is critical because it determines whether a face is included or left out of the results presented to the user.
A lower match threshold will return more matches, with a greater chance of misidentification, known as a false positive. A higher match threshold will return fewer matches, with the possibility that a potential “match” is missed, known as a false negative.
Click and drag the slider to see the importance of match thresholds
Beyond facial recognition
Sometimes facial recognition systems are described as including facial characterization (also called facial analysis) systems, which detect facial attributes in an image, and then sort the faces by categories such as gender, race, or age. These systems are not part of facial recognition systems because they are not used to verify or predict an identity.
