But while it’s important to debate how facial recognition tech can and should be used, it’s equally important to note there are multiple way to track a person’s face. A better understanding of how it works can help educate people about how we should be thinking about using it, and making rules for it.

Two types of facial recognition

First off, there are two different approaches that allow companies to employ facial recognition, and one is far more expansive than the other. The first generation of facial recognition technology relies on two popular methods of identifying a face, and is what powers the Churchix software that recognizes faces in church. One of these methods relies on criteria like the distance between your eyes, the measurements of your nose, lips and other facial features and matches them against an existing database.

The other method used by this first generation software looks at points of interest on the face, and tracks how the pixels in a photograph cluster to form a nose. In both of these models matching a face is similar to matching a fingerprint. You’re looking for how closely certain characteristics line up to find a match. To get a good match using these models, you need a full-frontal picture of a face, good lighting and a database to compare your “faceprints” to.

[youtube https://www.youtube.com/watch?v=cDq9P-VUr0I&w=560&h=315]

In the case of Churchix, the church is using a database of members’ names and their photos to build a tiny file of data that is then compared with the data generated by the pictures coming from a camera. When there is a match, the name pops up from the existing database. But if a person or photo isn’t in the database, the system can’t return a result. Likewise, if a person’s face is partially obscured, or distorted via a grin, the system cannot return a result.

The second method of facial recognition is much more powerful, and is what Facebook, Microsoft and Google are focusing on. This version of facial recognition is based off of machine learning and efforts to train computers to recognize objects more generally. Facebook is using this type of facial recognition for its Moments feature. Because it is far more flexible, the computer doesn’t need the entire face to recognize someone, nor does it rely solely on characteristics like the distance of facial features. These models are trained using databases of faces to understand what a person actually looks like and then can match a face even without knowing who the person is.

Facebook is also conducting research to recognize people based not only on their face but also their outfits and their other body parts (it can tell if a leg belongs to a man or a woman, for example, and use that to aid in identification). Using machine learning it can combine a variety of inferences made by different computer vision algorithms to identify a person, even from behind. It is now able to identify a person with 83% accuracy even when they aren’t facing a camera.

[youtube https://www.youtube.com/watch?v=l8V0jh62zvE&w=560&h=315]

How scary is this?
In the case of the first type of technology, people can avoid being identified by keeping their head down—literally. Moshe Greenshpan, the president of Skakash, the company that sells the Churchix software, explains that he recommends churches put attention-getting devices on cameras to encourage people to look at them, so the camera can capture the full-frontal view of a face. One can also use makeup to confuse the camera in a dystopian view of contouring where you use makeup to make your features appear different. It’s all the rage in Hollywood and on fashion blogs.

Other things to do are to make huge facial expressions that distort your mouth, to help confuse the software. A grinning Kim Kardashian is the enemy of this type of software.

However, Greenshpan explains that he is investigating using machine learning in his software and expects to have that influence his algorithm in a year or two. That changes things, because then the pictures won’t need to be as clear and the software can match your face against any existing photographs of you that have your name attached. That means it could use an API call (a software-to-software command) to a site like LinkedIn or a place that gathers mug shots in an effort to pull up not just a copy of your face, but a detailed profile of your identity.

And that has both privacy and security implications. On the privacy side, the fears are well documented. At its core facial recognition software takes all the benefits of digital technology—making something instantly and easily searchable from anywhere and forever—and applies them to your face. This means a record of your physical actions in the real world becomes searchable in the digital world. When it comes to the first-generation technology you need a database that equates people and names, as well as high-quality images of the people you hope to match.

But with the type of facial recognition that Facebook, and even Greenshpan is proposing, it becomes much more invasive. This also has implications for security. One of the ways some churches use the Churchix software is to prevent known criminals from attending church events. In that case the church uses a database of people it wants to exclude from events and looks for them. Casinos or banks could use this the same way. However, the church has to work from an existing local database of known offenders. The software can’t recognize unknown threats that come from further afield.

But over time, as the machine learning is applied the recognition gets much better. Plus, as the technology improves and gets faster and cheaper, any facial recognition software can cast a wider net in search of a person’s identity. These wider nets could also deliver more information about the proposed match. So while today it almost feels as if the war against facial recognition is lost—Facebook says it can identify the person in one picture out of 800 million on its network in less than 5 seconds—the technology can become both more pervasive and more powerful.

Given this, it’s probably time to start discussing how we want to protect privacy and how we can use this technology to improve our lives.