This Simple Technique Made Me Invisible to Two Major Facial Recognition Systems

Thomas Smith

https://onezero.medium.com

Jan 28, 2021

Image: Design Cells/Getty Images

As millions of demonstrators took to the streets last summer to protest police violence and the killing of George Floyd, government agencies wasted no time in surveilling them with facial recognition software. As authorities began to recommend mask-wearing to combat Covid-19, many activists saw a silver lining. A major study from the National Institute of Standards and Technology (NIST) published in July 2020 showed that masks fooled certain facial recognition systems up to 50% of the time, suggesting that masks might offer protesters some level of protection from mass surveillance.

That got me wondering: What’s the best way to use masks and other everyday accessories to fool facial recognition systems? I decided to find out.

Gothamist reports that the NYPD used facial recognition (and possibly photos from social media) to pursue Black Lives Matter activist Derrick Ingram. There have been several reports of false arrests resulting from use of the technologies, especially against people of color. I’ve personally tested facial recognition tools like Clearview AI, and I’ve seen their power firsthand. Widespread use of facial recognition against protesters has led to calls to ban the technology, and legislative efforts to do so are underway across the country.

By adding a simple pair of sunglasses to my masked face, I had effectively made myself invisible to the system.

Unwilling to wait for laws that might regulate facial recognition’s use, many protesters and activists have begun to take matters into their own hands, devising clever ways to thwart facial recognition systems with elaborate “Dazzle” makeup, customized QR-code-like stickers, and tools to digitally cloak photos. Unfortunately, many of these strategies don’t work. And techniques like Dazzle makeup take a long time to apply and make users look conspicuous. Masks, on the other hand, are easy to take on and off and can be worn publicly without drawing undue attention to their wearers.

The NIST study analyzing masks’ impact on facial recognition systems was comprehensive, looking at 89 different facial recognition platforms. But it had some issues. It didn’t use real masks — the researchers added digital masks to border-crossing photos and used the altered photos to test various facial recognition systems. It also revealed a lot of disparity in success rates. Some systems were fooled 50% of the time by masks, while others were fooled only 5% of the time. To find out which mask designs did the best job of fooling facial recognition software, I ran my own experiment.

For my tests, I decided to use the face-comparing function of Face++ from Megvii. Face++ is a popular provider of facial recognition services. You can access the platform through an API, and comparing two faces costs only $0.002, making the technology accessible to nearly anyone. Face++ is also reportedly good at detecting masked faces. This is likely because the company is based in China, where mask-wearing was common even before the pandemic, so it was likely trained on images of masked people years before masking became common in the West.

I started by uploading a generic photo of my face to Face++’s platform, along with a second, nearly identical photo taken in the exact same light. These are nearly ideal conditions for a facial recognition platform to make a comparison.

Unsurprisingly, Face++ did well, determining that the two photos depicted the same person with 95.071% confidence. That gave me a baseline for the system’s performance.

I then began to test it with different common face masks. I started with a photo of me wearing a generic blue disposable mask. Again, I photographed myself under the same lighting conditions and at the same angle.

Remarkably, wearing a mask did relatively little to fool Face++. Its confidence score dropped to 74.765%, which is a major reduction but still a high confidence value, given that more than half of my face was covered. This jibes with Face++’s pedigree, as well as research showing that facial recognition companies are already responding to the pandemic by beefing up their systems’ ability to identify masked faces.

Next, I tried different mask designs. I expected that wearing an N95 mask would worsen the system’s results, since the mask covers more of my face. But it actually improved results to 77.59% accuracy.

Wearing my son’s Paw Patrol mask bumped the results up to 82.421%, likely because the mask was too small for me and revealed more of my face. I also tried wearing a mask with a picture of Tom Cruise stapled to the front in an effort to confuse Face++ by presenting it with one face nested on top of another, a technique that others have used successfully. It didn’t work — the system found my face and returned a 79.191% confidence score. Besides, I wouldn’t exactly call that look “inconspicuous.”

Next, I tested a custom-designed adversarial mask pattern, created using a Python program that built a histogram of gradients specifically tuned to confuse facial recognition algorithms. This also failed and actually increased the system’s accuracy to 83.864%. By this point, I was getting frustrated. Face++ seemed able to handle nearly anything I threw at it. The harder I worked to obscure my face, the better it got at detecting me.

As a last-ditch effort, I decided to try something that seemed too simple to possibly work. I wore my generic blue disposable mask, but I paired it with the sunglasses I wear on a daily basis. Running my photo through Face++, I was startled by the results. The combination of a mask and sunglasses didn’t just reduce the system’s performance — it rendered Face++ completely unable to find any faces in the photo.

By adding a simple pair of sunglasses to my masked face, I had effectively made myself invisible to the system.

This seemed too good to be true, so I tried the images with another popular facial recognition software provider, FaceX. The FaceX API returned a 70.39% confidence score on my masked face, which is comparable to Face++’s accuracy. But it showed only an 11.18% confidence score when I added sunglasses, leading the system to conclude, “The faces belong to different people.”

Why would adding sunglasses to my face result in such dramatic results? To answer that, it helps to take a closer look at how platforms like Face++ identify faces. The systems are generally trained to identify a variety of facial landmarks, like the position of a person’s eyes, nose, mouth, hairline, and the outline of their face.

Here are the landmarks that Face++ identified on an unmasked image of my face (represented by blue dots).

Adding a mask degrades the results — but not completely. Even with a bulky N95 on my face, Face++ is still able to see my eyes and eyebrows perfectly. It can also detect the edges of my face from my cheekbones and follow those down below the mask with reasonable accuracy. It’s even able to make some relatively accurate educated guesses about the position of my nose and mouth under the mask.

Wearing sunglasses, though, denies the system the dense and valuable data about my eyes and eyebrows. And the sides of my sunglasses likely block enough of my cheekbones that the system is unable to derive much information from those, either. Meanwhile, the mask continues to block my mouth and nose and obscure the area around my chin. Combined, a mask and sunglasses appear to deny the systems so much data that they’re unable to make an accurate facial comparison (or even determine that a face is present in the photo, in the case of Face++), and they fail.

This is a remarkable result, especially since a generic disposable mask and sunglasses has been my standard going-out outfit for pretty much the entire pandemic. No one on the street would bat an eye if they saw me wearing this combo — yet it apparently renders me invisible to facial recognition algorithms that otherwise have no problem finding my masked face.

How long will this technique work to fool facial recognition algorithms? That’s unclear. Facial recognition as a field is constantly advancing, and given the technology’s utility in investigating serious crimes, in addition to profiling innocents, it’s likely that facial recognition providers will ultimately find a way to adapt their algorithms to the mask-and-sunglasses combo. Sunglasses are also banned in some public places, and most people wouldn’t wear them indoors anyway. (The technique doesn’t work with clear glasses — only opaque sunglasses.) It’s also unclear whether the same technique would fool the algorithms used by police and government agencies, which may be different than publicly available ones. And even without facial recognition, these agencies have plenty of other ways to track you.

But at the moment, and at least with the platforms I tested, combining a mask and sunglasses appears to be a remarkably powerful and simple way to remain anonymous — even in the face of mass surveillance.