How Do Snapchat And Instagram Filters Work?

Social media filters from Snapchat and Instagram that change a user’s facial features work using a technology called Augmented Reality.

I turned myself into a dog, a vampire, and a beauty queen with a full face of makeup today. In the space of mere seconds, I was able to transform completely, without ever lifting my head off the bed. That’s the kind of magic that Snapchat, Instagram, and other newfangled apps bring to us these days.

snapchat-kid_t20_09lpjB

Snapchat filters let us add all sorts of silly things to our faces (Photo Credit : twenty20)

Filters that alter our facial features in real-time are nothing new, but they keep getting better and more realistic. However, what goes on behind the scenes of these filters? What trick of technology is actually running the show?

Where did it come from?

Facial recognition and modification is not a new invention. Facebook was finding and tagging faces in images a long time ago. It is the scale and the processing speed—the ability to do this in real-time—that is new.

Young,Woman,Picked,Out,By,Face,Detection,Or,Facial,Recognition

Facebook was finding and tagging faces in images long before face-altering filters. (Photo Credit : Axel Bueckert/Shutterstock)

The social media filter craze began back in 2015 when Snapchat acquired Ukrainian startup Looksery. At the time, they were the only player in the market with the capability to enhance live videos and images.

Now, everyone has this ability in their bag of digital tricks. Filters can now modify your facial features with almost eerie accuracy, and can even detect the world around you—making dancing cats appear on the pavement in front of you with little more than a tap.

Snapchat world lenses can make dancing unicorns appear on your keyboard if you wish.

Snapchat world lenses can make dancing unicorns appear on your keyboard if you wish.

A technology called Augmented Reality (AR) is responsible for all these new features. Snapchat, Instagram and Facebook have their own version of these filters, but they’re almost all the same thing. Some describe it all as ‘Social AR’, but more correctly, it’s a combination of computer vision, AR, and Artificial Intelligence (AI).

Augmented Reality

Augmented Reality is a technology that allows real-world images to be enhanced and modified. You can combine the virtual world with the real world, overlaying one on the other.

Augmented,Reality,Marketing,Concept.,Hand,Holding,Smart,Phone,Use,Ar

AR finds applications in navigation, marketing, education, and other sectors, besides entertaining us with filters. (Photo Credit : Zapp2Photo/Shutterstock)

AR is an advanced technology that has capabilities extending far beyond dog filters, but for the time being, we’re talking about these basic filters. In its basic form, AR overlays images on the live video  being captured by a camera. For your social media needs, your smartphone camera usually serves this purpose.

How do the filters work?

First, the computer needs to figure out where your face is. Then it maps out the features of your specific face, distorting and applying effects to your appearance.

  • Detection: Teaching computers to detect and recognize faces and facial features is no simple feat.

crowd

It’s harder for a computer to identify a face than it is for you to find a specific person in this crowd. (Photo Credit : Today)

  • The computer converts the image to greyscale to make it simpler to understand. It then analyzes the color values of the pixels in a picture and recognizes patterns of contrast. Something called the Viola-Jones object detection framework is used for this.
  • Mapping: Active Shape Models (ASM) are used to find specific people and tag facial features. Machine learning models are trained using data where those features have already been mapped. A face has a set of points marked to define its features, forming a Point Distribution Model (PDM).
  • Basically, you feed the computer a hundred images and tell them where the eyes are in every picture. The computer then becomes capable of pinpointing eyes in a fresh image.
  • When you capture your own face using your camera, the computer places the points of an average face around where it detected your face to be. These points are adjusted according to their knowledge of how a face is “supposed” to appear. It then creates a ‘mesh’ of your face from these points.

Face recognition men. The polygonal wireframe of the head of a man demonstrates face recognition technology. A lot of points and lines for recognizing a person's personality. Vector illustration.

A mesh of your face is created from the points mapped. (Photo Credit : GermanVectorPro/Shutterstock)

  • Modification: The 3D mesh of your face can move with you. That’s why the lipstick filter you apply doesn’t get shaken off when you move your face to one side. This mesh can be distorted to create desired alterations, i.e., it can make your face look puffy, your nose look pointy, or your eyes look bigger.
  • Images are overlaid to create specific effects, like makeup. The image of the lipstick has points that are meant to match the points that indicate the position of your lips, which generates a very accurate positioning of the image.

And thus, voila! You’re puking rainbows!

AUGUMENTED REALITY; RULES meme

Conclusion

Now perhaps you better understand how social media AR filters work. They use what they already know about faces to detect and map your face. Then they can do what they want with it, including make it into weird and silly shapes.

As the technology gets better in the future, AR filters will be so good at emulating reality that it will be impossible to tell what’s real and what’s not. For now, however, let’s just have some fun with bunny ears and colored hair.

Help us make this article better
About the Author

Kavya has a Bachelors degree in Mechanical Engineering from APJ Abdul Kalam Technological University, Kerala, India. She loves to read and draw in her spare time. A firm believer that a jack of all trades is better than a master of one. She wants to explore the world, learn new things, and write about them.

.
Science ABC YouTube Videos

  1. Cellular Respiration: How Do Cell Get Energy?Cellular Respiration: How Do Cell Get Energy?
  2. Multiverse Theory Explained: Does the Multiverse Really Exist? Truth of Multiple RealitiesMultiverse Theory Explained: Does the Multiverse Really Exist? Truth of Multiple Realities
  3. What Exactly is Spacetime? Explained in Ridiculously Simple WordsWhat Exactly is Spacetime? Explained in Ridiculously Simple Words
  4. What Are The Different Atomic Models? Dalton, Rutherford, Bohr and Heisenberg Models ExplainedWhat Are The Different Atomic Models? Dalton, Rutherford, Bohr and Heisenberg Models Explained
  5. Why Is Blood Drawn From Veins And Not From Arteries?Why Is Blood Drawn From Veins And Not From Arteries?
  6. Emotions and the Brain: What is the limbic system?Emotions and the Brain: What is the limbic system?
  7. Dark Matter Explained: What Exactly is Dark Matter? | A Beginner’s Guide to Dark MatterDark Matter Explained: What Exactly is Dark Matter? | A Beginner’s Guide to Dark Matter
  8. What Exactly is a Tesseract? (Hint: Not a Superhero Stone)What Exactly is a Tesseract? (Hint: Not a Superhero Stone)

Tags: