November, 2018

SNAPCHAT LIKE REAL-TIME FILTER

Created for image processing course.

DESCRIPTION

In this project we wanted to create a realtime filter like Snapchat with a sample filter. We have chose Marvel ™ Ghost Rider character, with fire effect when the user opens his mouth.

APIS AND DEPENDENCIES

We have chosen OpenCV to get array of facial landmarks points for each face. We developed the program using C++ and built it into a dynamic library to be used by Unity 3D.

Unity 3D calls the dynamic library functions to get the facial landmarks and use this data to calculate the skull position, rotation and mouth state (open/close).

THE APPROACH

OpenCV provides a function to get facial landmarks to each face as an array of 68 elements. This landmarks are in 2D space as the image. Each group of points are for a cretain facial feature.

Index:

0-16: Chin
17-26: Eye brows
27-35: Nose
36-47: Eyes
48-76: Mouth

Using this info about the face, rotation of the face looking (left/right) could be calcualted as a ratio between the nose and the cheek on each side. Or simple a ratio in the distances: distance(34,3):distance(34,15).

POST PROCESSING

Finally a post processing shader was created for the toon effect to mimic comic books. The shader first discritize the color pixel value into one of countable values then apply edge detection shader on the discritized image.