Message Boards Message Boards

Create Snapchat Filters in Wolfram Language using Trained Neural Networks

GROUPS:

Hello everyone,

Frequently, we refer to the phrase "10 lines of code" in our group. Considering 1 line to be approximately 140-150 characters, I challenged myself to code something that is useful in some 1500 characters. Around the same time, when I was thinking what can be done, I was video chatting with my friend, where just to be silly, I started playing with the filters on Messenger, and started explaining what it was doing behind the scenes. After a discussion on facial landmark detection, I was given a suggestion to code it up. So that completed my challenge: Can I code something in maximum 10 lines of code that would mimic snap chat filters? Let's find out in this post. We start with the famous flower-crown or in general crown type filters

Step 1: Find the face

Finding Faces is pretty easy in Wolfram Language using the FindFaces function. You need to trim the image to just extract the face from the entire image. This is because we want to feed just the face to the second step where we want to detect facial features. Make sure to keep the dimensions of the face bounding box stored, because we need to account for this when we are finally placing the filter.

face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]];
   facedim = Flatten[FindFaces[img], 1];

Step 2: Find Facial Features

Facial keypoints can be detected using a trained NetModel. This net predicts the locations of 68 2D keypoints (17 for face contour, 10 for eyebrows, 9 for nose, 12 for eyes, 20 for mouth) from a facial image.

netevaluation[img_] := Block[
  {heatmaps, posFlattened, posMat},
  heatmaps = 
   NetModel["2D Face Alignment Net Trained on 300W Large Pose Data"][
    img];
  posFlattened = 
   Map[First@Ordering[#, -1] &, Flatten[heatmaps, {{1}, {2, 3}}]];
  posMat = QuotientRemainder[posFlattened - 1, 64] + 1;
  1/64.*Map[{#[[2]], 64 - #[[1]] + 1} - 0.5 &, posMat]
  ]

Step 3: Determining the size of the filter

We need to determine the face width (not all faces will have the same width) to determine the size of the crown (essentially we will be resizing the crown to fit faces of different width). Facial width can be determined by taking the distance from the left-most edge of the left eye brow to the right most point of the right eye. To have a better understanding of which landmark point correspond to which part of the face, I plotted them separately and sort of had an index (not just the groups) for each point. Once you determine the distance, you need to convert it to the dimensions of the face image (because the landmark points are on the scale of 0 to 1), and then multiply by a factor to determine how large the crown should be with respect to your face width. Playing with different types of crown filter, I realised that this factor need to be changed for each crown type and for a "generic crown" code I kept it a parameter

facewidth = ManhattanDistance[landmarks[[1]], landmarks[[17]]];
dimimg = ImageDimensions@face;
filterresize = 
  ImageResize[
   filter, {facewidth*resize1*dimimg[[1]], 
    facewidth*resize2*dimimg[[1]]}];

Step 4: Place filter at the appropriate position on the original image

Now that the size of the filter is determined, so in step 4, we need to place the filter on the face of the original image. We need to consider two things, one, where the filter needs to be placed, so with respect to which landmark, and two, how to put the filter on the original image (remember the original image is different from the trimmed face image on which we are doing all facial landmark detection). We don't have any landmark which is for the forehead, so I took the landmark for the centermost point of the eye and then added a shift to place the filter on the forehead. Again, how far should I place it, how much should it be scaled, is kept as a parameter, and is usually fixed for one kind of crown (can change from different crowns). Next just add the positions obtained from FindFaces, to place the filter on the original image

a = facedim[[1, 1]] + (landmarks[[23]][[1]] - pos1)*dimimg[[1]];
b = facedim[[1, 2]] + (landmarks[[23]][[2]] + pos2)*dimimg[[1]];
ImageCompose[img, filterresize, {a, b}]

Step 5: Make a function

Just combine all steps to make it into a single function

crown[img_, filter_, resize1_, resize2_, pos1_, pos2_] := 
 Block[{face, facedim, landmarks, facewidth, dimimg, filterresize, a, 
   b},
  face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]];
  facedim = Flatten[FindFaces[img], 1];
  landmarks = netevaluation[face];
  facewidth = ManhattanDistance[landmarks[[1]], landmarks[[17]]];
  dimimg = ImageDimensions@face;
  filterresize = 
   ImageResize[
    filter, {facewidth*resize1*dimimg[[1]], 
     facewidth*resize2*dimimg[[1]]}];
  a = facedim[[1, 1]] + (landmarks[[23]][[1]] - pos1)*dimimg[[1]];
  b = facedim[[1, 2]] + (landmarks[[23]][[2]] + pos2)*dimimg[[1]];
  ImageCompose[img, filterresize, {a, b}]
  ]

Adding the number of characters used in netevaluation function and the crown function, we have 800 characters, that is less than 6 lines code and potentially the technology for an application that was sold on September 2015 for $150m. So why was it worth $150m? The technology needs to run on mobile devices, and needs to be fast enough to capture real time changes in the images (video chatting). So let us go ahead and convert it to real time!

Step 6: Making it in Real Time

Dynamic[crown[CurrentImage[],(*filter*), 1.5, 0.75, 0.1, 0.25]]

Let us take the challenge and make it capture images in real time, detect the faces, and place the filter. Wrapping it with dynamic enables dynamic detection of faces, and CurrentImage will take your photo using your device camera. In this particular example, with the parameters, I used a the flower filter which is uploaded here:

enter image description here

All images of the crowns are first pre-processed so that all the background is removed (or made transparent).

Use cases

Since I was listening to Perfect duet while coding, here is the code for placing flower crown on a list of Beyonce images, and a king crown (some randomly chosen filter) for Ed Sheeran images. Again I just randomly chose these crowns to show in the post, I have some other heart crowns as well (if someone wants them). It is very easy to customize any filter, just make sure you remove the background, and set parameters right once, for each kind of crown.

eds = WebImageSearch["EdSheeran", "Images", "MaxItems" -> 10]
crown[#,(*filter2*), 1.75, 0.7, 0.1, 0.35] & /@ eds

Again the filter2 is a king-crown image that I randomly downloaded and then processed to be used. It looks like this

enter image description here

Okay, so how does Ed(s) look with the king crowns?

enter image description here

And how does Beyonce(s) look with the flower ones?

enter image description here

Conclusion

A second kind of filter with dog face/bunny face with multiple faces is almost done, within the 10 lines limit of course! I will share it either as a comment to this one or make a separate post. I am extending the idea to create distortions, and pose detections (feature changes tracking). Please provide any suggestions or algorithms you might have come across.

POSTED BY: Tuseeta Banerjee
Answer
4 months ago

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

POSTED BY: Moderation Team
Answer
4 months ago

Here is the link to the second post, which talks about bunny type filters and how you generalize these filters to multiple faces. http://community.wolfram.com/groups/-/m/t/1313034?p_p_auth=VPMz9d3W

POSTED BY: Tuseeta Banerjee
Answer
2 months ago

Thanks for Sharing this post!

POSTED BY: Siddhi Sabnis
Answer
15 days ago

Group Abstract Group Abstract