Message Boards Message Boards

GROUPS:

Bunny-face Snapchat filters for multiple faces using trained neural network

Posted 4 months ago
951 Views
|
4 Replies
|
12 Total Likes
|

Hello,

This is the second part of the custom snapchat filter, I decided to publish the bunny/dog types. This post is going to start from the previous post. So I am not going to explain the concepts discussed in the previous posts, of facial detection, facial features detection, creating a custom filter, determining the filter size automatically using trained neural networks. In this post, we will start with how we can place dog/bunny types filter and how to extrapolate it to more than one face. Multiple face detection and filter creation is exemplified using the bunny filter.

Facial Feature Detection

This part of the code is same as in previous post, so I am not explaining much, however, it needs to be evaluated for the filter functions to work.

netevaluation[img_] := Block[
  {heatmaps, posFlattened, posMat},
  heatmaps = 
   NetModel["2D Face Alignment Net Trained on 300W Large Pose Data"][
    img];
  posFlattened = 
   Map[First@Ordering[#, -1] &, Flatten[heatmaps, {{1}, {2, 3}}]];
  posMat = QuotientRemainder[posFlattened - 1, 64] + 1;
  1/64.*Map[{#[[2]], 64 - #[[1]] + 1} - 0.5 &, posMat]
  ]

Dog Face Filters

So let's start with dog-face filters and ask the question, how is is different from the crown filters? Well, for placing the crown filters we needed just one facial feature to place it, adjusted the filter size according to the size of the face. However, if we try to do the same with the dog face filters, the size of the dog face, the orientation of the dog face and obviously the lay-out of the ears, nose and tongue might not exactly match with the human face image your provide. So for this kind of filter, I imported a dog face filter from the web, and separated out the ears, tongue and nose. That involved background removal, and systematic cropping till you get them properly cropped. I am not including the code for that, because it would require you to customize depending on what filter you are using. We perform resizing of these filters in a very similar way that we discussed in the previous post, so I will leave that detail out. The new catch for this filter is that we need to place the right ear, left ear, tongue, and nose at appropriate landmarks for each of the facial part. Again, setting the position of nose did not require much parametrizing, however tongue and ears required a little bit. Putting all the filter parts and the parameters and the facial features in the original image, the function looks like:

dogface[img_, nose_, tongue_, leftear_, rightear_, resize1_, pos1_, 
  pos2_, pos3_] := 
 Block[{face, facedim, landmarks, facewidth, dimimg, noseresize, 
   tongueresize, leftearresize, rightearresize, nosea, noseb, tonguea,
    tongueb, lefteara, leftearb, righteara, rightearb},
  face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]];
  facedim = Flatten[FindFaces[img], 1];
  landmarks = netevaluation[face];
  dimimg = ImageDimensions@face;
  {noseresize, tongueresize, leftearresize, rightearresize} = 
   ImageResize[#, dimimg[[1]]*resize1] & /@ {nose, tongue, leftear, 
     rightear};
  nosea = facedim[[1, 1]] + (landmarks[[31]][[1]])*dimimg[[1]];
  noseb = facedim[[1, 2]] + (landmarks[[31]][[2]])*dimimg[[1]];
  tonguea = facedim[[1, 1]] + (landmarks[[67]][[1]])*dimimg[[1]];
  tongueb = 
   facedim[[1, 2]] + (landmarks[[67]][[2]] - pos3)*dimimg[[1]];
  lefteara = 
   facedim[[1, 1]] + (landmarks[[18]][[1]] - pos1)*dimimg[[1]];
  leftearb = 
   facedim[[1, 2]] + (landmarks[[18]][[2]] + pos2)*dimimg[[1]];
  righteara = 
   facedim[[1, 1]] + (landmarks[[27]][[1]] + pos1)*dimimg[[1]];
  rightearb = 
   facedim[[1, 2]] + (landmarks[[27]][[2]] + pos2)*dimimg[[1]];
  ImageCompose[
   img, {noseresize, tongueresize, leftearresize, 
    rightearresize}, {{nosea, noseb}, {tonguea, tongueb}, {lefteara, 
     leftearb}, {righteara, rightearb}}]
  ]

enter image description here

Just a disclaimer, these are randomly downloaded images from the internet.

Bunny Face Filters

The bunny face filters are similar to the dog face ones, except that they do not have the tongue. It had similar cropping, background removal to be done to be used to create the filters. So the function for bunny face filter is almost same as the dog face one, except the missing tongue. Here it is

bunnyface[img_, nose_, leftear_, rightear_, resize1_, pos1_, pos2_] :=
  Block[{face, facedim, landmarks, facewidth, dimimg, noseresize, 
   tongueresize, leftearresize, rightearresize, nosea, noseb, tonguea,
    tongueb, lefteara, leftearb, righteara, rightearb},
  face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]];
  facedim = Flatten[FindFaces[img], 1];
  landmarks = netevaluation[face];
  dimimg = ImageDimensions@face;
  {noseresize, leftearresize, rightearresize} = 
   ImageResize[#, dimimg[[1]]*resize1] & /@ {nose, leftear, 
     rightear};
  nosea = facedim[[1, 1]] + (landmarks[[31]][[1]])*dimimg[[1]];
  noseb = facedim[[1, 2]] + (landmarks[[31]][[2]])*dimimg[[1]];
  lefteara = 
   facedim[[1, 1]] + (landmarks[[18]][[1]] - pos1)*dimimg[[1]];
  leftearb = 
   facedim[[1, 2]] + (landmarks[[18]][[2]] + pos2)*dimimg[[1]];
  righteara = 
   facedim[[1, 1]] + (landmarks[[27]][[1]] + pos1)*dimimg[[1]];
  rightearb = 
   facedim[[1, 2]] + (landmarks[[27]][[2]] + pos2)*dimimg[[1]];
  ImageCompose[
   img, {noseresize, leftearresize, 
    rightearresize}, {{nosea, noseb}, {lefteara, 
     leftearb}, {righteara, rightearb}}
   ]]

enter image description here

Multiple Faces: Bunny Faces for the entire family on Easter

The final step is to detect multiple faces, make crops around the multiple faces, detect landmarks separately for each of the crops. We need to make resize and place the filters is each of these faces. Remember that all faces in a picture may not have the same size, so we need to map/redo the resizing for each of the cropped faces. Once the filters are resized, the individual pieces of the filter needs to be placed separately at the landmarks detected for each of the face. There is some mapping, array manipulation required to make sure that each of the section of the filters is appropriately created and placed. The function for multiple faces bunny filter is as follows:

bunnyface[img_, nose_, leftear_, rightear_, resize1_, pos1_, pos2_] :=
  Block[{crops, points, facedim, landmarks, facewidth, dimimg, 
   noseresize, tongueresize, leftearresize, rightearresize, nosea, 
   noseb, tonguea, tongueb, lefteara, leftearb, righteara, 
   rightearb},
  crops = ImageTrim[img, #] & /@ FindFaces[img];
  landmarks = netevaluation /@ crops;
  facedim = FindFaces[img];
  dimimg = ImageDimensions /@ crops;
  noseresize = 
   ImageResize[nose, dimimg[[#]][[1]]*resize1] & /@ 
    Range[Length[crops]];
  leftearresize = 
   ImageResize[rightear, dimimg[[#]][[1]]*resize1] & /@ 
    Range[Length[crops]];
  rightearresize = 
   ImageResize[leftear, dimimg[[#]][[1]]*resize1] & /@ 
    Range[Length[crops]];
  nosea = 
   facedim[[#]][[1, 1]] + (landmarks[[#]][[31]][[1]])*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  noseb = 
   facedim[[#]][[1, 2]] + (landmarks[[#]][[31]][[2]])*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  lefteara = 
   facedim[[#]][[1, 1]] + (landmarks[[#]][[18]][[1]] - pos1)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  leftearb = 
   facedim[[#]][[1, 2]] + (landmarks[[#]][[18]][[2]] + pos2)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  righteara = 
   facedim[[#]][[1, 1]] + (landmarks[[#]][[27]][[1]] + pos1)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  rightearb = 
   facedim[[#]][[1, 2]] + (landmarks[[#]][[27]][[2]] + pos2)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  ImageCompose[img, 
   Flatten@{noseresize, leftearresize, rightearresize}, 
   Partition[
    Flatten@{Riffle[nosea, noseb], Riffle[lefteara, leftearb], 
      Riffle[righteara, rightearb]}, 2]]
  ]

So final step, let's check it for a family picture (found from the documentation center): enter image description here

Conclusion

Again, these are just examples on how you would code these types of Snapchat filters. These helper functions are designed in such a way that you can customize your filters (something similar to what is used by Snapchat for creating Geofilters). You can create your own filter type, with any tool you want and how much ever creative you can get. The best part of this customization is that you can have as much complicated filter you want, as long as you are creating in an appropriate background, the filter can be cropped and placed appropriately.

The last part of these series of posts would be with facial distortions, fun backgrounds etc. Any suggestions/algorithms for real time facial distortions will be greatly appreciated.

4 Replies
Posted 4 months ago

How Easter appropriate! Thanks for sharing, this is very cool.

Yeah, that's why I published it on Easter Sunday! :)

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

Posted 2 months ago

Great Post.Thanks for Sharing!

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract