Message Boards

WOLFRAM COMMUNITY

14373 Views

5 Replies

13 Total Likes

View groups...

Follow this post

Share this post:

GROUPS:

Staff Picks Data Science Image Processing Recreation Visual Arts Graphics and Visualization Wolfram Language Machine Learning Neural Networks Artificial Intelligence

Bunny-face Snapchat filters for multiple faces using trained neural network

Test Account

Posted 7 years ago

Peek-a-boo, I decided to publish the bunny/dog types. In this post, we will start with how we can place dog/bunny types filter and how to extrapolate it to more than one face. Multiple face detection and filter creation is exemplified using the bunny filter. Facial Feature Detection This part of the code is same as in previous post, so I am not explaining much, however, it needs to be evaluated for the filter functions to work. netevaluation[img_] := Block[ {heatmaps, posFlattened, posMat}, heatmaps = NetModel["2D Face Alignment Net Trained on 300W Large Pose Data"][ img]; posFlattened = Map[First@Ordering[#, -1] &, Flatten[heatmaps, {{1}, {2, 3}}]]; posMat = QuotientRemainder[posFlattened - 1, 64] + 1; 1/64.Map[{#[[2]], 64 - #[[1]] + 1} - 0.5 &, posMat] ] Dog Face Filters So let's start with dog-face filters. So for this kind of filter, a dog face filter was imported from the web, and separated out the ears, tongue and nose. That involved background removal, and systematic cropping till you get them properly cropped. I am not including the code for that, because it would require you to customize depending on what filter you are using. The new catch for this filter is that we need to place the right ear, left ear, tongue, and nose at appropriate landmarks for each of the facial part. Again, setting the position of nose did not require much parametrizing, however tongue and ears required a little bit. Putting all the filter parts and the parameters and the facial features in the original image, the function looks like: dogface[img_, nose_, tongue_, leftear_, rightear_, resize1_, pos1_, pos2_, pos3_] := Block[{face, facedim, landmarks, facewidth, dimimg, noseresize, tongueresize, leftearresize, rightearresize, nosea, noseb, tonguea, tongueb, lefteara, leftearb, righteara, rightearb}, face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]]; facedim = Flatten[FindFaces[img], 1]; landmarks = netevaluation[face]; dimimg = ImageDimensions@face; {noseresize, tongueresize, leftearresize, rightearresize} = ImageResize[#, dimimg[[1]]resize1] & /@ {nose, tongue, leftear, rightear}; nosea = facedim[[1, 1]] + (landmarks[[31]][[1]])dimimg[[1]]; noseb = facedim[[1, 2]] + (landmarks[[31]][[2]])dimimg[[1]]; tonguea = facedim[[1, 1]] + (landmarks[[67]][[1]])dimimg[[1]]; tongueb = facedim[[1, 2]] + (landmarks[[67]][[2]] - pos3)dimimg[[1]]; lefteara = facedim[[1, 1]] + (landmarks[[18]][[1]] - pos1)dimimg[[1]]; leftearb = facedim[[1, 2]] + (landmarks[[18]][[2]] + pos2)dimimg[[1]]; righteara = facedim[[1, 1]] + (landmarks[[27]][[1]] + pos1)dimimg[[1]]; rightearb = facedim[[1, 2]] + (landmarks[[27]][[2]] + pos2)dimimg[[1]]; ImageCompose[ img, {noseresize, tongueresize, leftearresize, rightearresize}, {{nosea, noseb}, {tonguea, tongueb}, {lefteara, leftearb}, {righteara, rightearb}}] ] Just a disclaimer, these are randomly downloaded images from the internet. Bunny Face Filters The bunny face filters are similar to the dog face ones, except that they do not have the tongue. It had similar cropping, background removal to be done to be used to create the filters. So the function for bunny face filter is almost same as the dog face one, except the missing tongue. Here it is bunnyface[img_, nose_, leftear_, rightear_, resize1_, pos1_, pos2_] := Block[{face, facedim, landmarks, facewidth, dimimg, noseresize, tongueresize, leftearresize, rightearresize, nosea, noseb, tonguea, tongueb, lefteara, leftearb, righteara, rightearb}, face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]]; facedim = Flatten[FindFaces[img], 1]; landmarks = netevaluation[face]; dimimg = ImageDimensions@face; {noseresize, leftearresize, rightearresize} = ImageResize[#, dimimg[[1]]resize1] & /@ {nose, leftear, rightear}; nosea = facedim[[1, 1]] + (landmarks[[31]][[1]])dimimg[[1]]; noseb = facedim[[1, 2]] + (landmarks[[31]][[2]])dimimg[[1]]; lefteara = facedim[[1, 1]] + (landmarks[[18]][[1]] - pos1)dimimg[[1]]; leftearb = facedim[[1, 2]] + (landmarks[[18]][[2]] + pos2)dimimg[[1]]; righteara = facedim[[1, 1]] + (landmarks[[27]][[1]] + pos1)dimimg[[1]]; rightearb = facedim[[1, 2]] + (landmarks[[27]][[2]] + pos2)dimimg[[1]]; ImageCompose[ img, {noseresize, leftearresize, rightearresize}, {{nosea, noseb}, {lefteara, leftearb}, {righteara, rightearb}} ]] Multiple Faces: Bunny Faces for the entire family on Easter The final step is to detect multiple faces, make crops around the multiple faces, detect landmarks separately for each of the crops. We need to make resize and place the filters is each of these faces. Remember that all faces in a picture may not have the same size, so we need to map/redo the resizing for each of the cropped faces. Once the filters are resized, the individual pieces of the filter needs to be placed separately at the landmarks detected for each of the face. There is some mapping, array manipulation required to make sure that each of the section of the filters is appropriately created and placed. The function for multiple faces bunny filter is as follows: bunnyface[img_, nose_, leftear_, rightear_, resize1_, pos1_, pos2_] := Block[{crops, points, facedim, landmarks, facewidth, dimimg, noseresize, tongueresize, leftearresize, rightearresize, nosea, noseb, tonguea, tongueb, lefteara, leftearb, righteara, rightearb}, crops = ImageTrim[img, #] & /@ FindFaces[img]; landmarks = netevaluation /@ crops; facedim = FindFaces[img]; dimimg = ImageDimensions /@ crops; noseresize = ImageResize[nose, dimimg[[#]][[1]]resize1] & /@ Range[Length[crops]]; leftearresize = ImageResize[rightear, dimimg[[#]][[1]]resize1] & /@ Range[Length[crops]]; rightearresize = ImageResize[leftear, dimimg[[#]][[1]]resize1] & /@ Range[Length[crops]]; nosea = facedim[[#]][[1, 1]] + (landmarks[[#]][[31]][[1]])* dimimg[[#]][[1]] & /@ Range[Length[crops]]; noseb = facedim[[#]][[1, 2]] + (landmarks[[#]][[31]][[2]])* dimimg[[#]][[1]] & /@ Range[Length[crops]]; lefteara = facedim[[#]][[1, 1]] + (landmarks[[#]][[18]][[1]] - pos1)* dimimg[[#]][[1]] & /@ Range[Length[crops]]; leftearb = facedim[[#]][[1, 2]] + (landmarks[[#]][[18]][[2]] + pos2)* dimimg[[#]][[1]] & /@ Range[Length[crops]]; righteara = facedim[[#]][[1, 1]] + (landmarks[[#]][[27]][[1]] + pos1)* dimimg[[#]][[1]] & /@ Range[Length[crops]]; rightearb = facedim[[#]][[1, 2]] + (landmarks[[#]][[27]][[2]] + pos2)* dimimg[[#]][[1]] & /@ Range[Length[crops]]; ImageCompose[img, Flatten@{noseresize, leftearresize, rightearresize}, Partition[ Flatten@{Riffle[nosea, noseb], Riffle[lefteara, leftearb], Riffle[righteara, rightearb]}, 2]] ] So final step, let's check it for a family picture (found from the documentation center): Conclusion Again, these are just examples on how you would code these types of Snapchat filters. These helper functions are designed in such a way that you can customize your filters (something similar to what is used by Snapchat for creating Geofilters). You can create your own filter type, with any tool you want and how much ever creative you can get. The best part of this customization is that you can have as much complicated filter you want, as long as you are creating in an appropriate background, the filter can be cropped and placed appropriately. The last part of these series of posts would be with facial distortions, fun backgrounds etc. Any suggestions/algorithms for real time facial distortions will be greatly appreciated.

Peek-a-boo,

I decided to publish the bunny/dog types. In this post, we will start with how we can place dog/bunny types filter and how to extrapolate it to more than one face. Multiple face detection and filter creation is exemplified using the bunny filter.

Facial Feature Detection

This part of the code is same as in previous post, so I am not explaining much, however, it needs to be evaluated for the filter functions to work.

netevaluation[img_] := Block[
  {heatmaps, posFlattened, posMat},
  heatmaps = 
   NetModel["2D Face Alignment Net Trained on 300W Large Pose Data"][
    img];
  posFlattened = 
   Map[First@Ordering[#, -1] &, Flatten[heatmaps, {{1}, {2, 3}}]];
  posMat = QuotientRemainder[posFlattened - 1, 64] + 1;
  1/64.*Map[{#[[2]], 64 - #[[1]] + 1} - 0.5 &, posMat]
  ]

Dog Face Filters

So let's start with dog-face filters. So for this kind of filter, a dog face filter was imported from the web, and separated out the ears, tongue and nose. That involved background removal, and systematic cropping till you get them properly cropped. I am not including the code for that, because it would require you to customize depending on what filter you are using.

The new catch for this filter is that we need to place the right ear, left ear, tongue, and nose at appropriate landmarks for each of the facial part. Again, setting the position of nose did not require much parametrizing, however tongue and ears required a little bit. Putting all the filter parts and the parameters and the facial features in the original image, the function looks like:

dogface[img_, nose_, tongue_, leftear_, rightear_, resize1_, pos1_, 
  pos2_, pos3_] := 
 Block[{face, facedim, landmarks, facewidth, dimimg, noseresize, 
   tongueresize, leftearresize, rightearresize, nosea, noseb, tonguea,
    tongueb, lefteara, leftearb, righteara, rightearb},
  face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]];
  facedim = Flatten[FindFaces[img], 1];
  landmarks = netevaluation[face];
  dimimg = ImageDimensions@face;
  {noseresize, tongueresize, leftearresize, rightearresize} = 
   ImageResize[#, dimimg[[1]]*resize1] & /@ {nose, tongue, leftear, 
     rightear};
  nosea = facedim[[1, 1]] + (landmarks[[31]][[1]])*dimimg[[1]];
  noseb = facedim[[1, 2]] + (landmarks[[31]][[2]])*dimimg[[1]];
  tonguea = facedim[[1, 1]] + (landmarks[[67]][[1]])*dimimg[[1]];
  tongueb = 
   facedim[[1, 2]] + (landmarks[[67]][[2]] - pos3)*dimimg[[1]];
  lefteara = 
   facedim[[1, 1]] + (landmarks[[18]][[1]] - pos1)*dimimg[[1]];
  leftearb = 
   facedim[[1, 2]] + (landmarks[[18]][[2]] + pos2)*dimimg[[1]];
  righteara = 
   facedim[[1, 1]] + (landmarks[[27]][[1]] + pos1)*dimimg[[1]];
  rightearb = 
   facedim[[1, 2]] + (landmarks[[27]][[2]] + pos2)*dimimg[[1]];
  ImageCompose[
   img, {noseresize, tongueresize, leftearresize, 
    rightearresize}, {{nosea, noseb}, {tonguea, tongueb}, {lefteara, 
     leftearb}, {righteara, rightearb}}]
  ]

Just a disclaimer, these are randomly downloaded images from the internet.

Bunny Face Filters

The bunny face filters are similar to the dog face ones, except that they do not have the tongue. It had similar cropping, background removal to be done to be used to create the filters. So the function for bunny face filter is almost same as the dog face one, except the missing tongue. Here it is

bunnyface[img_, nose_, leftear_, rightear_, resize1_, pos1_, pos2_] :=
  Block[{face, facedim, landmarks, facewidth, dimimg, noseresize, 
   tongueresize, leftearresize, rightearresize, nosea, noseb, tonguea,
    tongueb, lefteara, leftearb, righteara, rightearb},
  face = (ImageTrim[img, #] & /@ FindFaces[img])[[1]];
  facedim = Flatten[FindFaces[img], 1];
  landmarks = netevaluation[face];
  dimimg = ImageDimensions@face;
  {noseresize, leftearresize, rightearresize} = 
   ImageResize[#, dimimg[[1]]*resize1] & /@ {nose, leftear, 
     rightear};
  nosea = facedim[[1, 1]] + (landmarks[[31]][[1]])*dimimg[[1]];
  noseb = facedim[[1, 2]] + (landmarks[[31]][[2]])*dimimg[[1]];
  lefteara = 
   facedim[[1, 1]] + (landmarks[[18]][[1]] - pos1)*dimimg[[1]];
  leftearb = 
   facedim[[1, 2]] + (landmarks[[18]][[2]] + pos2)*dimimg[[1]];
  righteara = 
   facedim[[1, 1]] + (landmarks[[27]][[1]] + pos1)*dimimg[[1]];
  rightearb = 
   facedim[[1, 2]] + (landmarks[[27]][[2]] + pos2)*dimimg[[1]];
  ImageCompose[
   img, {noseresize, leftearresize, 
    rightearresize}, {{nosea, noseb}, {lefteara, 
     leftearb}, {righteara, rightearb}}
   ]]

Multiple Faces: Bunny Faces for the entire family on Easter

The final step is to detect multiple faces, make crops around the multiple faces, detect landmarks separately for each of the crops. We need to make resize and place the filters is each of these faces. Remember that all faces in a picture may not have the same size, so we need to map/redo the resizing for each of the cropped faces. Once the filters are resized, the individual pieces of the filter needs to be placed separately at the landmarks detected for each of the face. There is some mapping, array manipulation required to make sure that each of the section of the filters is appropriately created and placed. The function for multiple faces bunny filter is as follows:

bunnyface[img_, nose_, leftear_, rightear_, resize1_, pos1_, pos2_] :=
  Block[{crops, points, facedim, landmarks, facewidth, dimimg, 
   noseresize, tongueresize, leftearresize, rightearresize, nosea, 
   noseb, tonguea, tongueb, lefteara, leftearb, righteara, 
   rightearb},
  crops = ImageTrim[img, #] & /@ FindFaces[img];
  landmarks = netevaluation /@ crops;
  facedim = FindFaces[img];
  dimimg = ImageDimensions /@ crops;
  noseresize = 
   ImageResize[nose, dimimg[[#]][[1]]*resize1] & /@ 
    Range[Length[crops]];
  leftearresize = 
   ImageResize[rightear, dimimg[[#]][[1]]*resize1] & /@ 
    Range[Length[crops]];
  rightearresize = 
   ImageResize[leftear, dimimg[[#]][[1]]*resize1] & /@ 
    Range[Length[crops]];
  nosea = 
   facedim[[#]][[1, 1]] + (landmarks[[#]][[31]][[1]])*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  noseb = 
   facedim[[#]][[1, 2]] + (landmarks[[#]][[31]][[2]])*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  lefteara = 
   facedim[[#]][[1, 1]] + (landmarks[[#]][[18]][[1]] - pos1)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  leftearb = 
   facedim[[#]][[1, 2]] + (landmarks[[#]][[18]][[2]] + pos2)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  righteara = 
   facedim[[#]][[1, 1]] + (landmarks[[#]][[27]][[1]] + pos1)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  rightearb = 
   facedim[[#]][[1, 2]] + (landmarks[[#]][[27]][[2]] + pos2)*
       dimimg[[#]][[1]] & /@ Range[Length[crops]];
  ImageCompose[img, 
   Flatten@{noseresize, leftearresize, rightearresize}, 
   Partition[
    Flatten@{Riffle[nosea, noseb], Riffle[lefteara, leftearb], 
      Riffle[righteara, rightearb]}, 2]]
  ]

So final step, let's check it for a family picture (found from the documentation center):

Conclusion

Again, these are just examples on how you would code these types of Snapchat filters. These helper functions are designed in such a way that you can customize your filters (something similar to what is used by Snapchat for creating Geofilters). You can create your own filter type, with any tool you want and how much ever creative you can get. The best part of this customization is that you can have as much complicated filter you want, as long as you are creating in an appropriate background, the filter can be cropped and placed appropriately.

The last part of these series of posts would be with facial distortions, fun backgrounds etc. Any suggestions/algorithms for real time facial distortions will be greatly appreciated.

POSTED BY: Test Account