Message Boards Message Boards

Identify emotions from a drawn face using machine learning?

Posted 5 years ago

Can you advise me what kind of machine learning to achieve? ... I had thought, for example, to make a face and, by varying the expression through the "manipulate" function (I do not know if it is possible to do so), the machine will tell me if it is sad, happy, etc. This is the code that allows me to manipulate the face:

Manipulate[
 eyeMat = {{1/(eyeRadius - pupilRadius/2), 0}, {0, 1/(0.15 + eyes - pupilRadius/2)}};
 If[Norm[eyeMat.(pup - eyeCenter[[left]])] < 1, 
  pupNow = pup - eyeCenter[[left]];];
 If[Norm[eyeMat.(pup - eyeCenter[[right]])] < 1, 
  pupNow = pup - eyeCenter[[right]];];
 Graphics[{face, eye[left, eyes], eye[right, eyes], Blue, 
   pupil[left, pupNow, eyes], pupil[right, pupNow, eyes], Black, 
   browDraw[left, brows, eyes], browDraw[right, brows, eyes], 
   Inset[mouthDraw[mouth], {0, -0.5}]}, 
  ImageSize -> {400, 450}], {{brows, {-Pi/20, 0}}, {-0.6, 0}, {0.6, 
   0.15}, ControlPlacement -> Left}, {{eyes, 0}, -0.07, 0.07, 
  ControlPlacement -> Left, 
  ControlType -> VerticalSlider}, {{mouth, 0.15}, -0.401, 0.4, 0.01, 
  ControlPlacement -> Left, 
  ControlType -> VerticalSlider}, {{pup, {0, 0}}, Locator, 
  Appearance -> None}, 
 Initialization :> (face = Circle[{0, 0}, {1, 1.2}]; pupNow = {0, 0};
   left = 1; right = 2;
   eyeRadius = 0.18; eyeCenter = {{-0.4, 0.15}, {0.4, 0.15}};
   pupilRadius = 0.09;
   browUp = 0.25; browW = 0.2; browAng = Pi/20;
   eye[side_, eccen_] := {Black, 
     Circle[eyeCenter[[side]], {eyeRadius + 0.05, eyeRadius + eccen}]};
   pupil[side_, pup_, 
     eccen_] := {Disk[eyeCenter[[side]] + pup + {0, pup[[2]] eccen}, 
      pupilRadius + Max[0, eccen/3]], Black, 
     Disk[eyeCenter[[side]] + pup + {0, pup[[2]] eccen}, 0.03]};
   browDraw[side_, {browAng_, browLift_}, eccen_] := 
    Rotate[{Thickness[0.01], 
      Line[{{eyeCenter[[side]] + {-browW, 
           browLift + browUp + 0.5 eccen}, 
         eyeCenter[[side]] + {browW, 
           browLift + browUp + 0.5 eccen}}}]}, 2 (side - 1.5) browAng];
   mouthDraw[s_] := 
    ParametricPlot[{Cos[u], -s Sin[u]}, {u, Pi/6, Pi - Pi/6}, 
     Axes -> False, PlotStyle -> {Red, Thickness[0.02]}, 
     PlotRange -> All];)]

Now, How can I teach the machine to say what kind of expression the face has (sad, happy, etc.)? Help please.

POSTED BY: Pasquale Rossi
5 Replies

Making the interface using the Chernoff faces package.

Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/ChernoffFaces.m"]

Manipulate[
 Column[{g = 
    ChernoffFace[<|"ForeheadShape" -> face[[1]], 
      "EyesVerticalPosition" -> face[[2]], "EyeSize" -> eyes[[1]], 
      "EyeSlant" -> eyes[[2]], "LeftEyebrowSlant" -> brows[[2]], 
      "LeftEyebrowTrim" -> brows[[1]], "MouthSmile" -> mouth[[2]], 
      "MouthWidth" -> mouth[[1]]|>, ImageSize -> {450, 450}], 
   Classify["FacialExpression", g]}],
 {{face, {0.5, 0.5}}, {0, 0}, {1, 1}, ControlPlacement -> Left},
 {{eyes, {0.5, 0.5}}, {0, 0}, {1, 1}, ControlPlacement -> Left},
 {{brows, {0.5, 0.5}}, {0, 0}, {1, 1}, ControlPlacement -> Left},
 {{mouth, {0.5, 0.5}}, {0, 0}, {1, 1}, ControlPlacement -> Left}
 ]

enter image description here

POSTED BY: Anton Antonov
Posted 5 years ago

The in-build facial expression classifier was trained on human faces so it does not do a good job on simple drawings. I changed the displayed part of the Manipulate to

Column[{g = 
   Graphics[{face, eye[left, eyes], eye[right, eyes], Blue, 
     pupil[left, pupNow, eyes], pupil[right, pupNow, eyes], Black, 
     browDraw[left, brows, eyes], browDraw[right, brows, eyes], 
     Inset[mouthDraw[mouth], {0, -0.5}]}, ImageSize -> {400, 450}], 
  Classify["FacialExpression", g]}]

and Increased the thickness of the lips and eyebrows. Here some examples of the results.

To get better results you would have to generate a dataset of classified drawings and train a neural network on them.

enter image description here

enter image description here

enter image description here

POSTED BY: Rohit Namjoshi
Posted 5 years ago

Thanks for the help, could you tell me how I could make a DataSet?

POSTED BY: Pasquale Rossi
Posted 5 years ago

One option is to add a dropdown selection control to the manipulate that is populated with the various emotions you want to classify, e.g. Happy, Sad, Afraid, Angry, Neutral..., and a Save button that exports the Graphic to a file in a folder named after the selected emotion. Then run the Manipulate, adjust the facial feature controls, select the appropriate emotion from the dropdown and click the save button. Repeat until you have several examples for each emotion. Then use maybe 80% of the classified data for training a neural network and the remaining 20% for testing. Consult the documentation to learn how to build and train a network.

POSTED BY: Rohit Namjoshi
Posted 5 years ago

Could you please kindly illustrate me something? .... I do not know where to start.

POSTED BY: Pasquale Rossi
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract