Message Boards

GROUPS:

Staff Picks Data Science Graphics and Visualization Wolfram Language Machine Learning Wolfram Summer School Computational Humanities Wolfram Function Repository Neural Networks

Diya Elizabeth

[WIS22] Emotion classification from face datasets

Diya Elizabeth, Indian Institute of Science Education and Research, Thiruvananthapuram

Posted 16 days ago

292 Views

1 Reply

4 Total Likes

Follow this post

Emotion classification from face datasets
by Diya Elizabeth

In this project, we use deep learning to make an emotion recognition convolution neural network by customizing the EfficientNet model pretrained on the ImageNe dataset. We used the FER-2013 dataset available in the Wolfram repository. Seven classes of emotions are considered in the dataset: happy, sad, angry, surprise, disgust, fear, and neutral. We try out different methods to tackle class imbalance. This model can be applied in teaching autistic children, young children in general, and people blind to facial expressions. It can also be used to train robots to interact more efficiently with humans.

Data Exploration

The data set used here is FER-2013. It has 35887 greyscale images of faces with expressions categorized into one of the seven categories: happy, sad, angry, surprise, disgust, fear, neutral. It was made under all challenging conditions, such as varying lighting, different head movements, and variances in facial features due to ethnicity, age, gender, facial hair, and glasses. This is important to not create a bias in our model.

Obtaining the data

ro=ResourceObject["FER-2013"]data=ResourceData[ro];

Out[]=

ResourceObject

Name: FER-2013 »
Type: DataResource
Description: The Facial Expression Recognition 2013 (FER-2013) Dataset



Overview of the data

classes=DeleteDuplicates[Values[data]];dataPerClass=Map[Function[{class},Select[data,SameQ[Values@#,class]&]],classes];

In[]:=

Counts[Values[data]]BarChart[Counts[Values[data]],ChartLabels->Keys@Counts[Values[data]],ChartStyle->"DarkRainbow",PlotLabel->"Distribution of classes in the dataset",AxesLabel->{Automatic,"Number of images"}]

Out[]=



happiness

8989,

neutral

6198,

sadness

6077,

anger

4953,

surprise

4002,

disgust

547,

fear

5121

Out[]=

Data distribution of classes of train and test sets

We choose 4000 random images from each class for our sample dataset. 70% of this sample dataset is chosen randomly as train dataset and the other 30% as test dataset. The function RandomSample is used to ensure that the data was chosen randomly to ensure an accurate representation of the original dataset in the train and test sets.

sampleData=Flatten@Map[RandomSample[#,UpTo[4000]]&,dataPerClass];{train,test}=TakeDrop[RandomSample[sampleData],Floor[0.7*Length[sampleData]]];

In[]:=

dataPerClass=Map[Function[{class},Select[data,SameQ[Values@#,class]&]],classes];sampleData=Flatten@Map[RandomSample[#,UpTo[4000]]&,dataPerClass];{train,test}=TakeDrop[RandomSample[sampleData],Floor[0.7*Length[sampleData]]];

In[]:=

BarChart[Counts[Values[sampleData]],ChartLabels->Keys@Counts[Values[sampleData]],ChartStyle->"DarkRainbow",PlotLabel->"Distribution of classes in the sample data",AxesLabel->{Automatic,"Number of images"}]BarChart[Counts[Values[train]],ChartLabels->Keys@Counts[Values[train]],ChartStyle->"DarkRainbow",PlotLabel->"Distribution of classes in the train set",AxesLabel->{Automatic,"Number of images"}]BarChart[Counts[Values[test]],ChartLabels->Keys@Counts[Values[test]],ChartStyle->"DarkRainbow",PlotLabel->"Distribution of classes in the test set",AxesLabel->{Automatic,"Number of images"}]

Out[]=

Transfer Learning with EfficientNet

To create a neural network, rather than create a new network from scratch, which may take days or weeks to train, we can use a pretrained model as a starting point and customize it for our needs. EfficientNet trained over the dataset ‘ImageNet’ can be adapted to the ‘FER-2013’ dataset as shown below:

Creating the neural network

We start with calling EfficientNet model pretrained on the dataset ‘ImageNet’.

In[]:=

NetModel["EfficientNet Trained on ImageNet"]

Out[]=

NetChain



Input

image

array

(size: 3×224×224)

stem_conv

ConvolutionLayer

array

(size: 32×112×112)

stem_bn

BatchNormalizationLayer

array

(size: 32×112×112)

stem_activation

LogisticSigmoid[

]

array

(size: 32×112×112)

block1a

NetChain (6 nodes)

array

(size: 16×112×112)

block2a

NetChain (9 nodes)

array

(size: 24×56×56)

block2b

NetGraph (11 nodes)

array

(size: 24×56×56)

block3a

NetChain (9 nodes)

array

(size: 40×28×28)

block3b

NetGraph (11 nodes)

array

(size: 40×28×28)

block4a

NetChain (9 nodes)

array

(size: 80×14×14)

block4b

NetGraph (11 nodes)

array

(size: 80×14×14)

block4c

NetGraph (11 nodes)

array

(size: 80×14×14)

block5a

NetChain (9 nodes)

array

(size: 112×14×14)

block5b

NetGraph (11 nodes)

array

(size: 112×14×14)

block5c

NetGraph (11 nodes)

array

(size: 112×14×14)

block6a

NetChain (9 nodes)

array

(size: 192×7×7)

block6b

NetGraph (11 nodes)

array

(size: 192×7×7)

block6c

NetGraph (11 nodes)

array

(size: 192×7×7)

block6d

NetGraph (11 nodes)

array

(size: 192×7×7)

block7a

NetChain (9 nodes)

array

(size: 320×7×7)

top_conv

ConvolutionLayer

array

(size: 1280×7×7)

top_bn

BatchNormalizationLayer

array

(size: 1280×7×7)

top_activation

LogisticSigmoid[

]

array

(size: 1280×7×7)

avg_pool

AggregationLayer

vector

(size: 1280)

top_dropout

DropoutLayer

vector

(size: 1280)

predictions

LinearLayer

vector

(size: 1000)

predictions_activation

SoftmaxLayer

vector

(size: 1000)

Output

class



In order to train a new model, we can use this architecture. We remove the last Linear and Softmax layers and add our own layers as required to customize the net to our dataset.

In[]:=

net=NetTake[NetModel["EfficientNet Trained on ImageNet"],{1,-3}]

Out[]=

NetChain



Input port:	image
Output port:	vector (size: 1280)



Customizing neural network

After the last 2 layers of the model EfficientNet are removed, we construct a customized neural network by adding new layers. This allows the model to learn new features form our dataset.

In[]:=

newNet=NetChain[<|"pretrainedNet"->net,"linear1"->LinearLayer[64],"dp"->DropoutLayer[0.2],"bn"->BatchNormalizationLayer[],"linear2"->LinearLayer[7],"softmax"->SoftmaxLayer[]|>,"Output"->NetDecoder[{"Class",classes}]]

Out[]=

NetChain



uniniti

aliz

Input port:	image
Output port:	class



The layers contribute as follows:

◼

LinearLayer: Adds complexity to the neural network.

◼

DropoutLayer: Prevents overfitting by randomly putting off the neurons while training.

◼

BatchNormalizationLayer: Normalizes its input data by learning the data mean and variance.

◼

SoftMaxLayer: Contributes the activation function layer for multiclass classification.

Handling data unbalance using different methodologies

It is evident from the graphs that the emotion class ‘disgust’ has a significantly less amount of data compared to the other classes. Most machine learning algorithms are based on the assumption that data is equally distributed among all classes in the data set and hence, when training the learning develops a bias towards the majority classes. We will look at a few different ways in which we can resolve this issue.

Undersampling

In the Undersampling method, we remove excess number of samples from the majority classes so all classes end up with about the same amount of sample data. Here, the class ‘disgust’ only has 547 samples. By taking 500 samples from each class for our sample data, each of our classes end up with the same amount of data.

In[]:=

sampleData1=Flatten@Map[RandomSample[#,UpTo[500]]&,dataPerClass];BarChart[Counts[Values[sampleData1]],ChartLabels->Keys@Counts[Values[sampleData1]],ChartStyle->"DarkRainbow",PlotLabel->"Distribution of classes for the sample data",AxesLabel->{Automatic,"Number of images"}]

Out[]=

In[]:=

{train1,test1}=TakeDrop[RandomSample[sampleData1],Floor[0.7*Length[sampleData1]]];

BarChart[Counts[Values[train1]],ChartLabels->Keys@Counts[Values[train1]],ChartStyle->"DarkRainbow",PlotLabel->"Distribution of classes for the new train",AxesLabel->{Automatic,"Number of images"}]BarChart[Counts[Values[test1]],ChartLabels->Keys@Counts[Values[test1]],ChartStyle->"DarkRainbow",PlotLabel->"Distribution of classes for the new test",AxesLabel->{Automatic,"Number of images"}]

Out[]=

Training the net

The model trains with a BatchSize 24 and 2 training rounds.

In[]:=

trainedNet=NetTrain[newNet,train,ValidationSet->test,BatchSize24,MaxTrainingRounds2]

Out[]=

NetChain



Input port:	image
Output port:	class



Net evaluation

To compare how our different methods work, we shall compute the confusion matrices, accuracies, and F1 scores of each.

Confusion matrix:

In[]:=

NetMeasurements[trainedNet,test,"ConfusionMatrixPlot"]

Out[]=

actual class
	predicted class

F1 score:

In[]:=

NetMeasurements[trainedNet,test,"F1Score"]

Out[]=



happiness

0.677316,

neutral

0.440678,

sadness

0.268199,

anger

0.289256,

surprise

0.705085,

disgust

0.650602,

fear

0.349835

Accuracy:

In[]:=

NetMeasurements[trainedNet,test,"Accuracy"]

Out[]=

0.494286

Weighted Class Training

In this method, we weigh the loss computed for different samples depending on whether they belong to the majority or minor classes.

Here, we use Inverse of Number of Samples: we weigh the samples as the inverse of the class frequency for their respective classes and then normalize them over the seven classes. Sample weight,

, where c is the class umber, n is the number of samples in the batch

In[]:=

weightsOfSamples=1/Counts[Values[train]];weightsOfSamples=N[weightsOfSamples/Total[weightsOfSamples]*7]

Out[]=



neutral

0.521368,

happiness

0.517286,

anger

0.517839,

sadness

0.51363,

fear

0.52381,

surprise

0.519505,

disgust

3.88656

In[]:=

trainingWeights=Map[weightsOfSamples[#]&,Values@train];

Customizing neural network

In[]:=

weightedCrossEntropy=NetGraph[<|"time"->ThreadingLayer[Times],"loss"->CrossEntropyLossLayer["Index"]|>,{{NetPort["Weight"],"loss"}->"time"}]

Out[]=

NetGraph



Number of inputs:	3
Output port:	real



In[]:=

netWithLoss=NetGraph[{newNet,weightedCrossEntropy},{1->NetPort[2,"Input"],2->NetPort["Loss"]},"Target"->NetEncoder[{"Class",classes}]]

Out[]=

NetGraph



uniniti

aliz

Number of inputs:	3
Loss port:	real



Training the net

netTrainedWithLoss=NetTrain[netWithLoss,<|"Weight"->trainingWeights,"Input"->Keys@train,"Target"->Values@train|>,ValidationSet->Scaled[0.3],BatchSize24,MaxTrainingRounds2];trainedNet2=NetReplacePart[NetExtract[netTrainedWithLoss,1],{"Output"->NetDecoder[{"Class",classes}],"Input"->NetEncoder[{"Image",224}]}]

Out[]=

NetChain



Input port:	image
Output port:	class



Net Evaluation

Confusion matrix:

NetMeasurements[trainedNet2,test,"ConfusionMatrixPlot"]

Out[]=

actual class
	predicted class

F1 score:

NetMeasurements[trainedNet2,test,"F1Score"]

Out[]=



happiness

0.71316,

neutral

0.470343,

sadness

0.443711,

anger

0.441403,

surprise

0.652563,

disgust

0.388889,

fear

0.162015

Accuracy:

NetMeasurements[trainedNet2,test,"Accuracy"]

Out[]=

0.498982

Focal Loss Training

Focal Loss can be used when there is an extreme imbalance between the majority and minority classes. It is a modification of alpha-balanced Cross Entropy Loss. ,

, where p_t is the probability of belonging to the class (0,1) and \alpha_t is the weight for each class. Large class imbalances overwhelms cross entropy loss and dominate the gradient. Though \alpha balances the importance of positive/negative examples, it does not differentiate between easy/hard examples. We add a modulating factor, (1-p_t)^{\gamma} and a focusing parameter, \gamma to the cross entropy loss to produce the focal loss:

In[]:=

focalLoss=With[{alpha=0.25,gamma=2.0},NetGraph[{CrossEntropyLossLayer["Index"],FunctionLayer[alpha*Power[1-Exp[-#],gamma]*#&]},{{NetPort["Input"],NetPort["Target"]}->1,1->2,2->NetPort["Output"]}]]

Training the net

trainedNet3=NetTrain[newNet,train,ValidationSet->Scaled[0.2],BatchSize24,LossFunctionfocalLoss,MaxTrainingRounds2]

Out[]=

NetChain



Input port:	image
Output port:	class



Net Evaluation

Confusion matrix:

NetMeasurements[trainedNet3,test,"ConfusionMatrixPlot"]

Out[]=

actual class
	predicted class

F1 score:

NetMeasurements[trainedNet3,test,"F1Score"]

Out[]=



happiness

0.768477,

neutral

0.462827,

sadness

0.464583,

anger

0.519053,

surprise

0.710257,

disgust

0.322581,

fear

0.414187

Accuracy:

NetMeasurements[trainedNet3,test,"Accuracy"]

Out[]=

0.558316

Concluding remarks

By comparing the results from the three methods we used to address the data imbalance in the dataset, the accuracies obtained are as follow: Undersampling gives us 0.494286 accuracy, using FocalLoss gives us 0.558316 and weighted sampling, 0.498982. We can see that using FocalLoss gave us the highest accuracy and F1 scores. The weighted class model performed the next best. The accuracies could be improved in the future with more training rounds and network layers.

Keywords

◼

Undersampling

◼

Weighted sampling

◼

Focal Loss

Acknowledgment

There are many people I would like to thank for their support. I would first like to thank Siria Sadeddin for being my mentor, her immense help and guidance allowed me to learn and use many new skills which I could use to complete this project. I would also like to express my sincere gratitude to Tuseeta Banerjee and Mads Bahrami for their time and consideration, the TAs for being available whenever I was in trouble, and Aravind Hanasoge and the entire Wolfram team for conducting this program.

References

◼

Sadeddin, Siria. “Face mask detection: classifying image data.” Wolfram Community. 2021.