Message Boards

GROUPS:

Staff Picks Biological Sciences Data Science Image Processing Medical Sciences Wolfram Language Machine Learning

Siria Sadeddin

Face mask detection: classifying image data

Siria Sadeddin, Wolfram Research Inc.

Posted 8 months ago

2327 Views

2 Replies

12 Total Likes

Follow this post

MODERATOR NOTE: coronavirus resources & updates: https://wolfr.am/coronavirus

Siria Sadeddin

The use of face mask have become an important topic in recent days, automatic face mask detection can help to prevent the spread of COVID-19 in public spaces, detecting people who is not wearing a mask. Image classification using Convolutional Neural Networks and pretrained models have shown high performance on a broad of different problems, and more recently when detecting the use of face mask on a person. We will use a pretrained Convolutional Neural Network for identifying wether a person is using or not a face mask based on a cropped image of their face. For this project, ResNet-50 pretrained on ImageNet competition data has been joined to a 300 linear layer network and trained on a balanced dataset of 1000 face images. This model showed 99% accuracy on the validation data which consisted of about 800 face images.

Data Exploration

We will use a data set containing 12000 cropped images of faces wearing mask and not wearing mask. The public data set is available here:
https://www.kaggle.com/ashishjangra27/face-mask-12k-images-dataset/activity.

This data set is well balanced with the following statistics:

◼

Train: 10000 training images (5000 with mask/without mask )

◼

Test: ~1000 tests images (500 with mask/without mask )

◼

Validation: ~800 validation (400 with mask/without mask )

Each of this folders contain two subfolders:

◼

WithMask (images of faces wearing a mask)

◼

WithoutMask (images of faces not wearing a mask)

The images will look as follows:

In[]:=

Data load

In[]:=

dir="C:/Users/ECF0124A/Downloads/Face Mask Dataset";

We want to create separated datasets for train, test and validate the model. In order to do this, we wont load all the images (it is a lot of data), instead of that, we will provide the model the image files paths associated with their corresponding class.
For the data loading we will create a function named loadFiles, this function will create an association map between the files and the corresponding image class (WithMask or WithoutMask).

In[]:=

loadFiles[dir_]:=Map[File[#]FileNameTake[#,{-2}]&,FileNames["*.png",dir,Infinity]];

Applying loadFiles function over the Train, Test and Validation folders we will obtain the association maps for each of this datasets:

In[]:=

trainingData=loadFiles[FileNameJoin[{dir,"Train"}]];testData=loadFiles[FileNameJoin[{dir,"Test"}]];valData=loadFiles[FileNameJoin[{dir,"Validation"}]];

View of some images

Lets view some of the images inside the Validation folder using the ImageCollage function:

In[]:=

ImageCollage[Map[Import,RandomSample[FileNames["*.png",FileNameJoin[{dir,"Validation"}],Infinity],75]],Automatic,{600,200}]

Out[]=

We can see the dataset has cropped squared images of faces with different resolution and face position, we observe it has different people ethnicity, gender and ages (this is important when training models based on peoples images, we don’t want to have to have those kind of bias in our model ). It also includes some comics, this can give the model the possibility of classifying comics too.

Data distribution of classes (with mask/without mask) of train, test and validation sets

We will check the data distribution of classes of each dataset, making sure the data is balanced as we expected it to be:

Train data distribution of classes

Out[]=

Test data distribution of classes

Out[]=

Validation data distribution of classes

Out[]=

We were able to verify that the datasets are balanced for train, test and validation.

Data Randomization

We make sure the data is randomized, in that way, each batch will have equally distributed classes and the learning process will be smooth. The function RandomSample will be used for this task.

In[]:=

trainingDataRandomized=trainingData//RandomSample;testDataRandomized=testData//RandomSample;valDataRandomized=valData//RandomSample;

Create the Neural Network

We will start with a ResNet-50 pretrained model, this model was trained over a dataset called ImageNet http://www.image-net.org/ which contains more than 1 million images and 1000 different classes. The resulting model weights can be used for other computer vision tasks, allowing the new models to learn faster.

In[]:=

NetModel["ResNet-50 Trained on ImageNet Competition Data"]

Out[]=

NetChain



Input port:	image
Output port:	class



From this architecture we will need (in order to train a new model with a different number of dataset classes) to remove the last LinearLayer and SoftmaxLayer, those layers correspond to the ImageNet dataset number of classes. Later we will add our own customized layers corresponding to the number of classes in our dataset (2 for this specific task).

Pretrained model

We first import the pretrained ResNet-50 model and remove the last 2 layers:

In[]:=

tempNet=Take[NetModel["ResNet-50 Trained on ImageNet Competition Data"],{1,-3}]

Out[]=

NetChain



Input port:	image
Output port:	vector (size: 2048)



Customized neural network

The pretrained model ResNet-50 is called and the last 2 layers are removed, allowing the construction of a customized neural network and define the number of classes needed for this specific task.
After the convolutional pretrained ResNet-50 Dropout layers and LinearLayers are added. This allows the model learning new features apart of the transferred ones (the pretrained weights) and get a more complex architecture. Each of the new layers contribute as follows:

◼

LinearLayer: Adds complexity to the neural network

◼

DropoutLayer: Prevents overfitting by randomly putting off the neurons while training

◼

SoftMaxLayer: Activation function layer for the classification task

In[]:=

newNet=NetChain[<|"pretrainedNet"tempNet,"Dropout1"->DropoutLayer[0.2],"linear"LinearLayer[300],"Dropout2"->DropoutLayer[0.2],"linear2"LinearLayer[2],"softmax"SoftmaxLayer[]|>,"Output"NetDecoder[{"Class",{"WithMask","WithoutMask"}}]]

Out[]=

NetChain



uniniti

aliz

Input port:	image
Output port:	class



Train the model

The model will be trained with a BatchSize of 16 and 2 rounds, as metric monitoring we use: “ConfusionMatrixPlot”,”Accuracy”,”Precision” and ”Recall”.

In[]:=

trainedNet=NetTrain[newNet,trainingDataRandomized,ValidationSettestDataRandomized,TrainingProgressMeasurements{"ConfusionMatrixPlot","Accuracy","Precision","Recall"},BatchSize16,MaxTrainingRounds2]

Net evaluation

Different model evaluation metrics shows the performance of the model over the validation data:

Confusion matrix:

In[]:=

NetMeasurements[trainedNet,valDataRandomized,"ConfusionMatrixPlot"]

Out[]=

actual class
	predicted class

ROC curve:

In[]:=

NetMeasurements[trainedNet,valDataRandomized,"ROCCurvePlot"]

Out[]=

recall
	false positive rate

Accuracy:

NetMeasurements[trainedNet,valDataRandomized,"Accuracy"]

Out[]=

0.9875

F1 Score:

NetMeasurements[trainedNet,valDataRandomized,"F1Score"]

Out[]=

WithMask0.987531,WithoutMask0.987469

Precision and recall:

NetMeasurements[trainedNet,valDataRandomized,{<|"Measurement""Precision","ClassAveraging""Macro"|>,<|"Measurement""Recall","ClassAveraging""Macro"|>}]

Out[]=

{0.987512,0.9875}

Basic usage:

Guess face mask use of a person from a photograph:

Example 1:

We will import one image of a person using a face mask from the validation set and apply the trainedNet over it:

Importing the image of a person with face mask:

In[]:=

imgMask=Import[FileNames["*.png",FileNameJoin[{dir,"Validation","WithMask"}],Infinity][[1]]]

Out[]=

Applying the trained net over the image:

In[]:=

trainedNet



Out[]=

WithMask

Obtain the probabilities:

In[]:=

trainedNet

,"Probabilities"

Out[]=

WithMask0.639089,WithoutMask0.360911

Example 2:

We will import one image of a person that is not using a mask from the validation set and apply the trainedNet over it:

Importing the image of a person without face mask :

In[]:=

imgNoMask=Import[FileNames["*.png",FileNameJoin[{dir,"Validation","WithoutMask"}],Infinity][[1]]]

Out[]=

Applying the trained net over the image:

In[]:=

trainedNet



Out[]=

WithoutMask

Obtain the probabilities:

In[]:=

trainedNet

,"Probabilities"

Out[]=

WithMask3.3725×

-10

,WithoutMask1.

Applications

We will now test the model over images that are not included in any of the train, test or validation sets and show how the trained model can be used for any persons image, specially when the face or faces are not cropped.

Example 1:

This net was designed to work with cropped images of faces. If the photograph is not a cropped image of a face, the results may be unexpected:

In[]:=

img=

;

Applying the trained net over the image:

In[]:=

trainedNet[img]

Out[]=

WithMask

Crop the photograph:

In[]:=

crop=ImageTrim[img,#]&/@FindFaces[img]

Out[]=





Guess the use of mask of a person from the cropped image:

In[]:=

trainedNet[crop]

Out[]=

{"WithoutMask"}

Example 2:

We can use the trained model on pictures with more than one face, on the next example we have a picture with two people using a face mask, we first find the faces and crop them, then we use the function Map to apply the trainedNet over each of the faces:

In[]:=

img2=

;

In[]:=

crop2=ImageTrim[img2,#]&/@FindFaces[img2]

Out[]=





In[]:=

Map[trainedNet,crop2]

Out[]=

{WithMask,WithMask}

Example 3:

Again, we use the trained model on pictures with more than one face, on the next example we have a picture with two people, one use the face mask and the other don’t, we first find the faces and crop them, then we use the function Map to apply the trainedNet over each of the faces:

In[]:=

img3=

;

In[]:=

crop3=ImageTrim[img3,#]&/@FindFaces[img3]

Out[]=





In[]:=

Map[trainedNet,crop3]

Out[]=

{WithoutMask,WithMask}

Results

We have seen how transfer learning with ResNet-50 architecture can be used to solve a computer vision task and recognize the use of face mask form an cropped image of a face. Training over a dataset of 10000 images we were capable of obtaining a model with 99% of precision and recall over the validation data set, we also tested the model over a few random images taken from the internet, we noticed however that the model can fail when the resolution of the image is low.

POSTED BY: Siria Sadeddin

Answer

2 Replies

Sort By:

Moderation Team

Moderation Team, WOLFRAM

Posted 8 months ago

-- you have earned *Featured Contributor Badge* Your exceptional post has been selected for our editorial column *Staff Picks* http://wolfr.am/StaffPicks and Your Profile is now distinguished by a *Featured Contributor Badge* and is displayed on the Featured Contributor Board. Thank you!

POSTED BY: Moderation Team

Answer

Giulio Alessandrini

Giulio Alessandrini, Wolfram Research Inc.

Posted 8 months ago

Very nice fine tuning! One small bit of code simplification could be FindFaces[image, "Image", PaddingSize -> Scaled[0.1]] instead of trimming in a separate steps. This way you can also add an arbitrary padding taken from the original image. You can even take advantage of the built-in listability of the NN framework to write trainedNet[FindFaces[image, "Image"]] that will evaluate the net in batch mode and will be faster when `image` contains multiple faces.

POSTED BY: Giulio Alessandrini

Answer

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Group Abstract

Feedback