Parental kinship recognition

Introduction

Facial images convey many important human characteristics, such as identity, gender, expression, age, and ethnicity. Compared with these face analysis tasks, facial kinship verification is a relatively new research topic in face analysis and only some attempts have been made over the past few years. However, this new research topic has several potential applications such as family album organization, image annotation, social media analysis, and missing children/parents search.

Objective

The objective of the project I realized in the Wolfram Summer School 2018 was to identify parental relationships using faces photographs.

Dataset

The database used is the KinFaceW includes around 50 % Caucasians, 40 % Asians, 7 % African Americans, and 3 % others. KinFaceW-II contains 250 pairs for each category ("Father-Son","Father-Daughter","Mother-Son","Mother-Daughter").

fatherSon = Import["Assets/KinFaceW-II/images/father-son/fs*"];
fatherDaughter = Import["Assets/KinFaceW-II/images/father-dau/fd*"];
motherSon = Import["Assets/KinFaceW-II/images/mother-son/ms*"];
motherDaughter = Import["Assets/KinFaceW-II/images/mother-dau/md*"];

Augmented data

The following function was applied to crop the faces and augment the dataset.

createAugmentedImages[image_Image]:=Map[ImageCrop[image,50,#]&,{Left,Right,Bottom,Top,Center}]

Feature extraction

I use ResNet 101 feature extractor to build a multilayer transfer learning neural network.

convP = NetModel["ResNet-101 Trained on Augmented CASIA-WebFace Data"];

Features were extracted before the training in order to reduce the computational usage.

fmodel = NetGraph[{convP, convP, 
                                CatenateLayer[]}, 
                                {NetPort["Parent"] -> 1 -> 3, NetPort["Child"] -> 2 -> 3}];

lnet = NetInitialize[fmodel] 

vectorsAugmentedFatherSon = lnet[dataAugmentedFatherSon];
vectorsAugmentedFatherDau = lnet[dataAugmentedFatherDau];
vectorsAugmentedMotherSon = lnet[dataAugmentedMotherSon];
vectorsAugmentedMotherDau = lnet[dataAugmentedMotherDau];

Neural Network

vectorsNet = NetGraph[{LinearLayer[256], LogisticSigmoid , LinearLayer[], 
                                    SoftmaxLayer[]}, {NetPort["Input"] -> 1, 1 -> 2 -> 3 -> 4}, 
                                    "Input" -> {4096}, 
                                   "Output" -> NetDecoder[{"Class", 
                                                            {"FatherSon", "FatherDau", "MotherDau", "MotherSon"}}]];
vectorsNet = NetInitialize[vectorsNet]

fmodelValidationAugmented = NetGraph[{convP, convP, CatenateLayer[], 
                                                                vectorModelAugmented}, {NetPort["Parent"] -> 1 -> 3, NetPort["Child"] -> 2 -> 3, 3 -> 4}, 
                                                               "Output" ->NetDecoder[{"Class",
                                                                                     {"FatherSon", "FatherDau", "MotherDau", "MotherSon"}}]];

netValidationAugmented = NetInitialize[fmodelValidationAugmented]

After 100 rounds of trainings the results are:

cm = ClassifierMeasurements[netValidationAugmented, test]
cm["Accuracy"]
cm["ConfusionMatrixPlot"]

An accuracy of 87% was achieved, that is not bad for state of the art kinship recognition algorithms.

Finally, I did an extra validation just to check that the trained net is working.

Future work

• Increase the dataset in order to obtain better results.
• Train the model for siblings and grandparents
• Determine the best traits for classification.