Suggestion algorithms trained on passed personal preferences are extensively used by various automated assistants. Their purpose could be, for example, marketing of the best possible product choices based on previous purchases, suggestion of new places to visit, semantically organizing photo albums, and many other similar tasks. One of the cornerstones of such algorithms is semantic image distance. The project goal will be building such a measure with help of Wolfram Language neural network tools and dictionary labels with hierarchical semantic structure.

High level operation of the application is the following.

1. Recognize produced images and get the probability of each recognized result.
2. Get the leaf vertices of the knowledge graph which corresponds to recognized image results.
3. Pick the top 10 most probable result vertices of each image and calculate the distance of each vertex of the first image with vertices of the second image.
4. Multiply the distance result of the step 3 with their corresponding probabilities.
5. Sum all the results from step 4 to get overall distance between images.

For the knowledge graph a subset of WorldNet graph database has been used. The following code is responsible to fetch and prune the WorldNet graph database.

    createWordnetGraph[parents_]:= Module[{edges},
        edges = (First@# -> Last@#)& /@parents;
        edges = DeleteCases[edges, Null];
        Graph[edges]
    ];
    generateGraph[]:=Module[
        {wordnetParents,wordnetgraph,networkGraph,ids,currentSynset,extendedGraph},
        wordnetParents=Import["http://www.image-net.org/archive/wordnet.is_a.txt","Data"];
        wordnetgraph=createWordnetGraph[wordnetParents]

    (*This graph has been further pruned using the list of synsets known to ImageIdentify.
    The code to do that is internal and cannot be published here. *)

    ];

    $networkGraph=generateGraph[];
    $undirectedNetworkGraph=Graph[VertexList[$networkGraph],EdgeList[$networkGraph]/.DirectedEdge->UndirectedEdge];

At the bottom you can see the diagram of graph that is being used to find out the distances between images.

SetProperty[$undirectedNetworkGraph,
    {EdgeStyle->Directive[White,Opacity[.8]],
    Background->Black,
    VertexStyle->Directive[Red,Opacity[.5]],
    VertexSize->.001}]

The following code is responsible to calculate the distance between two vertices of the graph. First it finds the common ancestor of two vertices, then it sums the distance between first vertex and common ancestor to the second vertex and common ancestor to get the distance between two vertices.

calculateDistance[vertex1_, vertex2_]:=Module[{id1,id2,formatedVertex1,formatedVertex2,in1,in2,commanVertices,firstCommanVertex,commonToVertex1Distance,commonToVertex2Distance,distance},

in1=VertexInComponent[$networkGraph,vertex1];
in2=VertexInComponent[$networkGraph,vertex2];
commanVertices=Cases[in1,x_/;MemberQ[in2,x]];
If[Length[commanVertices]>0,(
    firstCommanVertex=First[MaximalBy[commanVertices,Length@VertexInComponent[$networkGraph,#]&]];
    commonToVertex1Distance=GraphDistance[$undirectedNetworkGraph,vertex1, firstCommanVertex];
    commonToVertex2Distance=GraphDistance[$undirectedNetworkGraph,vertex2, firstCommanVertex];
    distance=commonToVertex1Distance+commonToVertex2Distance;
    distance
    ),
Infinity]

];

The picDistance function is the main function of the application. It returns the distance between two images. The smaller the return value the more similar the images are. It uses the above calculateDistance function to calculate the distances between vertices. Then it multiplies the distances with the probabilities.

picDistance[p1_,p2_]:=Module[{result1,result2,result1Keys,result2Keys,result1Values,result2Values,distance},

(*result1 and result2 contain associations of vertices and their correspondent probabilities for a given image.
Internal tools has been used to obtain theses values.*)

result1=Sort[result1,Greater];
result2=Sort[result2,Greater];
result1=result1[[1;;10]];
result2=result2[[1;;10]];
result1Keys=Keys[result1];
result2Keys=Keys[result2];
result1Values=Values[result1];
result2Values=Values[result2];

distance=0;
itterFunc[result1Item_,result1ItemIndex_]:=MapIndexed[
Set[distance,distance+calculateDistance[result1Item, #1]*result1Values[[result1ItemIndex]]*result2Values[[First[#2]]]]
&,result2Keys];

MapIndexed[itterFunc[#1,First[#2]]&,result1Keys];

If[distance==0,Infinity,distance]
];

The following code is a simple test case for picDisttance function. Notice the distance between pictures of the dog and the cat is much higher than the pictures of two cats.

Here is simple demonstration of application. It compares picture of a wolf with 5 random images.

Attachments: