Marco,

I am quite impressed with the automation you managed to attain in that classification problem. You were handed lemons, and made, well, limes. Though I would have preferred lemonade, I am impressed all the same. Thanks for showing that machine learning example along with the image processing and the like that went with it.

I'll take epsilon credit for having provided some motivation to put in the work you did. I am hoping people take note of the methods and that it becomes time well spent.

Dear Luis,

I am afraid Daniel is quite right: They are limes. In an attempt to reproduce Daniel's thought process, I have used Mathematica 10's new machine learning feature to mimic this. I downloaded 50 images for a number of fruits from google and saved them in directories on my desktop. I then made lists of images:

bananas = Import["~/Desktop/Fruits/bananas/" <> #] & /@ Import["~/Desktop/Fruits/bananas/"];
apples = Import["~/Desktop/Fruits/apples/" <> #] & /@ Import["~/Desktop/Fruits/apples/"];
pears = Import["~/Desktop/Fruits/pears/" <> #] & /@ Import["~/Desktop/Fruits/pears/"];
cherries = Import["~/Desktop/Fruits/cherries/" <> #] & /@ Import["~/Desktop/Fruits/cherries/"];
strawberries = Import["~/Desktop/Fruits/strawberries/" <> #] & /@ Import["~/Desktop/Fruits/strawberries/"];
grapes = Import["~/Desktop/Fruits/grapes/" <> #] & /@ Import["~/Desktop/Fruits/grapes/"];
pineapples = Import["~/Desktop/Fruits/pineapples/" <> #] & /@ Import["~/Desktop/Fruits/pineapples/"];
melons = Import["~/Desktop/Fruits/melons/" <> #] & /@ Import["~/Desktop/Fruits/melons/"];
avocados = Import["~/Desktop/Fruits/avocados/" <> #] & /@ Import["~/Desktop/Fruits/avocados/"];
limes = Import["~/Desktop/Fruits/limes/" <> #] & /@ Import["~/Desktop/Fruits/limes/"];
lemons = Import["~/Desktop/Fruits/lemons/" <> #] & /@ Import["~/Desktop/Fruits/lemons/"];
oranges = Import["~/Desktop/Fruits/oranges/" <> #] & /@ Import["~/Desktop/Fruits/oranges/"];

I then trained my classifier:

  c = Classify[
       Flatten[{
# -> "banana(s)" & /@ bananas,
# -> "apple(s)" & /@ apples, 
# -> "pear(s)" & /@ pears, 
# -> "cherries(s)" & /@ cherries, 
# -> "strawberrie(s)" & /@ strawberries, 
# -> "grape(s)" & /@ grapes, 
# -> "pineapple(s)" & /@ pineapples, 
# -> "melon(s)" & /@ melons, 
# -> "avocado(s)" & /@ avocados, 
# -> "limes(s)" & /@ limes, 
# -> "lemon(s)" & /@ lemons, 
# -> "orange(s)" & /@ oranges}]];

For 50 images each this takes quite a while. You might want to start with only limes and lemons which are the critical items here:

c = Classify[Flatten[{# -> "limes(s)" & /@ limes, # -> "lemon(s)" & /@ lemons}]];

I then cropped the unknown fruit out of your image, which I called "gridimage" and classified it:

c[ImageTrim[gridimage, {{94.5`, 348.5`}, {9.5`, 442.5`}}]]

The result is:

"limes(s)"

so I am now quite sure that Daniel is in fact right. How sure am I? Well,

c[ImageTrim[gridimage, {{94.5`, 348.5`}, {9.5`, 442.5`}}], "Probabilities"]

gives

<|"lemon(s)" -> 6.28905*10^-9, "limes(s)" -> 1.|>

So I guess we can be really sure that Daniel is right. The next step is quite obvious: We need to cut the original image into its constituents and then apply the machine learning to recognise what fruit it is. We should then be able to automise the entire problem, so that next time nobody needs to type in the matrix by hand. :-)

Cheers, Marco

PS: That happens when someone has too much time on their hands. The training data is about 7MB. If anyone wants to try that out for themselves, look me up my community profile and send me an email.

If anyone is interested I also made the grid myself with images from google:

rules = {"banana" -> a, "strawberry" -> b, "grapes" -> c, 
  "apple" -> d, "pineapple" -> e, "pear" -> f, "melon" -> g, 
  "avocado" -> h, "cherry" -> i, "lime" -> j};

matrix = Flatten[{{{a}, {b}, {c}, {d}, {e}, {f}}, {{g}, {h}, {i}, \
{b}, {f}, {d}}, {{h}, {f}, {h}, {f}, {h}, {f}}, {{e}, {i}, {b}, {a}, \
{g}, {c}}, {{j}, {j}, {j}, {j}, {j}, {j}}, {{d}, {a}, {e}, {i}, {a}, \
{b}}, {{f}, {g}, {c}, {b}, {g}, {i}}, {{e}, {h}, {a}, {i}, {f}, \
{c}}}, {1, 3}];
A = {26, 26, 24, 22, 36, 30, 19, 23};
B = {35, 38, 36, 34, 31, 32};

These two commands get you the images:

result = Import["https://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=\" <> #, "JSON"] & /@ rules[[All, 1]];

imgs = ImageResize[#, {100, 100}] & /@ Table[Import["url" /. ("results" /. ("responseData" /. result[[m]]))[[1]]], {m,1, Length[rules]}]

I use the google api to get them. Problem is if I want to show you the result I probably infringe half a dozen times the copy-right laws. So I have modified the first of the two commands:

result = Import["https://ajax.googleapis.com/ajax/services/search/images?v=1.0&as_rights=cc_publicdomain&q=" <> #, "JSON"] & /@ rules[[All, 1]];

This gives me only public domain images. If I then use

Grid[Transpose[Append[Transpose[Append[(matrix /. rulesimgs), B]], Append[A, ""]]],Frame -> All]

I get the following image:

You will notice that the apple is a bit weird. If you execute the non-public domain version of the command you get a proper apple.

Dear Sam,

thank you very much. I checked which method is uses: it is NearestNeighbors.

Thanks again, Marco

Regarding the colour of a lemon... what about using the images I got from Google (see post below!) and then use either

Graphics[{RGBColor[FindClusters[Flatten[Table[DominantColors[RemoveBackground[lemons[[q]]]], {q, 1, 72}]] /. RGBColor -> List][[1]] // Mean], Rectangle[{0, 0}, {253, 199}]}]

which gives an RGB of {0.946313, 0.80004, 0.151472} which is

or

color = Mean[FindClusters[Flatten[DominantColors[#, 10, Masking -> ColorNegate@RegionBinarize[#, {{1, 1}}, .1]] & /@ lemons] /. RGBColor -> List][1]]

which gives RGB {0.900287, 0.724746, 0.108707} which is

The main difference is how I remove the background. The first version uses the new RemoveBackground function and gives things like:

The second one uses the older version to mask the background. That looks like this:

which seems to do a bit worse. None of this is really good, because the background removal is not perfect yet and also the lighting is very different in the individual images. It might also be difficult to try to find a mathematical answer to the question what is the colour of a lemon anyway.

Anyway, this colour is quite different from the one of the limes.

Mean[FindClusters[Flatten[Table[DominantColors[RemoveBackground[limes[[q]]]], {q, 1, Length[limes]}]] /. RGBColor -> List][[2]]]

which gives {0.356328, 0.491976, 0.139417} or

Graphics[{RGBColor[Mean[FindClusters[Flatten[Table[DominantColors[RemoveBackground[limes[[q]]]], {q, 1, Length[limes]}]] /. RGBColor -> List][[2]]]], Rectangle[{0, 0}, {253, 199}]}]

which gives

Cheers,

Marco

Yes, sorry. I am a total theoretician. In my head they were neither limes nor lemons, just "j".

;-)

Marco

Hi,

rules = {"banana" -> a, "strawberry" -> b, "grapes" -> c,  "apple" -> d, "pineapple" -> e, "pear" -> f, "melon" -> g , 
  "avocado" -> h, "cherry" -> i, "lemon" -> j}

the matrix of fruits is:

matrix = Flatten[{{{a}, {b}, {c}, {d}, {e}, {f}}, {{g}, {h}, {i}, \
{b}, {f}, {d}}, {{h}, {f}, {h}, {f}, {h}, {f}}, {{e}, {i}, {b}, {a}, \
{g}, {c}}, {{j}, {j}, {j}, {j}, {j}, {j}}, {{d}, {a}, {e}, {i}, {a}, \
{b}}, {{f}, {g}, {c}, {b}, {g}, {i}}, {{e}, {h}, {a}, {i}, {f}, \
{c}}}, {1, 3}]

We then need the sums at the right and the bottom:

A = {26, 26, 24, 22, 36, 30, 19, 23};
B = {35, 38, 36, 34, 31, 32};

Up to now it was just setting up the problem. Here is the solution:

Solve[Flatten[{Total /@ matrix == A, (Total /@ Transpose[matrix]) ==B}], {a, b, c, d, e, f, g, h, i, j}] /. (Reverse /@ rules)

banana->8, strawberry->3, grapes->5, apple->9, pineapple->0, pear->1, melon->4, avocado->7, cherry->2, lemon->6

I might have made a mistake typing this. It seems to be more an exercise in typing than in Mathematica. It would certainly have helped to have your notebook with the matrix in it.

Cheers, Marco