First of all many thanks to Marco Thiel for the data and inspiration to get going on this topic.(Fruit Grid Sums) i've built various classifier functions for all 12 types of fruit images. As part of exploring I tried Classification Measurements.

cmeasures = ClassifierMeasurements[classNN, fruitTraingSet]
cmeasures["ConfusionMatrix"] // TableForm

The matrix looks fine. Now let us do

cmeasures["ConfusionMatrixPlot"]

That's not so good, instead of 12x12 matrix it's 7.5 x 7.5.

N.B. Even when built for maximum accuracy the function confuses bananas for lemons (strongly influenced by color) and confuses limes for avocados. Each classifier function takes about 180MB with only 7MB of images for training.

Maybe this is relevant to copyright, the downloaded images have different file names. After a while I realized that some images were duplicated but not the names. Thus DeleteDuplicates[] on lists of images cut the number down.