What I got from the discussion is that batch normalization scales a batch to be zero mean and 1 variance such that the scaling of the original data is irrelevant. So if an image is 0 background and 1 foreground or vise versa, or even 100 background and 200 foreground the batch normalization learns how to scale the data such that the network will alway behaves the same if the information in the images is similar.

Thanks a lot. Very useful stuff.

Additional to the big effect of Image-NetEncoder as you highlighted, the initial FlattenLayer also seems to be ~10% slower than a direct call to Flatten. So if I put this together I get on my machine:

In[4]:= nn = 
 NetChain[{64, Ramp, 10, SoftmaxLayer[]}, 
  "Output" -> NetDecoder[{"Class", Range[0, 9]}], "Input" -> 28*28]

Out[4]= NetChain[ <> ]

In[5]:= AbsoluteTiming[
 itrain3 = (1. - Flatten@ImageData@First@#) -> Last@# & /@ 
   trainingData;
 itest3 = (1. - Flatten@ImageData@First@#) -> Last@# & /@ testData;]

Out[5]= {1.59009, Null}

In[6]:= AbsoluteTiming[
 tn = NetTrain[nn, itrain3, BatchSize -> 100, MaxTrainingRounds -> 4]]

Out[6]= {5.67012, NetChain[ <> ]}

In[7]:= ClassifierMeasurements[tn, itest3, "Accuracy"]

Out[7]= 0.9652

Training is now faster than under Keras/Tensorflow! Accuracy about even. I like that, because I prefer working with MMA to Python/IPython/Jupyter/Keras.

Relating the inversion, I was first puzzled, that a bijective mapping, which leaves the entropy unchanged, could have any effect at all. But it seems plausible, that the zero plays its special role by making terms vanish and simplifying the computation. So maybe Wolfram should change the curated MNIST data to its original form with a zero background, which would make things comparable to other systems. Also if the Image-NetEncoder cannot made faster, at least a "Possible Issues"-Section under the NetEncoder help would be nice.