Hi,

I have a dataset of 192 binary inputs (0/1) and corresponding outputs. I have compressed the inputs to 24 8-bit integers, so that the entire dataset can fit into RAM which greatly speeds up training of the neural network. Obviously, before a batch is trained the 24 8-bit integers have to be expanded to 192 integers. I managed to figure out how to do this in Tensorflow using the 'map' functionality of datasets, and now I want to do the same in Mathematica. It turns out you can achieve this using NetEncoder (see FunctionLayer error compilerr: Cannot interpret as a network). When I now train the neural network I get a 'Processing data for "Input" port' dialog box before training starts, and looking at the numbers shown in 'input .. of ..' it seems that it is processing data for the validation set. What does 'Processing' do? Is it perhaps expanding the validation set completely? If so, as I do not get a similar dialog box for the training set, I guess it works for the training set as intended: only the batches are expanded as needed, not the full training set. Is it possible to also expand the validation set in batches?

Regards, GW