I think you should use a generator function, for example:

enc = NetEncoder[{"Function", Flatten[IntegerDigits[#, 2, 8]] &, {192}}]
b = ByteArray[Table[RandomInteger[{0, 255}], 10 * 24]]
genTrain = Function[ArrayReshape[Normal[b[[1 ;; 24 * #BatchSize]]], {#BatchSize, 24}]]
enc[genTrain[<|"BatchSize" -> 2|>]]

As the dataset resides in RAM I could use RandomSample to select a batch, but am I right that if I want to batch the dataset linearly I have to specify the "RoundLength" option of NetTrain as:

Length[b] / (24 * BatchSize)

and add a "Round" argument to the generator function:

genTrain = Function[ArrayReshape[Normal[b[[(#Round - 1)* 24 * #BatchSize + 1 ;; #Round * 24 * #BatchSize - 1]]], {#BatchSize, 24}]]

Regards, GW

Hi,

Thank you for your reply. After reading the dataset into RAM my dataset exists as a linear array of multiples of 24 bytes in RAM. Suppose the batch size is 65536. For the next batch I have to take the next block of (65536, 24) bytes of my compressed dataset, apply NetEncoder to the batch and train the network on the batch. However, it is unclear to me how I can configure NetTrain to do that. FYI, in Tensorflow I read the compressed dataset as a two-dimensional array in RAM, I turn the two-dimensional array into a batched dataset and apply a map function to the dataset to batch-unpack the 24 bytes (see also https://discuss.tensorflow.org/t/can-you-bit-pack-and-then-unpack-binary-inputs/19224). Tensorflow's model.fit will then loop over the batched dataset.

Regards, GW