Group Abstract

Message Boards

WOLFRAM COMMUNITY

3.7K Views

3 Replies

0 Total Likes

View groups...

Follow this post

Share this post:

GROUPS:

Data Science Wolfram Language Neural Networks

NetTrain exceeds size of dataset when using a generator function

Gijsbert Wiesenekker

Posted 2 years ago

POSTED BY: Gijsbert Wiesenekker

3 Replies

Sort By:

Mauro Bertani

Posted 2 years ago

Hi wiesenekker, I'm not an expert but here trainfeatures[[(#Round - 1)* 24 * #BatchSize + 1 ;; #Round * 24 * #BatchSize ]]] Trainfeatures[[99 * 24 * 10 + 1]]=trainfeatures[[23761]] When Length[trainfeatures]=2400 Batchsize is an error. Maybe I think Bye

POSTED BY: Mauro Bertani

Gijsbert Wiesenekker

Posted 2 years ago

The following generator function randomly selects a batch rather then relying on the 'Round' number: batchsize = 10 nbatches = Length[trainfeatures]/ (npack * batchsize) genTrain = Function[ With[{ibatch = RandomInteger[{1, nbatches}]}, <\| "Input" -> Map[Flatten, IntegerDigits[ ArrayReshape[ Normal[trainfeatures[[(ibatch - 1)* 24 * #BatchSize + 1 ;; ibatch * 24 * #BatchSize ]]], {#BatchSize, 24}], 2, 8]], "Output" -> trainlabels[[(ibatch - 1) * #BatchSize + 1 ;; ibatch* #BatchSize ]]\|>]] With this generator function the error is gone. Regards, GW

The following generator function randomly selects a batch rather then relying on the 'Round' number:

batchsize = 10
nbatches = Length[trainfeatures]/ (npack  * batchsize)
genTrain = 
 Function[
  With[{ibatch = RandomInteger[{1, nbatches}]}, <|
    "Input" -> 
     Map[Flatten, 
      IntegerDigits[
       ArrayReshape[
        Normal[trainfeatures[[(ibatch - 1)* 24 * #BatchSize + 1 ;; 
            ibatch * 24 * #BatchSize ]]], {#BatchSize, 24}], 2, 8]], 
    "Output" -> 
     trainlabels[[(ibatch - 1) * #BatchSize + 1 ;; 
        ibatch* #BatchSize ]]|>]]

With this generator function the error is gone.

Regards, GW

POSTED BY: Gijsbert Wiesenekker

Gijsbert Wiesenekker

Posted 2 years ago

Also, 'RoundLength' turns out to be 'the number of samples that is expected to be seen during a Round (epoch)', so it should be equal to the number of training samples, 'ntrain' in this case: trained = NetTrain[net, {genTrain, "RoundLength" -> ntrain}, All, BatchSize -> batchsize, TargetDevice -> "GPU"] With 'ntrain' set to 100000000 and 'batchsize' to 65536 you get 1525 batches. The Training Progress shows 'round 1//10, batch x/1525)' as expected. Regards, GW

POSTED BY: Gijsbert Wiesenekker

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Feedback