Message Boards Message Boards

Input vector length limitations for NetTrain?

I am attempting to train an ANN using NetTrain and see errors that I may be related to the size of the training inputs. The training data consists of 245 rows of 1004 input data elements and 245 rows of 2 output elements. When I use a smaller training data set (23 by 1004 inputs and 23 by 2 outputs) NetTrain runs without problems. However, when I use the full training set [245 by 1004 inputs and 245 by 2 outputs) I receive an error. The input data files do not appear to be corrupted in any way. Is there an obvious problem with this procedure? Any help is appreciated. Clark

In[19]:= testinputs = Import["SimpleLivingScanData2IN.csv"]
In[20]:= testoutputs = Import["SimpleLivingScanData2OUT.csv"];
In[27]:= Dimensions[testinputs]
Out[27]= {245, 1004}
In[38]:= Dimensions[testoutputs]
Out[38]= {245, 2}
In[30]:= trainingcnet = 
  Flatten@Table[testinputs[[i, All]] -> testoutputs[[i, All]], {i, 1, 23}];
In[31]:= trainingcnet[[23, All]];
In[32]:= Dimensions[trainingcnet]
Out[32]= {23}
In[36]:= cnet = NetInitialize[
  NetChain[{LinearLayer[20], LinearLayer[20], 2}, "Input" -> 1004]]
In[64]:= trainedcnet[testinputs[[20, All]]]
    testoutputs[[20, All]]
Out[64]= {0.100103, 13.9911}
Out[65]= {0.1, 14.1421}

Now experiment with BatchSize
In[66]:= trainingcnet2 = 
  Flatten@Table[testinputs[[i, All]] -> testoutputs[[i, All]], {i, 1, 245}]; 
In[39]:= cnet2 = NetInitialize[
  NetChain[{LinearLayer[20], LinearLayer[20], 2}, "Input" -> 1004]]
In[41]:= trainedcnet2 = 
 NetTrain[cnet2, trainingcnet2, BatchSize -> 20, MaxTrainingRounds -> 100000, 
  TrainingProgressReporting -> "Print"]
During evaluation of In[41]:= NetTrain::invtdata: Training data should be an association of lists, or a rule from input to output examples.

Out[41]= $Failed
POSTED BY: Clark Landis

Hi. I'm using Mathematica 11.1.1 (on Linux), and don't get the error message that you refer to when using the full dataset. Instead I got this error: NetTrain::invindim: Data provided to port "Input" should be a list of length-1004 vectors.

I also checked the data and there's no discernible problem with the data (except that there are 49 occurrences - of 245 instances - of exactly the same input, which perhaps makes it hard to train the model). However, the error goes away if I use Chop on the input data, i.e.

Chop@Import["SimpleLivingScanData2IN.csv"]

So it seems that there is some problem somewhere in the data when reading the input.

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract