Use NetTrain[net,f] to read in a large set of rules (Is there an example?)

Questions
, ,
1

Hello,
Is there an example of using NetTrain[net,f] to read in a large file (1.8 million lines). The file is in text format with lines of the form

{17.,1.,34.,6.,14.,21.}->{26.,2.,53.,13.,20.,45.},
{16.,2.,3.,21.,17.,10.}->{13.,21.,49.,47.,51.,27.},
{4.,38.,40.,20.,5.,49.}->{15.,32.,51.,28.,50.,24.},
{37.,52.,29.,15.,39.,5.}->{34.,53.,25.,21.,47.,12.}

.
.
.

Thank You

Michel

2

Much easier and faster if each line did not have the , at the end. Strip them off using sed or awk which would be much faster than using Mathematica. Then

$fileStream = OpenRead["data.txt"];
NetTrain[net, ReadList[$fileStream, Expression, 1000]] (* batches of 1000 *)
Close[$fileStream]

Training is probably going to take a while with such a large dataset and depending on the complexity of the model. So, if you have not already done so, you should train / test on a random subset of the data.

3

Michel, I hope this will help if you cannot delete the , at the end.

in = Import["data.txt", "Words"];
f = ToExpression@StringTrim[#, ","] & /@ in

f.jpg
f.jpg663×111 20.4 KB

4

Okazaki-san,

I assumed Michel is concerned that 1.8M training samples may not fit into available memory and was looking for a solution that loaded samples in smaller batches. Some techniques for dealing with this are documented here.

If that is not an issue then certainly your solution is more straightforward.

5

Thank you Okazaki-san

Very much appreciated this

6

Thank You Rohit Namjoshi

What you offered, is what I needed.

Thank You