Thank you very much for this posting. Have been waiting for over a decade for this kind of MMA application. What I wish is a pdf book to teach using MMA on texts. Rock on.

Dear Mengyi,

I have just completed training your network on a data set that is about 10 times larger than yours (and might have a better overall quality, because I did not use Wikipedia anymore).

This is the network I used:

net4 = NetChain[{
    embeddingLayer,
    ElementwiseLayer[Tanh],
    LongShortTermMemoryLayer[100],
    NetBidirectionalOperator[{LongShortTermMemoryLayer[40], 
      GatedRecurrentLayer[40]}],
    NetBidirectionalOperator[{GatedRecurrentLayer[30], 
      LongShortTermMemoryLayer[30]}],
    LongShortTermMemoryLayer[30],
    NetBidirectionalOperator[{LongShortTermMemoryLayer[10], 
      GatedRecurrentLayer[10]}],
    LongShortTermMemoryLayer[10],
    NetMapOperator[LinearLayer[3]], 
    SoftmaxLayer["Input" -> {"Varying", 3}]}, 
   "Output" -> NetDecoder[{"Class", {"a", "b", "c"}}]
   ];

And here are the results:

It turns out that all values got somewhat better, especially the comma values gained.

My actual corpus is at least 5-10 times larger than what I have been using, but I will need another GPU setup and/or other tricks to be able to run it on the entire corpus (10000 books of Gutenberg + other texts).

I am happy to make the trained network (163 MB) available upon request. If you are outside Germany, you can also easily download the Gutenberg dump.

Cheers, Marco

This is weird. As far as I can tell my data looks just like yours. But it must be something about the data; if I use the net and just NetInitialize it, everything seems to work fine. So it must be about the training.

Now, I am getting somewhere. It turns out that if I delete the Validation set and only use the first 358 test dataset it is running. The set 359 is offending. In the dataset the string has 199 words and there are 200 letters in the classification set.

If I clean this up by

trainingSetclean = Select[trainingSet, (Length[StringSplit[#[[1]]]] == 200 && Length[#[[2]]] == 200) &];

and

validationSetclean = Select[validationSet, (Length[StringSplit[#[[1]]]] == 200 && Length[#[[2]]] == 200) &];

it appears to work.

Thank you for helping to solve this.

I'll report back if I find get anything interesting.

Cheers,

Marco

Hi,

I must be overlooking something here. I think that I have got your code (with trivial modifications) and I always get warnings when I execute

NetTrain[net4, trainingSet, All, ValidationSet -> validationSet, TargetDevice -> "GPU"]

The warning is:

Can you post one training set data element? It is late here so I am probably not seeing something trivial. The same message occurs with all networks you suggest.

Cheers,

Marco

Hello Marco,

Thanks for your reply! I ran that on my laptop for a whole night, approximately 9 hours. Actually, I'm now working on a 10 times larger dataset and plan to run it with GPU. I believe this will produce a great improvement.

Mengyi