Message Boards Message Boards

0
|
5444 Views
|
3 Replies
|
2 Total Likes
View groups...
Share
Share this post:

Possible issue in NetTrain? The loss of validation/training set

Posted 4 years ago

Hi guys, I'm very curious why the loss of validation set is even lower than the loss of training set when I use NetTrain. Say, on this page,

https://reference.wolfram.com/language/tutorial/NeuralNetworksSequenceLearning.html#1094728277

for the Q&A RNN trained on the bAbI QA Dataset, the loss of validation set shouldn't be lower than the loss of training set, according to Goodfellow's DL book. Right?

Is it possible that these 2 sets are mistakenly labeled in the NetTrain function when it tries to plot the learning curve?

loss of validation and training

POSTED BY: HH C
3 Replies
Posted 4 years ago

Crossposted here.

POSTED BY: Rohit Namjoshi

It is not a bug, this behaviour is explained by the dropout. The training loss is computed with dropout enabled (as in DropoutLayer[...][..., NetEvaluationMode -> "Train"]). The validation loss is computed with dropout disabled (as in DropoutLayer[...][..., NetEvaluationMode -> "Test"]).

Posted 4 years ago

Thanks.

POSTED BY: HH C
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract