Group Abstract

Message Boards

WOLFRAM COMMUNITY

5.1K Views

3 Replies

7 Total Likes

View groups...

Follow this post

Share this post:

GROUPS:

How To score a model built using BERT?

Dalila Benachenhou

Posted 3 years ago

I've been trying to use your example using BERT for tweets classification (simple Kaggle dataset) trainembeddings = NetModel["BERT Trained on BookCorpus and English Wikipedia Data"][ forBERT[[All, 1]], TargetDevice -> "CPU"] -> forBERT[[All, 2]]; classifierhead = NetChain[{DropoutLayer[], NetMapOperator[2], AggregationLayer[Max, 1], SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", {"negative", "positive"}}]] bertresults = NetTrain[classifierhead, trainembeddings, All, TargetDevice -> "CPU", MaxTrainingRounds -> 50] But, after I build the model, I've no idea how to score with new tweets, and I cannot see how to do it in your example.

I've been trying to use your example using BERT for tweets classification (simple Kaggle dataset)

trainembeddings =    NetModel["BERT Trained on BookCorpus and English Wikipedia Data"][
    forBERT[[All, 1]], TargetDevice -> "CPU"] -> forBERT[[All, 2]];

classifierhead = NetChain[{DropoutLayer[], NetMapOperator[2], 
                                       AggregationLayer[Max, 1], SoftmaxLayer[]}, 
                                      "Output" -> NetDecoder[{"Class", {"negative", "positive"}}]]
bertresults = NetTrain[classifierhead, trainembeddings, All, TargetDevice -> "CPU",
   MaxTrainingRounds -> 50]

But, after I build the model, I've no idea how to score with new tweets, and I cannot see how to do it in your example.

POSTED BY: Dalila Benachenhou

3 Replies

Sort By:

Jérôme Louradour

Jérôme Louradour, Wolfram Research

Posted 3 years ago

It's true that the example in section "Train a classifier model with the subword embeddings" should be completed to show how to "reconstruct" the full classifier and apply it. The part that is missing for you is the following: Extract the trained classifier: trainedhead = bertresults["TrainedNet"] Join it with BERT embeddings: bertclassifier = NetJoin[ NetModel["BERT Trained on BookCorpus and Wikipedia Data"], trainedhead] Apply it to some text: bertclassifier["I liked it"] Out[9]= "positive" bertclassifier["I hated it", "Probabilities"] Out[10]= <\|"negative" -> 0.979603, "positive" -> 0.0203966\|>