How To score a model built using BERT?

Questions
, ,
1

I’ve been trying to use your example using BERT for tweets classification (simple Kaggle dataset)

trainembeddings =    NetModel["BERT Trained on BookCorpus and English Wikipedia Data"][
    forBERT[[All, 1]], TargetDevice -> "CPU"] -> forBERT[[All, 2]];

classifierhead = NetChain[{DropoutLayer[], NetMapOperator[2], 
                                       AggregationLayer[Max, 1], SoftmaxLayer[]}, 
                                      "Output" -> NetDecoder[{"Class", {"negative", "positive"}}]]
bertresults = NetTrain[classifierhead, trainembeddings, All, TargetDevice -> "CPU",
   MaxTrainingRounds -> 50]

But, after I build the model, I’ve no idea how to score with new tweets, and I cannot see how to do it in your example.

2

It’s true that the example in section “Train a classifier model with the subword embeddings” should be completed to show how to “reconstruct” the full classifier and apply it.

The part that is missing for you is the following:

Extract the trained classifier:

trainedhead = bertresults["TrainedNet"]

Screenshotfrom2022-04-3011-17-15_1.png

Join it with BERT embeddings:

bertclassifier = NetJoin[
   NetModel["BERT Trained on BookCorpus and Wikipedia Data"], 
   trainedhead]

Screenshotfrom2022-04-3011-17-15_2.png

Apply it to some text:

bertclassifier["I liked it"]

Out[9]= "positive"

bertclassifier["I hated it", "Probabilities"]

Out[10]= <|"negative" -> 0.979603, "positive" -> 0.0203966|>
3

Thank You for the prompt Response.

4

How To score a model built using BERT?