Hello everyone
I’m trying to fine-tune GPT-2 (https://resources.wolframcloud.com/NeuralNetRepository/resources/GPT-2-Transformer-Trained-on-WebText-Data)
I tried training it like this:
gpt = NetModel[{"GPT-2 Transformer Trained on WebText Data",
"Task" -> "LanguageModeling", "Size" -> "345M"}]
gpt = NetTrain[gpt, {"This is an example"}]
But it didn’t work, can someone explain me how to train transformers in Mathematica?
Unfortunately, training transformers is slightly more complicated. You would need to add code for the loss function.
You could instead train a simple (non-transformer) text classifier in Mathematica using the Classify function:
classifier = Classify[{“This is a happy example”->1, “This is a negative or bad example” → 0}]
If you really wanted to train transformers (and do so without implementing a lot of code yourself), you could check out the HiggingFace transformer github repo - which is a Python library implementing training of transformers.