Excellent introduction ... thanks!

After reading "What Is ChatGPT Doing … and Why Does It Work?" I really wanted to explore the networks, better understand what they do and play around with them in Mathematica. This post gave me everything in one go... which is awesome!

Happy to hear that. :)

I'm preparing another post on the original vanilla encoder-decoder transformer architecture (Attention is all you need paper from 2017).

It also might be of your interest this other recent post on Few Shot Learning using GPT-3 and WL: https://community.wolfram.com/groups/-/m/t/2848741

I find using the OpenAILink within the wolfram notebook kinda fun. And the ability to save them in an orderly fashion super useful. I can almost write sort of a graphic novella. Here's my first try with the defaults

However, the OpenAICompletion did not follow the 7 word and 5 line rule that I gave. it worked fine on the openAI website but not with the packet command. Here's the output from chatGPT

I tried to change the OpenAIMethod with no luck

Thanks a lot for this excellent post which stimulated me to try to understand more of the internals of the network.

In this context, I came up with two questions which I was not able to figure out by myself:

1) How can I visualize the word embedding space for the Shakespeare vocabulary in order to test if words which are "semantically near" show up nearby in the word embedding map, similar to what Stephen Wolfram showed in his blogpost "What is ChatGPT doing..."? I have the feeling that it should be quite simple, but all my attempts to access the embedding layer of the trained network failed, probably due to my very limited understanding of the tools of net surgery.
2) Is it easily possible to visualize the attention matrix for a particular input sentence?

Any help is appreciated...