@Arben , I attended the course live on Friday and am going through the other sessions through the recordings. Good job with the poll questions. They worked great in the BigMarker replays. I was able to understand the poll questions as you asked them. Also, you consistently shared the results on your screen; people listening that way could play along. It's a good quiz, too.

I loved your discussion about particles in the universe on Day 1. It tickled me immensely to see an instructor provide a calculation that uses [Wolfram's estimate of] the number of particles in the universe.

That discussion reminded me of a short story by Stanislaw Lem in the book The Cyberiad. In the collection, a pair of "constructor" robots create fantastic machines. In the sixth sally, they create a demon of the second kind to work their way out of a problem. The story references a demon of the first kind -- Maxwell's Demon. Maxwell's demon was about thermodynamics; this second demon was about information. For a book that was created over 50 years ago, Lem's machine in this story is a rather astonishing vision of an LLM. Their demon was using a tiny bit of stale air -- a tiny number of particles -- as the source of its information; it could not possibly work as described in the story. OTOH, Maxwell's demon couldn't possibly work, either. Both hypotheticals are quite interesting; they are fine little thought experiments. I don't know if you -- or anyone -- will learn anything from Lem's short story, but you should be amused. Ask an AI to find you a copy.

Hmm... annoying!! I absolutely dropped it in there yesterday. I'll make sure it works this time.

As far as DeepSeek goes, as best I can tell there are indeed some architectural improvements, but the overall idea and structure remain the same and most of the improvements are in the training process. This article from MIT Technology Review is insightful! https://www.technologyreview.com/2025/01/31/1110740/how-deepseek-ripped-up-the-ai-playbook-and-why-everyones-going-to-follow-it/

Added!! Sorry about that!

Arben, Thanks 10^6 for a wonderful course! Question: Will you be posting an updated Q&A Digest along with an updated notebook for Day 5? We would appreciate it very much.

I was wondering in the last class two topics related to line of research in Wolfram:

1) Is Wolfram Research developing their own Algorithmic Differentiation framework for programs, a.k.a (parametrized)functions, that depends on loops, conditionals, recursion, and (non)smooth elemental functions? I commented in class this is between numerical and symbolic differentiation. By the way, this is the standard in Scientific Machine Learning and the backbone to make computationally efficient optimizers like ADAM (and their subsequent versions).

2) In case that (1) is negative, are you more focused on developing the discrete techniques to emulate neural networks capacity via Rule Arrays applied to cellular automata?

I don't! If you're referring to the Q&A digests, that's because those are just logs of the live chats from the sessions rather than email or otherwise "official"/formal communications. I think you'll find that the number of people who capitalized the first word of their messages back on AIM in 1997 was also quite small—though to your point, I do imagine that that number is shrinking over time as people have more access to instant back-and-forth communication. Language does change over time, and while I am largely a stickler for rules in a visceral sense I'm certainly not a prescriptivist. (If I were, I might point out that you use a hyphen rather than an em-dash in your last sentence, and add a novel space before the ellipses :). Language—written and spoken—always changes, particularly when exacerbated by the movement to new mediums and entry tools like keyboards where it's easier to type a hyphen than an en- or em-dash.)

Hi Gerald—I've taken some time to respond to this one because I'm not sure there's much I can say in my position here. I have personal opinions on this, but we'd kind of get into the weeds more than I think we ought to on Community. I have two main things that I think I can say:

1. LLMs really are quite good at a lot of things, as we've seen. However, some companies suggesting that they're SO powerful that we need to be extremely worried about them "taking over" (et c.) needs to be understood in its proper context—namely, ask "who benefits?" I think that it's highly unlikely that most western governments outright ban or even strongly regulate this sort of stuff, so when a company says this kind of thing, I think the only reasonably-expectable outcome is that they create a public image of their product that they are trying to sell as being unbelievably powerful.

2. At the very least, I think that in the short and medium term the larger danger to society posed by LLMs is the potential for actual, sourced knowledge to be "evaporated". LLMs can learn "the gist" of facts about the world, but they don't know them and generally don't retain them word for word or have the ability to reliably produce sources. As more and more people rely on "what AI said" in terms of forming their opinions, beliefs, and views about certain facts of the world, I think that we run the risk of some bad outcomes. (An unexpected one I just saw recently—somebody who must have been rather young asked for help from other people on a forum, but their request was formatting exactly like a query to an LLM: "Explain it this and this way, and DON'T do this or I won't believe you." Scary!) You have to take this with a grain of salt because I'm an employee here at Wolfram telling you this, but this very issue is why computational language and LLM integration with tools like Wolfram Language is really important: it provides a way for users to be sure that what they're getting from the LLM is related in a concrete way to the real world.

I don't think that there's a 1:1 discussion from Stephen on this topic, but you're likely to find some posts of interest here: https://writings.stephenwolfram.com/category/artificial-intelligence/

Gradient decent optimization procedures can get trapped at falsw (local) minimums. How does Chat GPT avoid or correct false mionimums? Aside Incredible clas, many thanks Arbin.

Also, the idea of converging to a global minimiser of the loss function would not be ideal. We do not want our model to overfit the outputs of the training set, as in the supervised learning scenario.

However, the "descent" direction that an optimiser computes should not be only descent. Why? Because the more we explore the space of the parameters is much better to find suitable "contextual" fits to the output space. By exploring as many neighborhoods of local minimizers (while converging to local minimum) as possible, we could improve the training by starting our search from a local minimizer that worked for certain context. However, the optimizer would need to be smart enough to scape again from that local minimizer.