Arben, On Wednesday I had asked, "Is DeepSeek doing something fundamentally different, or did they find a way to do it more efficiently?" You responded, "I have to say I'm not sure on that front—I'll try to look into this tomorrow and post about it in the Community thread." Have you had a chance to look into this yet?

Also, I am not seeing a Digest_Day5 in the Daily Q&A Digests.

Arben, Thanks 10^6 for a wonderful course! Question: Will you be posting an updated Q&A Digest along with an updated notebook for Day 5? We would appreciate it very much.

I was wondering in the last class two topics related to line of research in Wolfram:

1) Is Wolfram Research developing their own Algorithmic Differentiation framework for programs, a.k.a (parametrized)functions, that depends on loops, conditionals, recursion, and (non)smooth elemental functions? I commented in class this is between numerical and symbolic differentiation. By the way, this is the standard in Scientific Machine Learning and the backbone to make computationally efficient optimizers like ADAM (and their subsequent versions).

2) In case that (1) is negative, are you more focused on developing the discrete techniques to emulate neural networks capacity via Rule Arrays applied to cellular automata?

I don't! If you're referring to the Q&A digests, that's because those are just logs of the live chats from the sessions rather than email or otherwise "official"/formal communications. I think you'll find that the number of people who capitalized the first word of their messages back on AIM in 1997 was also quite small—though to your point, I do imagine that that number is shrinking over time as people have more access to instant back-and-forth communication. Language does change over time, and while I am largely a stickler for rules in a visceral sense I'm certainly not a prescriptivist. (If I were, I might point out that you use a hyphen rather than an em-dash in your last sentence, and add a novel space before the ellipses :). Language—written and spoken—always changes, particularly when exacerbated by the movement to new mediums and entry tools like keyboards where it's easier to type a hyphen than an en- or em-dash.)

Hi Gerald—I've taken some time to respond to this one because I'm not sure there's much I can say in my position here. I have personal opinions on this, but we'd kind of get into the weeds more than I think we ought to on Community. I have two main things that I think I can say:

1. LLMs really are quite good at a lot of things, as we've seen. However, some companies suggesting that they're SO powerful that we need to be extremely worried about them "taking over" (et c.) needs to be understood in its proper context—namely, ask "who benefits?" I think that it's highly unlikely that most western governments outright ban or even strongly regulate this sort of stuff, so when a company says this kind of thing, I think the only reasonably-expectable outcome is that they create a public image of their product that they are trying to sell as being unbelievably powerful.

2. At the very least, I think that in the short and medium term the larger danger to society posed by LLMs is the potential for actual, sourced knowledge to be "evaporated". LLMs can learn "the gist" of facts about the world, but they don't know them and generally don't retain them word for word or have the ability to reliably produce sources. As more and more people rely on "what AI said" in terms of forming their opinions, beliefs, and views about certain facts of the world, I think that we run the risk of some bad outcomes. (An unexpected one I just saw recently—somebody who must have been rather young asked for help from other people on a forum, but their request was formatting exactly like a query to an LLM: "Explain it this and this way, and DON'T do this or I won't believe you." Scary!) You have to take this with a grain of salt because I'm an employee here at Wolfram telling you this, but this very issue is why computational language and LLM integration with tools like Wolfram Language is really important: it provides a way for users to be sure that what they're getting from the LLM is related in a concrete way to the real world.

I don't think that there's a 1:1 discussion from Stephen on this topic, but you're likely to find some posts of interest here: https://writings.stephenwolfram.com/category/artificial-intelligence/

Gradient decent optimization procedures can get trapped at falsw (local) minimums. How does Chat GPT avoid or correct false mionimums? Aside Incredible clas, many thanks Arbin.

Also, the idea of converging to a global minimiser of the loss function would not be ideal. We do not want our model to overfit the outputs of the training set, as in the supervised learning scenario.

However, the "descent" direction that an optimiser computes should not be only descent. Why? Because the more we explore the space of the parameters is much better to find suitable "contextual" fits to the output space. By exploring as many neighborhoods of local minimizers (while converging to local minimum) as possible, we could improve the training by starting our search from a local minimizer that worked for certain context. However, the optimizer would need to be smart enough to scape again from that local minimizer.