Thank you, Arben!

I would also like to point out that this does not just apply to us poor home users.

Any research project which involves repetitive operations on data (imagine sentiment analysis on a million samples) would have its budget blown out of the water by a more expensive model.

Once one has a viable prototype solution, try it with a Nano model. These are dirt cheap in comparison to even the Minis.

GPT-4.1 Nano is presently the one to test against. Once Wolfram has fixed their GPT-5 and GPT-5.1 interface code, these Nano versions are even cheaper.

Hey Phil! I also watch WWDC ;). I won't be running anything locally for this series, but maybe in the future. (Though I did just get a new laptop from our IT department earlier this year and won't be due for an upgrade for a while, so I won't be benefitting from the M5 improvements! But I really can't complain.)

As far as whether it's any different to run things locally, functionally it isn't. It's more secure, obviously, and relatedly it's more private, but the actual functioning of the thing is not really particularly different. The time it takes to send and receive information to/from LLMs is really not that big a component in the total runtime because it's generally all just text. The extra time taken by network communication may in fact be canceled out by the fact that large companies are already running their setups on specialized hardware, but even if this isn't quite correct it's probably close enough as makes no difference. (It's possible that there are delays based on queuing of their massive numbers of users, though!)

"Neural accelerators" -- optimally efficient matrix multiplication -- have just been added to the GPUs in Apple's M5 processors. The M5 Pro, M5 Max, and M5 Ultra chips should be mightily impressive for high-octane AI researchers and users. At least machines with M5 Pro and M5 Max should be available in the first part of 2026. It sounds like Apple will suggest multiple Mac Studio boxes with a Thunderbolt 5 "bus" for its highest-end customers to distribute computational tasks to multiple machines. The Wolfram Language core numeric computation engine could be seriously enhanced over the next software update or two.

A question for @Arben: it seems as if the Wolfram AI Course Assistant would be ideal for any questions in a Wolfram course -- including double-checking answers to Wolfram quiz questions. As a member of the faculty, are you encouraged or discouraged by the way students could use these marvelous tools?

Hi all! Hope you enjoyed the study group. There were a few lingering survey responses that I wanted to address here, as promised.

Please cover some methods to use personal reference documents in formats other than ASCII text with LLM functions. For example, create an LLM function to summarizes a collection of PDF files located in a specified directory. Another item would be a way direct an LLM function to synthesize its response based on a specified collection of documents in common document formats, (i.e XLSX, PDF, CSV, ODT, etc)

Today's discussion of LLMGraph hopefully provided some helpful insight for this question, but I also want to note that in my particular graph, I used Wolfram Language to grab only the PDF files, but WL can import basically any kind of thing you're interested in.

You can see all available Import formats by running:

$ImportFormats

Depending on what you're importing, it will often be useful when feeding it to an LLM to format it as markdown text. Remember that you can do this by calling this resource function:

ResourceFunction["ExportMarkdownString"]

As an applied mathematician with a background in signal processing, simulation, project management,, and Protege Ontology , I hope to learn enough to experiment with Agentic simulation of simple organizational governance policies via interacting Wolfram LLMFunctions—"coordinated" via ontologies . Is that hopelessly overly ambitious for a newcomer. to the Wolfram environment? My goal is to do some applied category theory work in the area of semiotics and the dynamics of organizational evolution. I hope that aim does not sound too abstract.

As someone trained as a physicist, I can't say I have a full grasp of every part of the goal here, but I don't think it should be beyond your grasp even as a newcomer. You've seen how to combine WL with LLMs in a rather arbitrary way, now, so if you have any questions about "how can I do XYZ with WL?" you can always feel free to ask.

Using 100s of documents along with a prompt. Anyway to make this efficient, so documents aren't processed by AI engine over again for each prompt.

The naive method that I used in today's lecture (LLMGraph with a "ListableLLMFunction" is probably not extensible to hundreds of documents, but my understanding is that you should be able to manage this without much issue with semantic search indices and vector databases. Gay Wilson from Wolfram has written up a short example workflow here.

There were a good number of questions about running local models, as well. This is not something I'm very familiar with, but maybe this notebook from Daniel—also here at Wolfram—can function as a jumping off point.

I have in mind a "projection" from all the extensive Mathematica Documentation, as well as Community threads --- a large corpus to be filtered and structured against personal technological experience and personal objectives, suitably described. It sounds like a tall but fascinating order, which I would imagine pursuing incrementally. What do you think?

This is basically Notebook Assistant—it has access to the full Wolfram documentation system, but also many of our repositories. I believe that we're working on adding Wolfram U notebooks and information from Community as well. It's maybe not quite as personal, but I'm not sure how we'd really objectively evaluate and thus implement that.

I would appreciate a bit of guidance on how to use the very powerful graphical features that you showed, which present visualizations of the flow of computation. I believe that these would help me to explore, and get a better grasp of, the basic Transformer architecture -- which is beyond the scope of this Study Group.

I recommend checking out our What is ChatGPT Doing... and Why Does It Work? study group. This series is based heavily on Stephen Wolfram's blog post (which was made into a book) and was adapted and delivered by yours truly.

There was also a series of questions about LLMTool that gets the implementation a bit backward, so I wanted to use those questions to offer some clarity.

1. How can an LLMTool be directed to use a model that is on the local machine or hosted on a local server?

2. Can an LLMTool be directed to use multiple models simultaneously?

3. Can an LLMTool specify a collection of documents in a specific directory for the model to use when preparing the response ? If so, what are the issues with respect to OCR and various file formats?

It's models that use tools, not vice versa. Models with access to tools will "decide" based on the context of their token generation given some input that it's time to feed some portion of their tokens as textual input to a tool. The tool does whatever computation it was meant to do, then provides the results back as a string to the model. The model then tokenizes that string and can use it as part of its "input" for generating future tokens.

It looks like "Reasoning" is an available setting for LLMEvaluator, but there seems to be a bug in it right now. I believe that it's fixed in our next major release and they're working on backporting it into a paclet update that will work in earlier versions. (And once 5.1 is working, that defaults to setting "Reasoning" to "None" anyway, which is not an option on the gpt-5 models [which default to "Medium"].)

This is my current understanding, at least, but I can't claim to be fully authoritative on this.

Minutes after writing the above post, OpenAI released the GPT-5.1 model series APIs.

For Mathematica (or any coding application), the premier OpenAI model is unquestionably GPT-5.1 Codex. It's designed for exceptional instruction following and code generation. While the full model is intended for high-end planning, architecture, design, refactoring, etc, the Mini model excels at coding.

And the Mini version is cheap. Using the above formula, the price per million tokens is (3 * 0.25 + 1 * 2.00) / 4 or $0.6875.

Unfortunately, while it can now be selected in Mathematica Preferences / AI Settings and used with the LLM APIs (Model name "gpt-5.1-codex-mini"), it will immediately fail (see attached image).

It seems Wolfram is still using the now-deprecated OpenAI "completions" API rather that the modern "responses" API.

Crap!

Attachments:

GPT5.1 API Fail.png

Re: getting a list of models, here's how you can do that:

In[139]:= services = Keys[(LLMServices`LLMServiceInformation[])[[1]]]

You can then grab the models associated with a given service with:

In[142]:= Wolfram`Chatbook`Common`getServiceModelList["OpenAI"]

And just the model names explicitly with:

#Name & /@ Wolfram`Chatbook`Common`getServiceModelList["OpenAI"]

Hi Carl—thank you! This is actually more of a social issue than a technical one, which is not a thing I can say too frequently when asked about error messages :).

Basically, Lena—the image in question—has long been used for image processing examples. However, the image is cropped from a Playboy centerfold and thus many people have thought that maybe different images should be used for testing. You can read about the image and its history at this wikipedia link.

A few updates back, Wolfram decided that we would probably remove Lena as a test image some time in the future, so while the image is currently still available we pop up an "error" message that's really more of a warning that any code you write that involves the image might not work in the future as the image will eventually not be accessible though ExampleData["TestImage","Lena"].

I think that this context is probably not baked into the models used for Notebook Assistant, and in trying to fix an "error" that wasn't actually an error it started meandering and getting things wrong. For example, it tried to pull the "Iris" dataset for a machine learning example... but in Wolfram Language, we call that "FisherIris", not just "Iris", so it didn't work. If you run ExampleData["MachineLearning"], you'll see it in there.

Hey Stephen—could you run the following: ` PacletInstall["Wolfram/Chatbook", UpdatePacletSites -> True] and fully quit and reopen the app and see if it starts to work? It looks like the bug you're reporting was known about in an older version of the Chatbook paclet that powers all of the Chat cell functionality but was fixed a little while back.

(I'm actually going to just recommend everybody run this at tomorrow's session to head off any issues!)

So think I found problem. If you have dynamic content in the notebook and don't "approve it" in the banner showing at top of the window, then the forward slash does not work in a chat cell.

Thanks Paul,

I agree. I too tried it on a different computer without any success. It would be nice to at least show a message that says "Hey, it's us, don't waste time, we will be back as soon as we can"

I am sure an LLM can come up with something to that effect..

Carl

Hi, Carl. Here's what Grok suggested. "Flush caches" was the only thing I saw that you probably haven't already tried. The response is repeated because I prompted it to "think harder". HTH. https://x.com/i/grok?conversation=1990170094241214480

"https://www.perplexity.ai/search/tell-me-a-funny-joke-no-one-ha-cKpP47u6TNageZo465bXCQ#0"

One-second response with GPT-5.1. I tried it multiple times.

This on the first day of the 5.1 API release - and it will only get better.

OK, last LLM question of the evening. I promise!

Arben... Is there a way to hook into the (actual raw text) request-response conversation between Mathematica and the LLM service provider? Or, is there a way to get a log of this?

For anybody who was interested in extracting the tokens for a given input from GPT2 (or any other net model that tokenizes that you can get into a notebook, you just need to do this):

tokenizer = 
 NetExtract[NetModel["GPT2 Transformer Trained on WebText Data"], 
  "Input"]

Then, you can run tokenizer["whatever text here"] and it will produce the tokens. For example:

In[591]:= tokenizer["Hello there! My name is Arben. Hello hello hello hello hello goodbye goodbye."]

Out[591]= {15241, 357, 50257, 1756, 1183, 63, 688, 11467, 50244, 18180, 23493, 23493, 23493, 23493, 24574, 24574, 50244}

The webpage I was asking about today was the Wolfram Support Document #62525, How can I start using Chat Notebooks in Wolfram products? It was announced in the chat panel on Monday by (I believe) Arben. I located it in my browser history. I thought that "Grok" was misspelled on that webpage; I hadn't realized that "Groq" was actually a completely different AI than xAI's product. AIs are easy, but product naming is complicated. :)

The Wolfram LLM Benchmarking Project is a fascinating little project. As one of those many who went through the EIWL test framework, I viewed it as an idiosyncratic set of questions. I'm fascinated the best score from an AI was 64.7% -- especially when the quiz answers are available through the pages of the published books of EIWL. The test framework is doing some sophisticated pattern-matching, and sometimes its choices were completely arbitrary. I (imperfectly) recall one of the exercise's "correct" answers had to do with the names one chose for variables. I'd have to plow through that mass of exercises to find the specifics. If someone internally at Wolfram thought this was a good way to systematically benchmark all AIs, I'd love to hear a detailed explanation why.

The generated Shakespearian AI output on Monday reminded me of one of the original "computational essays". The DeCSS Haiku was created in 2001 as a means of protesting the arrest of another programmer for creating the original DeCSS software. The code would allow decryption of a DVD: something Hollywood tried to prohibit in the late 1990s. Seth Schoen created the 465 stanza haiku to create "art" that was precise computational instructions for re-implementing the DeCSS. In this blog post, he says it took him about 13 hours to make almost all of the haiku. AIs could probably repeat his effort in a few minutes. AIs could also generate code in the language of choice to re-generate a DeCSS implementation from that original 465-stanza haiku. OTOH, current AIs are a terrible choice for either project.

Actually, Paul, could you run this and report back? PacletInstall["Wolfram/Chatbook"] We can't reproduce it in 14.3, or pre-release versions, so it might be that your paclet is out of date and hasn't auto-updated yet.

Thanks for letting me know—I'll get this filed with the appropriate team, in case they're not already aware.

Ah, yes! Many interactive elements in notebooks subtly depend on dynamic updating's being enabled.

So it does work in a new notebook. Didn't even restart Wolfram Mathematica. But wont' work in exiting notebook that I just created yesterday. Very strange.

In Day 1 of the class (Monday), the use of the forward slash key within a chat is discussed. It's purpose is to enter Wolfram Language in the chat. On my Mac, running 15.7.2 and Wolfram 14.3, When I push the forward slash key (above return), it is not entered. And so nothing happens. Any ideas? I have TextExpander turned off. I can enter forward slash in other applications.