Hi, Carl. Here's what Grok suggested. "Flush caches" was the only thing I saw that you probably haven't already tried. The response is repeated because I prompted it to "think harder". HTH. https://x.com/i/grok?conversation=1990170094241214480

I have experienced the same thing on two different computers. I think the service is down and has been for several days.

Now that Wolfram is now a serious "services" company, they need status pages that are updated 24/7 with the status of all key cloud services and repositories.

I've wasted several hours on this, including filing a report with Wolfram.

Hi all! Hope you enjoyed the study group. There were a few lingering survey responses that I wanted to address here, as promised.

Please cover some methods to use personal reference documents in formats other than ASCII text with LLM functions. For example, create an LLM function to summarizes a collection of PDF files located in a specified directory. Another item would be a way direct an LLM function to synthesize its response based on a specified collection of documents in common document formats, (i.e XLSX, PDF, CSV, ODT, etc)

Today's discussion of LLMGraph hopefully provided some helpful insight for this question, but I also want to note that in my particular graph, I used Wolfram Language to grab only the PDF files, but WL can import basically any kind of thing you're interested in.

You can see all available Import formats by running:

$ImportFormats

Depending on what you're importing, it will often be useful when feeding it to an LLM to format it as markdown text. Remember that you can do this by calling this resource function:

ResourceFunction["ExportMarkdownString"]

As an applied mathematician with a background in signal processing, simulation, project management,, and Protege Ontology , I hope to learn enough to experiment with Agentic simulation of simple organizational governance policies via interacting Wolfram LLMFunctions—"coordinated" via ontologies . Is that hopelessly overly ambitious for a newcomer. to the Wolfram environment? My goal is to do some applied category theory work in the area of semiotics and the dynamics of organizational evolution. I hope that aim does not sound too abstract.

As someone trained as a physicist, I can't say I have a full grasp of every part of the goal here, but I don't think it should be beyond your grasp even as a newcomer. You've seen how to combine WL with LLMs in a rather arbitrary way, now, so if you have any questions about "how can I do XYZ with WL?" you can always feel free to ask.

Using 100s of documents along with a prompt. Anyway to make this efficient, so documents aren't processed by AI engine over again for each prompt.

The naive method that I used in today's lecture (LLMGraph with a "ListableLLMFunction" is probably not extensible to hundreds of documents, but my understanding is that you should be able to manage this without much issue with semantic search indices and vector databases. Gay Wilson from Wolfram has written up a short example workflow here.

There were a good number of questions about running local models, as well. This is not something I'm very familiar with, but maybe this notebook from Daniel—also here at Wolfram—can function as a jumping off point.

I have in mind a "projection" from all the extensive Mathematica Documentation, as well as Community threads --- a large corpus to be filtered and structured against personal technological experience and personal objectives, suitably described. It sounds like a tall but fascinating order, which I would imagine pursuing incrementally. What do you think?

This is basically Notebook Assistant—it has access to the full Wolfram documentation system, but also many of our repositories. I believe that we're working on adding Wolfram U notebooks and information from Community as well. It's maybe not quite as personal, but I'm not sure how we'd really objectively evaluate and thus implement that.

I would appreciate a bit of guidance on how to use the very powerful graphical features that you showed, which present visualizations of the flow of computation. I believe that these would help me to explore, and get a better grasp of, the basic Transformer architecture -- which is beyond the scope of this Study Group.

I recommend checking out our What is ChatGPT Doing... and Why Does It Work? study group. This series is based heavily on Stephen Wolfram's blog post (which was made into a book) and was adapted and delivered by yours truly.

There was also a series of questions about LLMTool that gets the implementation a bit backward, so I wanted to use those questions to offer some clarity.

1. How can an LLMTool be directed to use a model that is on the local machine or hosted on a local server?

2. Can an LLMTool be directed to use multiple models simultaneously?

3. Can an LLMTool specify a collection of documents in a specific directory for the model to use when preparing the response ? If so, what are the issues with respect to OCR and various file formats?

It's models that use tools, not vice versa. Models with access to tools will "decide" based on the context of their token generation given some input that it's time to feed some portion of their tokens as textual input to a tool. The tool does whatever computation it was meant to do, then provides the results back as a string to the model. The model then tokenizes that string and can use it as part of its "input" for generating future tokens.

As far as I know, all of the available information we can provide is produced if you give All as a second argument to basically any LLM* function. If it isn't there, I think we must not have it accessible.

Minutes after writing the above post, OpenAI released the GPT-5.1 model series APIs.

For Mathematica (or any coding application), the premier OpenAI model is unquestionably GPT-5.1 Codex. It's designed for exceptional instruction following and code generation. While the full model is intended for high-end planning, architecture, design, refactoring, etc, the Mini model excels at coding.

And the Mini version is cheap. Using the above formula, the price per million tokens is (3 * 0.25 + 1 * 2.00) / 4 or $0.6875.

Unfortunately, while it can now be selected in Mathematica Preferences / AI Settings and used with the LLM APIs (Model name "gpt-5.1-codex-mini"), it will immediately fail (see attached image).

It seems Wolfram is still using the now-deprecated OpenAI "completions" API rather that the modern "responses" API.

Crap!

Attachments:

GPT5.1 API Fail.png

Thank you, Arben!

I would also like to point out that this does not just apply to us poor home users.

Any research project which involves repetitive operations on data (imagine sentiment analysis on a million samples) would have its budget blown out of the water by a more expensive model.

Once one has a viable prototype solution, try it with a Nano model. These are dirt cheap in comparison to even the Minis.

GPT-4.1 Nano is presently the one to test against. Once Wolfram has fixed their GPT-5 and GPT-5.1 interface code, these Nano versions are even cheaper.

Arben... Thanks for this. Really helpful.

Something I wish to point out. Perhaps this should be a support request but, for LLM stuff, they never know what I'm talking about.

When I do a fresh start of Mathematics (new 14.3 download and clean install with packlet update on Windows 11) and then execute your code, I get the error shown in "getServiceModelList.png". It then gives me the LLM model association. This will only happen once. I have to restart Mathematica or quit the kernel to repeat the error.

The same error occurs the first time I open Preferences / AI Settings in a kernel session, as shown in "AI Settings.png".

Attachments:

AI Settings.png

getServiceModelList.png

It looks like "Reasoning" is an available setting for LLMEvaluator, but there seems to be a bug in it right now. I believe that it's fixed in our next major release and they're working on backporting it into a paclet update that will work in earlier versions. (And once 5.1 is working, that defaults to setting "Reasoning" to "None" anyway, which is not an option on the gpt-5 models [which default to "Medium"].)

This is my current understanding, at least, but I can't claim to be fully authoritative on this.

"https://www.perplexity.ai/search/tell-me-a-funny-joke-no-one-ha-cKpP47u6TNageZo465bXCQ#0"

One-second response with GPT-5.1. I tried it multiple times.

This on the first day of the 5.1 API release - and it will only get better.

The webpage I was asking about today was the Wolfram Support Document #62525, How can I start using Chat Notebooks in Wolfram products? It was announced in the chat panel on Monday by (I believe) Arben. I located it in my browser history. I thought that "Grok" was misspelled on that webpage; I hadn't realized that "Groq" was actually a completely different AI than xAI's product. AIs are easy, but product naming is complicated. :)

The Wolfram LLM Benchmarking Project is a fascinating little project. As one of those many who went through the EIWL test framework, I viewed it as an idiosyncratic set of questions. I'm fascinated the best score from an AI was 64.7% -- especially when the quiz answers are available through the pages of the published books of EIWL. The test framework is doing some sophisticated pattern-matching, and sometimes its choices were completely arbitrary. I (imperfectly) recall one of the exercise's "correct" answers had to do with the names one chose for variables. I'd have to plow through that mass of exercises to find the specifics. If someone internally at Wolfram thought this was a good way to systematically benchmark all AIs, I'd love to hear a detailed explanation why.

The generated Shakespearian AI output on Monday reminded me of one of the original "computational essays". The DeCSS Haiku was created in 2001 as a means of protesting the arrest of another programmer for creating the original DeCSS software. The code would allow decryption of a DVD: something Hollywood tried to prohibit in the late 1990s. Seth Schoen created the 465 stanza haiku to create "art" that was precise computational instructions for re-implementing the DeCSS. In this blog post, he says it took him about 13 hours to make almost all of the haiku. AIs could probably repeat his effort in a few minutes. AIs could also generate code in the language of choice to re-generate a DeCSS implementation from that original 465-stanza haiku. OTOH, current AIs are a terrible choice for either project.

Hey Arben, As always, when you teach a class, I sign up. I thought I had an example here but it was premature. But sometimes the notebook assistant get's lost trying to invent functions that don't exist. (when I am being lazy and trying to get the assistant to write some code for me) It seems like an inspiration for writing a function that does what it thinks should exist, but what do you do generally when that happens?

Will you include a demo running a local LLM during this course? Is there anything special about accessing an LLM locally, or does that look the same as accessing remotely?

I believe that 2026 will be the first year that local LLM usage becomes widespread. With their A19 Pro and M5 processors, Apple has straightened out a major kink in their architecture. When Apple announced the iPhone 17 models at their September 9 press event, they included a brief discussion of the A19 Pro's new GPU accelerator in the event:

We have been at the forefront of AI acceleration since we first introduced the neural engine eight years ago. We later brought machine-learning accelerators to our CPUs. And while our GPU has always been great at AI compute, we’re now taking a big step forward, building neural accelerators into each GPU core, delivering up to three times the peak GPU compute of A18 Pro. This is MacBook Pro levels of compute in an iPhone, perfect for GPU-intensive AI workloads.

That was a bit vague. Fortunately, the "Petar Tech" YT channel explained more in the video Demystifying Apple's AMX AI accelerator: when and why it outperforms the GPU. In a nutshell, matrix multiplication -- a cornerstone of LLM training and token generation -- happens atomically in this chip. Historically, the elements of the two matrices being multiplied are fetched from memory multiple times. Petar notes in his video commentary: Because nowadays, the energy needed to transfer data significantly exceeds the energy needed to perform computations on the data. Apparently, NVIDIA has been doing this for several generations on their Tensor Cores; Apple is just catching up on this particular processor feature.

Apple has incorporated the same per-GPU accelerators in its M5 Processors that were released for the low-end MacBook Pro and iPad Pro models this month. In one early test, "Time to First Token" was measured about 3.6x faster than a M4 processor. That baseline M5 processor is limited to 32GB of unified memory, but the "M5 Pro" and "M5 Max" chip computers (both available early 2026) should have up to 128GB of unified memory available. Now that Apple has had time to percolate this change through the years of chip layout and fab, we should be able to run some general-purpose LLMs locally starting soon. All of Apple's M5 processor computers should be far more attractive than earlier processors for any sort of AI development or usage.

Hey Phil! I also watch WWDC ;). I won't be running anything locally for this series, but maybe in the future. (Though I did just get a new laptop from our IT department earlier this year and won't be due for an upgrade for a while, so I won't be benefitting from the M5 improvements! But I really can't complain.)

As far as whether it's any different to run things locally, functionally it isn't. It's more secure, obviously, and relatedly it's more private, but the actual functioning of the thing is not really particularly different. The time it takes to send and receive information to/from LLMs is really not that big a component in the total runtime because it's generally all just text. The extra time taken by network communication may in fact be canceled out by the fact that large companies are already running their setups on specialized hardware, but even if this isn't quite correct it's probably close enough as makes no difference. (It's possible that there are delays based on queuing of their massive numbers of users, though!)

"Neural accelerators" -- optimally efficient matrix multiplication -- have just been added to the GPUs in Apple's M5 processors. The M5 Pro, M5 Max, and M5 Ultra chips should be mightily impressive for high-octane AI researchers and users. At least machines with M5 Pro and M5 Max should be available in the first part of 2026. It sounds like Apple will suggest multiple Mac Studio boxes with a Thunderbolt 5 "bus" for its highest-end customers to distribute computational tasks to multiple machines. The Wolfram Language core numeric computation engine could be seriously enhanced over the next software update or two.

A question for @Arben: it seems as if the Wolfram AI Course Assistant would be ideal for any questions in a Wolfram course -- including double-checking answers to Wolfram quiz questions. As a member of the faculty, are you encouraged or discouraged by the way students could use these marvelous tools?

Dark mode and chat cells and forward slash do have some problems with readability

This is being looked into :)

Hi Arben. My notebooks seem to be stuck going to ChatGPT. How do I undo the? Do I need to get the notebook advisor?

You know, I was flustered this morning by the account issue and called that forward slash when it's explicitly backslash :). But—as you mention—it is the key above Enter/Return on most keyboards.

My only thought as to why it might not work is an obscure/arcane stylesheet setting. If you open a fresh notebook with cmd+N, immediately open a Chat cell, and type \ in there, does it work? If not, you may want to contact support@wolfram.com because I'm not sure why that would be...

So think I found problem. If you have dynamic content in the notebook and don't "approve it" in the banner showing at top of the window, then the forward slash does not work in a chat cell.

Working at November 10, 2025 at 090351_AM.