George and others. Of course, the so-called hallucinations are problems that must be brought to an absolute minimum. Unlike some people, I have a lot of patience to put up with obvious errors and find a way of getting at the truth. How do I know the difference? I don't always, but fortunately hallucinations don't seem to be based in difficulty! Sometimes, I can ask a competing Ai in a different context. Sometimes ask another for a proof while putting it off as my own naive idea. Sometimes do a little research. Sometimes the hallucination just makes no sense!

As far as "reasoning limitations go," like proving something clearly provable, but I've never seen written out and am too lazy to think of on my own, giving the Notebook assistant hints from my best reasoning as I go and asking for a step or two at a time seems to work well so far.

Here is an example of where we both just had to hold each other's hand to scratch out a proof that was sufficient for me:

I think it would be useful to have user input on what the Notebook Assistant should do, and this may help the software engineers craft a tool that is useful in the real world.

The discussion is not interested in how these goals are attained. I may be skeptical about the general utility of LLMs, but that is really not the point. Wolfram Language and its ecosystem has grown large and complicated enough that even the most experienced users can use help.

Some of the items on this wish list have been suggested by demos of the current iteration, plus some of my own exploration, so I think that they can be realized.

The main issue in any of this is what 'reliable' really means in context. For example, LLMs may produce useable output 90% of the time for general use, but 90% is not sufficient for mission critical tasks, and I count producing code as mission critical.

So here is my first stab at a list:

1. Discovery The assistant should be able to find useful functions in both the built-in language and function repositories. That is, if I ask the assistant to make a function or answer a question, it will (always) make use of built-in functions or vetted functions in the function repository. Stephen has commented that the language has grown so large that even he cannot remember all the functions.

2. Validation This wish has several parts. The first is that any code it presents should have been validated before the user sees it. Validation comes at several levels, of course. The lowest level, and one that the code should always meet, is syntax validation. (The code will not create any red or orange boxes, for example.) The next level is that the code should pass some basic unit tests. This, I realize is a non-trivial task, but I think it is doable with the current level of technology, and probably not requiring machine learning of any kind. The code should be checked to see if it actually answers the user's question. I think that this is the intent of the Notebook assistant, and may be difficult to achieve without technology beyond LLMs. Again, there are degrees of compliance, and getting 'close' for suitable definition of 'close' may be good enough. I understand that there will probably be some iteration needed to get a satisfactory solution, so the object is to get close enough so that the amount of back-and-forth is minimized.

3. Debugging This wish is related to the second one, except there the object is validation of code that the assistant creates, where here the object is to fix or improve the user's code. Clearly, if the assistant can do this task with the user's code, it should be able to do the task with its own code. [In my experience, this is the most frustrating aspect of the current Notebook Assistant: it can't fix simple errors in its own code, let alone more subtle problems with my code. ]

4. Boilerplate operations Stephen has hinted at this task. Once I have some function working, I should be able to ask the assistant to prepare it for the function repository, for example. There are a whole lot of other tasks of a similar nature.

That's it for now. I would really appreciate comments and additions.

Wolfram Language (indeed any advanced software system) needs this level of technology to deal with increasingly complicated tasks. Beginners, as well as advanced users need help, although of different types. I think that the community can help with the design as well a being beta testers for new systems.

From comment - "Bottom line: the current iteration of Notebook Assistant is a pricy toy rather than a practical tool for most WL users."

Totally agree with you - I am a Hobbyist, been using Mathematica for approx. 20 years - I immediately "invested" in the annual subscription Notebook Assistant on release in December and could not get it to give me any meaningful result without syntax errors - very disappointed.

As a MINIMUM requirement - Notebook Assistant SHOULD return code the works and does NOT have syntax errors. I also subscribe to ChatGPT Plus and Gemini advanced to help me with my coding and syntax errors - Both of those were working better than Wolframs Notebook Assistant to correct syntax and produce working code. That said I recently upgraded to Mathematica 14.2 and the Notebook Assistant is working better. So glad other people have been feeling similar frustration...

I expected that an expensive paid additional add-on like Wolfram's Notebook Assistant would have produced perfect syntax error free code - initially I was very disappointed, but I think it maybe better now - let's see...