This is impressive technology when it works. The benefits are so significant that it is worth the effort to make it work.

I have been experimenting with Notebook Assistant for a while, and it often gives useful results. When it does not, it can sometimes be 'led' to doing something useful interactively. Sometimes, it is simply off the wall.

As an experienced user, I can often help the assistant, but even when this works, it usually takes more time and effort than simply doing the work itself.

It is possible that the assistant works well in a restricted domain of expertise, and my experiments are not in this domain. However, trying the same input that Stephen Wolfram used in his demo, I not only got different answers (expected, due to the stochastic nature of of the technology), but the hallucinations were far from anything that could be fixed interactively.

In my opinion, for this to be a valuable technology, it needs to do the right thing almost all the time. Just what 'almost' means in this context is probably subjective, but consensus can establish a pragmatic lower limit.

In an educational setting, I think that the technology needs to do the right thing all the time.

All LLMs seem to have a problem with 'hallucination'. Whether this issue can be eliminated is an open question. Based on my own experience and work with language and hermeneutics, I would guess that this is a fundamental flaw in the underlying design -- probably in the way language is regarded by the designers.

Recent events have shown that the cost of R&D in this area has dropped, and I am hopeful that a group with a different understanding of language can develop a more useful model -- perhaps one that is not also racist and sexist, and given to making things up.

Having said that, this technology is quite promising, and is an indication that it can be beneficial. I just hope that it is not oversold (the way AI has been for every other bubble), or worse, the defects are embraced as "features".