I was trying to see what Chat would provide with respect to analyzing text in Mathematica It responded with:

pdfFilePath="path/to/your/file.pdf";
text=Import[pdfFilePath,"Plaintext"];
summary=TextSummary[text];
summary

When I asked about what TextSummary[text] was it seem it seemed to be hallucinating with respect to making up an imaginary Mathematica function, i.e. TextSummary[]. Am I correct in that this is a hallucination and if so, how can I prompt to avoid this? When I asked about TextSummary it responded with

Sure! The ChatCodeInlineTemplate[TextSummary] function in Mathematica is used to generate a summary of a given text. It takes a string or a list of strings as input and returns a concise summary of the text. The summary is generated using natural language processing techniques, such as extracting important sentences and phrases from the input text.

Is there a Mathematica function called "TextSummary?

Seems to be a major failure in GPT -- when it doesn't "know" something, it makes stuff up. Dealing with an arts/humanities type discussion, it is so often wrong, it is useful only as a curiosity.

However, I am sure that with a bit of tweaking, LLMs could be the next step in coding. When I started coding (early 1970s), OSes did little more than manage the file system. NO LAPAK, etc., and you had to write essentially everything. This has evolved (good thing) with things like the Macintosh Tools, etc. Every advance in developer environments has been to have the computer do more of the boiler-plate, for an ever-expanding definition of 'boilerplate'.

Depending on what you mean by "that", it is exactly the issue. I have been following AI since the 1970s, so the issues are nothing new to me.

All ChatGPT 'knows', if we can even use that term, is how to select the next word, given the previous ones. Truth, or accuracy are not even a consideration. In a sense it is always just "making stuff up", since it has no understanding (in the human sense) of what it is doing. What is surprising is that it does pretty well for the prose equivalent of elevator music.

Other than an increase in sophistication (and hardware requirements), we are not much beyond where we were in the 1970s. I remember implementing Eliza on my Apple ][, and I had some grad students who were convinced that the program was intelligent, even after showing them the source code. People hoping to make a buck from LLMs seem to be the easiest to convince.

All of this is an example of the overreach of formalism (and yes, I know what that means). Try reading McGilchrist's The Matter with Things for a discussion in way more depth than I have time for here.

Your point seems to be that ChatGPT makes mistakes (hallucinates) because it does not have access to information (in the Shannon sense). That was exactly my point. If a LLM (it probably does not need to be that large) had access to, for example, only examples of well-written WL code (e.g., the internals of Mathematica itself), then it would do a pretty good job at creating WL code -- especially for problems it had seen something similar to before.

I really suggest that you read the book.

We may be talking past each other. I'll look at the video. Read the book.

Thanks, I will take a look at them. I have a lot of copious free time.

Remember that I have been through 4(?) waves of hype concerning AI, and each the proponents of each one were just as earnest and expert as the current batch.

I still recommend the book.

Hi Michael,

Thank you for your excellent post.

I have an interesting challenge for the computational capabilities of ChatGPT-4, which is to come up with a 1-line WL program that is capable of winning the Wolfram 1-liner competition (this isn't about winning the competition, obviously; it's just an aspirational standard).

This is a very difficult challenge, not so much due to the program-length constraint, but because it is so open-ended and it requires us to think about what constitutes "interesting" or "surprising" output.

So far, my own efforts have been a resounding failure: ChatGPT triumphantly produces the chart of a Sine curve, or something at that level, even when I try to provide extensive guidance in the prompt.

My sense is that you are the man to take on this challenge, given the levels of creativity and expertise required. I hope you will give it some thought.