I thought I would test Notebook Assistant (NA) on something I actually thought it could do accurately. I have attached the Chat Notebook. I wanted it to develop code for a discrete FFT to multiply two large numbers. This is a well worn path. There is an enormous amount of code that was presumably part of the training set for ChatGPT so I thought it would spit out useful code. As background when I worked in Defence Science in Australia 50 years ago I was working with FFTs regularly for various things. The Great Internet Mersenne Prime Search (GIMPS) used a highly efficient assembly language FFT written by George Woltman that underpinned the massive calculations involved in that project. As you can see Notebook Assistant went to the Automatic Teller of Talent to make a withdrawal and found it had insufficient funds. I then gave it a Mathematica module I had written years ago which I knew worked and let it convince itself that my code worked. I asked what version of ChatGPT ir was and it was only version 4 so I asked ChatGPT 5,2 (which begs the question of why Wolfram is charging the same price for Version 2 in NA) whether my code worked and it said "no". After some Socratic badinage it admitted it was wrong and worked out why. I asked Claude Sonnet 4.5 whether my code would work and got a provisional thumbs up for "small" numbers but not ones of at least 1000 digits so I challenged it and it produced 2 1000 digits numbers which it multiplied using Python's architecture and got the same answer as my code - see attached pdf of the chat.
Anyone doing serious stuff needs to be very careful with these LLMs. I have found NA useful for what I consider "hole digging" type issues that do not require really sophisticated analysis. I have actually used ChatGPT on some reasonably sophisticated analytical problems and it has provided useful insights but you have to be very careful and check every step. Commercially I can't see why Wolfram can charge for ChatGPT version 4 in NA rather than the latest version - I'm paying twice and I may actually drop NA although the convenience of NA is seductive.
When you see breathless articles about Claude Sonnet 4.6 producing 100,000 lines of compiler code there is a disconnect between that and the sort of reality that I experienced on a really well worn problem.
Attachments:
