Hi Alec,
Thank you for this very thoughtful reply.
I 100% agree with you that
Programming languages are fundamentally about representing an abstract computer that developers know how to talk to and which is ideally easier to talk to than the underlying hardware. Language models have demonstrated an ability to... well... model language...
I would add that computer languages are (or are supposed to be) much more highly structured than natural languages and therefore easier to model. Plus, as I keep repeating, in the case of WL the language itself is computable and is architected in a tree-like structure that can be traversed programatically.
From what I have read, I also think this is likely to be true:
I would bet over 95% of accepted results required further modification from an experienced developer to actually solve the problem at hand.
However, I think the capabilities of ChatGPT/Co-pilot may be greater than you suggest. Here is one brief anecdote from a source I trust, who is a highly experienced quant. He was playing with ChatGPT over the holidays and asked it to build a Monte Carlo simulation of a Geometric Brownian Motion process. He was highly impressed with the Python code that ChatGPT produced in a couple of minutes which, he told me, would have taken him a couple of days to program himself. I didn't have the heart to tell him I could have probably done the same in WL in under a minute! The point is: he had the domain knowledge to assess that the output was correct and it saved him a bunch of time. The fact that it was a couple of hundred lines of Python code vs 1-2 lines of WL is neither here nor there, if they both work and take about the same time to produce. That's the "threat" that NL interface platforms like ChatGPT represent.
Your next point is well-made and I agree with much of it:
Beyond that, the difficult part of programming is not typing the syntax. I think it is instead figuring out what problem you want to solve and how to integrate that solution into the larger system in a coherent way.
The real issue often is the part about figuring out the problem and how to integrate the solution, and you illustrate that very well with you Wolfram Alpha examples. But here, too, NLP interfaces can help the user iterate towards a correct formulation of the problem, which is indeed what happens with your WA examples.
But my complaint is that we are well past the point where Wolfram Alpha should be able to figure out the answer to the instruction:
- generate a list of random ketones
just as easily as it does for the instruction:
- generate 5 random ketones
It's frustrating that in the first case WA isn't capable of generating any kind of response. It should be able to (I am quite sure that ChatGPT would). So the issue is that the NL interface for Wolfram Alpha currently just isn't good enough (and there really isn't one for Mathematica/WL in general).
Secondly, I do not agree that programming syntax is not an issue. I believe that for the WL it is very much an issue, especially for beginners. Furthermore, even for experienced users like myself, there are scores of functions I almost never use, or am even aware of - other users have even begun to complain about the bloating of the language with duplicative functionality. It would be great to have an NLP interface that is able to deploy any and all WL functions and programming concepts and show me their syntax and utility - I would learn a huge amount that way.
I believe that, in general, WR is currently underperforming in terms of producing ML-oriented functionality. This applies not only to the issue discussed here, which I would say is a key missing ingredient from the Wolfram tech stack, but also more mundane but critically important ML architectural features such as Transformers and frameworks for Reinforcement Learning which Matlab, for example, has offered in least the last four releases. It's past time for WR to step up to this challenge.
Wolfram started out with a considerable competitive advantage in terms of its thoughtful application of a NL interface to a curated set of high quality computable data in Wolfram Alpha. But that was years ago and the lead has been squandered. Competitors are going to eat WR's lunch if it doesn't respond soon.
If it were up to me I would be prepared to divert some large percentage (say 20%) of dev resources from other projects to get this done - it's far more important than ongoing projects such as the Wolfram compiler, for instance.
Release 14 should be about Machine Learning and a new NL interface for the WL. Everything else can take a back seat, for now.
