I think this nicely illustrates some of the strengths and weakness of ChatGPT, as I understand them.
Firstly, the answer given has some elements that appear correct and clearly suggest some grasp of the WL syntax. Consequently, the answer seems plausible, to someone with limited knowledge of WL.
Secondly, however, the answer contains several basic errors. For instance,
Total[{{1, 2, 3}, {2, 3, 4}}]
will not produce the answer 15, as claimed, but rather:
{3, 5, 7}
More importantly, the bot fails to spot the root cause of the problem, which is the missing & at the end of the anonymous function, as in:
Total[# - 1] & /@ {{1, 2, 3}, {2, 3, 4}}
{3, 6}
So, in summary, ChatGPT's answer is plausible, but completely incorrect and it fails to identify the real issue.
Now, imagine that instead of a toy programming problem like this you were faced with finding a bug in a much more complex code base, perhaps involving thousands of lines of WL code. In those circumstances it might be very difficult indeed to validate the answer given; or worse, the given answer might appear to work, but only for some limited set of circumstances (which only becomes apparent much later, perhaps not until the system makes it into production).
An even worse scenario: imagine using ChatGPT to try to figure out a proof of some complex mathematical theorem. The proof could be highly complex and take a very long time to validate (as indeed it sometimes does with theorem proofs produced by professional mathematicians).
And then, finally, imagine the consequences of using a tool like ChatGPT to diagnose and suggest remedies for "real world" problems. Sounds implausible: but school children using this tool might be fooled into believing that the plausible but flawed solutions offered by the bot are somehow valid.
That could quickly become a very slippery road to find ourselves walking down.
