This article has a nice sense of humor. And we can all be glad that A.I. isn't better than us at math yet.
The original programmers are aware of the issue:
ChatGPT sometimes writes plausible-sounding but incorrect or
nonsensical answers. Fixing this issue is challenging, as: (1) during
RL training, there’s currently no source of truth; (2) training the
model to be more cautious causes it to decline questions that it can
answer correctly; and (3) supervised training misleads the model
because the ideal answer depends on what the model knows, rather than
what the human demonstrator knows.
Someone or some article also claimed that ChatGPT could write Haiku poems, so I asked for a few and
found that ChatGPT could reliably create seemingly reasonable examples. However, I also noticed something odd.
In every poem the 季語 (kigo, season word) came on the final line, and was quite obnoxiously the season itself. This shows a lack of creativity, since most Haiku poets have studied lists of kigo, and would try and avoid such bluntness. Saying "Spring" or "Autumn" is essentially wasting a word, when you already start out with so few!
The other issue was that ChatGPT couldn't get anything nearly like 切れ字 (Kireji, "cutting word"). Perhaps this isn't to the fault of the programmers behind the show. Given the description available on wikipedia, it seems even humans have difficulty understanding or agreeing what is meant.
It would be interesting to see if a pre-trained model could ever write a good Haiku poem. I guess a basic question is whether the notions of kigo and kireji can be inferred from pre-existing samples. I'd like to think there's a bit of deductive theory behind it, but who knows what we'll find. The technology is already quite skilled at conversation, even if it does seem to blather at times.
Congrats, to OpenAI and Wolfram | Alpha, for developing interesting products with different strengths! Looking forward to seeing how "source of truth" issue is solved, and not just in math.