Without any jargon.
There is a book title: The Notation is not the Music. I have not read it yet (mostly on Renaissance music), but the title is an idea that any classically trained musician would agree with.
You can make analogues with many other arts: the words are not the poetry, the shapes are not the art, etc., etc.
Importantly, you can make the assertion that the notation is not the maths.
There is a way of looking at maths, called formalism, that essentially turns this around -- the notation is the maths. It is the core principle behind the rewrite rules that is the basis of Mathematica.
I prefer to think of maths differently -- along with a lot of other mathematicians -- in a more metaphorical or intuitionist manner. One might have thought that Gödel would have struck a fatal blow to formalism, but, in fact, it is a useful way of looking at maths, as long as it is not the only way.
This way of looking at maths is also much easier to express in code.
From what I can tell, LLMs are all formalist in design -- the words are the ideas.
When I learn to play a new piece -- true of most musicians -- I go through a stage where I can play the notes, and then there is a much more difficult stage of being able to play the music. LLMs seem to be happy with reaching the first stage, which for a suitable definition of success, they may be close to.
However, if we as a culture redefine 'intelligence" and language to match the definitions implicit in LLMs, we will lose something important.
How all this will play out is uncertain, especially since there is a lot of money involved (and the world-view of the designers of LLMs are congruent with the dominant neoliberal view of economics).
For my part, I chose to take a more holistic and metaphorical view of reality.
