Interesting question! Are there studies on the benefits of the more general edge cases: (1) Learning to program, regardless of language, and perhaps not even with maths as a primary focus, and (2) using software to "look at" maths, plot things, solve stuff, but more in the "graphing calculator" sense than in the "programming" sense. Both of these aspects may give you ideas about how to proceed here, and there must be some data on that already.

And as you said, there also needs to be a way to put these benefits in perspective and compare them with other activities, because school time is a limited resource.

(By the way, "traditional maths" can teach a very different perspective and skillset from computational thinking, so I really wouldn't throw either one overboard so quickly. You know... it's fine to enjoy the book and the movie.)

Side-stepping the naming of languages and returning to the original prompt, I am putting together a education platform to gather data and answer these types of research questions, http://www.codeseal.org . The terminology being used lately to describe the type of cognitive problem solving strategies learned through training in computer programming is "computational thinking". There are centers at places like Carnegie Mellon studying the topic in general https://www.cs.cmu.edu/~15110-s13/Wing06-ct.pdf. My work is focused quite explicitly on looking at how learning to program within the context of a particular discipline affects learning, e.g. "does modeling a polymer improve one's understanding of polymer formation better than deriving the mathematical formulas?", as opposed to the bigger question of "does programming in general affect how one thinks?". You pose another interesting question which is "does programming in different languages affect the learning outcomes for students in different ways?".

Good points from Bianca and Richard. Maybe your questions highlight some paths for experimentation while exposing some possible problems. The main one I think is the choice of programming language, and even the questions like what is a program and are there different ways of thinking.

I am not an expert and hardly familiar with educational research but found some old articles from the 1980s. This one is about the cognitive effects of learning computer programming. They examined Logo, the claims made about it, and concluded that it neither helped learn rigorous thinking (traditional math) nor was there evidence that it helped mathematical explorations (something the computer advocates were claiming). The latter could be the thing to test about Wolfram education.

It is worth noting that "what is a program" has evolved quite a bit. Nowadays we think of all the different ways of interacting, from traditional code to dynamic graphical manipulations to natural language Wolfram Alpha queries, that all of these count as programming. You also have the simple programs like cellular automata. One thing this means is the notion that bugs in a program is not really relevant like it was (or like it is with other programming languages).

Maybe it is naïve to say this, but I would claim that rigorous thinking is not the exclusive domain of mathematics. I am not sure, but it seems to me that different programming languages promote different types of thinking. (Anyone disagree?) Maybe the right place thing to test is for someone exposed to Wolfram Education in its various guises, whether they can independently explore a problem. This is the main claim (I think it is) that learning this language will let you explore solutions to problems you could not solve before.

If the subjects did not know how they were being tested, one measure could be the volume of material they produced, and another measure could be originality (high scores for distinct results). Of course, it would be easy to cheat those measures, but a bigger problem is that there could not be a control group who does not know how Wolfram Language and hence could not produce any material (except on paper). Likewise, it would not be a fair test to give a math problem to students who can use Wolfram Language and those who cannot. It would be like a road race where one guy gets a car.

It's hard to find a fair comparison because learning to program in a versatile thing like Wolfram Language impacts so many other potential activities. (music?)

Very interesting, @Kyle Keane, that platform deserves a separate announcement, great work! I'd love to see you sharing your experience building it in a separate thread.

i just had to google those two guys to find out that they created Python and Ruby, respectively (i'm a theoretical physicist and those are the nerds i know). you can't compare the two s'es (steve jobs and stephen wolfram) - btw, i assume you know that steve suggested the use of the name Mathematica to stephen

totally irrelevant story. i was at the first Mathematica Conference (the reception buffet served shrimp - or prawns - the size of your hand) which i recall being held in Redwood City, or maybe it was in SF) and i was walking around the meeting hall and i saw steve and stephen conversing in a small room. my instinctive reaction was 'how can those two guys breathe with their combined egos taking up all of the oxygen in the room' LOL

jobs had charisma and the ability to create a 'reality distortion field'. it was incredible just to be in his vicinity. stephen's public 'gift' is that he exudes intelligence. i've known stephen since 1986 - when Mathematica was being created at the U of I - but i didn't talk to him for many years out of trepidation that i would say something that he would think was stupid. (and i don't get intellectually intimidated easily, not even by my nobel prize winning friends).

anyway to return to the theme of this discussion, i think that WL is just now beginning to reaching beyond STEM nerds to non-STEM nerds as Big Data and Digital Humanities become increasingly important. this is part of an ongoing revolution. i expect that the search for 'cause' that characterizes science will be (is being) replaced in large measure, by the search for 'correlation'. this drives us (were?) scientists nuts but that's their problem (not mine - i don't think we can ever understand reality; we can only model it).

and i think WL's pattern matching capabilities will become increasingly important. human thought is characterized by analogical reasoning which is really a type of pattern matching (by semantic rather than syntax... or do i have that backwards)? without question, the pattern-matching capabilities in WL are unsurpassed in any other language that i know of. i realize that pattern matching in WL is relatively slow (and discouraged by some excellent WL programmers) but i follow stephen's dictum expressed 30 years ago " you shouldn't develop [or use - my view] software based on hardware limitations because hardware keeps getting better and faster and you don't want it to outgrow the capabilities of the software". besides using pattern matching in WL is just too much fun (even more than making nested anonymous functions which i love to do) not to not utilize.

here's something to think about - it has recently been found that women do not avoid going into STEM fields specifically but rather they avoid going into fields that are perceived as requiring brilliance or inherent ability rather than hard work (google gender field brilliance). yet every time that stephen gives a lecture (e.g. at SXSW), the advertisements for and discussions of his talk emphasize that he is (was?) a brilliant theoretical physicist and a child prodigy (apparently there is no age limit on having been a child prodigy being a citable credential - note: it's NOT stephen's fault that others mention it when they refer to him. they even even credit him for writing scientific books as a youth that were not published (i wish i was given credit for the scientific work that i did but didn't publish). this cult of celebrity pervades the scientific community as much as it does the entertainment community. but i wonder if this emphasis on stephen's personal brilliance/genius might actually be counter-productive in promoting WL. and his live demos of WL coding might add to this problem, causing some people to conclude after watching stephen hack WL code, that they can't master WL themselves (we know that isn't true. i've taught WL programming to literally thousands of people (men and women) - close to 10,000 at this point - and it is far easier to become competent programming in WL than in ANY other programming language (at least when it is properly taught).