After reading about this article on testing the educational benefits of chess, I was wondering what would constitute a decent experiment for the benefits of learning Wolfram Language?
Certainly there is a lot of common sense behind Wolfram's educational initiatives and Computer based maths. Teach some useful skills with powerful software, possibly even in place of traditional mathematical education. Among the range of alternatives to traditional education the possibility here is positive. More fun than memorizing facts and logical rules. One could just drop math altogether, but instead this is a way to still do math, just in a modern sort of way.
I am not alone with anecdotal evidence that learning Wolfram language can make you smarter. I've seen a high school dropout who did not know the line-slope formula (or any formula) pick up Wolfram Language and become very productive. But how could one measure it? It would seem wrong to measure something like the learning of formulas that the student no longer needs to learn.
The article by Sala and Gobet offers perhaps a bit of caution comes from its observation that the improvements might not be as great as other things, like learning music.
Interesting question! Are there studies on the benefits of the more general edge cases: (1) Learning to program, regardless of language, and perhaps not even with maths as a primary focus, and (2) using software to "look at" maths, plot things, solve stuff, but more in the "graphing calculator" sense than in the "programming" sense. Both of these aspects may give you ideas about how to proceed here, and there must be some data on that already.
And as you said, there also needs to be a way to put these benefits in perspective and compare them with other activities, because school time is a limited resource.
(By the way, "traditional maths" can teach a very different perspective and skillset from computational thinking, so I really wouldn't throw either one overboard so quickly. You know... it's fine to enjoy the book and the movie.)
Side-stepping the naming of languages and returning to the original prompt, I am putting together a education platform to gather data and answer these types of research questions, http://www.codeseal.org . The terminology being used lately to describe the type of cognitive problem solving strategies learned through training in computer programming is "computational thinking". There are centers at places like Carnegie Mellon studying the topic in general https://www.cs.cmu.edu/~15110-s13/Wing06-ct.pdf. My work is focused quite explicitly on looking at how learning to program within the context of a particular discipline affects learning, e.g. "does modeling a polymer improve one's understanding of polymer formation better than deriving the mathematical formulas?", as opposed to the bigger question of "does programming in general affect how one thinks?". You pose another interesting question which is "does programming in different languages affect the learning outcomes for students in different ways?".
Very interesting, @Kyle Keane, that platform deserves a separate announcement, great work! I'd love to see you sharing your experience building it in a separate thread.
On the importance of computer programming to cognitive ability: Jobs was asked what practical purpose he and other early Apple employees envisioned for programming personal computers. "It didn't have to do with using them for anything practical; we used it as a way to mirror our thought process. I think everyone should learn how to program a computer, because it teaches you how to think. I view computer science as a liberal art, something everyone should learn to do."
Good points from Bianca and Richard. Maybe your questions highlight some paths for experimentation while exposing some possible problems. The main one I think is the choice of programming language, and even the questions like what is a program and are there different ways of thinking.
I am not an expert and hardly familiar with educational research but found some old articles from the 1980s. This one is about the cognitive effects of learning computer programming. They examined Logo, the claims made about it, and concluded that it neither helped learn rigorous thinking (traditional math) nor was there evidence that it helped mathematical explorations (something the computer advocates were claiming). The latter could be the thing to test about Wolfram education.
It is worth noting that "what is a program" has evolved quite a bit. Nowadays we think of all the different ways of interacting, from traditional code to dynamic graphical manipulations to natural language Wolfram Alpha queries, that all of these count as programming. You also have the simple programs like cellular automata. One thing this means is the notion that bugs in a program is not really relevant like it was (or like it is with other programming languages).
Maybe it is naïve to say this, but I would claim that rigorous thinking is not the exclusive domain of mathematics. I am not sure, but it seems to me that different programming languages promote different types of thinking. (Anyone disagree?) Maybe the right place thing to test is for someone exposed to Wolfram Education in its various guises, whether they can independently explore a problem. This is the main claim (I think it is) that learning this language will let you explore solutions to problems you could not solve before.
If the subjects did not know how they were being tested, one measure could be the volume of material they produced, and another measure could be originality (high scores for distinct results). Of course, it would be easy to cheat those measures, but a bigger problem is that there could not be a control group who does not know how Wolfram Language and hence could not produce any material (except on paper). Likewise, it would not be a fair test to give a math problem to students who can use Wolfram Language and those who cannot. It would be like a road race where one guy gets a car.
It's hard to find a fair comparison because learning to program in a versatile thing like Wolfram Language impacts so many other potential activities. (music?)
Your summary of the case of Logo seems interesting and somewhat discouraging, although I don't have time to read the full paper. Anyway, that was over 30 years ago, and I'd expect the increased accessibility of technology in everyday life to play a big role here. I think it's actually getting easier to get people interested in "making the computer do stuff", but that's (1) just my guess and (2) not even a well-founded guess considering I'm younger than that paper.
You also mention the changing perception of what constitutes programming. I think you're overestimating this, or at least it's only true for people who are already immersed in Wolfram technology. It's worth noting that the Wolfram Language is fighting two separate battles at the moment. (Sorry about the anthropomorphization of WL, I think this is not exclusively on WRI as a company, so I'll just pretend that the wolf is conscious.) On the one hand, WL wants to be counted as a real and proper programming language, and on the other hand, it wants to change the definition of "programming language" at the same time. Outside the wolfish ecosystem, "interacting with the computer by talking normal English" is most certainly not considered "programming", it's "using a nifty (wannabe-)AI program someone else wrote". (Just like "I'm programming my video tape recorder" was never really "programming" - not to a programmer at least.) Also, I believe a lot of people still don't usually consider "using Mathematica" a form of programming, or only in the sense that using a graphing calculator is "programming". (And I still maintain the opinion that this is one of the intrinsic problems proprietary languages will never shake entirely, but that's neither here nor there.) It's comparatively easy to be an established programming language and then try to broaden the scope of the term. It's also not so hard to establish yourself as a new programming language if you already look like one. Trying both at the same time is ambitious, and I like it, but the process is only just beginning. And it won't bring about a paradigm shift if either one of the two aspects falls behind. ("Mathematica can do natural language processing" and "WL is a programming language", each taken just on its own, are both no big deal conceptionally.)
Anyway, where I was going with that last paragraph is this: If we use Mathematica in school, will students even think that they've learned to program, will they be able to transfer their knowledge to other languages easily, and will other people (particularly potential employers) acknowledge that they can program if they do things like (Ctrl+=)NYC["population"] and call it a web app? Or is the more realistic approach that we try to enable computational problem solving in everyone, especially people who don't want to be anywhere near the technology sector, perhaps even well outside the STEM fields altogether? That seems somewhat more feasible at the moment, and STEM-phobic students might actually learn more maths this way than through a traditional approach. I'm not trying to argue for or against anything here, I genuinely don't know what the bigger picture could be expected to look like a few years down the road.
As for this statement: "Maybe it is naïve to say this, but I would claim that rigorous thinking is not the exclusive domain of mathematics. I am not sure, but it seems to me that different programming languages promote different types of thinking. (Anyone disagree?)" Totally agree with all of this.
And finally, you say that "This is the main claim (I think it is) that learning this language will let you explore solutions to problems you could not solve before." I think I know what you mean, but I had to chuckle a bit. Isn't that the main claim of all forms of education? What else is education supposed to do? But you're right, the ideal measurement of benefits of WL in education must be something along the lines of "what does this enable people to do further down the line". I think that's actually a reasonable measurement for education in general (if you want to go beyond employment rate and average salary as an indicator of success). But I don't have any concrete ideas on how to do this either (considering I still don't want to actually dig into the literature on this).
Just as an aside, you also said that "One thing this means is the notion that bugs in a program is not really relevant like it was (or like it is with other programming languages)." This is interesting; in which way are bugs in WL less relevant than in other languages? Because I've been hunting some of these critters just last week, and they sure looked relevant enough to me!
WL is most definitely a programming language. the fact that you can or may be able to do other things (e.g. computations within Mathematica) in no way takes away from WL being a programming language. and the fact that it is difficult, and sometimes impossible, to transfer what you can program in WL to other programing languages is irrelevant. As a term rewriting system, WL using different programming constructs and methods of evaluation than programming languages that follow different paradigms but that doesn't make WL any less of a programing language. i personally think that the first instantiation of WL within the Mathematica application, the confusion created by (mis)identifying Mathematica with WL, the ongoing inability to be able to create WL notebooks rather than Mathematica notebooks (you should be able to create a WL notebook (having the wolf logo) within Mathematica), the statements made that WL is a multi-paradigmatic language may have been useful in getting WL used initially - remembering that in 'the old days' programming was something done by physical scientists and engineers (STEM people), not by liberal arts types or even mathematicians, and Stephen, recognizing this, stated that he was using Mathematica as a trojan horse to smuggle in a new programming language to people who had no interest in learning a programming language per se - has caused some problems in getting WL recognized as a programming language (somewhat like having Donald Trump run for the Republican presidential nomination may have brought new people into the Republican party but at the cost of fundamentally changing the definition of what the Republican identification represents, for better and for worse). In fact, i think that it is the recent influx of people interested in doing big data analysis, internet (smartphone and tablet) app development and other 'non-technical' tasks that is now allowing WL to come out from hiding beneath the skirts of STEM and stand on its own as a programming language.
Richard, I agree complete with what you're saying, of course WL is a programming language. It's just that from personal experience, I'm not so convinced that that's the public perception just yet. So if that's still a work in progress, then it probably hasn't yet caused a broader paradigm shift. Maybe I'm just too pessimistic though.
You can already create WL packages and scripts, maybe the notebook is coming at some point?
here's something to think about - it has recently been found that women do not avoid going into STEM fields specifically but rather they avoid going into fields that are perceived as requiring brilliance or inherent ability rather than hard work (google gender field brilliance). yet every time that stephen gives a lecture (e.g. at SXSW), the advertisements for and discussions of his talk emphasize that he is (was?) a brilliant theoretical physicist and a child prodigy (apparently there is no age limit on having been a child prodigy being a citable credential - note: it's NOT stephen's fault that others mention it when they refer to him. they even even credit him for writing scientific books as a youth that were not published (i wish i was given credit for the scientific work that i did but didn't publish). this cult of celebrity pervades the scientific community as much as it does the entertainment community. but i wonder if this emphasis on stephen's personal brilliance/genius might actually be counter-productive in promoting WL. and his live demos of WL coding might add to this problem, causing some people to conclude after watching stephen hack WL code, that they can't master WL themselves (we know that isn't true. i've taught WL programming to literally thousands of people (men and women) - close to 10,000 at this point - and it is far easier to become competent programming in WL than in ANY other programming language (at least when it is properly taught).
That's an interesting and unexpected perspective. Lots of programming languages have very charismatic and brilliant leading figures, I've always considered that a good thing. After all, a mediocre language designer without any vision is highly unlikely to develop an intuitive and usable language (let alone a beautiful one).
as far as i know, almost no programing language is identified with a specific individual (except maybe ken inversion for APL). and few people name the programming language they develop after themselves (i could be wrong - maybe bob dylan named the Dylan programming language after he created it when he wasn't writing songs LOL).
I don't know, people like Guido van Rossum or Yukihiro Matsumoto are pretty well-known, but you're right about the naming of course. (Although the naming of WL (a) made sense because the company already had that name and (b) there was a blog post asking/polling for suggestions and WL won among the users.) Other technological companies also have this type of well-known leaders without becoming intimidating, Steve Jobs certainly never scared anyone away from an Apple product. Also, consider how much popular opinion loves people like Elon Musk... Charismatic leading figures are a generally positive and inspirational thing in the tech world, and they're allowed to be all sorts of nerdy.
Anyway, I don't think it will matter much in an educational context. Students are exposed to exceptional people all the time because those are the people whose work we study and whose inventions/discoveries we use. Students would have to be scared all the time if this was really a major issue. Besides, you can teach a programming language just fine without talking about its developers at all.
Like you said, WL is a very accessible and non-scary language for people without an extensive STEM background, so in the interest of Todd's original question it probably has a lot of potential for actually producing benefits beyond mathematics education.
i just had to google those two guys to find out that they created Python and Ruby, respectively (i'm a theoretical physicist and those are the nerds i know). you can't compare the two s'es (steve jobs and stephen wolfram) - btw, i assume you know that steve suggested the use of the name Mathematica to stephen
totally irrelevant story. i was at the first Mathematica Conference (the reception buffet served shrimp - or prawns - the size of your hand) which i recall being held in Redwood City, or maybe it was in SF) and i was walking around the meeting hall and i saw steve and stephen conversing in a small room. my instinctive reaction was 'how can those two guys breathe with their combined egos taking up all of the oxygen in the room' LOL
jobs had charisma and the ability to create a 'reality distortion field'. it was incredible just to be in his vicinity. stephen's public 'gift' is that he exudes intelligence. i've known stephen since 1986 - when Mathematica was being created at the U of I - but i didn't talk to him for many years out of trepidation that i would say something that he would think was stupid. (and i don't get intellectually intimidated easily, not even by my nobel prize winning friends).
anyway to return to the theme of this discussion, i think that WL is just now beginning to reaching beyond STEM nerds to non-STEM nerds as Big Data and Digital Humanities become increasingly important. this is part of an ongoing revolution. i expect that the search for 'cause' that characterizes science will be (is being) replaced in large measure, by the search for 'correlation'. this drives us (were?) scientists nuts but that's their problem (not mine - i don't think we can ever understand reality; we can only model it).
and i think WL's pattern matching capabilities will become increasingly important. human thought is characterized by analogical reasoning which is really a type of pattern matching (by semantic rather than syntax... or do i have that backwards)? without question, the pattern-matching capabilities in WL are unsurpassed in any other language that i know of. i realize that pattern matching in WL is relatively slow (and discouraged by some excellent WL programmers) but i follow stephen's dictum expressed 30 years ago " you shouldn't develop [or use - my view] software based on hardware limitations because hardware keeps getting better and faster and you don't want it to outgrow the capabilities of the software". besides using pattern matching in WL is just too much fun (even more than making nested anonymous functions which i love to do) not to not utilize.
I wasn't comparing anyone to anyone else at all, just dropping a few more datapoints for my statement that in general, people are attracted to (and not scared away from) technological products when they're really impressed by a preeminent leading figure.
So... back to the topic at hand, then: If we want to find a good way to measure the impact of WL, we probably need some statistics beyond the usual anecdotes. Who is actually using the language, and what are people doing with it? Especially outside of academic contexts, because that's the world many students want to be prepared for. Can we see some concrete quantitative evidence for WL becoming important outside of STEM, and/or outside of academia? (Obviously not a question you or I can answer conclusively, but someone at WRI must have a bit of data on this... Maybe sales numbers and support requests to start with?)
we probably need some statistics beyond the usual anecdotes.
we probably need some statistics beyond the usual anecdotes.
The TIOBE Index for May 2016 sees
$Mathematica$ on a place 50+. It is a programming community index. If we concede that Nobel Laureats as well as Fields Medallists ... and Mr Wolfram ... are using Mathematica or WL, we have a statement about the fitness of
$Mathematica$ to do things such people usually do.
A language must not be everybody's darling but it should encourage original ways to get things done -- to phrase it as inaccurate as possible.
A propos Fields medallists, read the funny "Hello World!" experiences by Timothy Gowers for recreation.