A great discussion, which obviously touched a nerve. For context, I am an ecologist, and one of at most a handful of ecologists who use the WL. I attempt to do almost everything in it, and succeed about 95% of the time.

Some of the more recent posts touched on Wolfram's computational data. It certainly blows people away when you use it in a demo: it seems like magic. But there is an analog to the 'production quality' issue: the built-in data have a black-box vibe to them, whereby tracing those data to a source can be challenging (or at least time-consuming). For this reason I would hesitate to use anything from the Wolfram data repository in an academic publication (my version of 'production'). Which makes it more of a tease than anything.

Overall, the issue seems to be one of breadth. Because the language is so capable in so many areas, Wolfram has built it out almost exponentially into different realms. (Stephen has proudly graphed this on more than one occasion.) It's why it is so fun to play with, and develop ideas. But to make each of those areas state-of-the art you would need equivalent growth in personnel, and that isn't realistic. Which is why so many application areas remain somewhere between 50% and 90% developed. It's a shame because the WL has more potential to be a 'universal tool' than any other I have seen.

The real strength of the platform as it is now lies not in any one application area (except maybe a few really well-developed areas, e.g., symbolic math and physics), but in the ability to move seamlessly between many. This is how I use it: I can ingest multiple large spatial datasets, process them, fit a statistical model, simulate something based in the fitted parameters, then visualize them on a 3D map, all in one notebook. There are tools that would do each part better and/or faster, but I would have to a) install and know them all, and b) keep saving and loading files.

But I suspect most potential users come looking for abilities in just one or two areas, and compare those to their narrow but highly optimized specialist alternatives.

Most users of phylogenetic software do not understand what it is actually doing, nor therefore the potential for inferential errors, let alone actual algorithmic or coding mistakes. Which is part of a larger discussion that goes well beyond phylogenetics. But how to solve the problem?

I'm curious, Gareth, if you feel isolated in your academic arena because so few other ecologists use the WL? Does this mean that others really can't peer review your work (since they don't know how to use WL) and you cannot peer review theirs, when they perhaps use other common ("free") tools like R, Stan & Python?

I invested in WL a long time ago, and resist learning other languages because WL is so productive. Even so, I would be hard pressed to recommend WL to my biology students. After all, what employer really wants WL skills?

The "R" package contains both unsound individual modules plus procedures that combine otherwise valid modules into an erroneous computation chain. Somewhere along the line, biology professors started equating "knowing how to program" with "knowing what to program".

Gareth, such an effort would involve eliminating several existing functionalities - so potential users would have to be educated as to why that is the case.

In my perspective, this suite of functions would be invoked by WL and perhaps a WL GUI app for novices; i.e. the R scripting interface would be eliminated.

Further, I'm a fan of commercially supported scientific software so I'd like to see Wolfram involved. This would mean that some of the software development monies went Wolfram instead of graduate students. This would also ensure the package did not become another free-for-all collection of erroneous modules.

Just to be clear - I have no grievances with community-based "non-computational" software. For example: LaTeX.

For the immediate future I'm heavily invested in a genomic ID and ancestry project for fruiting perennials. I'm willing to consider your proposal though. For further discussions feel free to contact me via the email posted on the webpage on my Profile.

Mathematica supports LaTeX with TeXForm, and if you press Ctrl+4, you can type TeX into Mathematica. Stephen Wolfram describes in his blog post where he announces the release of Mathematica 12.2 how you can enter TeX into a notebook.

Hi David, Thank you for your well organized, thoughtful post. I agree with much of it. Like you, I am happy to see the emergence of useful structures such as AxisObject. However, I have concerns about the reduction of "wrapper" functions such as the ListPlot family you mentioned. One is the effect on legacy codes of substantial length at site-licensed institutions - both public and private. The second concern is for the user base from the non-mathematical arts who have learned just enough WL to be productive. For example, consider applications in comparative semantics. A third concern is with function names that are synonyms. Consider Union and DeleteDuplicates. Should Wolfram attempt to force some users to increase their set theory vocabulary, or conversely force the avid set theory user base to change long standing mathematical terminology? I think not. So while the long list of function names deserves review, I believe there are many criteria to consider.

This is a very well thought-out reply with several interesting points.

Now I may use Excel to create a well-formed CSV file, which I import to create a Dataset and analyze it in an Mathematica notebook. I spent a lot of time to master Datasets and associations and now it pays off.

Dave may well be right that the trade-off between using Excel vs Mathematica for pivot-table type analysis comes down to a question of familiarity and dexterity with the WL functionality. Mathematica has a steeper learning curve than Excel, but once you have mastered tools such as datasets and associations you can do much more in MMA than Excel, of course. And there is a bonus in being able to do everything in one application rather than switching between two.

I believe this is what WR is aiming for:

If you have Mathematica, you shouldn’t need any other specialized application to do your research.

which leads logically to Dave's conclusion that:

WR should focus on finishing and polishing e.g. statistics and time series analysis, machine learning and a number of other application areas, so it becomes the go-to tool for research.

I agree in principle that:

If Mathematica becomes the indispensable tool for researchers, then more transparency about stability, reliability and accuracy becomes paramount.

However, that is always going to be something of a challenge for a development platform built with proprietary software. Similar issues may arise w.r.t. Dave's other appeal for more documentation of implemented methods.

I, too,

see a lot of redundancies i.e. new functions that do things slightly different from similar functions.

But I am less convinced than Dave that this may be cluttering up the language unnecessarily - the jury's still out on that question, I think.

I find myself in complete agreement with Dave's final conclusion that:

the Wolfram Language could become the computational ecosystem to develop, test and deploy code for all things computable. We have to some miles to go there.

Hi Tom,

I think a technically educated person should be writing with a word processor, doing reports with a spreadsheet, and thinking with a technical computing language as an extension of their thought process.

This strikes me as too prescriptive. Fifty years ago you would have had to write, perhaps: "a technically educated person should be writing with a fountain pen, doing reports using a pocket calculator and hardly thinking at all about technical computing languages, unless they are computer scientists".

The point is: time moves on. If you work at Wolfram Research I imagine you (try to) do everything in WL, whether it be writing text, producing some analysis, or programming an application. Certainly that's my understanding of Stephen Wolfram's expectation, anyway. For myself I do a large proportion of my work in WL, including carrying out analysis, creating presentations, and programming applications. It's very convenient to use a common UI and tech stack for all these varied tasks: it save shifting from one application to another, having to be concerned about file format compatibility, etc.

A technical computing language is not the right tool for producing commercial computer applications.

I understand your point, but I expect that there are legions of Fortran programmers, to say nothing of the good people at WR and Mathworks who would heartily disagree with you. And indeed on the pages of this post several examples are given of commercial applications developed in WL (including one or two of my own).

To your point, however, it's obviously a significant challenge to develop a commercial application in WL, even though it can be and has been done. I certainly wouldn't regard the development of commercial applications as a forte of Mathematica, in they way that, for instance, symbolic programming clearly is. Which goes to the topic of this post - i.e. what is Mathematica for? And I would have to concur that the development of commercial applications is not one of those things - although I am quite sure that Stephen Wolfram and others at WR would strongly disagree.

My answer: “it’s an extension of one’s thought process.”

This is a bold claim for any programming language/technology platform, including Mathematica. Could one not make a similar claim about C++, Java, or Julia? I suspect that, unless you are a professional developer, probably not. Mathematica has certain unique features that make it a natural fit for the rapid exploration of ideas, in a way that certainly isn't true of most other computer languages that I am aware of.

I think that your description encapsulates what Stephen Wolfram may be aiming for: a kind of cognitive sand-pit offering a plethora of computational tools that provide a unique way to explore ideas, whether experimentally, graphically, or analytically, that will assist the process of developing them into functional algorithms.

You may want to revise your 'way-back' time. Lotus did not come out until 1983. The HP-35 debuted in 1972, but the programmable HP-65 came out in 1974 -- at a whopping $795 -- equivalent to $4k in today's dollars. HP portables no earlier than 1987.

I started coding in 1972, and it was on a timesharing PDP-5. No personal computers at all -- Altair 8800 debuted in 1974. I got an HP-35 shortly after it came out, which meant that I could ditch my slide rule and log/trig tables. Most of us who learned to code in those days didn't take CS, because most schools didn't offer them. (I was a maths major, so I did have numerical analysis.)

So, 50 years ago (1972), I was writing on a typewriter (an IBM Selectric if I did so at work), doing calculations with my slide rule or walking to the Medical School computer facility -- or by hand, and when I did learn to code, used Assembly and BASIC. C came out in 1972, but I did not have access to a computer the compiler (etc.) would run on until the late 1980s.

SO, no need to make the 1970s 'better' than it was. It was the wild frontier.

I think a technically educated person should be writing with a word processor, doing reports with a spreadsheet, and thinking with a technical computing language as an extension of their thought process.

I observe that people don’t think about things much. If they do, they use a spreadsheet. And that may be too complicated for some. The market for technical computing language may be limited to what I think of as “practicing PhD’s”. Mathematica makes it practical to think about a problem in terms of differential equations, for example, but one first needs some understanding of calculus and not everyone does.

I have recommended Mathematica to others but so far none have been able to use it well enough for it to be an extension of their thought process. It’s a steep learning curve. I have been using Mathematica since version 1.

A technical computing language is not the right tool for producing commercial computer applications. They are two very different things. This is not good or bad, just two different things. And tools for commercial computer applications are well developed. I had an application where I was writing c-code for a microcomputer. It needed a digital filter. I used Mathematica to simulate the data and to design a digital filter. Expressing the filter as c-code was trivial.

My typical Mathematica notebook often starts with some simple notes on a topic. It extends to some “proof of concept” analysis. Then expands from there. This was the development pattern in a recent activity where I extracted telemetry data from a GoPro MP4 file and used it to analyze sailing performance.

My answer: “it’s an extension of one’s though process.”

Years ago, we may have understood that an analysis might be theoretically doable, but we did not have tools to do it. Now with a technical computer language as an extension of one’s thought process we can. The possibilities for expanding understanding become endless. Perhaps we should put more emphasis on teaching a technical computing language. It might enable people to use, for example, calculus in a meaningful way.

Hi Wener,

I am not sure I agree, for two reasons.

Firstly, the question was "What is Mathematica for", not "What is the Wolfram programming language for". i.e the question was intended to be a little more general. Indeed, as you yourself have pointed out, there is an

impenetrable jumble of products and licenses

i.e it's getting harder to identify the precise boundary lines between each product offering. I believe this is termed "confusion marketing". i.e. it's a deliberate marketing tactic.

Secondly, a major thrust in recent years has been to embed computable data / entities and related functionality within the WL, for example in geographics. I believe Stephen's vision for "Mathematica" is as a blend of computable data and programming capability. Indeed, access to Wolfram Alpha is itself embedded in Mathematica, as you know.

I recently programmed a prototype of a game that makes extensive use of computational geographics data. It would have taken many times longer to program that in e.g. Python, as a great deal of time would have been expended in data retrieval and wrangling. In Mathematica, it was relatively easy. That's a good example of why it makes sense to include computable data in the technology stack.

I think Mathematica is good for writing computational essays. Stephen Wolfram has a blog post on this.

I agree that Wolfram Alpha is becoming indispensable. I think Wolfram Alpha is used by math students, math teachers, science students, and science teachers.

I am a Wikipedia editor and I think Wikipedia is a good reference.

Maybe it's different in your country. I am in Germany. And I am not only talking about students in maths or engineering, but social and many other sciences that have at least to do much with statistics and probabilities.

The prices for student Mathematica desktop licenses in Germany:

I think the ability to do symbolic math is underrated. I'm a hobbyist programmer, and I once encountered a problem that really did require to do symbolic math. I really, really didn't want to do the calculations with pen and paper so I bought a Mathematica license. Since then, I've learned to appreciate the capabilities of the language and the elegance of its design.

I think one thing that Mathematica does well is solving differential equations including differential-algebraic equations with DSolve.

Mathematica has been the platform of choice for sensors-to-animation production by the film industry for decades. A colleague of mine is a system administrator at one of the sites.

The famous example are the animations added to the actors in Pirates of the Caribbean.

There is a video of the making of Pirates of the Caribbean, although it's been years since I watched it. As for references to specific sites: that is a matter of intellectual property.

Hi George, thanks for taking the time to comment in detail. You have plenty of relevant experience with WL and have come to some clear views as to its strengths and limitations.

First, I agree with this comment:

There is also the tendency to leave stuff at the 99% done level and move on to the next shiny object. There are simply too many bugs. There are too many similar functions that do pretty much the same thing, but which have subtle differences in the API or performance, and the documentation does not make that clear at all.

I also largely agree that:

The notebook interface is great for this type of investigation and interactive exploration. However, it is rubbish at making stand-alone apps.

But WL is not completely rubbish at production systems. I am aware of at least one product, which I believe was developed 100% in WL that works extremely well, even by the standard of a production environment. I, too, have produced a small WL application that has shown itself to be very robust (although it's bit of a cheat as its just WL wrapper for a C++ system, really).

I also struggle with this view:

It is also far too hard to do computer-science or complicated programming.

I've written some pretty complicated programs in WL - not in the sense that developers typically mean, which often just refers to tens of thousands of lines of code related to a UI for example, but in terms of core mathematical, statistical and econometric concepts, which (in my view) is much more important (and complicated) stuff.

Yes, WL is a large ocean and for the new swimmer it can appear confusing and intimidating. But is not that hard, because of WL's nature as an interactive, functional programming language. The trick is simply to start by formulating some small component of the system you want to build, test it interactively to ensure it works as intended and then move on to the next component. Then you string all the components together, one by one.

I think the difficulty is that in languages like C, Fortran, Python, etc, the structure of the language often dictates a single, obvious way to tackle a problem. WL's diversity and flexibility means that there are often multiple plausible approaches one could take. In WL, I often spend ages thinking about which method might be best and experimenting with alternatives in order to arrive at the best solution. Sometimes, as I do that, I feel I am wasting time - but this is usually more than made up during the implementation stage, which tends to progress much faster if I have spent time at the outset thinking more deeply about the problem I am trying to solve.

Finally:

1. Fix the bugs.
2. Market Mathematica (Wolfram|One, etc.) for what it does best -- interactive computation.
3. Fully implement the Wolfram engine

Perhaps as you say the solution to the challenge facing WR is not to try to "fix" MMA to enable it to do production work, but to focus on the Wolfram engine instead...

Hi George,

as far as I know the Wolfram Engine is already ported to iOS for the App "Wolfram Player" but not embeddable in a custom application.

https://apps.apple.com/us/app/wolfram-player/id1059014516

I think the complaints about cost are misplaced. Wolfram products are in high demand and so I don't believe the price is going to come down.

My thoughts on this topic, besides that it should recieve more public discussion, are twofold.

First, I am not sure whether I will buy Mathematica once I exit academia. As long as I have a 'free' license though, I'm hooked. I think both mathematics and computer science students have a hard time approaching MMA even though it has tremendous potential to benefit both kinds of people†. I like to tell them that MMA is the worlds best software rapid prototyping tool; or when I'm being cynical, that MMA is both good for nothing and good at literally everything††. Frivolous yet powerful things are often the most compelling and fun...

Second, can MMA ever have a practical use†††? I think besides small niches, there is exactly one possibility. I haven't played with it's C code generation very much, but it seems like it's mainly limited to producing 'zombie routines' that must hook into a kernel. The symbolic nature of WL admits, for instance, a search over all possible algorithms to achieve a task. I think a richer library of code generation tools/examples plus an expanded code generation interface (i.e. x11 calls, various windows dlls; think of existing gpu and ffmpeg functionality) could start to make WL useful for 'actual code development'.

†In many lower division math classes, the entirety of homework can be automated in MMA. Document scanning, equation parsing, equation solving with steps, typesetting and printing. Literally the whole process: it's a thing of beauty. Applied computer science students, once relieved of worrying about the nontrivial trivialities of modern software, can appreciate the beauty of theory.

††Good for nothing in that it's useless for producing anything dynamic (i.e. production software), and good for everything in that it's capabilities are wiiiidespread and richly interconnected and as a result can produce amazing static visualizations, sounds, videos etc. Most academic publications are entirely static--most of scholarship is just prototyping ideas--such that MMA is the perfect scholarly workbench.

†††Again, practical in the sense of providing a directly valuable service--there's some economics word I'm looking for. One might say I'm asking for the well crafted code that goes into the kernel to be freely given up. Instead, I'm suggesting that the symbolic power of WL could be suited for automatedly writing lots of code--I would only be able to give examples if I were to embark on this as a project.

You make some good points but maybe your premise is wrong. Is Facebook dying?

Google trends might be deceiving. As a frequent WL user I never search for WL!
I think the WR is addressing many of the isssues you mentioned: creating standalone software - Wolfram cloud and WebMathematica -- WL add-on for Excel -- etc. I do agree that some applications need more finishing and polishing. As an Engineer I appreciate the addition of FEA to Mathemetica but it still needs a little more. The statistics capabilities are tops but why not add design of experiment capability. The presenter notebooks are fantastic if only I could use my custom theme background images like power point can. Nevertheless Mathematica remains to be my preferred tool to solve complex problems symbolically and numerically, simulate systems and present results.

I agree Jonathan and I am also worried about the long term future.

In my opinion the following issues have to be addressed first:

1.) Polishing and bug fixing of existing features

• Inconsistencies in TimeSeries
• Bugs and missing features in Machine Learning (Batch Prediction, GPU-support, Support for Apple Silicon and ML engine, …)
• Performance of Dataset, …
• Improve Developer Tools (update Workbench, support for other editors - might be coming as far as I know)
• Fixing long standing issues in various kinds of functions. Very often when I try something new, I stumble over bugs which give an unpolished impression of the product and the features run out of gas after they are 90% finished.

2.) Generally „Big Data“ and Data Processing has to be handled better

• fast converter / importing routine for numbers, datetime, …
• Out of core storage
• Supporting CSV - Files with „;“ deliminator
• support for secure web sockets wss - Protocol
• => focus on performance for processing huge amount of data
• …

It is really surprising what basic features are missing in this area. Python is far ahead.

3.) Deployment for Production

• as far as I know efforts are made in this area (Application Server) but nothing really useable in an easy way has been released so far
• Licensing has to be pain free, no cloud credits or something like this but permanent license
• Deployment / packaging of an app has to be easy and stable
• Deployment on own infrastructure
• support long running services (eg. Bot)
• Automatic error recovery, restart, logging, …

4.) Predictable bug fixes and releases

• when a bug is reported it is simple unpredictable for a customer if it gets fixed at all and when it gets fixed
• This is especially annoying for regressions. Fixes have to be pushed out quicker.
• I wouldn‘t suggest WL for a critical app in an enterprise because of this
• as long as this is not worked out, WL is only useable for research and experimentation in my opinion

5.) Create and promote examples for real world use cases in enterprises (IoT, Predicitive Maintenance, Big Data, ML Recommendation engine, Trading Bot, …) outside of research and development. Whenever I talk to somebody about MMA nobody thinks about production usage of it. This is probably the hardest part to build up awareness.

That said I really like the concept and idea of Mathematica and I want it to last for a long time and be as stable as possible. I enjoy discovering new things and areas where I had no idea about before.

Jonathan, could you please elaborate on what you mean by:

Anything requiring deployment in production

Does it mean the possibility to convert a Mathematica application so it can be used by a non-MMA user? Or a close to bug free MMA application for a MMA user (a "bug" may reside in the applications algorithm, not necessarily in MMA itself)? Or something else.

I recently spent almost two weeks to install keras(the most star count deep learning framework on github) to my gcp compute engine linux debian machine, keras now bundled with tensorflow2, only can run inside docker container, docker itself is cloud native technology, and I also spent great portion of time to install the gpu support in docker, now everything seems ok to go. The Capability to package app inside docker to run seems the way to go in cloud era.

Hi Rohit,

Thank you for the information. I will take a look at it as soon as there is a trial version available. There was also a presentation about WAS at WWDC 2020. Hopefully it will be easy accessible => pricing information available online without contacting sales, trial license for download, …

Hi Kapio,

I know about redmine. Actually I am a heavy user :) and it is definitely a step in the right direction. Direct contact with QA and sometimes with the developers helps a lot. Still what I would really appreciate is that development cycles are dedicated for bug fix only releases like version 12.1.1. Just polishing the existing stuff. There are also a lot of things fixed in regular releases but because of the new features almost always some regressions end up in the release.

Wenguang,

I happen to agree with you. But what would you say are the specific data processing capabilities that Mathematica lacks, or is inferior to Python, from your experience? Can you give some examples?

I always enjoy using Mathematica for serious and less serious work. Don't want to spend all time on finding the right python libraries and check there vitality etc when most absolutely necessary. I think Mathematica is fantastic and thoroughly enjoy listening to Stephen wolfram's discussions with his outstanding team. I mean I don't know any other cto doing that. I also believe that Wolfram is really looking for "stuff" that is actually useful for his customers. Other languages have also great features and sometimes better syntax but that doesn't matter. The flexibility is incredible. I also enjoy the documentation and all the crosslink to other fields that might be useful. You always learn from it. No other languages has given me more pleasure programming than this Wolfram language. I once played with low code software mendix in combination with sap software. I didn't really like it. Low code for sure but many many many clicks and different windows.. Not for me.

Jonathan, the answer is very simple:

"Mathematica is for Stephen Wolfram to do his NKS-stuff on."

Other features are put in from time-to-time and as is deemed necessary to make Mathematica sufficiently saleable to support Stephen so he doesn't have to get some other less enjoyable occupation while he pursues NKS.

;-)

B

Hi Raspi, in some ways I think you are an ideal Mathematica user - a total convert: MMA with everything! And I get it - it's a wonderful sandpit environment for every kind of experimentation.

But in broad terms, my thinking is that this is what Mathematica really is: a means of doing fast research, experimentation and prototyping - but not necessarily for production systems.

If you are not a professional developer that probably doesn't matter at all - everything is an experiment, or recreation! But for a professional developer, getting systems into production and making sure they are battle-ready is a critical part of what you do for a living. And its in this context that I see the greatest challenge for Mathematica - making WL applications that are sufficiently bullet-proof to hand over to an end user (rather than, say, a research co-worker).

BTW what do you use Raspberry Pi for?

You mention using your RPi to access Mathematica remotely. But cant that be done just as easily and more effectively by accessing MMA on your PC via remote desktop when you are travelling)? Is there something specific about the RPi implementation of MMA that makes it more accessible for remote work?

I have found Mathematica essential in production settings for at least 20 years. Most of those were before the Wolfram Engine (and the more recent cloud offereings). I have re-analyzed some of the applications I mention below, and we could have cost-effectively used the Wolfram Engine if it had been available. Stability of the kernel has also dramatically improved.

I have used Mathematica in at least three different companies. In a consulting environment, we did all our in-house analyses using Mathematica. Notebooks would call large databases, issue SQL commands, and bring in the data for further processing and visualizations. Most algorithms we needed had to be developed from scratch, and Mathematica made it quick to go from reading a paper or a pencil and paper calculation, to something tested and working on data. The graphics would usually get some tweaking in Illustrator, but would often be close to camera ready. We also used Mathematica to create reference algorithms to be implemented in C++ or Java. Many of these where not numerical algorithms.

At my next company we used Mathematica directly as a compute engine. We had developed a very performant cyber security system that identified executing malicious code by observing the stream of system calls. All the machine learning steps were done in Mathematica (this was before the free Wolfram Engine). It allowed us to have version 1 of our system up way faster than if we had implemented them directly. We experienced memory leaks and random hangs that forced us to periodically kill and restart the compute engine, but we were able to engineer it so that those issues remained invisible to the user. Over time, because of costs, we replaced the Mathematica subsystem with Java code.

In my current work we still develop most of the algorithms in Mathematica. We are building a system that use Mathematica for some core data generation; reference algorithms are still in Mathematica. We also use it as a thinking tool (see my blog post). Having all that curated data at one's fingertips is great.

I feel costs have prevented a wider adoption. Wolfram Research now offers low cost versions for non-commercial applications, which should encourage wider adoption, but I sometimes worry that its market may be shrinking.