Thought I'd post to see if anyone has run Benchmark[] on Apple silicon (M1, M2 or M3) for MMA v 13.2 and 13.3. Interested to see if there has been any further optimization.

MacBook Pro with M1 Pro chip, 32 GB memory, running MacOS 13.0 and Mathematica 13.1.0.0. I just got a WolfMark Score of 7.14. This is significantly better than the score of 3.37 I got on this same machine last December.

John Shonder

Benchmark results under macOS Monterey 12.5 on Mac Studio M1 Max (64GB RAM, 10 core): [corrected: results shown now are after a reboot (previous ones were after heavy system usage]

WolframMark score: 4.41 After LaunchKernels: 12

I am a physics professor at a university with the same Mathematica "Max Licensed Process" count of 8, and I am trying to decide whether to purchase a Macbook Pro with an M1 Max or a Mac Studio with an M1 Ultra. Any additional data on how Mathematica is distributing load across all 20 CPU cores would be extremely helpful. In addition, can you provide any information on the graphics processing side? For instance, is there a noticeable difference in the speed with which Mathematica renders complex graphics? Thanks for any information you can provide. --Mark

Running the BenchmarkReport[] on Mathematica 13.0.1 on my Mac Studio with M1 Ultra, I get a measly Benchmark Result of 1.58. However, if I execute LaunchKernels[] and then rerun BenchmarkReport[], I get a whopping Benchmark Result of 27.21.

Has anyone else had problems with Mathematica being sluggish, unstable and very prone to freezing and crashing ungracefully? I'm using Mathematica 13.0.0.0 on a M1 MacBook Air with macOS Monterey. I have had Mathematica crash spontaneously with memory errors many times, usually losing me data. I experience 2-3 crashes per week and usually I'm not doing anything too crazy, to be clear most crashes are just freezes and only about 1/6 crashes or so are memory crashes .

I'm updating to 13.0.1 tonight so hopefully that will fix this.

Thank you!

For Wolfram Desktop users: the equivalent of ~/Library/Mathematica is ~/Library/WolframDesktop, see https://reference.wolfram.com/language/tutorial/ConfigurationFiles.html.

I was not aware that 13.0.1 had been released. I am downloading now and will report back what I find.

I am seeing the same results with 13.0.1 that I saw with 13.0.0. I get scores in the 3.0 to 3.5 range.

What would be the best MacBook Pro for the following kind of calculations, an M1 Max w/ 64GB RAM or a maxed out Intel (something like 2.4 Ghz 8-core i9, 64GB RAM)?

Here's an example with small numbers, namely 3 and 5. The goal is to determine the minimal polynomials of certain root sums. Consider the polynomials

 x ^ 3 - x - 1    and x ^ 5 - x - 1 .

We find their largest real roots by using

x / . NSolve [ y==0 , x , WorkingPrecision->90 ] ,

first with y the first above polynomial, and then with y the the second one. We then add these two roots. (They are 1.324...... and 1.167...... with sum 2.492........ .)

To get the minimal polynomial of their sum (call it s ) use

(MinimalPolynomial [ RootApproximant [ s ] ] / . # 1 - > x ) [[1]] .

In this case we find that the probable minimal polynomial is

-19 + 19 x + 12 x ^ 2 + ,,,, - 5 x ^ 12 - 5 x ^ 13 + x ^ 15

In fact, it is the minimal polynomial.

I would want to do calculations of more or less this type with numbers larger than 3 and 5. This simple direct approach will not work on any computer once the numbers are too big, but what available MacBook Pro would allow me to go as far as possible?

Anyone watching CPU usage while the tests are running? Apple's claims about speed are based in part on software that uses multiple cores, but Mathematica's tests are largely single-threaded things, no?

Thanks. I tried the same thing -- got a benchmark score of 13.81

Pretty impressive, and points to the advantage of using multiple cores. The new M1 (all of them) chips have multiple GPU cores. It would be interesting to see what would happen if we could force Mathematica to use all of those as well.

In his talk on Monday, Stephen mentioned that they were working on optimizing Mathematica for the new Apple Hardware. Not sure what that entails, but if the neural net/ AI stuff can be made to work on the built-in hardware instead of relying on NVIDIA, it would be something, indeed.

My new M1 Pro MacBook Pro scores an impressive 3.37. Slightly higher than the score for the M1 Max posted above. I think this score is highly dependent on what else is going on on your machine. For example, for the first couple of days there was a lot of activity from Spotlight, as it catalogued my drives in the background.

Machine Name: mercury System: Mac OS X ARM (64-bit) Date: December 16, 2021 Wolfram Language Version: 13.0.0 Benchmark Result: 3.37

Attachments:

WolframMark Report.nb

Benchmark on my brand new MacBook Pro 16: M1Max 64 Gbytes RAM:

3.32 Powered 3.41 Battery

Running version 13.0, of course. (I started a new session for each.)

According to Stephen in his rollout talk, they are still working on optimization.

Also, I have a short function that looks for new and changed packets, and there were w whole bunch that were downloaded since Monday. Some of these may affect execution speed.

Ah, so v13 did improve the scores for tests 4 and 14, dramatically so. But, they are still somewhat the weak links overall. Now we need v13 scores for the most recent Intel MacBook Pro, as a comparison.

Was the power cord connected? This can have a large impact.

Yes, that result is not good at all. Mine was certainly better, but I am still disappointed that it wasn’t even better. I have a 2019 Intel MBP that I can install Mathematica 13 on, but I moved the license to the new machine, so I don’t know that it will still run the benchmark. I will try soon and let you all know what I find.

Extracted the timings from Gary's notebook and from my notebook to see which tests performed worse.

The significant ones are Numerical Integration, SVD, and Solving a Linear System. What does that suggest? Some problem with floating point computations and matrices on my machine?

Here is a YouTube video I found on this topic:

Math Professor reviews M1 MacBook Pro!

Certainly not definitive, but interesting nonetheless.

John Shonder

I ran benchmark on both MacBook Air M1 and MacBook Pro M1 Max, the funny thing is that Macbook Air's score is like 30% higher than MacBook Pro M1 Max, I suspect it's something to do with 2 less efficiency cores. But then both ain't as fast as I expected.

Interesting. Most of your test scores are class-leading, but you got very poor scores in Tests 4 (eigenvalues) and 14 (SVD). That suggests some specific libraries that haven't been optimized for Apple Silicon. Some other reports on StackExchange don't show that particular pattern, which makes me think that performance may be changing with each point upgrade. I suggest waiting for v13 before drawing any conclusions.

Here are some results running Mathematica 12.3.1 natively on the M1 MacBook Air (https://mathematica.stackexchange.com/questions/234881/benchmarking-with-mathematica-v-12-for-up-to-date-comparison-across-different-ma/250736#250736) and M1 MacBook Pro (https://mathematica.stackexchange.com/questions/234950/benchmarks-for-new-m1-macbooks/250779#250779).

Both benchmark at 3.2. That's pretty far off the 4.48 Murray reported for his core i9 iMac (though with a slightly different version—12.1.1; I don't know how much diff. that makes). Is it possible Mathematica's code isn't yet well-optimized for Apple Silicon? As I understand it, Mathematica's performance on Intel benefits from MKL. Perhaps there's not yet an equivalent library available for Apple Silicon.

Of course, ideally you'd want to assess the M1's performance with calculations representative of your own workload, rather than relying on a synthetic benchmark.

I have a new M1 MacBook Air. Running the last nightly build for the Apple Silicon version, I got a benchmark of 3.21. not bad for a computer intended for light duty.

Hoping to refresh this thread.

Has anyone benchmarked the just-released 12.3 on an ARM M1 chip?

Thank you.

Yes, the M1 is significantly faster than the i7-3770 listed in the Mathematica Benchmark report, but that processor was released in 2012.

Thus I wasn't considering that score, but rather what Murray Eisenberg obtained with the 8-core i9-9900K in his 2019 iMac. And based on that, the M1's comparative performance for MMA is somewhat slower than expected (even accounting for Rosetta 2), relative to its comparative performance thus far on other benchmarks.

Specifically, the score Murray obtained with the i9-9900K is 50% higher than what you obtained with an M1 Air. By contrast, some (though not all) benchmarks comparing the single-core performance of the M1 to that of the (even faster) i9-10900K have the M1 as faster when the benchmark is run natively, and approximately comparable when run under Rosetta 2. Thus it's natural to want to ensure that this discrepancy isn't because Murray's i9 was able to use all 8 cores.

More broadly, my own interest/curiosity is in assessing the performance of this new technology from Apple and, to do this, I think it's most meaningful to focus on per-core performance, since the current 4-performance-core limitation is merely a characteristic of these first models. Apple will be likely be offering higher-end systems with more cores in 2021.

It was interesting to see the benchmark you posted—thanks for doing that. But, to get a meaningful assessment of MMA's per-core performance on AS, we'll probably need to wait until WRI produces a build that runs natively on AS.

These CPUs which you compared with are too old, I don't believe that.

How many cores does the Benchmark function use?

The M1 has only four high-performance cores, while most desktop/laptop CPU's have more; thus, to maintain comparability with benchmark results from other CPU's, you'd want the Benchmark function to be limited to four as well (and you'd also want it to be using the high-performance cores only).

Alternately, one could do comparative testing by timing individual single-threaded commands.

Hello Jacob,

very promising results - could you give the benchmark another try while the MacBook is hot? E.g after watching a HD Youtube Video for some minutes?

As you are surely aware, the CPU gets throttled when too hot; maximum speed is only available for a short time, until the aluminum body heats up.

Therefore, the high benchmark scoring might be a little misleading: Surely true when the Laptop is idle and the computation is short. But for longer tasks, Classify or Predict of large Datasets, the CPU might run into the temperature-throttle. Such an issue would not be detected by the Benchmark test which is only a few seconds in duration.

Could you "heat up" the MacBook and run the benchmark again? Or maybe run the Benchmark 100 times, and give the result of the last run?

Greetings.

Oliver, I did as you suggested and ran the Benchmark function 100 times. The last "BenchmarkResult" reports to be 3.051. While the function was on repeat, the bottom of the MacBook Air became noticeably more warm and the cumulative CPU utilization held steady at around 73%.

Murray, the aforementioned WolframMark benchmark test was ran on an M1 MacBook Air with the 8-core GPU and 16GB of unified memory.

That would be interesting for me as well. I assume that right now, Mathematica has to use the Rosetta 2 Layer to run on non-x64-architecture?

An easy "benchmark" would be to have them install Mathematica in an Apple Store, run some pre-defined code, and measure the time it needs to perform that task.