Anyone watching CPU usage while the tests are running? Apple's claims about speed are based in part on software that uses multiple cores, but Mathematica's tests are largely single-threaded things, no?

Thanks. I tried the same thing -- got a benchmark score of 13.81

Pretty impressive, and points to the advantage of using multiple cores. The new M1 (all of them) chips have multiple GPU cores. It would be interesting to see what would happen if we could force Mathematica to use all of those as well.

In his talk on Monday, Stephen mentioned that they were working on optimizing Mathematica for the new Apple Hardware. Not sure what that entails, but if the neural net/ AI stuff can be made to work on the built-in hardware instead of relying on NVIDIA, it would be something, indeed.

My new M1 Pro MacBook Pro scores an impressive 3.37. Slightly higher than the score for the M1 Max posted above. I think this score is highly dependent on what else is going on on your machine. For example, for the first couple of days there was a lot of activity from Spotlight, as it catalogued my drives in the background.

Machine Name: mercury System: Mac OS X ARM (64-bit) Date: December 16, 2021 Wolfram Language Version: 13.0.0 Benchmark Result: 3.37

Attachments:

WolframMark Report.nb

Benchmark on my brand new MacBook Pro 16: M1Max 64 Gbytes RAM:

3.32 Powered 3.41 Battery

Running version 13.0, of course. (I started a new session for each.)

According to Stephen in his rollout talk, they are still working on optimization.

Also, I have a short function that looks for new and changed packets, and there were w whole bunch that were downloaded since Monday. Some of these may affect execution speed.

Ah, so v13 did improve the scores for tests 4 and 14, dramatically so. But, they are still somewhat the weak links overall. Now we need v13 scores for the most recent Intel MacBook Pro, as a comparison.

Was the power cord connected? This can have a large impact.

Yes, that result is not good at all. Mine was certainly better, but I am still disappointed that it wasn’t even better. I have a 2019 Intel MBP that I can install Mathematica 13 on, but I moved the license to the new machine, so I don’t know that it will still run the benchmark. I will try soon and let you all know what I find.

Extracted the timings from Gary's notebook and from my notebook to see which tests performed worse.

The significant ones are Numerical Integration, SVD, and Solving a Linear System. What does that suggest? Some problem with floating point computations and matrices on my machine?

Here is a YouTube video I found on this topic:

Math Professor reviews M1 MacBook Pro!

Certainly not definitive, but interesting nonetheless.

John Shonder

I ran benchmark on both MacBook Air M1 and MacBook Pro M1 Max, the funny thing is that Macbook Air's score is like 30% higher than MacBook Pro M1 Max, I suspect it's something to do with 2 less efficiency cores. But then both ain't as fast as I expected.

Here are some results running Mathematica 12.3.1 natively on the M1 MacBook Air (https://mathematica.stackexchange.com/questions/234881/benchmarking-with-mathematica-v-12-for-up-to-date-comparison-across-different-ma/250736#250736) and M1 MacBook Pro (https://mathematica.stackexchange.com/questions/234950/benchmarks-for-new-m1-macbooks/250779#250779).

Both benchmark at 3.2. That's pretty far off the 4.48 Murray reported for his core i9 iMac (though with a slightly different version—12.1.1; I don't know how much diff. that makes). Is it possible Mathematica's code isn't yet well-optimized for Apple Silicon? As I understand it, Mathematica's performance on Intel benefits from MKL. Perhaps there's not yet an equivalent library available for Apple Silicon.

Of course, ideally you'd want to assess the M1's performance with calculations representative of your own workload, rather than relying on a synthetic benchmark.

Hoping to refresh this thread.

Has anyone benchmarked the just-released 12.3 on an ARM M1 chip?

Thank you.

Yes, the M1 is significantly faster than the i7-3770 listed in the Mathematica Benchmark report, but that processor was released in 2012.

Thus I wasn't considering that score, but rather what Murray Eisenberg obtained with the 8-core i9-9900K in his 2019 iMac. And based on that, the M1's comparative performance for MMA is somewhat slower than expected (even accounting for Rosetta 2), relative to its comparative performance thus far on other benchmarks.

Specifically, the score Murray obtained with the i9-9900K is 50% higher than what you obtained with an M1 Air. By contrast, some (though not all) benchmarks comparing the single-core performance of the M1 to that of the (even faster) i9-10900K have the M1 as faster when the benchmark is run natively, and approximately comparable when run under Rosetta 2. Thus it's natural to want to ensure that this discrepancy isn't because Murray's i9 was able to use all 8 cores.

More broadly, my own interest/curiosity is in assessing the performance of this new technology from Apple and, to do this, I think it's most meaningful to focus on per-core performance, since the current 4-performance-core limitation is merely a characteristic of these first models. Apple will be likely be offering higher-end systems with more cores in 2021.

It was interesting to see the benchmark you posted—thanks for doing that. But, to get a meaningful assessment of MMA's per-core performance on AS, we'll probably need to wait until WRI produces a build that runs natively on AS.

These CPUs which you compared with are too old, I don't believe that.

How many cores does the Benchmark function use?

The M1 has only four high-performance cores, while most desktop/laptop CPU's have more; thus, to maintain comparability with benchmark results from other CPU's, you'd want the Benchmark function to be limited to four as well (and you'd also want it to be using the high-performance cores only).

Alternately, one could do comparative testing by timing individual single-threaded commands.

Hello Jacob,

very promising results - could you give the benchmark another try while the MacBook is hot? E.g after watching a HD Youtube Video for some minutes?

As you are surely aware, the CPU gets throttled when too hot; maximum speed is only available for a short time, until the aluminum body heats up.

Therefore, the high benchmark scoring might be a little misleading: Surely true when the Laptop is idle and the computation is short. But for longer tasks, Classify or Predict of large Datasets, the CPU might run into the temperature-throttle. Such an issue would not be detected by the Benchmark test which is only a few seconds in duration.

Could you "heat up" the MacBook and run the benchmark again? Or maybe run the Benchmark 100 times, and give the result of the last run?

Greetings.

Oliver, I did as you suggested and ran the Benchmark function 100 times. The last "BenchmarkResult" reports to be 3.051. While the function was on repeat, the bottom of the MacBook Air became noticeably more warm and the cumulative CPU utilization held steady at around 73%.

Murray, the aforementioned WolframMark benchmark test was ran on an M1 MacBook Air with the 8-core GPU and 16GB of unified memory.

That would be interesting for me as well. I assume that right now, Mathematica has to use the Rosetta 2 Layer to run on non-x64-architecture?

An easy "benchmark" would be to have them install Mathematica in an Apple Store, run some pre-defined code, and measure the time it needs to perform that task.