Message Boards Message Boards

14
|
53927 Views
|
62 Replies
|
51 Total Likes
View groups...
Share
Share this post:

Mathematica benchmarks for new M1 MacBooks?

Are there any benchmarks yet for Mathematica 12.1 on the new (13") M1 MacBook Air or (13") M1 MacBook Pro?

POSTED BY: Murray Eisenberg
62 Replies

Mathematica 13.3 under macOS Ventrua 13.4.1 on Mac Studio, M1 Max, 64GB RAM:

benchmark

POSTED BY: Murray Eisenberg

Thought I'd post to see if anyone has run Benchmark[] on Apple silicon (M1, M2 or M3) for MMA v 13.2 and 13.3. Interested to see if there has been any further optimization.

POSTED BY: Gareth Russell

As it happens I don't have Apple silicon to test, so I ran various older versions on my underpowered Intel iMac, running repeatedly to get a sense of variability

v12.0: Consistently around 2.7, high of 2.91 v13.0: Consistently around 2.88, high of 2.94 v13.2: Consistently around 2.6, high of 2.61 v13.3: Consistently around 3.0, high of 3.22!

Of course it's possible the test has changed, but at first glance it looks as if there have been some recent optimizations for the Intel code at least.

POSTED BY: Gareth Russell

MacBook Pro with M1 Pro chip, 32 GB memory, running MacOS 13.0 and Mathematica 13.1.0.0. I just got a WolfMark Score of 7.14. This is significantly better than the score of 3.37 I got on this same machine last December.

John Shonder

System Timing

POSTED BY: John Shonder
Posted 1 year ago

Benchmark results under macOS Ventura 13.0 on MacBook Pro 16 (mid 2019, 2.3GHz 8-core Intel i9, 32GB RAM), running Mathematica 13.1 with WolframMark Score: 4.05.

Attachment

Attachment

POSTED BY: Bolun Chen

Benchmark results under macOS Monterey 12.5 on Mac Studio M1 Max (64GB RAM, 10 core): [corrected: results shown now are after a reboot (previous ones were after heavy system usage]

WolframMark score: 4.41 After LaunchKernels: 12

WolfamMark - Mac Studio M1 Max

WolfrmMark - Mac Studio M1 Max - after LaunchKernels[]

POSTED BY: Murray Eisenberg
Posted 2 years ago

Some have reported getting much higher WolframMark benchmark scores when using LaunchKernels[].

Note that, when you do this, even though you do get a higher benchmark score, the actual amount of time it takes to complete the benchmark (i.e., the wall clock time) is much higher.

This suggests that, when LaunchKernels[] is used, it is launching multiple benchmarks and running them simultaneously (that's why it takes longer), and then adding the scores (or combining them in some way). [It appears that some of the WolframMark tasks are single-core, and some are multi-core; so for the latter you would be running multiple multi-core tasks.]

I.e., when you use LaunchKernels[] with the benchmark, it appears it's not running a single benchmark test on more cores, but rather running more benchmark tests at once. Thus the higher score seen with LaunchKernels[] doesn't appear to be meaningful.

POSTED BY: Aron Yoffe

I am a physics professor at a university with the same Mathematica "Max Licensed Process" count of 8, and I am trying to decide whether to purchase a Macbook Pro with an M1 Max or a Mac Studio with an M1 Ultra. Any additional data on how Mathematica is distributing load across all 20 CPU cores would be extremely helpful. In addition, can you provide any information on the graphics processing side? For instance, is there a noticeable difference in the speed with which Mathematica renders complex graphics? Thanks for any information you can provide. --Mark

POSTED BY: Mark Lusk
Posted 2 years ago

My question is whether I am right in assuming that you do not have the standard professional Mathematica license which I believe allows only 8 simultaneous kernels.

I'm running a site license of Mathematica provided by my University, which has a "Max Licensed Process" count of 8. However, it appears my LaunchKernels[] function is starting up 20 Kernels, and the BenchmarkReport[] appears to be exercising all 20 Kernels.

POSTED BY: Jacob Evans
Posted 2 years ago

Running the BenchmarkReport[] on Mathematica 13.0.1 on my Mac Studio with M1 Ultra, I get a measly Benchmark Result of 1.58. However, if I execute LaunchKernels[] and then rerun BenchmarkReport[], I get a whopping Benchmark Result of 27.21.

POSTED BY: Jacob Evans

Your 27.21 is 1.97x the 13.81 reported for the M1 Max a few posts above, which shows how good the scaling of Apple silicon is (i.e. 2x memory bandwidth on the Ultra vs the Max). My question is whether I am right in assuming that you do not have the standard professional Mathematica license which I believe allows only 8 simultaneous kernels. I ask because if the standard license somehow allows full utilization of the Ultra, that would accelerate my decision to move from my current iMac Pro to a Mac Studio Ultra.

As far as your 1.58 goes, that seems inexplicable unless MacOS somehow allocated the task to a single efficiency core.

POSTED BY: Eric Stromquist
Posted 2 years ago

Has anyone else had problems with Mathematica being sluggish, unstable and very prone to freezing and crashing ungracefully? I'm using Mathematica 13.0.0.0 on a M1 MacBook Air with macOS Monterey. I have had Mathematica crash spontaneously with memory errors many times, usually losing me data. I experience 2-3 crashes per week and usually I'm not doing anything too crazy, to be clear most crashes are just freezes and only about 1/6 crashes or so are memory crashes .

I'm updating to 13.0.1 tonight so hopefully that will fix this.

POSTED BY: Olav Fosse

No issues at all with either of my Apple Silicon MacBooks (Air and Pro), even the nightly builds for 13.1 seem to be stable.

You may need to 'reset' your copy of Mathematica by deleting the ~/Library/Mathematica folder. When you restart, a new copy will be made. You will want to save a copy of any custom stylesheets, packages, etc. elsewhere, because they will be deleted as well.

There is a tech note somewhere that describes this, but I can't find it at the moment.

This folder can pick up a lot of cruft or become compromised over time. When I remember, I do this for every major release.

Posted 2 years ago

Thank you!

For Wolfram Desktop users: the equivalent of ~/Library/Mathematica is ~/Library/WolframDesktop, see https://reference.wolfram.com/language/tutorial/ConfigurationFiles.html.

POSTED BY: Olav Fosse
Posted 2 years ago

UPDATE: I think Wolfram Desktop got more stable after updating and removing ~/Library/WolframDesktop, but I just had it randomly crash on me while navigating the docs.

UPDATE: Another crash, this time when moving a notebook from my MacBook monitor to the external monitor

POSTED BY: Olav Fosse

I was not aware that 13.0.1 had been released. I am downloading now and will report back what I find.

POSTED BY: Gary Gray

Anyone checked whether the 13.0.1 update changes things?

POSTED BY: Gareth Russell
Posted 2 years ago

I am seeing the same results with 13.0.1 that I saw with 13.0.0. I get scores in the 3.0 to 3.5 range.

POSTED BY: Updating Name

I've found that the M1 MacBooks are faster all around than the Intel MacBooks. The GeekBench benchmarks show the M1 faster in both single and multiple core operations. For single core operations, It is one of the fastest processors ever made -- it only gets nudged out by a handful of desktop/server processors.

In Mathematica the benchmarks are similar except for two operations in the benchmark suite (see posts above -- Eigenvalues of a Matrix and Singular Value Decomposition). I believe that is is likely because Wolfram has not fully updated the benchmark test and those operations are likely using emulated code. I have found that on actual real-world problems, my M1 trounces my previous generation Intel machine. Your operation is single core and does not use any unusual MMA functions so I think it is likely to run faster on the M1.

Lastly, even in emulation, I have found that the M1 is not bad -- it sometimes even runs faster than the older processors in emulation. I hope this helps.

Regards,

Neil

POSTED BY: Neil Singer

What would be the best MacBook Pro for the following kind of calculations, an M1 Max w/ 64GB RAM or a maxed out Intel (something like 2.4 Ghz 8-core i9, 64GB RAM)?

Here's an example with small numbers, namely 3 and 5. The goal is to determine the minimal polynomials of certain root sums. Consider the polynomials

 x ^ 3 - x - 1    and x ^ 5 - x - 1 .

We find their largest real roots by using

x / . NSolve [ y==0 , x , WorkingPrecision->90 ] ,

first with y the first above polynomial, and then with y the the second one. We then add these two roots. (They are 1.324...... and 1.167...... with sum 2.492........ .)

To get the minimal polynomial of their sum (call it s ) use

(MinimalPolynomial [ RootApproximant [ s ] ] / . # 1 - > x ) [[1]] .

In this case we find that the probable minimal polynomial is

-19 + 19 x + 12 x ^ 2 + ,,,, - 5 x ^ 12 - 5 x ^ 13 + x ^ 15

In fact, it is the minimal polynomial.

I would want to do calculations of more or less this type with numbers larger than 3 and 5. This simple direct approach will not work on any computer once the numbers are too big, but what available MacBook Pro would allow me to go as far as possible?

POSTED BY: David Stolarsky
Posted 2 years ago

Hi David,

Can you provide an example of large exponents? I used this to test various values of exponents

ClearAll[mp, x];
mp[e1_Integer, e2_Integer] := Module[
  {p1 = x^e1 - x - 1,
   p2 = x^e2 - x - 1,
   s},
  s = x /. 
       First@
        NSolve[# == 0, x, Reals, WorkingPrecision -> 90] & /@ {p1, p2} // Total;
  MinimalPolynomial[RootApproximant[s]][x]
  ]

For exponents <= 10^15 it finishes in under 1.5 sec. For exponents >= 10^16 after ~40 sec. I get

Throw::sysexc: Uncaught SystemException returned to top level. SystemException["MemoryAllocationFailure"]

I am using a MBP M1 Max with 64GB of RAM.

POSTED BY: Rohit Namjoshi

Anyone watching CPU usage while the tests are running? Apple's claims about speed are based in part on software that uses multiple cores, but Mathematica's tests are largely single-threaded things, no?

POSTED BY: Gareth Russell

I think one can activate the other kernels with LaunchKernels[], and then run the benchmarking. With all 8 kernels activated on my device, I get a score of 11.71. To activate n kernels where n is an integer less than or equal to the number of cores (say, 3), the command is LaunchKernels[3].

POSTED BY: Sudeepan Datta

Thanks. I tried the same thing -- got a benchmark score of 13.81

Pretty impressive, and points to the advantage of using multiple cores. The new M1 (all of them) chips have multiple GPU cores. It would be interesting to see what would happen if we could force Mathematica to use all of those as well.

In his talk on Monday, Stephen mentioned that they were working on optimizing Mathematica for the new Apple Hardware. Not sure what that entails, but if the neural net/ AI stuff can be made to work on the built-in hardware instead of relying on NVIDIA, it would be something, indeed.

Until I just read these last two posts I never realized that LaunchKernels[] was required to exploit parallelism in benchmarking (now slapping palm against forehead). I just did this with my 2017 iMac Pro with a 3GHz 10-core Intel Xeon W. Running 13.0.1 with the standard 8 licensed cores, the score improved from 3.94 to 13.43. Considering mine is a desktop machine with a large thermal envelope, your 13.81 on a MacBook Pro is looking pretty good.

POSTED BY: Eric Stromquist

My new M1 Pro MacBook Pro scores an impressive 3.37. Slightly higher than the score for the M1 Max posted above. I think this score is highly dependent on what else is going on on your machine. For example, for the first couple of days there was a lot of activity from Spotlight, as it catalogued my drives in the background.

Machine Name: mercury System: Mac OS X ARM (64-bit) Date: December 16, 2021 Wolfram Language Version: 13.0.0 Benchmark Result: 3.37

Attachments:
POSTED BY: John Shonder

My 14-inch Macbook Pro (M1 Pro, the base 8-core variant, with 16 GB RAM) scores 2.63. Nothing else significant (as in terms of taking up processor attention) was running in the background, and the battery was fully charged, with the magsafe still plugged in, glowing green.

I was expecting it to make it above 3, at the very least.

Here's the notebook:

POSTED BY: Sudeepan Datta

Benchmark on my brand new MacBook Pro 16: M1Max 64 Gbytes RAM:

3.32 Powered 3.41 Battery

Running version 13.0, of course. (I started a new session for each.)

According to Stephen in his rollout talk, they are still working on optimization.

Also, I have a short function that looks for new and changed packets, and there were w whole bunch that were downloaded since Monday. Some of these may affect execution speed.

I downloaded and installed Mathematica 13 on my M1 Max MacBook Pro and ran the benchmark test again. The results are much better than I was seeing with Mathematica 12.3.1. I have attached the resulting notebook.

POSTED BY: Gary Gray

Ah, so v13 did improve the scores for tests 4 and 14, dramatically so. But, they are still somewhat the weak links overall. Now we need v13 scores for the most recent Intel MacBook Pro, as a comparison.

POSTED BY: Gareth Russell
Posted 2 years ago

Hi Gary,

The results on my M1 Max MacBook Pro are significantly worse than yours

System: Mac OS X ARM (64-bit)

Date: December 15, 2021

Wolfram Language Version: 13.0.0

Benchmark Result: 2.33

$Version
(* "13.0.0 for Mac OS X ARM (64-bit) (December 3, 2021)" *)
POSTED BY: Rohit Namjoshi

Was the power cord connected? This can have a large impact.

POSTED BY: Gustavo Delfino
Posted 2 years ago

Hi Gustavo,

Yes, the power cord was connected and the battery was at 100%. Tried without the power cord connected and the change was insignificant.

POSTED BY: Rohit Namjoshi

Yes, that result is not good at all. Mine was certainly better, but I am still disappointed that it wasn’t even better. I have a 2019 Intel MBP that I can install Mathematica 13 on, but I moved the license to the new machine, so I don’t know that it will still run the benchmark. I will try soon and let you all know what I find.

POSTED BY: Gary Gray

I get 2.54 on a 2017 MBP!

System: Mac OS X x86 (64-bit)

Date: December 15, 2021

Wolfram Language Version: 13.0.0 Benchmark

Result: 2.54

So agreed it should be significantly better.

POSTED BY: Gareth Russell
Posted 2 years ago

Extracted the timings from Gary's notebook and from my notebook to see which tests performed worse.

enter image description here

The significant ones are Numerical Integration, SVD, and Solving a Linear System. What does that suggest? Some problem with floating point computations and matrices on my machine?

POSTED BY: Rohit Namjoshi

What does that suggest? Some problem with floating point computations and matrices on my machine?

@Rohit Namjoshi This likely means the machine was not sufficiently idle during the benchmark. This can cause a slowdown if the OS schedules one of the computational threads on the same core that is running some kind of background task (for example, the Spotlight indexing could be implicated).

POSTED BY: Ilian Gachevski

Here is a YouTube video I found on this topic:

Math Professor reviews M1 MacBook Pro!

Certainly not definitive, but interesting nonetheless.

John Shonder

POSTED BY: John Shonder

Hi Kenneth. Can you share absolute values of the banchmaks? I wanted to compare them with emulated version on ARM64 on Windows 11 Mathematica 12.3.1 Regards Marcin Balcerzyk

POSTED BY: Marcin Balcerzyk
Posted 2 years ago

I ran benchmark on both MacBook Air M1 and MacBook Pro M1 Max, the funny thing is that Macbook Air's score is like 30% higher than MacBook Pro M1 Max, I suspect it's something to do with 2 less efficiency cores. But then both ain't as fast as I expected.

POSTED BY: Kenneth Ho

I just got a MacBook Pro with the M1 Max chip. It has 32GB of RAM, 10 cores (8 high performance, 2 efficiency). When I run Mathematica’s benchmark on my 2019 MBP, the benchmark result is 3.49. When I run it on the new MBP with the M1 Max, I get 1.86. I am running Mathematica 12.3.1 on both. The Intel version on the 2019 MBP and the ARM version on the 2021 MPB. Shouldn’t I be seeing a far better number than that?

POSTED BY: Gary Gray

Interesting. Most of your test scores are class-leading, but you got very poor scores in Tests 4 (eigenvalues) and 14 (SVD). That suggests some specific libraries that haven't been optimized for Apple Silicon. Some other reports on StackExchange don't show that particular pattern, which makes me think that performance may be changing with each point upgrade. I suggest waiting for v13 before drawing any conclusions.

POSTED BY: Gareth Russell

Hi. I have just made Surface Pro X with Windows 11 (build 10.0.22000.160) to run Mathematica 12.3.1, obviously in emulation mode. I tried BenchmarkReport[] and this machine got only 0.45, which is very low compared to 3.21 of M1 Mac running Mathematica 12.3.0 which is also emulation, not the native 12.3.1 one for Mac.

On my desktop Windows 11 system with intel i7-8700 with 3.2 GHz the benchmark is 2,8.
. Wolfram support shows that they resist strongly compiling Mathematica for Windows on ARM64. To make the Mathematica running I needed to uninstall Math Recognizer from Windows. This file is responsible for Inking Equation in Word (but not in PowerPoint nor OneNote). Another thing was to run the Display in default 200% magnification. Otherwise, Mathematica was either not showing Welcome Screen at all and crashing. Whatever is the benchmark now in Windows Surface Pro X, I am happy that Mathematica is working, although it is an unstable work.

I wonder how Mathematica on M1 Mac shows the Benchmark when moving from 12.3.0 to 12.3.1 (i.e. native ARM64). Regards MarcinB.

POSTED BY: Marcin Balcerzyk
Posted 2 years ago

Here are some results running Mathematica 12.3.1 natively on the M1 MacBook Air (https://mathematica.stackexchange.com/questions/234881/benchmarking-with-mathematica-v-12-for-up-to-date-comparison-across-different-ma/250736#250736) and M1 MacBook Pro (https://mathematica.stackexchange.com/questions/234950/benchmarks-for-new-m1-macbooks/250779#250779).

Both benchmark at 3.2. That's pretty far off the 4.48 Murray reported for his core i9 iMac (though with a slightly different version—12.1.1; I don't know how much diff. that makes). Is it possible Mathematica's code isn't yet well-optimized for Apple Silicon? As I understand it, Mathematica's performance on Intel benefits from MKL. Perhaps there's not yet an equivalent library available for Apple Silicon.

Of course, ideally you'd want to assess the M1's performance with calculations representative of your own workload, rather than relying on a synthetic benchmark.

POSTED BY: Aron Yoffe

You are Apple users, anybody tried to run Mathematica on Windows 10 on ARM64 with their x64 emulator? Does it start or install at all? You need Insider´s Build for that as of June 2021. Marcin

POSTED BY: Marcin Balcerzyk

I have a new M1 MacBook Air. Running the last nightly build for the Apple Silicon version, I got a benchmark of 3.21. not bad for a computer intended for light duty.

enter image description here

Posted 3 years ago

I just did, but didn't bother to run multiple times + I have been heavily multitasking.

Attachment

Attachments:
POSTED BY: Michal M

Hoping to refresh this thread.

Has anyone benchmarked the just-released 12.3 on an ARM M1 chip?

Thank you.

POSTED BY: W. Craig Carter
Posted 3 years ago
POSTED BY: Jacob Evans
Posted 3 years ago

Yes, the M1 is significantly faster than the i7-3770 listed in the Mathematica Benchmark report, but that processor was released in 2012.

Thus I wasn't considering that score, but rather what Murray Eisenberg obtained with the 8-core i9-9900K in his 2019 iMac. And based on that, the M1's comparative performance for MMA is somewhat slower than expected (even accounting for Rosetta 2), relative to its comparative performance thus far on other benchmarks.

Specifically, the score Murray obtained with the i9-9900K is 50% higher than what you obtained with an M1 Air. By contrast, some (though not all) benchmarks comparing the single-core performance of the M1 to that of the (even faster) i9-10900K have the M1 as faster when the benchmark is run natively, and approximately comparable when run under Rosetta 2. Thus it's natural to want to ensure that this discrepancy isn't because Murray's i9 was able to use all 8 cores.

More broadly, my own interest/curiosity is in assessing the performance of this new technology from Apple and, to do this, I think it's most meaningful to focus on per-core performance, since the current 4-performance-core limitation is merely a characteristic of these first models. Apple will be likely be offering higher-end systems with more cores in 2021.

It was interesting to see the benchmark you posted—thanks for doing that. But, to get a meaningful assessment of MMA's per-core performance on AS, we'll probably need to wait until WRI produces a build that runs natively on AS.

POSTED BY: Aron Yoffe
Posted 3 years ago

Gotcha. I see what you're saying. It'll be interesting to observe how the M1 performs once Mathematica has been updated to run natively on Apple Silicon. Anecdotally speaking, the non-native Microsoft Office apps seem to run slower on my M1 MacBook Air than they do on my 2012 12-core Mac Pro.

POSTED BY: Jacob Evans

These CPUs which you compared with are too old, I don't believe that.

And My mark is 8.0.

image

Posted 3 years ago

How many cores does the Benchmark function use?

The M1 has only four high-performance cores, while most desktop/laptop CPU's have more; thus, to maintain comparability with benchmark results from other CPU's, you'd want the Benchmark function to be limited to four as well (and you'd also want it to be using the high-performance cores only).

Alternately, one could do comparative testing by timing individual single-threaded commands.

POSTED BY: Aron Yoffe

Thank you!!

POSTED BY: Oliver Ruessing

Hello Jacob,

very promising results - could you give the benchmark another try while the MacBook is hot? E.g after watching a HD Youtube Video for some minutes?

As you are surely aware, the CPU gets throttled when too hot; maximum speed is only available for a short time, until the aluminum body heats up.

Therefore, the high benchmark scoring might be a little misleading: Surely true when the Laptop is idle and the computation is short. But for longer tasks, Classify or Predict of large Datasets, the CPU might run into the temperature-throttle. Such an issue would not be detected by the Benchmark test which is only a few seconds in duration.

Could you "heat up" the MacBook and run the benchmark again? Or maybe run the Benchmark 100 times, and give the result of the last run?

Greetings.

POSTED BY: Oliver Ruessing

Presumably such throttling does not occur with the M1 MacBook Pro, which has a fan rather than just the Air's passive cooling.

POSTED BY: Murray Eisenberg
Posted 3 years ago

Oliver, I did as you suggested and ran the Benchmark function 100 times. The last "BenchmarkResult" reports to be 3.051. While the function was on repeat, the bottom of the MacBook Air became noticeably more warm and the cumulative CPU utilization held steady at around 73%.

POSTED BY: Jacob Evans

Thank you for that benchmark!

Which version of the new M1 MacBook Air is this: which processor (8-core CPU and 7-core GPU or 8-core CPU and 8-core GPU, in case Mathematica takes advantage of the GPU, too)? How much "unified memory" (8GB or 16GB, in case that matters?

For a sobering comparison, for my 2019 27-inch iMac with 3.6 GHz 80Core i9 (64GB 2667 MHz DDR4 RAM) running Mathematica 12.1.1 under macOS Catalina 10.15.7, the WolframMark is 4.48. (If I buy a of new MacBook M1 Air or Pro, it will be secondary to the iMac, but I'll still want to run Mathematica on it at a tolerable speed.)

I'd like to see somebody's WolframMark on the new M1 MacBook Pro, too,

POSTED BY: Murray Eisenberg
Posted 3 years ago

Murray, the aforementioned WolframMark benchmark test was ran on an M1 MacBook Air with the 8-core GPU and 16GB of unified memory.

POSTED BY: Jacob Evans
Posted 3 years ago

After some cursory usage of Mathematica 12.1.1, I can report that it seems to run well on my new MacBook Air M1. Here's a screenshot of the WolframMark Results report:

WolframMark Results on M1 MacBook Air

Given the fact that Mathematica is running under x86 emulation, I'm. pretty impressed with the results.

POSTED BY: Jacob Evans

That would be interesting for me as well. I assume that right now, Mathematica has to use the Rosetta 2 Layer to run on non-x64-architecture?

An easy "benchmark" would be to have them install Mathematica in an Apple Store, run some pre-defined code, and measure the time it needs to perform that task.

POSTED BY: Oliver Ruessing
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract