Can you share the exact code you used to benchmark this?

I can reproduce this slowdown on OS X. I got 220 s (v9) vs 280 s (v10).

The Benchmarking says that MMA10 is faster - at least on my machine (MacBook Pro, 2.8GHz i7, 16GB RAM, SSD).

Needs["Benchmarking`"]
BenchmarkReport[]

gives

for Mathematica 9 and

for Mathematica 10. That means that the score went up by more or less 20%, which constitutes quite a nice speed gain. In version 9 have noticed that the same task is quite a bit slower on a Mac than on a Linux machine - I do not have access to Windows. So the speed increase could be due to an optimisation for OSX. We also benchmarked to more Macs, and a standard iMac got a speed gain of about 40%.

Here's the data in another, more detailed form:

{"MachineName" -> "marcos-mbp", "System" -> "Mac OS X x86 (64-bit)",
"BenchmarkName" -> "MathematicaMark9", "FullVersionNumber" -> "9.0.1", "Date" -> "July 11, 2014", "BenchmarkResult" -> 1.101, "TotalTime" -> 12.568, "Results" -> {{"Data Fitting", 0.493}, {"Digits of Pi", 0.65}, {"Discrete Fourier Transform", 0.82}, {"Eigenvalues of a Matrix", 0.665}, {"Elementary Functions", 1.141}, {"Gamma Function", 0.549}, {"Large Integer Multiplication", 0.552}, {"Matrix Arithmetic", 1.016}, {"Matrix Multiplication", 1.377}, {"Matrix Transpose", 0.802}, {"Numerical Integration", 0.663}, {"Polynomial Expansion", 0.119}, {"Random Number Sort", 1.561}, {"Singular Value Decomposition", 0.998}, {"Solving a Linear System", 1.162}}}

and

{"MachineName" -> "marcos-mbp", "System" -> "Mac OS X x86 (64-bit)",
"BenchmarkName" -> "WolframMark", "FullVersionNumber" -> "10.0.0",
"Date" -> "July 11, 2014", "BenchmarkResult" -> 1.236, "TotalTime" -> 11.197, "Results" -> {{"Data Fitting", 0.862}, {"Digits of Pi", 0.444}, {"Discrete Fourier Transform", 0.567}, {"Eigenvalues of a Matrix", 0.572}, {"Elementary Functions", 0.843}, {"Gamma Function", 0.585}, {"Large Integer Multiplication", 0.466}, {"Matrix Arithmetic", 2.012}, {"Matrix Multiplication", 0.616}, {"Matrix Transpose", 0.808}, {"Numerical Integration", 0.735}, {"Polynomial Expansion", 0.147}, {"Random Number Sort", 1.164}, {"Singular Value Decomposition", 0.78}, {"Solving a Linear System", 0.596}}}

We can plot this:

If this is correct Matrix arithmetic became slower, from 1.016 in MMA9 to 2.012 in MMA10. Many other functions became faster.

Cheers, M.

No, we do not really know whether they are identical, but the 3.07Ghz i7 950 (8 cores) is set to 1 in both benchmarks, i.e. the benchmarks are normalised to that machine. So there is a hope that they are the same.

More interestingly, it appears that all its individual timings for all individual tests are the same for the i7 in V9 and V10 , too, so that could suggest that the benchmarks are supposed to be the same. In fact, none of the machines that are in both benchmarks have changed the individual timings. So they are probably all the same tests run in V9. The comparison on the V10 benchmark are most likely compared to the machines in V9 - at least for the "older" machines. The top scorer in the V10 benchmark is the Mac Pro, which might have been tested under V10. I agree that much of this is mere speculation.

You are absolutely right, it would indeed be nice to know which bits are slower and why. It is strange that "Matrix Arithmetic" got quite a bit slower. Can anyone run the benchmark on something other than OSX? It appeared in MMA9 that OSX was a bit slower than Linux systems that had lower specifications. I think that we ran Windows in a sandbox (parallels or so) on a Mac and MM9 was faster there than on OSX.

I have tested a couple of my programs and they do appear to run faster in MMA10 than in MMA9.

Cheers, Marco

Hi Douglas,

yes, it appears that the "old machines" are not benchmarked with the new release, but if the tests are the same and I compare my own computer with both versions, i.e. MMA9 and MMA10, that should be valid, shouldn't it?

I meant the argument to suggest that the benchmark tests are probably the same for both versions, which is likely based on the numbers for the "standard machines". So, then it should be valid to compare the two versions 9/10 on the same machine.

M.

You can actually see the test code used in the benchmarking. The code for "Matrix Arithmetic" in Mathematica 10 is the following.

Module[{m}, 
 AbsoluteTiming[SeedRandom[1]; m = RandomReal[{}, {840, 840}]; 
  Do[(1.` + 0.5` m)^127, {50}];]]

On my laptop (MacBook Pro Late 2013, i5 2.4GHz), the output is {1.842847, Null} by Mathematica 10, while it is {1.115800, Null} by Mathematica 9. From run to run, the exact time changes, but it is certain that M10 runs slower than M9 for this code.

I have observed that the number of processing cores was detected to be 2 in M10, while it is 4 in M9. I changed the number of parallel kernels manually, but it didn't affect the output of the code above.

On the other hand, the "Matrix Multiplication"

Module[{m1, m2}, 
 AbsoluteTiming[SeedRandom[1]; m1 = RandomReal[{}, {1050, 1050}]; 
  m2 = RandomReal[{}, {1050, 1050}]; Do[m1.m2, {12}]]]

gives {0.581776, Null} by M10, and {1.421713, Null} by M9. Again, the exact time fluctuates a bit, but it is certain that M10 runs the multiplication faster than M9.

I ran the matrix multiply for M8 and M10. (M9 was uninstalled) M10 is marginally faster but repeating the tests a few times shows M8 can be better than the worst of M10.

Code attached. Chrome freezes every time I insert code so I'll show a sample picture.

I ran Timing and AbsoluteTiming, matrix sizes 1050 and 2050..

• Matrix 1050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 0.374402

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 0.436803

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 0.406

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 0.468

• Matrix 1050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 0.456026

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 0.493028

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 0.5020287

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 0.5210298

• Matrix 2050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 2.714417

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 2.839218

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 2.855

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 2.949

• Matrix 2050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 2.938168

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 3.058175

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 3.1111780

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 3.1811819

Attachments: