Message Boards Message Boards

2
|
19158 Views
|
13 Replies
|
7 Total Likes
View groups...
Share
Share this post:

Mathematica V10 execution time has grown up to 12% to 45% from V9

Posted 10 years ago

A simple example: RosettaCode's Universal Turing Machine (5-state, 2-symbol probable Busy Beaver machine from Wikipedia) In V9 time = 225 seconds, in V10 322 seconds (Windows i5 desktop).

Not much to do but hope for better performance. I think I hear more disk swapping but that's not measured (yet).

POSTED BY: Douglas Kubler
13 Replies
Posted 10 years ago

I ran the matrix multiply for M8 and M10. (M9 was uninstalled) M10 is marginally faster but repeating the tests a few times shows M8 can be better than the worst of M10.

Code attached. Chrome freezes every time I insert code so I'll show a sample picture.

enter image description here

I ran Timing and AbsoluteTiming, matrix sizes 1050 and 2050..

  • Matrix 1050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 0.374402

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 0.436803

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 0.406

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 0.468

  • Matrix 1050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 0.456026

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 0.493028

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 0.5020287

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 0.5210298

  • Matrix 2050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 2.714417

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 2.839218

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 2.855

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 2.949

  • Matrix 2050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 2.938168

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 3.058175

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 3.1111780

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 3.1811819

Attachments:
POSTED BY: Douglas Kubler
Posted 10 years ago

A simple example: RosettaCode's Universal Turing Machine (5-state, 2-symbol probable Busy Beaver machine from Wikipedia) In V9 time = 225 seconds, in V10 322 seconds (Windows i5 desktop).

Not much to do but hope for better performance. I think I hear more disk swapping but that's not measured (yet).

POSTED BY: Nev Nev
Posted 10 years ago

You can actually see the test code used in the benchmarking. The code for "Matrix Arithmetic" in Mathematica 10 is the following.

Module[{m}, 
 AbsoluteTiming[SeedRandom[1]; m = RandomReal[{}, {840, 840}]; 
  Do[(1.` + 0.5` m)^127, {50}];]]

On my laptop (MacBook Pro Late 2013, i5 2.4GHz), the output is {1.842847, Null} by Mathematica 10, while it is {1.115800, Null} by Mathematica 9. From run to run, the exact time changes, but it is certain that M10 runs slower than M9 for this code.

I have observed that the number of processing cores was detected to be 2 in M10, while it is 4 in M9. I changed the number of parallel kernels manually, but it didn't affect the output of the code above.

On the other hand, the "Matrix Multiplication"

Module[{m1, m2}, 
 AbsoluteTiming[SeedRandom[1]; m1 = RandomReal[{}, {1050, 1050}]; 
  m2 = RandomReal[{}, {1050, 1050}]; Do[m1.m2, {12}]]]

gives {0.581776, Null} by M10, and {1.421713, Null} by M9. Again, the exact time fluctuates a bit, but it is certain that M10 runs the multiplication faster than M9.

POSTED BY: Jeongwan Haah

The Benchmarking says that MMA10 is faster - at least on my machine (MacBook Pro, 2.8GHz i7, 16GB RAM, SSD).

Needs["Benchmarking`"]
BenchmarkReport[]

gives

enter image description here

for Mathematica 9 and

enter image description here

for Mathematica 10. That means that the score went up by more or less 20%, which constitutes quite a nice speed gain. In version 9 have noticed that the same task is quite a bit slower on a Mac than on a Linux machine - I do not have access to Windows. So the speed increase could be due to an optimisation for OSX. We also benchmarked to more Macs, and a standard iMac got a speed gain of about 40%.

Here's the data in another, more detailed form:

{"MachineName" -> "marcos-mbp", "System" -> "Mac OS X x86 (64-bit)",
"BenchmarkName" -> "MathematicaMark9", "FullVersionNumber" -> "9.0.1", "Date" -> "July 11, 2014", "BenchmarkResult" -> 1.101, "TotalTime" -> 12.568, "Results" -> {{"Data Fitting", 0.493}, {"Digits of Pi", 0.65}, {"Discrete Fourier Transform", 0.82}, {"Eigenvalues of a Matrix", 0.665}, {"Elementary Functions", 1.141}, {"Gamma Function", 0.549}, {"Large Integer Multiplication", 0.552}, {"Matrix Arithmetic", 1.016}, {"Matrix Multiplication", 1.377}, {"Matrix Transpose", 0.802}, {"Numerical Integration", 0.663}, {"Polynomial Expansion", 0.119}, {"Random Number Sort", 1.561}, {"Singular Value Decomposition", 0.998}, {"Solving a Linear System", 1.162}}}

and

{"MachineName" -> "marcos-mbp", "System" -> "Mac OS X x86 (64-bit)",
"BenchmarkName" -> "WolframMark", "FullVersionNumber" -> "10.0.0",
"Date" -> "July 11, 2014", "BenchmarkResult" -> 1.236, "TotalTime" -> 11.197, "Results" -> {{"Data Fitting", 0.862}, {"Digits of Pi", 0.444}, {"Discrete Fourier Transform", 0.567}, {"Eigenvalues of a Matrix", 0.572}, {"Elementary Functions", 0.843}, {"Gamma Function", 0.585}, {"Large Integer Multiplication", 0.466}, {"Matrix Arithmetic", 2.012}, {"Matrix Multiplication", 0.616}, {"Matrix Transpose", 0.808}, {"Numerical Integration", 0.735}, {"Polynomial Expansion", 0.147}, {"Random Number Sort", 1.164}, {"Singular Value Decomposition", 0.78}, {"Solving a Linear System", 0.596}}}

We can plot this:

enter image description here

If this is correct Matrix arithmetic became slower, from 1.016 in MMA9 to 2.012 in MMA10. Many other functions became faster.

Cheers, M.

POSTED BY: Marco Thiel

Indeed. Some of that is due to MKL getting in update (linear algebra, FFT, etc.) Do we know if the benchmarks included with v9 and v10 are identical though?

My general impression has been that most things are faster in v10, but some examples (like this one) show that some other things got slower. It would be nice to be able to track down what exactly is causing the slowdown.

POSTED BY: Szabolcs Horvát

No, we do not really know whether they are identical, but the 3.07Ghz i7 950 (8 cores) is set to 1 in both benchmarks, i.e. the benchmarks are normalised to that machine. So there is a hope that they are the same.

More interestingly, it appears that all its individual timings for all individual tests are the same for the i7 in V9 and V10 , too, so that could suggest that the benchmarks are supposed to be the same. In fact, none of the machines that are in both benchmarks have changed the individual timings. So they are probably all the same tests run in V9. The comparison on the V10 benchmark are most likely compared to the machines in V9 - at least for the "older" machines. The top scorer in the V10 benchmark is the Mac Pro, which might have been tested under V10. I agree that much of this is mere speculation.

You are absolutely right, it would indeed be nice to know which bits are slower and why. It is strange that "Matrix Arithmetic" got quite a bit slower. Can anyone run the benchmark on something other than OSX? It appeared in MMA9 that OSX was a bit slower than Linux systems that had lower specifications. I think that we ran Windows in a sandbox (parallels or so) on a Mac and MM9 was faster there than on OSX.

I have tested a couple of my programs and they do appear to run faster in MMA10 than in MMA9.

Cheers, Marco

POSTED BY: Marco Thiel
Posted 10 years ago

I gave up with the Benchmarks some time ago. The tests are not run for the new release so you're always comparing apples to oranges.

POSTED BY: Douglas Kubler

Hi Douglas,

yes, it appears that the "old machines" are not benchmarked with the new release, but if the tests are the same and I compare my own computer with both versions, i.e. MMA9 and MMA10, that should be valid, shouldn't it?

I meant the argument to suggest that the benchmark tests are probably the same for both versions, which is likely based on the numbers for the "standard machines". So, then it should be valid to compare the two versions 9/10 on the same machine.

M.

POSTED BY: Marco Thiel
Posted 10 years ago

Hi Marco,

Making a horizontal comparison with yourself is OK, but not a vertical comparison with other machines. It's deceptive to show my computer against any other model since no others have run M10. I took this up with support (going M8->M9) but met with apathy.

(benchmarks should be longer running anyway)

D.

POSTED BY: Douglas Kubler
Posted 10 years ago

I can also reproduce a longer processing time (about 20%) with my Windows 8.1 machine (4 cores/8 logical processors): 186s (V9) - 228s (v10)

POSTED BY: Luca M
POSTED BY: Szabolcs Horvát
Posted 10 years ago

The web page has the exact code I uploaded. Put Timing[ ] around any driver/test. The last one is the long-running example.

POSTED BY: Douglas Kubler

Can you share the exact code you used to benchmark this?

POSTED BY: Szabolcs Horvát
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract