Message Boards Message Boards

2
|
19024 Views
|
13 Replies
|
7 Total Likes
View groups...
Share
Share this post:

Mathematica V10 execution time has grown up to 12% to 45% from V9

Posted 10 years ago

A simple example: RosettaCode's Universal Turing Machine (5-state, 2-symbol probable Busy Beaver machine from Wikipedia) In V9 time = 225 seconds, in V10 322 seconds (Windows i5 desktop).

Not much to do but hope for better performance. I think I hear more disk swapping but that's not measured (yet).

POSTED BY: Douglas Kubler
13 Replies
Posted 10 years ago

I ran the matrix multiply for M8 and M10. (M9 was uninstalled) M10 is marginally faster but repeating the tests a few times shows M8 can be better than the worst of M10.

Code attached. Chrome freezes every time I insert code so I'll show a sample picture.

enter image description here

I ran Timing and AbsoluteTiming, matrix sizes 1050 and 2050..

  • Matrix 1050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 0.374402

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 0.436803

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 0.406

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 0.468

  • Matrix 1050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 0.456026

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 0.493028

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 0.5020287

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 0.5210298

  • Matrix 2050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 2.714417

10.0 for Microsoft Windows (64-bit) (June 29, 2014) Time = 2.839218

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 2.855

8.0 for Microsoft Windows (64-bit) (October 7, 2011) Time = 2.949

  • Matrix 2050

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 2.938168

10.0 for Microsoft Windows (64-bit) (June 29, 2014) (Abs)Time = 3.058175

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 3.1111780

8.0 for Microsoft Windows (64-bit) (October 7, 2011) (Abs)Time = 3.1811819

Attachments:
POSTED BY: Douglas Kubler
Posted 10 years ago

A simple example: RosettaCode's Universal Turing Machine (5-state, 2-symbol probable Busy Beaver machine from Wikipedia) In V9 time = 225 seconds, in V10 322 seconds (Windows i5 desktop).

Not much to do but hope for better performance. I think I hear more disk swapping but that's not measured (yet).

POSTED BY: Nev Nev
Posted 10 years ago

You can actually see the test code used in the benchmarking. The code for "Matrix Arithmetic" in Mathematica 10 is the following.

Module[{m}, 
 AbsoluteTiming[SeedRandom[1]; m = RandomReal[{}, {840, 840}]; 
  Do[(1.` + 0.5` m)^127, {50}];]]

On my laptop (MacBook Pro Late 2013, i5 2.4GHz), the output is {1.842847, Null} by Mathematica 10, while it is {1.115800, Null} by Mathematica 9. From run to run, the exact time changes, but it is certain that M10 runs slower than M9 for this code.

I have observed that the number of processing cores was detected to be 2 in M10, while it is 4 in M9. I changed the number of parallel kernels manually, but it didn't affect the output of the code above.

On the other hand, the "Matrix Multiplication"

Module[{m1, m2}, 
 AbsoluteTiming[SeedRandom[1]; m1 = RandomReal[{}, {1050, 1050}]; 
  m2 = RandomReal[{}, {1050, 1050}]; Do[m1.m2, {12}]]]

gives {0.581776, Null} by M10, and {1.421713, Null} by M9. Again, the exact time fluctuates a bit, but it is certain that M10 runs the multiplication faster than M9.

POSTED BY: Jeongwan Haah

The Benchmarking says that MMA10 is faster - at least on my machine (MacBook Pro, 2.8GHz i7, 16GB RAM, SSD).

Needs["Benchmarking`"]
BenchmarkReport[]

gives

enter image description here

for Mathematica 9 and

enter image description here

for Mathematica 10. That means that the score went up by more or less 20%, which constitutes quite a nice speed gain. In version 9 have noticed that the same task is quite a bit slower on a Mac than on a Linux machine - I do not have access to Windows. So the speed increase could be due to an optimisation for OSX. We also benchmarked to more Macs, and a standard iMac got a speed gain of about 40%.

Here's the data in another, more detailed form:

{"MachineName" -> "marcos-mbp", "System" -> "Mac OS X x86 (64-bit)",
"BenchmarkName" -> "MathematicaMark9", "FullVersionNumber" -> "9.0.1", "Date" -> "July 11, 2014", "BenchmarkResult" -> 1.101, "TotalTime" -> 12.568, "Results" -> {{"Data Fitting", 0.493}, {"Digits of Pi", 0.65}, {"Discrete Fourier Transform", 0.82}, {"Eigenvalues of a Matrix", 0.665}, {"Elementary Functions", 1.141}, {"Gamma Function", 0.549}, {"Large Integer Multiplication", 0.552}, {"Matrix Arithmetic", 1.016}, {"Matrix Multiplication", 1.377}, {"Matrix Transpose", 0.802}, {"Numerical Integration", 0.663}, {"Polynomial Expansion", 0.119}, {"Random Number Sort", 1.561}, {"Singular Value Decomposition", 0.998}, {"Solving a Linear System", 1.162}}}

and

{"MachineName" -> "marcos-mbp", "System" -> "Mac OS X x86 (64-bit)",
"BenchmarkName" -> "WolframMark", "FullVersionNumber" -> "10.0.0",
"Date" -> "July 11, 2014", "BenchmarkResult" -> 1.236, "TotalTime" -> 11.197, "Results" -> {{"Data Fitting", 0.862}, {"Digits of Pi", 0.444}, {"Discrete Fourier Transform", 0.567}, {"Eigenvalues of a Matrix", 0.572}, {"Elementary Functions", 0.843}, {"Gamma Function", 0.585}, {"Large Integer Multiplication", 0.466}, {"Matrix Arithmetic", 2.012}, {"Matrix Multiplication", 0.616}, {"Matrix Transpose", 0.808}, {"Numerical Integration", 0.735}, {"Polynomial Expansion", 0.147}, {"Random Number Sort", 1.164}, {"Singular Value Decomposition", 0.78}, {"Solving a Linear System", 0.596}}}

We can plot this:

enter image description here

If this is correct Matrix arithmetic became slower, from 1.016 in MMA9 to 2.012 in MMA10. Many other functions became faster.

Cheers, M.

POSTED BY: Marco Thiel

Indeed. Some of that is due to MKL getting in update (linear algebra, FFT, etc.) Do we know if the benchmarks included with v9 and v10 are identical though?

My general impression has been that most things are faster in v10, but some examples (like this one) show that some other things got slower. It would be nice to be able to track down what exactly is causing the slowdown.

POSTED BY: Szabolcs Horvát
POSTED BY: Marco Thiel
Posted 10 years ago

I gave up with the Benchmarks some time ago. The tests are not run for the new release so you're always comparing apples to oranges.

POSTED BY: Douglas Kubler

Hi Douglas,

yes, it appears that the "old machines" are not benchmarked with the new release, but if the tests are the same and I compare my own computer with both versions, i.e. MMA9 and MMA10, that should be valid, shouldn't it?

I meant the argument to suggest that the benchmark tests are probably the same for both versions, which is likely based on the numbers for the "standard machines". So, then it should be valid to compare the two versions 9/10 on the same machine.

M.

POSTED BY: Marco Thiel
Posted 10 years ago

Hi Marco,

Making a horizontal comparison with yourself is OK, but not a vertical comparison with other machines. It's deceptive to show my computer against any other model since no others have run M10. I took this up with support (going M8->M9) but met with apathy.

(benchmarks should be longer running anyway)

D.

POSTED BY: Douglas Kubler
Posted 10 years ago
POSTED BY: Luca M
POSTED BY: Szabolcs Horvát
Posted 10 years ago

The web page has the exact code I uploaded. Put Timing[ ] around any driver/test. The last one is the long-running example.

POSTED BY: Douglas Kubler

Can you share the exact code you used to benchmark this?

POSTED BY: Szabolcs Horvát
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract