What is the most efficient export/import format for persisting data?

Questions
, ,
1

I exported a List from a Mathematica 11.1.1.0 Notebook to .wdx format. Since this format is native to Mathematica and intended for data, I had expected it to be the most performant option, with fast export and import times and maintaining exaction resolution of numerics. Unfortunately, the load time to read the .wdx for a moderate size List was over an hour. The List holds a rectangular structure with 24,000 rows and 11,000 columns.

Is .wdx really the best / fastest format for exporting data in a Mathematica variable and then loading it back into a new session?

2

The most efficient format is .mx:

test = RandomReal[{-10, 10}, {24000, 11000}];
AbsoluteTiming[Export["big.mx", test];]

needs 15 seconds ( depending also on your hard drive speed I guess).
Loading it back in by

test=Import["big.mx"];

is pretty fast, too, less than a second.
I never ever use wdx.
Always .mx, which is also the fastest way known to me to collect larger results from parallel subkernels back to the master kernel …

3

Interesting question! I was not aware about the mentioned huge difference in I/O-speed. But here I just want/must cite the documentation:

? MX files cannot be exchanged between different operating systems or versions of the Wolfram System.

? [WDX] Stores arbitrary Wolfram Language expressions in a serialized, platform-independent form.

So, at least for me the .mx format would never ever be an option if data are meant to be persistent.

4

I remember hearing that since a certain version it is cross compatible, I have never tested that thoughÂ…

Other options are compressing it and writing it binary:

str=OpenWrite[file,BinaryFormat->True];
BinaryWrite[str,Compress[myexpression],"TerminatedString"];
Close[str];
5

But is is well-known that the documentation is incorrect.
It just works cross-platform.
The only thing which does work is to load a .mx file produced in a newer version of Mathematica into an oder one.

6

Maybe it is also an endian-issue? since most of us use little endian, the problem doesn’t show up a lot? just a thought…

7

Thanks for the note @Sander Huisman. In my problem case with .wdx, the export from was a GNU/Linux x86_64 (Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz) and the import was to a GNU/Linux x86_64 (Intel(R) Core™ i5-5300U CPU @ 2.30GHz) so both sides would have been using little-endian format. In addition, the version of Mathematica was the same on both systems.

8

WDX is designed to be cross-platform and backwards compatible and so on. MX supposedly isn’t, though there is evidence that makes it cross-platform for newer versions of Mathematica…

9

Hi, may I ask a question? When I export the data after parallel evaluation by .mx, I found it’s very slow! But under the same data amount I export it very fast in one-kernel calculation. Due to the calculation is very solw I have to use parallel, but get into trouble while exporting. So I wonder if you can give some advice? Thanks!