Group Abstract Group Abstract

Message Boards Message Boards

1
|
20.5K Views
|
8 Replies
|
8 Total Likes
View groups...
Share
Share this post:

What is the most efficient export/import format for persisting data?

Donald Pellegrino, Dow Chemical
Posted 9 years ago

I exported a List from a Mathematica 11.1.1.0 Notebook to .wdx format. Since this format is native to Mathematica and intended for data, I had expected it to be the most performant option, with fast export and import times and maintaining exaction resolution of numerics. Unfortunately, the load time to read the .wdx for a moderate size List was over an hour. The List holds a rectangular structure with 24,000 rows and 11,000 columns.

Is .wdx really the best / fastest format for exporting data in a Mathematica variable and then loading it back into a new session?
POSTED BY: Donald Pellegrino
8 Replies
Sort By:
POSTED BY: Rolf Mertig
POSTED BY: Sander Huisman

But is is well-known that the documentation is incorrect. It just works cross-platform. The only thing which does work is to load a .mx file produced in a newer version of Mathematica into an oder one.
POSTED BY: Rolf Mertig

Interesting question! I was not aware about the mentioned huge difference in I/O-speed. But here I just want/must cite the documentation:

? MX files cannot be exchanged between different operating systems or versions of the Wolfram System.

? [WDX] Stores arbitrary Wolfram Language expressions in a serialized, platform-independent form.

So, at least for me the .mx format would never ever be an option if data are meant to be persistent.
POSTED BY: Henrik Schachner
Tony Coliderse
Tony Coliderse
Posted 5 years ago

Hi, may I ask a question? When I export the data after parallel evaluation by .mx, I found it's very slow! But under the same data amount I export it very fast in one-kernel calculation. Due to the calculation is very solw I have to use parallel, but get into trouble while exporting. So I wonder if you can give some advice? Thanks!
POSTED BY: Tony Coliderse

WDX is designed to be cross-platform and backwards compatible and so on. MX supposedly isn't, though there is evidence that makes it cross-platform for newer versions of Mathematica…
POSTED BY: Sander Huisman
POSTED BY: Donald Pellegrino

Maybe it is also an endian-issue? since most of us use little endian, the problem doesn't show up a lot? just a thought…
POSTED BY: Sander Huisman
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard