Message Boards Message Boards

0
|
9335 Views
|
5 Replies
|
1 Total Likes
View groups...
Share
Share this post:

Importing large HDF5 files on Windows

Dear All,

I have a fully functioning script that boils down to:

Import[fn, {"HDF5", "Datasets", "/TRACKS_sort"}]

where fn is the filename, and "/TRACKS_sort" is one of the large datasets. The file is a MAT file from Matlab and weighs in at about 3.3 GB.

On Mac the above works fine and I can import the HDF5 file without a problem. However on Windows 7, it does not work. I can get all the names of the datasets without a problem. I can't however import one of those big datasets. This is both Mathematica 10.4.1.

The examples in the HDF5 documentation all work fine, so the HDF5.exe (the convertor used internally by Mathematica) works ok (at least partially). If one tries to import on windows one gets an error like:

" Unable to communicate with closed link LinkObject["C:\Program Files\Wolfram Research\Mathematica\10.4\SystemFiles\Converters\Binaries\Windows-x86-64\HDF5.exe")...]"

Which I think is just a message it spawns because HDF5 crashes?

Any idea?

Might be related to this thread

POSTED BY: Sander Huisman
5 Replies

Hello Sander,

we (at Wolfram) are currently rewriting Import and Export of HDF5 files. Our goal is to add plenty of new functionality and to fix all existing bugs, including the problem you described.

Can you, please, provide the specific file that causes this problem? Or if it is inconvenient for you to upload it, can you at least provide a Matlab script you used to generate this huge dataset?

Thank you.

POSTED BY: Rafal Chojna

Hi Rafal,

I will try to generate a file that shows the same problem very soon. The original file is several GBs so not very handy, but it can be (might be possible to be) minimised to show the same issue.

POSTED BY: Sander Huisman

Dear Rafal, could you send me your wolfram emailaddress? You can find my email address either in the wolfram databases or on my websites. Even better would be a facebook/skype so we can chat.

POSTED BY: Sander Huisman

Dear Rafal,

do you also plan to work on the HDF4 Import/Export. The older format is still widely used and I (and maybe others) often suffer from the incomplete implementation of HDF4 Im/Export.

POSTED BY: Markus Roellig

Hello Markus,

as for now, we do not plan to extend our Import/Export functionality of HDF4 files.

I am aware that this is not a perfect solution, but you may try the following tool from the HDF Group: h4toh5 to convert your files to HDF5 format and then use Mathematica Import/Export functionality.

POSTED BY: Rafal Chojna
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract