# HDF5 import partial dataset

**URL:** <https://community.wolfram.com/t/hdf5-import-partial-dataset/4846>\
**Category:** Questions\
**Tags:** chemistry, computer-science, data-science, engineering, signal-processing, software-development\
**Created:** [February 13, 2015, 9:57pm UTC](https://community.wolfram.com/t/hdf5-import-partial-dataset/4846 "2015-02-13T21:57:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![drozd](https://avatars.discourse-cdn.com/v4/letter/d/e68b1a/32.png) [@drozd](https://community.wolfram.com/u/drozd)\
**Post date:** [February 13, 2015, 9:57pm UTC](https://community.wolfram.com/t/hdf5-import-partial-dataset/4846/1 "2015-02-13T21:57:14Z")

</div>

Hi I am trying to import a dataset from an HDF5 file, but the dataset is ~3 gb in size, so I cannot just open it directly. I need to import parts of the dataset, reduce that part to a single number (take the sum of all its elements), keep that number, move to the next part, and so on.

It seems that I cannot open part of a dataset even if I use:  
Import[“file.h5”, {“HDF5”, “Datasets”,1,1,1}]]

This will return the first part, of the first part, of the first dataest, but it seems to still open the _entire_ dataset, which causes the operation to crash.

My though was to do something like:

```
data = Table[Total@Import["file.h5",{"HDF5","Datasets",1,1,n}]

```

But I am stuck at getting the Import[] to work. I get this error:

```
" Unable to communicate with closed link \
LinkObject[\!\(\"\\\"C:\\\\Program Files\\\\Wolfram \
Research\\\\Mathematica\\\\10.0\SystemFiles\Converters\
Binaries\\\\Windows-x86-64\\\\HDF5.exe\\\"\"\),7137,4"] "

```

I can open smaller datasets and parts of the smaller datasets with no problem.

Any thoughts on how to access just a part of a large dataset so it doesn’t crash the import process?

---

<div class="post-metadata">

**Author:** ![pwillis](https://avatars.discourse-cdn.com/v4/letter/p/8c91f0/32.png) [@pwillis](https://community.wolfram.com/u/pwillis)\
**Post date:** [February 21, 2015, 2:47am UTC](https://community.wolfram.com/t/hdf5-import-partial-dataset/4846/2 "2015-02-21T02:47:12Z")

</div>

For large HDF datasets it may be better to export the SD that you want using another tool.  
Then import the raw binary data into Mathematica.

I can suggest the following tool for exporting SD blocks from HDF4 and HDF5 files:  
‘gdal\_translate’ is one of the GDAL utilities and works great for this.  
[http://www.gdal.org/frmt\_hdf5.html](http://www.gdal.org/frmt_hdf5.html)

I am currently working on some code to step through the various dimensions of  
binary remote sensing files. I am trying to avoid loading the complete file in favor  
of only loading a single channel, frame, DN value, or spectral profile.

The lower level file I/O routines are your friend there.

Functions of interest are:  
OpenRead[]  
Skip[]  
Read[]  
Close[]

I did battle with Import[] for a while until it became obvious that I was loading way too much data.

I hope that helps.
