Quickly read HDF 5 file in python?

Question

I have an instrument that saves data (many traces from an analog-to-digital converter) as an HDF 5 file. How can I efficiently open this file in python? I have tried the following code, but it seems to take a very long time to extract the data.

Also, it reads the data in the wrong order: instead of reading 1,2,3, it reads 1,10,100,1000.

Any ideas?

Here is a link to the sample data file: https://drive.google.com/file/d/0B4bj1tX3AZxYVGJpZnk2cDNhMzg/edit?usp=sharing

And here is my super-slow code:

import h5py
import matplotlib.pyplot as plt
import numpy as np


f = h5py.File('sample.h5','r')

ks = f.keys()

for index,key in enumerate(ks[:10]):
    print index, key
    data = np.array(f[key].values())
    plt.plot(data.ravel())

plt.show()

Community · Accepted Answer · 2017-05-23 11:58:18Z

3

As far as the order of your data:

In [10]: f.keys()[:10]
Out[10]:
[u'Acquisition.1',
 u'Acquisition.10',
 u'Acquisition.100',
 u'Acquisition.1000',
 u'Acquisition.1001',
 u'Acquisition.1002',
 u'Acquisition.1003',
 u'Acquisition.1004',
 u'Acquisition.1005',
 u'Acquisition.1006']

This is the correct order for numbers that isn't left padded with zeros. It's doing its sort lexicographically, not numerically. See Python: list.sort() doesn't seem to work for a possible solution.

Second, you're killing your performance by rebuilding the array within the loop:

In [20]: d1 = f[u'Acquisition.990'].values()[0][:]

In [21]: d2 = np.array(f[u'Acquisition.990'].values())

In [22]: np.allclose(d1,d2)
Out[22]: True

In [23]: %timeit d1 = f[u'Acquisition.990'].values()[0][:]
1000 loops, best of 3: 401 µs per loop

In [24]: %timeit d2 = np.array(f[u'Acquisition.990'].values())
1 loops, best of 3: 1.77 s per loop

edited May 23, 2017 at 11:58

CommunityBot

11 silver badge

answered Apr 19, 2014 at 3:25

JoshAdel

69.1k27 gold badges146 silver badges146 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

DanHickstein Over a year ago

Thanks Josh! But plt.plot doesn't seem to want to plot the data unless I use the np.array() function. Is there another way to do this?

JoshAdel Over a year ago

@DanHickstein I updated my answer to fix the issue. This assumes that each f[key].values() contains only one item of interest, which seems to be the case from the data that I quickly looked at.

DanHickstein Over a year ago

Ah! I figured this out just a moment before I saw your answer pop up. Thanks :)

Collectives™ on Stack Overflow

Quickly read HDF 5 file in python?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related