Open file as a buffer in Python?

Question

I have a compressed HDF file (HDF.Z) and would like to open it like that:

from subprocess import Popen, PIPE
f = Popen(['zcat', 'myfile.HDF.Z'], stdout=PIPE).stdout

In order to get the data I need to use pyhdf:

from pyhdf.SD import SD, SDC
mydata = SD(f, SDC.READ)

However, this results in an error message:

*** TypeError: coercing to Unicode: need string or buffer, file found

Is there a way to open this file as a buffer to read it in? Btw: what is a buffer?

Since the file is extracted via zcat and opened with Popen I only have the stdout? — HyperCube
– HyperCube, Commented Feb 5, 2014 at 11:26

glglgl · Accepted Answer · 2014-02-05 11:24:08Z

1

At a short glance, I found no way to make it access an open stream.

You can do the following:

Another option is very system dependent: you could take the file handle and do

SD('/dev/fd/%d' % f.fileno(), SDC.READ)

bit this is very platform-dependent (Linux only) and, if SD() does mmap() by any chance, it will fail.

answered Feb 5, 2014 at 11:24

glglgl

91.5k13 gold badges157 silver badges230 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

The temporary unpacking of the file works, although this takes more time (considering the fact that I perform this action on hundreds of files).