Convert binary input stream to text mode

Question

I am trying to read a bzip2-compressed CSV file in Python 3.2. For an uncompressed CSV file, this works:

datafile = open('./file.csv', mode='rt')
data = csv.reader(datafile)
for e in data:    # works
    process(e)

The problem is that BZ2File only supports creating a binary stream, and in Python 3, csv.reader accepts only text streams. (The same issue occurs with gzip and zip files.)

datafile = bz2.BZ2File('./file.csv.bz2', mode='r')
data = csv.reader(datafile)
for e in data:    # error
    process(e)

In particular, the indicated line throws the exception _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?).

I've also tried data = csv.reader(codecs.EncodedFile(datafile, 'utf8')), but that doesn't fix the error.

How can I wrap the binary input stream so that it can be used in text mode??

pepr · Accepted Answer · 2012-05-09 08:48:41Z

5

This works for me:

import codecs, csv
f = codecs.open("file.csv", "r", "utf-8")
g = csv.reader(f)
for e in g:
    print(e)

In the case of BZ2:

import codecs, csv, bz2
f = bz2.BZ2File("./file.csv.bz2", mode="r")
c = codecs.iterdecode(f, "utf-8")
g = csv.reader(c)
for e in g:
    print(e)

edited May 9, 2012 at 8:48

pepr

21.1k15 gold badges83 silver badges148 bronze badges

answered May 9, 2012 at 4:22

vz0

33k7 gold badges47 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ignacio Vazquez-Abrams Over a year ago

And how exactly did you intend on using this with a bz2-compressed file?

pepr Over a year ago

@vz0: The last detail to be checked is if the interpretation of newlines works as expected (i.e. no interpretation). In other words, if the newline sequences can be contained inside the quoted values, an uncompressed CSV would use the open(fname, 'r', newlines=''). It is likely, it works correctly. Can you try?

Collectives™ on Stack Overflow

Convert binary input stream to text mode

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related