Use codecs to read file with correct encoding: TypeError

Question

I need to read from a file, linewise. Also also need to make sure the encoding is correctly handled.

I wrote the following code:

#!/bin/bash

import codecs

filename = "something.x10"

f = open(filename, 'r')
fEncoded = codecs.getreader("ISO-8859-15")(f)

totalLength = 0
for line in fEncoded:
  totalLength+=len(line)

print("Total Length is "+totalLength)

This code does not work on all files, on some files I get a

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    for line in fEncoded:
  File "/usr/lib/python3.2/codecs.py", line 623, in __next__
    line = self.readline()
  File "/usr/lib/python3.2/codecs.py", line 536, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib/python3.2/codecs.py", line 480, in read
    data = self.bytebuffer + newdata
TypeError: can't concat bytes to str

Im using python 3.3 and the script must work with this python version.

What am I doing wrong, I was not able to find out which files work and which not, even some plain ASCII files fail.

You say some plain ASCII files fail. Do you have one that is short enough to post here? — Kevin
– Kevin, Commented May 23, 2013 at 11:56

Thomas Fenzl · Accepted Answer · 2013-05-23 12:22:46Z

3

You are opening the file in non-binary mode. If you read from it, you get a string decoded according to your default encoding (http://docs.python.org/3/library/functions.html?highlight=open%20builtin#open).

codec's StreamReader needs a bytestream (http://docs.python.org/3/library/codecs#codecs.StreamReader)

So this should work:

import codecs

filename = "something.x10"

f = open(filename, 'rb')
f_decoded = codecs.getreader("ISO-8859-15")(f)

totalLength = 0
for line in f_decoded:
   total_length += len(line)

print("Total Length is "+total_length)

or you can use the encoding parameter on open:

 f_decoded = open(filename, mode='r', encoding='ISO-8859-15')

The reader returns decoded data, so I fixed your variable name. Also, consider pep8 as a guide for formatting and coding style.

edited May 23, 2013 at 12:22

answered May 23, 2013 at 12:09

Thomas Fenzl

4,4221 gold badge19 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Use codecs to read file with correct encoding: TypeError

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related