python csv encoding writerow

Question

Using web2py on windows 2008 server I have a following problem

I am creating csv document from json and when writting list to file i get the following error. It crashes on csv writerow

<type 'exceptions.UnicodeEncodeError'> 'ascii' codec can't encode character u'\\u010c'

It works ok on my computer. Windows 7 but on server I have an encoding problems

any suggestions? thank you

My code for creating file is the following

dataDict = json.loads(data.replace("'", "\""))
path = path
scriptName = os.path.join(path, id + 'script.txt')
file = open(scriptName, 'wb') 
output = csv.writer(file, delimiter='\t')

##Month hours
file.write("begin month_hours \r\n")
file.write("delavec    mesec    month_hours_min    month_hours_max\r\n")
for rec in dataDict["mandatory"]:
    output.writerow(rec)
file.write("\r\nend month_hours \r\n")

You are most likely using Python 3 locally, Python 2 on the server. — Martijn Pieters
– Martijn Pieters, Commented Aug 18, 2014 at 12:57
Any reason you're opening the file in 'wb' (write binary) instead of 'w' (write text)? — HSquirrel
– HSquirrel, Commented Aug 18, 2014 at 13:17
@Yebach: then on the server you are dealing with data that happens to be outside the ASCII range and you got lucky locally. — Martijn Pieters
– Martijn Pieters, Commented Aug 18, 2014 at 14:39
@HSquirrel: that's the correct handling in Python 2 for the CSV module, see that module's documentation. — Martijn Pieters
– Martijn Pieters, Commented Aug 18, 2014 at 14:40

Martijn Pieters · Accepted Answer · 2018-05-04 08:38:41Z

6

JSON strings are always Unicode values and in Python 2 need to be encoded when writing to a CSV file. If you don't do this explicitly, Python will use the ASCII codec. That's fine if all your data contains text in the ASCII range but fails when you encounter data beyond that.

Pick a different encoding and encode explicitly; UTF-8 is a good encoding to pick:

for rec in dataDict["mandatory"]:
    output.writerow([unicode(c).encode('utf8') for c in rec])

I first convert all values to unicode(), in case you have data in there that's not already a unicode() value; numbers or booleans or None for example. The result is then explicitly encoded to UTF-8.

edited May 4, 2018 at 8:38

answered Aug 18, 2014 at 14:51

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Wes Over a year ago

Note, in Python3, replace unicode() with str() above

Martijn Pieters Over a year ago

@Wes: In Python 3, you'd have opened the file in text mode and specified a codec, so this whole issue wouldn't even play. You would not need to encode data passed to the csv.writer() object at all.

kdazzle · Accepted Answer · 2021-12-03 19:06:25Z

1

It's been a while since this question was posted, so I'm guessing that the Python API has changed, but nowadays, you can actually open the file with an explicit encoding, which fixed a similar issue that I was having:

Changing this line in your code should do the trick. No need to make your code O(n2)ish with that inner loop.

open(scriptName, 'w', encoding='utf-8')

Or for the full example from the python docs:

import csv
with open('some.csv', newline='', encoding='utf-8') as f:
    reader = csv.reader(f)
    for row in reader:
        print(row)

edited Dec 3, 2021 at 19:06

answered Dec 3, 2021 at 19:01

kdazzle

4,3303 gold badges26 silver badges28 bronze badges

Collectives™ on Stack Overflow

python csv encoding writerow

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related