Python : UnicodeEncodeError when I use grep

Question

I am using a simple python script to get reservation results for my CID : simple.py:

data = {"minorRev":"current minorRev #","cid":"xxx","apiKey":"xxx","customerIpAddress":"  ","creationDateStart":"03/31/2013","}

url = 'http://someservice/services/rs/'                      
req = requests.get(url,params=data)                        
print req                                                                 
print req.text                                                                
print req.status_code

Now on the command prompt if I do python simple.py it runs perfectly and prints the req.text variable

However when I try to do

python simple.py | grep pattern

I get

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 1314: ordinal not in range(128)

read through this. Basically, when piping the output, sys.stdout.encoding==None — shx2
– shx2, Commented Apr 1, 2013 at 7:02

Álvaro Justen · Accepted Answer · 2013-04-01 09:27:07Z

print needs to encode the string before sending to stdout but when the process is in a pipe, the value of sys.stdout.encoding is None, so print receives an unicode object and then it tries to encode this object using the ascii codec -- if you have non-ASCII characters in this unicode object, an exception will be raised.

You can solve this problem encoding all unicode objects before sending it to the standard output (but you'll need to guess which codec to use). See these examples:

File wrong.py:

# coding: utf-8

print u'Álvaro'

Result:

alvaro@ideas:/tmp
$ python wrong.py 
Álvaro
alvaro@ideas:/tmp
$ python wrong.py | grep a
Traceback (most recent call last):
  File "wrong.py", line 3, in <module>
    print u'Álvaro'
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in position 0: ordinal not in range(128)

File right.py:

# coding: utf-8

print u'Álvaro'.encode('utf-8')
# unicode object encoded == `str` in Python 2

Result:

alvaro@ideas:/tmp
$ python right.py 
Álvaro
alvaro@ideas:/tmp
$ python right.py | grep a
Álvaro

jfs · Accepted Answer · 2016-01-30 11:39:34Z

7

If sys.stdout.isatty() is false (the output is redirected to a file/pipe) then configure PYTHONIOENCODING envvar outside your script. Always print Unicode, don't hardcode the character encoding of your environment inside your script:

$ PYTHONIOENCODING=utf-8 python simple.py | grep pattern

answered Jan 30, 2016 at 11:39

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Collectives™ on Stack Overflow

Python : UnicodeEncodeError when I use grep

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related