Pipe output of python script

Question

I'm running ./sample.py --url http://blah.com without error, though if I run ./sample.py --url http://blah.com | wc -l or similar I receive an error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u200f' in position 0: ordinal not in range(128)

How do I make a python script compatible with my terminal commands? I keep seeing reference to sys.stdin.isatty though its use case appears to be opposite.

What does sample.py look like? Are you just doing normal print statements? — Brendan Long
– Brendan Long, Commented Nov 20, 2012 at 20:35
You might try using xargs: ./sample.py --url http://blah.com | xargs wc -l — David
– David, Commented Nov 20, 2012 at 20:42

unutbu · Accepted Answer · 2012-11-20 20:47:17Z

6

When Python detects that it is printing to a terminal, sys.stdout.encoding is set to the encoding of the terminal. When you print a unicode, the unicode is encoded to a str using the sys.stdout.encoding.

When Python does not detect that it is printing to a terminal, sys.stdout.encoding is set to None. When you print a unicode, the ascii codec is used (at least in Python2). This will result in a UnicodeError if the unicode contains code points outside of 0-127.

One way to fix this is to explicitly encode your unicode before printing. That perhaps is the proper way, but it can be laborious if you have a lot of print statements scattered around.

Another way to fix this is to set the PYTHONIOENCODING environment variable to an appropriate encoding. For example,

PYTHONIOENCODING=utf-8

Then this encoding will be used instead of ascii when printing output to a file.

See the PrintFails wiki page for more information.

edited Nov 20, 2012 at 20:47

answered Nov 20, 2012 at 20:40

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

mbb Over a year ago

Thank you for that @unutbu. Thought #1 - follow this article from now on. #2 - Where is the right place to put this line for now?

unutbu Over a year ago

@mjb: How you should set the PYTHONIOENCODING environment variable depends on the machine's OS. It is done the same way as you set the PYTHONPATH environment variable. On Linux, you could put export PYTHONIOENCODING=utf-8 in your ~/.profile or ~/.bashrc file.

jfs Over a year ago

@mjb: for a single command in bash: PYTHONIOENCODING=utf-8 ./sample.py .... btw, User-perceived characters and codepoints are different things, though it is a topic for another article

sampson-chen · Accepted Answer · 2012-11-20 20:35:59Z

-1

Try:

(./sample.py --url http://blah.com) | wc -l

This spawns a subshell to run your python script then pipes the output from stdout to wc

answered Nov 20, 2012 at 20:35

sampson-chen

47.6k13 gold badges87 silver badges81 bronze badges

2 Comments

jfs Over a year ago

It won't work, try ( python -c 'print(u"tra\u00eetre")' ) | wc -l

Fahad Naeem Over a year ago

py file can only be executed with python interpretor at the start of sample.py

Collectives™ on Stack Overflow

Pipe output of python script

2 Answers 2

3 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related