Problem encoding accented characters with python

Question

I'm having trouble encoding accented characters in a URL using the python command line. Reducing my problem to the essential, this code:

>>> import urllib
>>> print urllib.urlencode({'foo' : raw_input('> ')})
> áéíóúñ

prints this in a mac command line:

foo=%C3%A1%C3%A9%C3%AD%C3%B3%C3%BA%C3%B1

but the same code prints this in windows' command line:

foo=%A0%82%A1%A2%A3%A4

The mac result is correct and the characters get encoded as needed; but in windows I get a bunch of gibberish.

I'm guessing the problem lies in the way windows encodes characters, but I haven't been able to find a solution; I'd be very grateful if you could help me. Thanks in advance!

Daniel Kluev · Accepted Answer · 2010-07-23 03:10:46Z

3

You can use explicit encoding to get consistent result.

>>> str = u"áéíóúñ"
>>> import urllib
>>> urllib.urlencode({'foo':str.encode('utf-8')})
'foo=%C3%A1%C3%A9%C3%AD%C3%B3%C3%BA%C3%B1'

However you need to ensure your string is in unicode first, so it may require decoding if its not, like raw_input().decode('latin1') or raw_input().decode('utf-8')

Input encoding depends on the locale of console, I believe, so its system-specific.

EDIT: unicode(str) should use locale encoding too to convert to unicode, so that could be a solution.

answered Jul 23, 2010 at 3:10

Daniel Kluev

11.4k2 gold badges38 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mark Tolonen · Accepted Answer · 2010-07-23 16:09:12Z

2

The Windows command line uses cp437 encoding in US Windows. You need utf-8:

>>> import sys
>>> sys.stdin.encoding
'cp437'
>>> print urllib.urlencode({'foo':raw_input('> ').decode('cp437').encode('utf8')})
> áéíóúñ
foo=%C3%A1%C3%A9%C3%AD%C3%B3%C3%BA%C3%B1

edited Jul 23, 2010 at 16:09

answered Jul 23, 2010 at 15:36

Mark Tolonen

181k26 gold badges182 silver badges278 bronze badges

Collectives™ on Stack Overflow

Problem encoding accented characters with python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related