Python encoding decoding how to

Question

I am decoding a string "öçÇşŞükrÜ" to some other charset.

I encountered some interesting problem and want to learn how I can solve it.

>>> "öçÇşŞükrÜ".decode("utf-8")
u'\xf6\xe7\xc7\u015f\u015e\xfckr\xdc'
>>> "öçÇşŞükrÜ".decode("utf-8").encode("utf-8")
'\xc3\xb6\xc3\xa7\xc3\x87\xc5\x9f\xc5\x9e\xc3\xbckr\xc3\x9c'


>>> import chardet
>>> chardet.detect('\xc3\xb6\xc3\xa7\xc3\x87\xc5\x9f\xc5\x9e\xc3\xbckr\xc3\x9c')
{'confidence': 0.99, 'encoding': 'utf-8'}
>>> chardet.detect("öçÇşŞükrÜ")
{'confidence': 0.99, 'encoding': 'utf-8'}

What I do not understand and can not find is how to retrieve the original string "öçÇşŞükrÜ" with an decode and encode process with Python ?

Read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!). Python files should be written in ASCII (although you can enter Unicode characters if you set the encoding). — Katriel
– Katriel, Commented Nov 16, 2012 at 13:13
Has been discussed many times before. Best resource: youtube.com/watch?v=sgHbC6udIqc — deceze
– deceze ♦, Commented Nov 16, 2012 at 13:14

AKX · Accepted Answer · 2012-11-16 13:16:34Z

1

I am decoding a string "öçÇşŞükrÜ" to some other charset.

No, you're decoding an encoded string into an in-memory Unicode representation. Sorry if that sounds harsh, but IMHO it's really important to get the basics right with Unicode and not just fumble around until something works. EDIT: The resources in the comments, https://www.youtube.com/watch?v=sgHbC6udIqc and http://www.joelonsoftware.com/articles/Unicode.html are also essential. Read and watch them. What you need to know is that nothing on disk, or in your terminal is Unicode. Everything is encoded, using one encoding or another.

If you are using Linux, you're lucky! (Us Windows devs... well, world of pain, etc.) Your console will most likely be UTF-8, so doing

"öçÇşŞükrÜ".decode("UTF-8")

will probably make sense.

Did you try

print "öçÇşŞükrÜ".decode("UTF-8").encode("UTF-8")

(that is, with print applied)? You should then get your original öçÇşŞükrÜ back.

answered Nov 16, 2012 at 13:16

AKX

171k16 gold badges146 silver badges228 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Skr Over a year ago

Thanks, now I know I need to study more on encodings. That solved my problem.

Collectives™ on Stack Overflow

Python encoding decoding how to

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related