Detect wrong character encodings using python

Question

I'm new to serious programming and I was trying to write a python program where I encountered strings in this form while reading from a file:

Îêåàí Åëüçè - Ìàéæå âåñíà
Ëÿïèñ Òðóáåöêîé - Ñâÿùåííûé Îãîíü

which is actually supposed to be in cyrillic (cp-1251), so this string is the victim of wrong encoding (I found it after long searching, with the help of this site:Universal Cyrillic Decoder)

Also using detect function in chardet module could find it

chardet.detect('Îêåàí Åëüçè - Ìàéæå âåñíà'.decode('utf-8').encode('windows-1252'))

which gives:
{'confidence': 0.7679697235616183, 'encoding': 'windows-1251'}

after doing the following I'm able to get the intended string

string.decode('utf-8').encode('windows-1252').decode('windows-1251').encode('utf-8')

which gives:

Океан Ельзи - Майже весна and
Коррозия Металла - Война Миров

respectively for the aforementioned strings.

My question is: Is there anyway to detect such strings? Here are some other strings which I haven't even found a way to correct:

Isao Sasaki - ¨¬¡Æ¨¬¡ÆAI¨¬¡Æ (A Different Farewell) (¡¾¢¬Cy¨ù¡¾ AU¡Æi)
Yoon K. Lee & Salzburg Kammerp - ³»¸¶À½
⁂‭晉䤠圠牥⁥⁡潂⁹䬨牡慭牴湯捩删浥硩䴠楡⥮
Ã�Ã�Ã³Ã´Ã¥ÃµÃ¡ Ã¼Ã´Ã© Ã¯ Ã£Ã�Ã¬Ã¯Ã² Ã�Ã´Ã¡Ã Ã¬Ã©Ã¡ Ã¡Ã°Ã«Ã� ÃµÃ°Ã¼Ã¨Ã¥Ã³Ã§.

Much grateful for your replies.

Lennart Regebro · Accepted Answer · 2011-01-18 16:09:52Z

4

Well, that Cyrillic string isn't in cp-1251. As you seem to have found out, it has been encoded "twice". Most likely somebody took a binary string in cp1251 believed it was in utf8 and encoded it in cp1252, or something like that.

No automatic check could figure that out.

>>> print 'Îêåàí Åëüçè - Ìàéæå âåñíà'.decode('utf8').encode('latin1').decode('cp1251')
Океан Ельзи - Майже весна

works. The latter look like UTF8, in that it supports both single and multibyte characters, but it's not UTF8. so again some sort of incorrect transformaton has been done. Going through all possible combinations until one work is probably the only possibility.

edited Jan 18, 2011 at 16:09

answered Jan 18, 2011 at 15:37

Lennart Regebro

173k45 gold badges230 silver badges254 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user579943 Over a year ago

Thank You very much, I guess I'll have to resort to tiresome trial and error and a bit of guesswork. Well, I'll post if I do find out more about this.

Collectives™ on Stack Overflow

Detect wrong character encodings using python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related