Python: MySQLdb library encoding issue

Question

I have a mysql db. I set charset to utf8;

...
  PRIMARY KEY  (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 | 
...

I connect to db in python with MySQLdb;

conn = MySQLdb.connect(host = "localhost",
                               passwd = "12345",
                               db = "db",
                               charset = 'utf8',
                               use_unicode=True)

When I execute a query, response is decoding with "windows-1254". Example response;

curr = conn.cursor(MySQLdb.cursors.DictCursor)
select_query = 'SELECT * FROM users'
curr.execute(select_query)

for ret in curr.fetchall():
    username = ret["username"]
    print "repr-username; ", repr(username)
    print "username; "username.encode("utf-8")
...

output is;

repr-username;  u'\xc5\u0178\xc3\xbckr\xc3\xbc\xc3\xa7a\xc4\u0178l\xc3\xbcli'
username;  ÅŸÃ¼krÃ¼Ã§aÄŸlÃ¼li

When I print username with "windows-1254" it works fine;

...
print "repr-username; ", repr(username)
print "username; ", username.encode("windows-1254")
...

Output is;

repl-username;  u'\xc5\u0178\xc3\xbckr\xc3\xbc\xc3\xa7a\xc4\u0178l\xc3\xbcli'
username;  şükrüçağlüli

When I try it with some other characters like cyrillic alphabet, decodeding is changed dinamicly. How can I prevent it?

Yes. This text has some turkish special characters like "şüçğ". — umut
– umut, Commented Aug 28, 2014 at 12:57
Terminal encoding ? Other idea: could you modify your test case to both INSERT and SELECT from Python. Does the problem persist ? — Sylvain Leroux
– Sylvain Leroux, Commented Aug 28, 2014 at 13:12
On my UTF-8 system u"şükrüçağlüli" == u'\u015f\xfckr\xfc\xe7a\u011fl\xfcli'. This is not what you have. Are you certain the data have been properly encoded at INSERT time ? — Sylvain Leroux
– Sylvain Leroux, Commented Aug 28, 2014 at 13:19

Vizjerei · Accepted Answer · 2014-08-28 13:43:06Z

3

I think the items where encoded wrong while INSERT to the database.

I recommend python-ftfy(from https://github.com/LuminosoInsight/python-ftfy) (helped me out in a simillar problem):

import ftfy

username = u'\xc5\u0178\xc3\xbckr\xc3\xbc\xc3\xa7a\xc4\u0178l\xc3\xbcli'
print ftfy.fix_text(username) # outputs şükrüçağlüli

answered Aug 28, 2014 at 13:43

Vizjerei

1,0309 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python: MySQLdb library encoding issue

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related