Getting error when INSERT into MySQL

Question

_mysql_exceptions.Warning: Incorrect string value: '\xE7\xB9\x81\xE9\xAB\x94...' for column 'html' at row 1


def getSource(theurl, moved = 0):
    if moved == 1:
        theurl = urllib2.urlopen(theurl).geturl()
    urlReq = urllib2.Request(theurl)
    urlReq.add_header('User-Agent',random.choice(agents))
    urlResponse = urllib2.urlopen(urlReq)
    htmlSource = urlResponse.read()
    return htmlSource

new_u = Url(source_url = source_url, source_url_short = source_url_short, source_url_hash = source_url_hash, html = htmlSource)
new_u.save()

Why is this happening? I am basically downloading URL of a page...and then saving it to a database using Django.

It only happens sometimes....and sometimes it works fine.

Edit: it seems like I have to set the database to UTF-8? What is the command to do that?

@alex: Ohhh...never worked with that system. May best guess is, that you don't escape the html-String, and he's trying to insert 'faulty' values. For PHP the functionality is called mysql_real_escape_string. — Bobby
– Bobby, Commented Nov 27, 2009 at 12:27

Community · Accepted Answer · 2017-05-23 12:19:33Z

3

You basically need to ensure proper a string encoding. E.g. the string you provide to django is not UTF-8 encoded and therefore some characters can't be resolved.

Some helpful advice on how to find the encoding of the requested page can be found here: urllib2 read to Unicode

edited May 23, 2017 at 12:19

CommunityBot

11 silver badge

answered Nov 27, 2009 at 12:32

miku

189k47 gold badges314 silver badges317 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

extraneon · Accepted Answer · 2009-11-27 13:31:03Z

0

There are 2 ways to go if you want to alter the character set in MySQL. First is the default of the database, see MySQL Alter database, and the second is per-table: MySQL Alter Table.

The database gives the default charset for, I believe, new tables. This can be overridden on a per-table basis, which you need to do since you already have tables. "utf8" is a supported character set.

Also have a look at Blog about UTF8 with django and MySQL.

answered Nov 27, 2009 at 13:31

extraneon

24.1k2 gold badges51 silver badges52 bronze badges

Collectives™ on Stack Overflow

Getting error when INSERT into MySQL

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related