Python mysqldb db.commit() not working

Question

I have some code like this

db = MySQLdb.connect(host="localhost", user="root", passwd="root", db="domains")
cursor = db.cursor()

def check_urls(res):
    pool = Pool(25)
    for row in res:
        pool.spawn(fetch, row[0], row[1])
    pool.join()

def fetch(*args):
    """code trimmed for brevity"""
    cursor.execute("""UPDATE com SET http_status=%s, is_checked=1 WHERE id=%s""",
                  (output.get('http_status', ""), id))


for _ in xrange(10000):
    cursor.execute("SELECT domain, id FROM com WHERE is_checked IS NULL LIMIT 100")
    result = cursor.fetchall()
    check_urls(result)
    db.commit()


cursor.close()
db.close()

My program get stuck at db.commit(). No values updated in the database. Can someone tell me whats wrong?.

Please note: My check_urls function has a for loop and each loop has one mysql update query.

Are you getting any errors? What does "stuck at db.commit()" mean? Is the program stuck or do you mean that part doesn't happen? — yuvi
– yuvi, Commented May 17, 2014 at 20:36
It appears as though you might be spawning 25,000 threads and joining the results back together just before the commit. Are you sure the commit is the problem? It's not that likely... — Ben
– Ben, Commented May 17, 2014 at 20:36
@yuvi No i'm not getting any errors. I checked my MySQL error logs. No errors. I mean its running only the first loop. Even that first loop no values get updated in the database — PrivateUser
– PrivateUser, Commented May 17, 2014 at 20:48
@Ben 25000 threads? I'm selecting only 100 domains and passing it into the pool. — PrivateUser
– PrivateUser, Commented May 17, 2014 at 20:51

valignatev · Accepted Answer · 2014-05-18 08:33:10Z

1

It seems like you should move your db.commit() into fetch function after cursor.execute. Another way you may use "global" keyword with cursor. Or, third way, just make cursor as parameter of functions.

answered May 18, 2014 at 8:33

valignatev

6,3468 gold badges41 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python mysqldb db.commit() not working

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related