How to transfer MySQL database to MongoDB using MongoEngine?

Question

I have an existing collection in MongoDB and I need to update certain documents by getting values from a MySQL table.

Currently, I am using the following python code to update the Collection using MongoEngine but it takes a lot of time since the dataset is very large (2m+ documents in the Collection, out of which 80k+ documents need to be updated)

query = "Select * from business"
cursor.execute(query)
for line in cursor:
    Movie.objects(movieid=line[0]).first().update(set__business=line[1])

While updating early documents (like first 5k documents) it takes very less amount of time, but when it starts to go beyond that, it takes forever. My assumption is that it starts finding the record from the beginning every time. It can be somehow optimized, since MySQL table records are ordered.

Is there a way to search in the collection so that the next loop starts from the last updated documentid? What is the best method to update MongoDB collections using MongoEngine?

(Additional info: inserting 2m+ records took about 2 hours whereas updating 80k records took 25 hours)

@JohnnyHK, Thanks, I did not realize that. If you wish to add it as an answer, I'll accept it. — Grimlock
– Grimlock, Commented Mar 12, 2017 at 16:06

JohnnyHK · Accepted Answer · 2017-03-12 16:09:06Z

1

Add an index on movieid to your collection to speed up your updates that key off that field.

answered Mar 12, 2017 at 16:09

JohnnyHK

313k70 gold badges632 silver badges479 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to transfer MySQL database to MongoDB using MongoEngine?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related