MongoDB iterating cursor using python

Question

I am trying to extract some data from a mongodatabase. My collection is structured in this way:

_id:5d306db32e98f83520ae90b7 entity_id:"360803e052b511e3bc11180373d69d01" entity_type_name_item:"TEST" entity_name_item:"Sea" data:"{"json"}" file_type:"list" entity_name:"000001"

I need to extract the json file in "data" and do some operation. The code is working but is really, really slow when I try to loop through the cursor:

    for entity in entity_list:
       cursor = db[coll_name].find({'entity_id':entity})

       for document in cursor:
           jdata = json.loads(document['data'])

Do you have any suggestion?

Have you tried to profile your code? It is not clear what is slow here, getting the data or parsing the json. If json parsing is slow, try a faster json parser (pypi.org/project/ujson). If mongo is slow, try to only fetch what you need using the projection params. pymongo.readthedocs.io/en/stable/api/pymongo/… . In addition, you can fetch all the entities in one query using the $in opetator (docs.mongodb.com/manual/reference/operator/query/in) db[coll_name].find({'entity_id': {'$in': entity_list}}) — user983716
– user983716, Commented May 8, 2020 at 12:41

dankondr · Accepted Answer · 2020-05-08 12:17:06Z

1

I believe that losing speed occurs in part where you make the query.

You can speed up this part using:

find({'entity_id': {'$in': entity_list}})

answered May 8, 2020 at 12:17

dankondr

517 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

MongoDB iterating cursor using python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related