Inserting data from csv file to mongodb

Question

Made a route in flask to read data from csv file and insert in mongodb. This is my first time writing python code so i'm trying few things what i want to do in my project.

@app.route('/adddata', methods=["GET"])
def add_data():
data = []
with open(csvfile) as file:
    filereader = csv.DictReader(file)
    for row in filereader:
        data.append(row)

usersList = mongo.db.users
usersList.insert_many(data)
return jsonify(data)



ERROR in app: Exception on /adddata [GET]
Traceback (most recent call last):
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site-packages/flask/app.py", 
line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site-packages/flask/app.py", 
line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site-packages/flask/app.py", 
line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site- 
packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site-packages/flask/app.py", 
line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site-packages/flask/app.py", 
line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "index.py", line 39, in add_data
return jsonify(d)
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site- 
packages/flask/json/__init__.py", line 370, in jsonify
dumps(data, indent=indent, separators=separators) + "\n",
File "/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site- 
packages/flask/json/__init__.py", line 211, in dumps
rv = _json.dumps(obj, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 
234, in dumps
return cls(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py", line 
199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py", line 
257, in iterencode
return _iterencode(o, 0)     File 
"/Users/aditya/Work/python/flaskMongo/venv/lib/python3.8/site- 
packages/flask/json/__init__.py", 
line 100, in default
return _json.JSONEncoder.default(self, o)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py", line 
179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ObjectId is not JSON serializable
127.0.0.1 - - [12/Jan/2020 01:19:44] "GET /adddata HTTP/1.1" 500 -

Data is inserting in mongodb but i am not getting response in browser. When i remove insert code block i am able to see response in my browser.

usersList = mongo.db.users
usersList.insert_many(data)

Generated 1000 random data from mockaroo

id,first_name,last_name,email,gender,ip_address
1,Boonie,Caldero,[email protected],Male,61.232.233.148
2,Tillie,Speck,[email protected],Female,214.179.103.117
3,Lotti,Mulqueen,[email protected],Female,28.90.88.221
4,Hilda,Arlett,[email protected],Female,81.29.176.212
5,Philbert,Dwelley,[email protected],Male,183.54.107.176
6,Anastasia,Faucett,[email protected],Female,79.121.189.231
7,Trev,Hakes,[email protected],Male,132.112.216.92
8,Kiersten,Siss,[email protected],Female,34.204.92.1
9,Scot,Donoghue,[email protected],Male,227.46.24.115
10,Anatollo,Urien,[email protected],Male,198.96.237.73

I found the error the reason is data list is being modified and every value contains _id field like '_id': ObjectId('5e1acfc63ebed78e17f266e6') after insert_many, But i don't understand why data list is being mutated. — Aditya.Naik
– Aditya.Naik, Commented Jan 12, 2020 at 8:01

Belly Buster · Accepted Answer · 2020-01-12 11:42:23Z

1

You've mostly answered your own question but to cover your issue, the pymongo drivers will always add an _id field to the data on insertion.

When a document is inserted a special key, "_id", is automatically added if the document doesn’t already contain an "_id" key. Reference

See this simple example. If you don't want the _id field, just pop it after the insert; alternatively you can take a copy of the data into a new variable before inserting it.

import pymongo

db = pymongo.MongoClient()['mycollections']
records = [{'a': 1}, {'b': 2}]
usersList = db.testcollection.insert_many(records)
print(f'Before pop() : {records}')
for record in records:
    record.pop('_id')
print(f'After pop()  : {records}')

gives:

Before pop() : [{'a': 1, '_id': ObjectId('5e1b055b1eaf204578766fc6')}, {'b': 2, '_id': ObjectId('5e1b055b1eaf204578766fc7')}]
After pop()  : [{'a': 1}, {'b': 2}]

answered Jan 12, 2020 at 11:42

Belly Buster

8,9142 gold badges12 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Aditya.Naik Over a year ago

Is it the drivers that add _id or it is being added by mongod ?

Belly Buster Over a year ago

MongoDB will always add an id field if not present, and the drivers will add it to document.

Collectives™ on Stack Overflow

Inserting data from csv file to mongodb

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related