How to upload a CSV file into MongoDB using FastAPI and Pandas

Question

I'm currently trying to have an upload function on my client side that takes a csv file and uploads it into the mongoDB. If anyone has any clue on how to help I'd really appreciate it!

Here's my current code

@router.post("/uploadfile", response_description="upload a file")
async def upload(request: Request,file: UploadFile = File(...)):
    contents = file.file.read()
    buffer = BytesIO(contents)
    df = pd.read_csv(buffer)
    buffer.close()
    file.file.close()
    df.to_dict(orient='records')
    await request.app.mongodb["SentenceModel"].insert_one(file.file)

I also have gotten the ability for the file to upload into mongo but only as one document with all of the data as objects when it should be 1700 documents.. Here's that code

    @router.post("/uploadfile", response_description="upload a file")
async def upload(request: Request,file: UploadFile = File(...)):
    csvReader = csv.DictReader(codecs.iterdecode(file.file, 'utf-8'))
    data = {}
    for rows in csvReader:             
        key = rows['rowNum'] 
        data[key] = rows  
    
    file.file.close()
    await request.app.mongodb["SentenceModel"].insert_one(data)

I've tried many other methods that i've seen posted around but found no luck with any... I have no problem uploading the data but I want the data to be inserted into mongo once uploaded

Does this answer your question? Insert a Pandas Dataframe into mongodb using PyMongo — ray
– ray, Commented Feb 24, 2023 at 1:10
On the second code block if I use .insert.many I get the error that can't insert empty data into db — Brunyan
– Brunyan, Commented Feb 24, 2023 at 1:14

Brunyan · Accepted Answer · 2023-02-24 01:26:21Z

0

I fixed it! for anyone wondering how what I did wrong... I didn't put my mongo request in my for loop....

@router.post("/uploadfile", response_description="upload a file")
async def upload(request: Request,file: UploadFile = File(...)):
    data = {}
    contents = file.file.read()
    buffer = StringIO(contents.decode('utf-8'))
    csvReader = csv.DictReader(buffer)
    for row in csvReader:  
        key = row['rowNum']
        data[key] = row  
        await request.app.mongodb["SentenceModel"].insert_one(row)

    buffer.close()
    file.file.close()

answered Feb 24, 2023 at 1:26

Brunyan

11 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

ray Over a year ago

This is performing one db call per record, which can be very costly. You may want to wrap your insertions in a bulk operation like this

Collectives™ on Stack Overflow

How to upload a CSV file into MongoDB using FastAPI and Pandas

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related