How to upload csv with FastAPI and process it in memory with Pandas?

Question

I am trying to upload a csv file with FastAPI and then load it into pandas.

import pandas as pd
import os
import io, base64

from fastapi import FastAPI, File, UploadFile, Form

app = FastAPI()

@app.post('/uploadfile/')
async def create_data_file(
        experiment: str = Form(...),
        file_type: str = Form(...),
        file_id: str = Form(...),
        data_file: UploadFile = File(...),
        ):
    
    #decoded = base64.b64decode(data_file.file)
    #decoded = io.StringIO(decoded.decode('utf-8'))
    
    print(pd.read_csv(data_file.file, sep='\t'))

    return {'filename': data_file.filename, 
            'experiment':experiment, 
            'file_type': file_type, 
            'file_id': file_id}

I tried using the file.file content directly or converting it with base64 or StringIO. I also tried codec. The error I get with the example code is

AttributeError: 'SpooledTemporaryFile' object has no attribute 'readable'

I would be curious to see an example of the file that failed. The method above works for me. (only difference is my test files are not tab-separated) — csum
– csum, Commented Feb 4, 2021 at 17:43

thib00 · Accepted Answer · 2021-04-30 08:47:32Z

7

Changing the encoding to what suits you best, I found this workaround:

from io import StringIO
 
pd.read_csv(StringIO(str(data_file.file.read(), 'utf-16')), encoding='utf-16')

answered Apr 30, 2021 at 8:47

thib00

711 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Soerendip · Accepted Answer · 2021-02-03 19:26:09Z

3

This is a workaround using libraries csv and codecs to create the records which then can be turned into a pandas dataframe:

def to_df(file):
    data = file.file
    data = csv.reader(codecs.iterdecode(data,'utf-8'), delimiter='\t')
    header = data.__next__()
    df = pd.DataFrame(data, columns=header)
    return df

edited Feb 3, 2021 at 19:26

answered Feb 3, 2021 at 19:06

Soerendip

9,29520 gold badges87 silver badges144 bronze badges

1 Comment

Howins Over a year ago

This works great on my side but I need to use the second row of my csv as my Dataframe header. How can I choose which row will be the header? If I do header = data.__next__() it will by default take the first row of my csv as header

Collectives™ on Stack Overflow

How to upload csv with FastAPI and process it in memory with Pandas?

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related