How to convert Pandas Dataframe to csv reader directly in python?

Question

I have a csv file on with millions of rows. I used to create a dictionary out the csv file like this

 with open('us_db.csv', 'rb') as f:
    data = csv.reader(f)
    for row in data:
       Create Dictionary based on a column

Now to filter the rows based on some conditions I use pandas Dataframe as it is super fast in these operations. I load the csv as pandas Dataframe do some filtering. Then I want to continue doing the above. I thought of using pandas df.iterrows() or df.itertuples() but it is really slow.

Is there a way to convert the pandas dataframe to csv.reader() directly so that I can continue to use the above code. If I use csv_rows = to_csv(), it gives a long string. Ofcourse, I can write out a csv and then read from it again. But I want to know if there is a way to skip the extra read and write to a file.

Bob Haffner · Accepted Answer · 2016-11-11 03:37:58Z

14

You could do something like this..

import numpy as np
import pandas as pd
from io import StringIO
import csv

#random dataframe
df = pd.DataFrame(np.random.randn(3,4))

buffer = StringIO()  #creating an empty buffer
df.to_csv(buffer)  #filling that buffer
buffer.seek(0) #set to the start of the stream

for row in csv.reader(buffer):
    #do stuff

answered Nov 11, 2016 at 3:37

Bob Haffner

8,5231 gold badge40 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Raja Over a year ago

Thanks. That worked. As I was using python2.7 I had to use BytesIO instead of StringIO() as I had some problems with utf-8 coding.

Qikai · Accepted Answer · 2016-11-11 09:23:16Z

0

Why don't you apply the Create Dictionary function to the target column? Something like:

df['column_name'] = df['column_name'].apply(Create Dictionary)

answered Nov 11, 2016 at 9:23

Qikai

346 bronze badges

1 Comment

Raja Over a year ago

I need the whole row to be available inside the function. Apply only send one value at a time. Not one row at a time. Thanks.

Collectives™ on Stack Overflow

How to convert Pandas Dataframe to csv reader directly in python?

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related