Python, Memory Error in making dataframe

Question

When I use pandas DataFrame, occuring the Memory Error.

data's row is 200000 and column is 30.(type: list) fieldnames1 has columns name.(type:list)

Error occured in:

df = pd.DataFrame(data,columns=[fieldnames1])

what should I do? (python version 2.7 32bit)

You ran out of RAM. Try to use less data for testing and a combination of more RAM, a 64bit OS and 64bit Python. — Klaus D.
– Klaus D., Commented Jul 6, 2015 at 9:03
Any reason you don't use 64-bit version of python? Also storing lists in a df is a bit strange — EdChum
– EdChum, Commented Jul 6, 2015 at 9:13

Community · Accepted Answer · 2017-05-23 10:30:35Z

3

As indicated by Klaus, you're running out of memory. The problem occurs when you try to pull the entire text to memory in one go.

As pointed out in this post by Wes McKinney, "a solution is to read the file in smaller pieces (use iterator=True, chunksize=1000) then concatenate then with pd.concat".

edited May 23, 2017 at 10:30

CommunityBot

11 silver badge

answered Jul 6, 2015 at 10:11

Julien Marrec

11.9k5 gold badges51 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Javeria · Accepted Answer · 2016-12-26 07:26:56Z

-1

You can try this line of code:

data=pd.DataFrame.from_csv("train.csv")

This is an alternate of read.csv but it returns Data frame object without giving any memory error P.S the size of the training data is around 73 mb

answered Dec 26, 2016 at 7:26

Javeria

1011 gold badge1 silver badge7 bronze badges

Collectives™ on Stack Overflow

Python, Memory Error in making dataframe

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related