more efficient way to construct DataFrame through loop function execution

Question

Now I have written a parser to extract the information from raw html source code, which could return them as a tuple, and I have to loop this function and use the return to construct a DataFrame (each loop's return as a row). Here's what I have done:

import pandas as pd
import leveldb
for key, value in db.RangeIter():
    html = db.Get(key)
    result = parser(html)
    df = df.append(pd.Series(result, index = index), ignore_index = True)

Note that parser and index are already defined, and db is a leveldb object which store all links and corresponding html source code. My problem is what's the more efficient way to construct that DataFrame? THANKS!

Do you want to keep the tuple in one column, or split among len(tuple) columns? If the former, you're probably better off just appending to a simple list, then converting that list to a series after the for loop. — elPastor
– elPastor, Commented May 10, 2017 at 11:20
@pshep123 Sorry for ambiguity, I try to make each return one row and each element under one column. So yes, there are total len(tuple) columns. — Jiahui Zhang
– Jiahui Zhang, Commented May 10, 2017 at 11:29

elPastor · Accepted Answer · 2017-05-10 12:34:31Z

1

I would create a dataframe before the loop starts, then append successive dataframes to that. Note that if result is a tuple, it needs to be converted to a list before being converted into a dataframe. And I assume your index is already a list. So:

df = pd.DataFrame()

for key, value in db.RangeIter(): 
    html = db.Get(key) 
    result = parser(html)
    df = df.append(pd.DataFrame(list(result), index = index).transpose())

df.reset_index(inplace = True)

This is not to say your parser could not more efficiently return data for the creation of a dataframe, but I'm working within the confines of a single returned tuple.

Also, depending on the number of elements in the tuple, it may be more efficient to create simple python lists within the loop then create dataframes from those lists when complete, but you don't state the tuple size.

edited May 10, 2017 at 12:34

answered May 10, 2017 at 11:53

elPastor

9,14411 gold badges59 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

more efficient way to construct DataFrame through loop function execution

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related