Pandas/Python append or concat dataframes being created in a for loop

Question

I have a process which I am able to loop through for values held in a list but it overwrites the final dataframe with each loop and I would like to append or concat the result of the loops into one dataframe. For example given below I can see 'dataframe' will populate initially with result of 'blah1', then when process finishes it has the result of 'blah2'

listtoloop = ['blah1','blah2']

for name in listtoloop:


    some process happens here resulting in

        dataframe = result of above process

stackoverflow.com/questions/28669482/…

Evan
– Evan

2022-02-03 19:12:32 +00:00
Commented Feb 3, 2022 at 19:12 — Evan
– Evan, Commented Feb 3, 2022 at 19:12

sjw · Accepted Answer · 2022-02-03 17:51:06Z

8

The typical pattern used for this is to create a list of DataFrames, and only at the end of the loop, concatenate them into a single DataFrame. This is usually much faster than appending new rows to the DataFrame after each step, as you are not constructing a new DataFrame on every iteration.

Something like this should work:

listtoloop = ['blah1','blah2']
dfs = []
for name in listtoloop:
    # some process happens here resulting in
    # dataframe = result of above process
    dfs.append(dataframe)
final = pd.concat(dfs, ignore_index=True)

answered Feb 3, 2022 at 17:51

sjw

6,5512 gold badges30 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Evan Over a year ago

Thank you, I think I managed to find another way similar, have also posted.

OldManSeph Over a year ago

@Evan that is what thesilkworm gave in their answer here.

Evan Over a year ago

Yeah, I mentioned I had found a similar way and posted the link to that way also. Thankyou both, I was continuing to research after I asked and our paths crossed.

OldManSeph · Accepted Answer · 2022-02-03 17:59:22Z

0

Put your results in a list and then append the list to the df making sure the list is in the same order as the df

listtoloop = ['blah1','blah2']
df = pd.DataFrame(columns="A","B")
for name in listtoloop:
    ## processes here

    to_append = [5, 6]
    df_length = len(df)
    df.loc[df_length] = to_append

edited Feb 3, 2022 at 17:59

answered Feb 3, 2022 at 17:50

OldManSeph

2,79020 silver badges25 bronze badges

1 Comment

Evan Over a year ago

Thankyou, I think I managed to find another way also, posted in answer.

Evan · Accepted Answer · 2022-02-03 19:03:39Z

0

data_you_need=pd.DataFrame()  
listtoloop = ['blah1','blah2']  
for name in listtoloop:     
    ##some process happens here resulting in          
    ##dataframe = result of above process    
    data_you_need=data_you_need.append(dataframe,ignore_index=True)

answered Feb 3, 2022 at 19:03

Evan

591 gold badge3 silver badges9 bronze badges

1 Comment

Sean Richards Over a year ago

The potential issue with this one is it results in a FutureWarning message. frame.append is deprecated and will be removed from pandas in a future version. However, you can append to a list no problem. Additionally, it's less performant.

Collectives™ on Stack Overflow

Pandas/Python append or concat dataframes being created in a for loop

3 Answers 3

3 Comments

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related