Python pandas fast way to Insert row into DF with missing values as Nan

Question

I am looking for a fast way to insert a df_row (dataframe with only 1 row, and column could be less than 24) into a df_master (dataframe with few thousand rows and 24 columns)

for example:

df_row
A B C D E F G .... H
1 2 3 4 5 6 7 .... 8

df_master
A B C D ........................... Y

I am looking for a fast way to insert df_row into df_master, and set NaN for the columns that do no exist in df_row.

Previously, I was doing

df_master = df_master.append(df_row)

But this method gets slower as df_master gets better.

Is there a fast way of doing inplace append?

jezrael · Accepted Answer · 2017-08-01 14:05:16Z

2

You can use setting with enlargement, but need convert df1 to Series - e.g. by selecting by iloc:

d = {'F': {0: 6}, 'D': {0: 4}, 'B': {0: 2}, 'C': {0: 3}, 
'A': {0: 1}, 'E': {0: 5}, 'G': {0: 7}, 'H': {0: 8}}
df = pd.DataFrame(d)
print (df)
   A  B  C  D  E  F  G  H
0  1  2  3  4  5  6  7  8

df1 = pd.DataFrame([[10,20,30,40]], columns=list('ACDE'))
print (df1)
    A   C   D   E
0  10  20  30  40

df.loc[len(df.index)] = df1.iloc[0]
print (df)
       A    B     C     D     E    F    G    H
 0   1.0  2.0   3.0   4.0   5.0  6.0  7.0  8.0
 1  10.0  NaN  20.0  30.0  40.0  NaN  NaN  NaN

edited Aug 1, 2017 at 14:05

answered Aug 1, 2017 at 14:00

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Chris Johnson Over a year ago

It looks like its not appending properly. I only see 1 record at the end of my function. I should be seeing 100

Chris Johnson Over a year ago

Is there a faster way to append? This method is still too slow... When the DF gets bigger (few hundred thousand records), its taking 5 seconds to insert 100 records.

jezrael Over a year ago

Hmmm, I think you need append only one row.

jezrael Over a year ago

If need append 100 row, better is use one big df with 100 rows and then append only once.

Chris Johnson Over a year ago

Yes, but what Im trying to say is, the speed of append depends on the size of the DF... Is there a way to append in constant speed?

|

Collectives™ on Stack Overflow

Python pandas fast way to Insert row into DF with missing values as Nan

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related