Pandas append dataframe without creating new columns

Question

I have two dataframes that look like this:

df1=
   A   B   
1  A1  B1
2  A2  B2
3  A3  B3

df2 = 
   A   C
4  A4  C4
5  A5  C5

I would like to append df2 to df1, like so:

   A   B   
1  A1  B1
2  A2  B2
3  A3  B3
4  A4  NaN
5  A5  NaN

(Note: I've edited the dataframes so that not all the columns in df1 are necessarily in df2)

Whether I use concat or append, the resulting dataframe I get would have a column called "C" with the first three rows filled with nan. I just want to keep the two original columns in df1, with the new values appended. Is there a way concatenate the dataframes without having to drop the extra column afterwards?

Can you edit to show what you want the final dataframe to look like, given the above example? I'm having a hard time visualizing it. — Jeff
– Jeff, Commented Jul 5, 2016 at 14:00

jezrael · Accepted Answer · 2016-07-05 15:12:57Z

3

You can first filter columns for appending by subset:

print (df2[['A']])
    A
4  A4
5  A5

print (pd.concat([df1, df2[['A']]]))
    A    B
1  A1   B1
2  A2   B2
3  A3   B3
4  A4  NaN
5  A5  NaN

print (df1.append(df2[['A']]))
    A    B
1  A1   B1
2  A2   B2
3  A3   B3
4  A4  NaN
5  A5  NaN

print (df2[['A','B']])
    A   B
4  A4  B4
5  A5  B5

print (pd.concat([df1, df2[['A','B']]]))
    A   B
1  A1  B1
2  A2  B2
3  A3  B3
4  A4  B4
5  A5  B5

Or:

print (df1.append(df2[['A','B']]))
    A   B
1  A1  B1
2  A2  B2
3  A3  B3
4  A4  B4
5  A5  B5

EDIT by comment:

If columns in df1 and df2 have different columns, use intersection:

print (df1)
    A   B  D
1  A1  B1  R
2  A2  B2  T
3  A3  B3  E

print (df2)
    A   B   C
4  A4  B4  C4
5  A5  B5  C5

print (df1.columns.intersection(df2.columns))
Index(['A', 'B'], dtype='object')

print (pd.concat([df1, df2[df1.columns.intersection(df2.columns)]]))
    A   B    D
1  A1  B1    R
2  A2  B2    T
3  A3  B3    E
4  A4  B4  NaN
5  A5  B5  NaN

edited Jul 5, 2016 at 15:12

answered Jul 5, 2016 at 14:02

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

ylangylang Over a year ago

Thanks. That works for the original way I asked the question. What should I do if not all the columns of df1 are in df2?

jezrael Over a year ago

Then you get NaN in columns which does not match like (df1.append(df2)). Do you need append other way?

ylangylang Over a year ago

The result you got looks perfectly good, but then you would have to do this for every column that's in df1 but not df2.

jezrael Over a year ago

If all columns of df1 are in df2, use print (pd.concat([df1, df2[df1.columns]]))

jezrael Over a year ago

If not, you can find common columns by intersection print (df1.columns.intersection(df2.columns))

ylangylang · Accepted Answer · 2016-07-05 14:12:37Z

0

Actually the solution is in an obscure corner of this page. Here's the code to use:

pd.concat([df1,df2],join_axes=[df1.columns])

answered Jul 5, 2016 at 14:12

ylangylang

3,53411 gold badges33 silver badges35 bronze badges

Collectives™ on Stack Overflow

Pandas append dataframe without creating new columns

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related