Pandas dataframe append function adding empty columns with NaN values

Question

I have the following piece of code:

graphdata = pd.DataFrame(columns=['MaturityGroup','SumOfNetAssetPercent'] )
print(graphdata.dtypes)
graphdata["SumOfNetAssetPercent"] = graphdata["SumOfNetAssetPercent"].astype(float)
print(graphdata.dtypes)
graphdata.info()
graphdata = graphdata.append([['Remainder', filterdata[FilterRemainder]['NetAssetPercent'].sum()]
                          , ['<1 Yr', filterdata[Filter0to1Year]['NetAssetPercent'].sum()]
                          , ['1 to <3 Yrs', filterdata[Filter1to3Year]['NetAssetPercent'].sum()]
                          , ['3 to <5 Yrs', filterdata[Filter3to5Year]['NetAssetPercent'].sum()]
                          , ['5 Yrs & Above', filterdata[Filter5YrNAbove]['NetAssetPercent'].sum()]]
                            , ignore_index=True)

print(graphdata)

This gives me the following output:

MaturityGroup           object
SumOfNetAssetPercent    object
dtype: object
MaturityGroup            object
SumOfNetAssetPercent    float64
dtype: object
<class 'pandas.core.frame.DataFrame'>
Index: 0 entries
Data columns (total 2 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   MaturityGroup         0 non-null      object 
 1   SumOfNetAssetPercent  0 non-null      float64
dtypes: float64(1), object(1)
memory usage: 0.0+ bytes
  MaturityGroup  SumOfNetAssetPercent              0          1
0           NaN                   NaN      Remainder   5.370009
1           NaN                   NaN          <1 Yr   7.295347
2           NaN                   NaN    1 to <3 Yrs  39.340446
3           NaN                   NaN    3 to <5 Yrs  12.524880
4           NaN                   NaN  5 Yrs & Above  35.472455

My question is, why 'append' adds two additional empty NaN column to my empty dataframe?

@MayankPorwal The problem isn't with filter data, coz when im using pd.DataFrame instead of graphdata.append it works fine with data being saved to the 2 defined columns with no additional ones being created.. However i need to run the code in a loop, hence wanted to keep appending to existing dataframe. — 7bluephoenix
– 7bluephoenix, Commented May 13, 2020 at 13:37
Another thing to mention is if the graphdata dataframe already has any existing data then graphdata.append works perfectly. It is only with the first time when it is empty the problem is arising. — 7bluephoenix
– 7bluephoenix, Commented May 13, 2020 at 13:56

run-out · Accepted Answer · 2020-05-14 01:36:31Z

The two rows are getting added because you are appending rows that don't have the same column names as the original dataframe.

For example:

df = pd.DataFrame(columns=["a", "b"])
print(df)
Empty DataFrame
Columns: [a, b]
Index: []

Create a datarame to append but without column names.

df_other = pd.DataFrame([["remainder", 5.3], ["more", 2.2], ["some", 66]])
print(df_other)
          0     1
0  remainder   5.3
1       more   2.2
2       some  66.0

Now append them...

df = df.append(df_other)
print(df)
     a    b          0     1
0  NaN  NaN  remainder   5.3
1  NaN  NaN       more   2.2
2  NaN  NaN       some  66.0

Now change the appending dataframe to include columns.

df_other = pd.DataFrame(
    [["remainder", 5.3], ["more", 2.2], ["some", 66]], columns=["a", "b"]
)
print(df_other)
          a     b
0  remainder   5.3
1       more   2.2
2       some  66.0

And the result is.

df = df.append(df_other)
print(df)
           a     b
0  remainder   5.3
1       more   2.2
2       some  66.0

Collectives™ on Stack Overflow

Pandas dataframe append function adding empty columns with NaN values

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related