replace a part of pandas data.frame by given array

Question

I have a pandas.DataFrame named fake_num:

  fake_num=pd.DataFrame([[1,2,3,4,np.nan,np.nan,np.nan],[1.1,1.2,1.3,1.4,1.6,1.8,2.5]]).T
  fake_num
    Out[4]: 
         0    1
    0  1.0  1.1
    1  2.0  1.2
    2  3.0  1.3
    3  4.0  1.4
    4  NaN  1.6
    5  NaN  1.8
    6  NaN  2.5

I am trying to fill the NaN values using a linear regression:

    from sklearn.linear_model import LinearRegression
    fdrop=fake_num.dropna(axis=0,how='any')
    lr=LinearRegression()
    lr.fit(np.array(fdrop.iloc[:,1]).reshape(-1, 1),np.array(fdrop.iloc[:,0]))
    lr.predict(np.array(fake_num[np.isnan(fake_num[0])][1]).reshape(-1, 1))
Out[5]: array([ 6.,  8., 15.])

The part I want to replace is fake_num[np.isnan(fake_num[0])][0]so what I want is:

    Out[6]: 
     0    1
0  1.0  1.1
1  2.0  1.2
2  3.0  1.3
3  4.0  1.4
4  6.0  1.6
5  8.0  1.8
6  5.0  2.5

While I tried:

fake_num[np.isnan(fake_num[0])][0]=lr.predict(np.array(fake_num[np.isnan(fake_num.iloc[:,0])].iloc[:,1]).reshape(-1, 1))
fake_num
__main__:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
Out[11]: 
     0    1
0  1.0  1.1
1  2.0  1.2
2  3.0  1.3
3  4.0  1.4
4  NaN  1.6
5  NaN  1.8
6  NaN  2.5

And

    fake_num[np.isnan(fake_num.loc[:,0])].loc[:,0]=lr.predict(np.array(fake_num[np.isnan(fake_num.iloc[:,0])].iloc[:,1]).reshape(-1, 1))
fake_num
D:\Users\shan xu\Anaconda3\lib\site-packages\pandas\core\indexing.py:630: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item_labels[indexer[info_axis]]] = value
Out[12]: 
     0    1
0  1.0  1.1
1  2.0  1.2
2  3.0  1.3
3  4.0  1.4
4  NaN  1.6
5  NaN  1.8

And

    fake_num[np.isnan(fake_num.iloc[:,0])].iloc[:,0]=lr.predict(np.array(fake_num[np.isnan(fake_num.iloc[:,0])].iloc[:,1]).reshape(-1, 1))
fake_num
D:\Users\shan xu\Anaconda3\lib\site-packages\pandas\core\indexing.py:630: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item_labels[indexer[info_axis]]] = value
Out[12]: 
     0    1
0  1.0  1.1
1  2.0  1.2
2  3.0  1.3
3  4.0  1.4
4  NaN  1.6
5  NaN  1.8

what should I do to replace part of the dataframe with some values give its posiition. BTW, Since I need more details elaboration, any good tools, take fill na values with simple predictions models using other all non-na rows and other columns as input? Something like missforest in R.

Possible duplicate of How to deal with SettingWithCopyWarning in Pandas? — Space Impact
– Space Impact, Commented Sep 7, 2018 at 6:11

cs95 · Accepted Answer · 2018-09-07 06:12:08Z

1

Just call fit, and then use loc to assign back.

v = fake_num.dropna()
lr.fit(v[[1]], v[[0]])

m = fake_num[0].isna()
fake_num.loc[m, [0]] = lr.predict(fake_num.loc[m, [1]])

fake_num
      0    1
0   1.0  1.1
1   2.0  1.2
2   3.0  1.3
3   4.0  1.4
4   6.0  1.6
5   8.0  1.8
6  15.0  2.5

answered Sep 7, 2018 at 6:12

cs95

406k106 gold badges744 silver badges797 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

replace a part of pandas data.frame by given array

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related