How to remove blank and null values from a list that passed into dataframe in python

Question

I have dataframe column like below.

df['lane']
AZ
NL

NaN
BL
AZ

My code

unique_lane = df['lane'].unique()
unique_lane = pd.DataFrame( list(zip(unique_lane)), columns =['unique_lane'])
t = ', '.join(unique_lane['unique_lane'].astype(str))

While I am passing unique list values blank('') or Null values should be removed from the list. The list t created should contain not blank or not Null values.

bigdata_null_zones = bigdata_null_zones[~bigdata_null_zones["lane"].isin([t])]

How can this be done in python?

jezrael · Accepted Answer · 2020-05-31 08:19:19Z

1

Sample data for test DataFrame from question:

df = pd.DataFrame({'lane':['AZ','NL','', np.nan, 'BL','AZ']})

Test for pass only misisng values or empty strings:

df = pd.DataFrame({'lane':['', np.nan]})
print (df)
  lane
0     
1  NaN

bigdata_null_zones = pd.DataFrame({'lane':['AZ','NL','AB', 'BL','AZ']})
print (bigdata_null_zones)
  lane
0   AZ
1   NL
2   AB
3   BL
4   AZ

After remove it get empty Series:

t = df['lane'].replace('',np.nan).dropna()
print (t)
Series([], Name: lane, dtype: float64)

So if pass get same values, because nothing filtered:

bigdata_null_zones[bigdata_null_zones["lane"].isin(t)]
print (bigdata_null_zones)
  lane
0   AZ
1   NL
2   AB
3   BL
4   AZ

If same DataFrame:

df = pd.DataFrame({'lane':['AZ','NL','', np.nan, 'BL','AZ'],
                   'col':range(6)})

print (df)
  lane  col
0   AZ    0
1   NL    1
2         2
3  NaN    3
4   BL    4
5   AZ    5

df1 = df.assign(lane= df['lane'].replace('',np.nan)).dropna(subset=['lane'])
print (df1)
  lane  col
0   AZ    0
1   NL    1
4   BL    4
5   AZ    5

edited May 31, 2020 at 8:19

answered May 31, 2020 at 8:07

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

sneha nair Over a year ago

bigdata_null_zones =bigdata_null_zones[~bigdata_null_zones["lane"].isin([t])]. Will this line of code break if I have no values inside t beacuse df[lane] had only blank or null values

jezrael Over a year ago

@s - Unfortunately not understand.

sneha nair Over a year ago

My orginal daframe df['lane'] has only null and blank values only. then I apply your code remove those using code t = df['lane'].replace('',np.nan).dropna(). After this I pass this inside the second dataframe. I am asking in such a case will the code break

jezrael Over a year ago

@snehanair - I can test, but I think not.

sneha nair Over a year ago

@jazrael, In that case t doesnt have any value so will the isin condition work Thats my question

|

Collectives™ on Stack Overflow

How to remove blank and null values from a list that passed into dataframe in python

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related