Remove strings based on condition pandas dataframe column

Question

I have the following data:

df = pd.DataFrame({ 'Column_A': [1,2,3,4],
                'Column_B': [["X1", "X2", "Y1"],
                            ["X3", "Y2"],
                            ["X4", "X5"],
                            ["X5", "Y3", "Y4"]],})

   Column_A      Column_B
0         1  [X1, X2, Y1]
1         2      [X3, Y2]
2         3      [X4, X5]
3         4  [X5, Y3, Y4]

I wish to remove all strings starting with Y in the second column. Desired output:

   Column_A  Column_B
0         1  [X1, X2]
1         2      [X3]
2         3  [X4, X5]
3         4      [X5]

jezrael · Accepted Answer · 2018-11-23 11:15:33Z

Use nested list comprehension with filtering with startswith:

df['Column_B'] = [[y for y in x if not y.startswith('Y')] for x in df['Column_B']]

apply alternative:

df['Column_B'] = df['Column_B'].apply(lambda x: [y for y in x if not y.startswith('Y')])

Or use filter:

df['Column_B'] = [list(filter(lambda y: not y.startswith('Y'), x)) for x in df['Column_B']]

print (df)
   Column_A  Column_B
0         1  [X1, X2]
1         2      [X3]
2         3  [X4, X5]
3         4      [X5]

Performance:

Depends of number of rows, number of values in lists and number of matched values:

#[40000 rows x 2 columns]
df = pd.concat([df] * 10000, ignore_index=True)
#print (df)


In [142]: %timeit df['Column_B'] = [[y for y in x if not y.startswith('Y')] for x in df['Column_B']]
23.7 ms ± 410 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [143]: %timeit df['Column_B'] = [list(filter(lambda y: not y.startswith('Y'), x)) for x in df['Column_B']]
36.5 ms ± 204 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [144]: %timeit df['Column_B'] = df['Column_B'].apply(lambda x: [y for y in x if not y.startswith('Y')])
30.4 ms ± 1.86 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Collectives™ on Stack Overflow

Remove strings based on condition pandas dataframe column

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related