How to replace substrings in strings in pandas dataframe

Question

I have a dataframe, and a list of strings that I want to remove from a column in that dataframe. But when I use the replace function those characters remain. Can someone please explain why this is the case?

bad_chars = ['?', '!', ',', ';', "'", '|', '-', '--', '(', ')', 
             '[', ']', '{', '}', ':', '&', '\n']

and to replace:

df2['page'] = df2['page'].replace(bad_chars, '')

when i print out df2:

for index, row in df2.iterrows():
    print( row['project'] + '\t' + '(' + row['page'] + ',' + str(row['viewCount']) + ')' + '\n'  )

en (The_Voice_(U.S._season_14),613)

jpp · Accepted Answer · 2018-04-14 17:30:59Z

3

One way is to escape your characters using re, then use pd.Series.str.replace.

import pandas as pd
import re

bad_chars = ['?', '!', ',', ';', "'", '|', '-', '--', '(', ')', 
             '[', ']', '{', '}', ':', '&', '\n']

df = pd.DataFrame({'page': ['hello?', 'problems|here', 'nothingwronghere', 'nobrackets[]']})

df['page'] = df['page'].str.replace('|'.join([re.escape(s) for s in bad_chars]), '')

print(df)

#                page
# 0             hello
# 1      problemshere
# 2  nothingwronghere
# 3        nobrackets

answered Apr 14, 2018 at 17:30

jpp

166k37 gold badges301 silver badges362 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rob Over a year ago

Thank you ver much jpp, that does it perfectly

mcard · Accepted Answer · 2018-04-14 17:42:36Z

0

Use .str.replace, and pass your strings as a single, pipeline-separated string. You can use re.escape() in order to escape regex characters from that string, as suggested by @jpp. I tweak his suggestion a bit by avoiding iteration:

import re 
df2['page'] = df2['page'].str.replace(re.escape('|'.join(bad_chars)), '')

edited Apr 14, 2018 at 17:42

answered Apr 14, 2018 at 17:21

mcard

6271 gold badge9 silver badges16 bronze badges

7 Comments

Rob Over a year ago

Cheers mcard, when I do that, I get the error : TypeError: unhashable type: 'list' If I replace the list variable with a literal example of an element in that list it works though. Is there a way to replace more than one string at the same time?

jpp Over a year ago

This does not work in the general case, e.g. what about regex characters such as | in your bad character list?

mcard Over a year ago

THey need to be escaped with a ´\´

jpp Over a year ago

So is the only way to manually go through the list of bad characters one by one to check for escapes? I'm sure there's a better way...

Rob Over a year ago

Thanks folks for talking a look, I'm accepting jpp's answer as it does exactly what I'm looking for in a generic way.

|

Collectives™ on Stack Overflow

How to replace substrings in strings in pandas dataframe

2 Answers 2

1 Comment

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related