Cannot replace special characters in a Python pandas dataframe

Question

I'm working with Python 3.5 in Windows. I have a dataframe where a 'titles' str type column contains titles of headlines, some of which have special characters such as â,€,˜.

I am trying to replace these with a space '' using pandas.replace. I have tried various iterations and nothing works. I am able to replace regular characters, but these special characters just don't seem to work.

The code runs without error, but the replacement simply does not occur, and instead the original title is returned. Below is what I have tried already. Any advice would be much appreciated.

df['clean_title'] = df['titles'].replace('€','',regex=True)
df['clean_titles'] = df['titles'].replace('€','')
df['clean_titles'] = df['titles'].str.replace('€','')

def clean_text(row):
   return re.sub('€','',str(row))
   return str(row).replace('€','')
df['clean_title'] = df['titles'].apply(clean_text)

I can't reproduce, your third example works for me. Can you post a sample of your dataframe? — user3483203
– user3483203, Commented Jun 13, 2018 at 21:46
let's say on title is 'â€˜BetterHashâ€™: Bitcoin Core Dev. Proposes New Protocols to Decentralize Bitcoin Mining' — user2238328
– user2238328, Commented Jun 13, 2018 at 21:54
This looks more like an encoding error. Therefore you better solve this at the encoding level. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Jun 13, 2018 at 21:57
I have tried various encodings such as utf-8 while importing the csv file using read_csv. Nothing works :( — user2238328
– user2238328, Commented Jun 13, 2018 at 22:05

sudonym · Accepted Answer · 2018-06-14 14:34:09Z

4

We can only assume that you refer to non-ASCI as 'special' characters.

To remove all non-ASCI characters in a pandas dataframe column, do the following:

df['clean_titles'] = df['titles'].str.replace(r'[^\x00-\x7f]', '')

Note that this is a scalable solution as it works for any non-ASCI char.

edited Jun 14, 2018 at 14:34

answered Jun 13, 2018 at 23:01

sudonym

4,0384 gold badges40 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

halfelf · Accepted Answer · 2022-11-30 07:30:32Z

0

How to remove escape sequence character in dataframe

Data.

product,rating pest,<br> test mouse,/
mousetest

Solution: scala Code

 val finaldf = df.withColumn("rating", regexp_replace(col("rating"), "\\\\", "/")).show()

edited Nov 30, 2022 at 7:30

halfelf

10.2k13 gold badges60 silver badges69 bronze badges

answered Nov 24, 2022 at 12:30

Tarkeshwar Prasad

294 bronze badges

Collectives™ on Stack Overflow

Cannot replace special characters in a Python pandas dataframe

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related