convert string in utf-8 format to unicode : Python

Question

I have a text file with lines like:

str = '0|Crazy Taxi\xe2\x84\xa2 City Rush^Truck Racing Super Gear^Candy Crush Soda Saga^Car Parking^BMX Kid^Hill Climb Racing^UNLimited Kareena Kapoor^3D Car Parking^Find My Android Phone!^Christmas Trains^Top Free Games^Telegram^Door Screen Lock^Adventure of Ted 2 - Free^Sonic Jump^'

I want to to remove "\xe2\x84\xa2" which i am able to do using the following code line:

print unicode(str,errors="ignore")

output = '0|Crazy Taxi City Rush^Truck Racing Super Gear^Candy Crush Soda Saga^Car Parking^BMX Kid^Hill Climb Racing^UNLimited Kareena Kapoor^3D Car Parking^Find My Android Phone!^Christmas Trains^Top Free Games^Telegram^Door Screen Lock^Adventure of Ted 2 - Free^Sonic Jump^'

But when I am running the same logic on complete files using below mentioned code:

with open('train_data_dump.txt', mode='r') as document:
    for line in document:
        print unicode(line,errors='ignore')

It is printing the line as it was before.

Feel free to ask If I am not clear enough in asking question and please help.

No Sir, It has text written after crwaling web pages seperated by "^". — Pappu Jha
– Pappu Jha, Commented Sep 3, 2015 at 11:58

pacholik · Accepted Answer · 2015-09-03 11:24:32Z

3

When you assign a variable from a file it is as if you assigned a raw string - the backslash is considered a normal letter. You need to decode the escaped chars at first.

unicode(i.decode("string_escape"), errors="ignore")

Python Specific Encodings

edited Sep 3, 2015 at 11:24

answered Sep 3, 2015 at 11:13

pacholik

9,0529 gold badges48 silver badges59 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ahasanul Haque Over a year ago

will you please explain how it works internally and why it differs? Even i am trying to find a solution of this.

Pappu Jha Over a year ago

Thank you Sir, It has worked for me. I was also thinking alike and was using unicode(i.strip(), errors="ignore") at my end. But that was not working. Most probably because strip removes white spaces not the backslash.

Collectives™ on Stack Overflow

convert string in utf-8 format to unicode : Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related