Encoding python utf-8 splitting

Question

I've got a variable:

age_expectations = dictionary['looking_for']['age']
print type(age_expectations), age_expectations

The output is:

<type 'unicode'> 22‑35

When I'm trying to split it with the dash I've got the following problem:

res = age_expectations.split('-')
print res

And the output look like:

[u'22\u201135']

Instead of:

["22", "35"]

What is the problem? I've tried many encoding and decoding but not really sure to understand how it's work. Does the problem come from the split?

Yeah I'm reading from a file, which contain data crawled on a website — mel
– mel, Commented Dec 6, 2016 at 13:04

Praveen · Accepted Answer · 2016-12-06 12:59:03Z

2

Use unicode to split the unicode like,

>>> u_code = u'\u0032\u0032\u2011\u0033\u0035'
>>> print u_code
22‑35
>>> u_code.split('-')
[u'22\u201135']
>>> u_code.split(u'\u2011')
[u'22', u'35']
>>>

answered Dec 6, 2016 at 12:59

Praveen

9,4534 gold badges38 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

bruno desthuilliers · Accepted Answer · 2016-12-06 12:59:42Z

1

As you can see from your code, the hyphen in your age_expectations variable is the unicode U+2011 character, not the standard "-" hyphen. You would have seen it from the start if you had printed the variable's representation instead:

>>> uu = u"22\u201135"
>>> print uu
22‑35
>>> print repr(uu)
u'22\u201135'
>>>

So you need to either replace the u"\u2011" character with a simple hyphen (if you can have any of them in your data) or just simply split the string on u"\u2011" (if you're sure you'll always get this as delimiter).

answered Dec 6, 2016 at 12:59

bruno desthuilliers

78.3k6 gold badges103 silver badges129 bronze badges

1 Comment

Matthias Over a year ago

If it's a non-breaking hyphen it might even be replaced by an empty string, but that depends on the context.

Collectives™ on Stack Overflow

Encoding python utf-8 splitting

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related