Unicode classes in regex with Python2

Question

Is it possible?

This code works in Python3:

In [1]: import re

In [2]: re.split(r'\W+', 'Les Misérables')
Out[2]: ['Les', 'Misérables']

But it does not work in Python2:

In [1]: import re

In [2]: re.split(r'\W+', u'Les Misérables')
Out[2]: [u'Les', u'Mis', u'rables']

This does not work either (tested on Linux with es_ES.UTF-8 locale):

In [1]: import locale

In [2]: locale.setlocale(locale.LC_ALL, 'es_ES.UTF-8')
Out[2]: 'es_ES.UTF-8'

In [3]: import re

In [4]: re.split(ur'\W+', u'Les Misérables', re.U|re.L)
Out[4]: [u'Les', u'Mis', u'rables']

Is there any way to get regex working with Unicode in Python2?

Note: The question is about getting Unicode-aware matches. I know I can rewrite the above regex to separate words using only ASCII classes.

georg · Accepted Answer · 2013-05-26 15:36:29Z

4

Your mistake is that you're adding flags on the wrong place (flags should be the 4th param).

>>> import re
>>> re.split(r'(?u)\W+', u'Les Misérables')
[u'Les', u'Mis\xe9rables']
>>> re.split(ur'\W+', u'Les Misérables', 0, re.U)
[u'Les', u'Mis\xe9rables']

To avoid these issues I'd recommend using inline flags (like (?u) above).

answered May 26, 2013 at 15:36

georg

216k57 gold badges324 silver badges401 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alicia Over a year ago

Right, I was mispositioning the Unicode flag. Didn't know the (?u) expression. Seems useful.

jfs Over a year ago

you could use flags=re.U keyword argument.

Collectives™ on Stack Overflow

Unicode classes in regex with Python2

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related