UTF in Python Regex

Question

I'm aware that Python 3 fixes a lot of UTF issues, I am not however able to use Python 3, I am using 2.5.1

I'm trying to regex a document but the document has UTF hyphens in it – rather than -. Python can't match these and if I put them in the regex it throws a wobbly.

How can I force Python to use a UTF string or in some way match a character such as that?

Thanks for your help

I deleted the duplicates. Not sure why my browser submitted 3 at once... — Teifion
– Teifion, Commented Dec 16, 2008 at 17:50
I find that Python had a very good (not perfect) but very good support for strings in various encoding (e.g. utf-8 (it is an encoding)) as ell as Unicode (Unicode is not an encoding) strings long before Python 3 therefore don't blame language; just ask a question if you don't know how to do ... . — jfs
– jfs, Commented Dec 16, 2008 at 18:16
I wanted to pre-empt someone telling me about Python 3 or asking if I was using it. Python 2.5 is still a wonderful language and I prefer it over PHP — Teifion
– Teifion, Commented Dec 16, 2008 at 18:32

Patrick McElhaney · Accepted Answer · 2008-12-16 19:06:28Z

7

You have to escape the character in question (–) and put a u in front of the string literal to make it a unicode string.

So, for example, this:

re.compile("–")

becomes this:

re.compile(u"\u2013")

edited Dec 16, 2008 at 19:06

answered Dec 16, 2008 at 18:01

Patrick McElhaney

59.6k41 gold badges138 silver badges170 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Teifion Over a year ago

I was putting an r before the string for raw string

rlafuente Over a year ago

You can also add 'ur' before the string so that it's raw and Unicode.

Patrick McElhaney · Accepted Answer · 2008-12-16 18:25:46Z

4

After a quick test and visit to PEP 0264: Defining Python Source Code Encodings, I see you may need to tell Python the whole file is UTF-8 encoded by adding adding a comment like this to the first line.

# encoding: utf-8

Here's the test file I created and ran on Python 2.5.1 / OS X 10.5.6

# encoding: utf-8
import re
x = re.compile("–") 
print x.search("xxx–x").start()

edited Dec 16, 2008 at 18:25

answered Dec 16, 2008 at 18:20

Patrick McElhaney

59.6k41 gold badges138 silver badges170 bronze badges

Comments

unbeknown · Accepted Answer · 2008-12-16 18:27:38Z

3

Don't use UTF-8 in a regular expression. UTF-8 is a multibyte encoding where some unicode code points are encoded by 2 or more bytes. You may match parts of your string that you didn't plan to match. Instead use unicode strings as suggested.

answered Dec 16, 2008 at 18:27

unbeknown

Collectives™ on Stack Overflow

UTF in Python Regex

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related