python unicode regex

Question

I'd like to replace the below regex with a unicode-friendly version that will catch things like http://➡.ws and other non-ascii IRIs. The purpose is to grab these out of users' text and encode and html-ize them into real links.

Python provides a re.UNICODE flag which changes the meaning of \w, but that's not super helpful in this case (that I can see) because it is defined as "alphanumeric characters and underscore" and not all of my below character classes include underscore.

domain_regex = re.compile(r"""
    (
        (https?://)
        (
            [0-9a-zA-Z]
            [0-9a-zA-Z_-]*
            \.
        )+
        [a-zA-Z]{2,4}
    )
    | # begins with an http scheme followed by a domain, or
    (
        (?<!   # negative look-behind
            [0-9a-zA-Z.@-]
        )
        (
            [0-9a-zA-Z]
            [0-9a-zA-Z_-]*
            \.
        )+
        # top-level domain names
        com|ca|net|org|edu|gov|biz|info|mobi|name|
        us|uk|fr|au|be|ch|de|es|eu|it|tv|cn|jp
    )
""", re.VERBOSE)

More non-ascii domains:

Bücher.ch -- (swiss-german "books". Currently down.)
http://παράδειγμα.δοκιμή
http://실례.테스트

This is a possible duplicate of stackoverflow.com/questions/1832893/… Let us know if you need any more help — buckley
– buckley, Commented Mar 22, 2012 at 22:08

Mark Byers · Accepted Answer · 2012-03-22 22:05:43Z

5

If you want to write "\w except underscore" you can do so using a negated character class:

[^\W_]

answered Mar 22, 2012 at 22:05

Mark Byers

844k202 gold badges1.6k silver badges1.5k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 10:29:21Z

0

As buckley noted, "Python regex matching Unicode properties" presents some alternatives to use regex + unicode in Python. If what you want is just alphanumeric, alphanumeric + underscore or letters only, maybe it's easier to stick with Mark Byers suggestion ([^\W_], \w and [^\W\d_] respectively, with re.UNICODE active; Edit: got the order wrong...).

Otherwise, look up which character classes are valid as a IRI part and either use a regex engine that supports unicode character classes, or - if you need a pure python solution - I'd suggest the code I provided in an answer to that question (or a similar solution).

edited May 23, 2017 at 10:29

CommunityBot

11 silver badge

answered Mar 22, 2012 at 22:14

mgibsonbr

22.1k7 gold badges73 silver badges115 bronze badges

Collectives™ on Stack Overflow

python unicode regex

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related