Python Regex not catching pattern

Question

i am basically scraping data from a particular page. I have this code:

regex = '<ul class="w462">(.*?)</ul>'

opener.open(baseurl)
urllib2.install_opener(opener)

... rest of code omitted ...

requestData = urllib2.urlopen(request)
htmlText = requestData.read()

pattern = re.compile(regex)
movies = re.findall(pattern, htmlText)

# Lines below will always returns empty.
if not movies:
    print "List is empty. Printing source instead...", "\n\n"
    print htmlText
else:
    print movies

content of htmlText:

<ul class="w462">

... bunch of <li>s (the content i want to retrieve).

</ul>

htmlText contains the correct source (i tried to ctrl+F it and i can verify that it contains the desired ul element. It just that my regex unable to get the desired content.

I have tried to use this instead:

movies = re.findall(r'<ul class="w462">(.*?)</ul>', htmlText)

Does anyone know what went wrong?

Anyways, your data you seem to find by doing control+F could have been created by some JavaScript, which I don't think regex can catch. (Don't quote me, I could be completely wrong). Consider selenium. I've never used it, but I think it's the right tool — TerryA
– TerryA, Commented Sep 20, 2013 at 4:17

Tim Peters · Accepted Answer · 2013-09-20 04:18:02Z

2

By default, . in a regexp matches any character except for a newline. So your regexp can't match anything that spans more than one line (that contains at least one newline).

Change the compilation line to:

pattern = re.compile(regex, re.DOTALL)

to change the meaning of .. With re.DOTALL, . will match any character (including newline).

answered Sep 20, 2013 at 4:18

Tim Peters

71.4k14 gold badges133 silver badges140 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Tim Peters Over a year ago

Ya, for fancy parsing you definitely want a real HTML parsing module, but for simple tasks like this regexps are fine. Don't heed the haters - LOL ;-)

user2618501 Over a year ago

Regular expressions are never appropriate in the context of parsing markup languages. NEVER.

Tim Peters Over a year ago

Heh - @username55 was stuck, and is now unstuck. This was a Python question: "practicality beats purity" ;-)

TerryA Over a year ago

@user2618501 Yes, they can be appropriate at sometimes. If you're dealing with limited HTML, it's fine. Stop being so pedantic over it :p

Collectives™ on Stack Overflow

Python Regex not catching pattern

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related