Error in regex formulation for web scraping in python

Question

I am trying to scrape some information from a website. I require 8 fields of information, I have got it for 5 fields, but 3 fields are always coming empty. I think there is some mistake with my regular expression formulation. I am doing it in python and I don't have to use BS. Here are the HTML fileds I need to scrape. This is example of one of the webpage.

enter code here

<td><span class="facultyName">John Matthew Falletta, MD</span>

<span class="primaryTitle">Professor of Pediatrics</span>

<span class="secondaryTitle">Professor in the School of Nursing</span>

<td><span class="label">Department:</span>
        &nbsp;&nbsp;
    </td><td>Pediatrics</td>

<td><span class="label">Division:</span>
        &nbsp;&nbsp;
    </td><td>Hematology/Oncology</td>

<td><span class="label">Address:</span></td><td>Box 2991<br>DUMC<br>Durham, NC &nbsp;27710   </td>

<td><span class="label">Phone:</span></td><td>
       (919)
       668-5111<br>

<td><span class="label">FAX:</span></td><td>                
        (919)
        688-5125</td>

Here is my code containing respective regular expressions for each type of tag:

enter code here

patFinderFullname = re.compile('<span class="facultyName">(.*)</span>')
patFinderPTitle = re.compile('<span class="primaryTitle">(.*)</span>')
patFinderSTitle = re.compile('<span class="secondaryTitle">(.*)</span>')
patFinderDepartment = re.compile('<span class="label">Department:</span>\s+&nbsp;&nbsp;\s+</td><td>(.*)</td>')
patFinderDivision = re.compile('<span class="label">Division:</span>\s+&nbsp;&nbsp;\s+</td><td>(.*)')

patFinderAddress = re.compile(' <span class="label">Address:</span>\s+(.*)\s+</td>')
patFinderPhone = re.compile('<span class="label">Phone:</span></td><td>\s*(.*?)\s*<br>')
patFinderFax = re.compile('<td><span class="label">FAX:</span>\s+</td><td>\s+(.*)</td>')

First five field results are coming correct, but the last three fields for Address, Phone and Fax are returning always empty. Can anyone point out what I am missing? Or what is wrong with the regular expressions for the last three fields. I have posted an earlier [1][question], but these problems arrived later to that, so I am asking it in a different question.

[1] : How to scrape html tags spread over multiple lines in python?

You are missing a closing bracket on your Division re.compile(). In fact the end of that expression doesn't look right at all, should you not have a closing </td>)' — Pep_8_Guardiola
– Pep_8_Guardiola, Commented Feb 15, 2013 at 7:30
Sorry, If I missed in DIvision, it must be a typo. I am having probles in the last three fields i.e Address, phone and fax. — Steve
– Steve, Commented Feb 15, 2013 at 7:31
I am learning web scraping, and I was familiar to python earlier, and it also has BS, so I thought it would be a good idea, to do it in python. And I have already got most of the result correct, so I would like to continue with this. — Steve
– Steve, Commented Feb 15, 2013 at 7:40
You should really invest in learning a parsing library, because every time your task changes your expressions become more or less useless. — root
– root, Commented Feb 15, 2013 at 7:45

Jason White · Accepted Answer · 2013-02-15 08:20:15Z

1

patFinderAddress = re.compile('<td><span class="label">Address:</span></td>.*?</td>'
patFinderPhone  = re.compile('<td><span class="label">Phone:</span>\s*</td><td>\s*^\s*.*\s*^\s*.*<br>',re.M)
patFinderFax = re.compile('<td><span class="label">FAX:</span>\s*</td><td>\s*^\s*.*\s*^\s*.*</td>',re.M)

Here's the some regexs that work with your data. The last two weren't working as the data spanned multiple lines. The first didn't work because it was wrong.

But, for html parsing, use an html parser as it's far more robust and gives you the data you want rather than this eyesore of html strings.

answered Feb 15, 2013 at 8:20

Jason White

6864 gold badges11 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Steve Over a year ago

I will learn how to learn how to use BS. I tried the above regex,they are displaying the whole data including all the tags.

Jason White Over a year ago

So are your regexs! If you want only the data, use a parser. No way in hell I'm crafting regexs to do a parsers job.

Steve Over a year ago

I am new to this, so I didn't know what's easy and what's not. Okay, I will learn and use BS. Thanks for your time.

Collectives™ on Stack Overflow

Error in regex formulation for web scraping in python

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related