Parsing html with Python

Question

I'm using BeautifulSoup to extract some data from a search result from this website http://www.cpso.on.ca/docsearch/default.aspx

Here's a sample of the HTML code that's been .prettify()

<tr>
 <td>
  <a class="doctor" href="details.aspx?view=1&amp;id= 72374">
   Smith, Jane
  </a>
  (#72374)
 </td>
 <td>
  Suite 042
  <br />
  21 Jump St
  <br />
  Toronto&nbsp;ON&nbsp;&nbsp;M4C 5T2
  <br />
  Phone:&nbsp;(555) 555-5555
  <br />
  Fax:&nbsp;(555) 555-555
 </td>
 <td align="center">
 </td>
</tr>

Essentially every <tr> block has 3 <td> blocks.

I want the output to be

Smith, Jane Suite 042 21 Jump St Toronto ON M4C 5T2

I also have to separate entries by a new line.

I'm having problem writing the address which is stored in the 2nd <td> block.

I'm also writing this to a file.

Here's what I have so far... it doesn't work :p

for tr in soup.findAll('tr'):
    #td1 = tr.td
    td2 = tr.td.nextSibling.nextSibling 

    for a in tr.findAll('a'):
        target.write(a.string)
        target.write(" ")

    for i in range(len(td2.contents)):
        if i != None:
            target.write(td2.contents[i].string)
            target.write(" ")
    target.write("\n")

Your first for loop is missing a :, and the inner loop isn't indented. Is that the actual code or a posting mistake? — Jacob
– Jacob, Commented Jan 11, 2012 at 1:57
Yes. my bad. Python is something I've just picked up to do some quick html parsing. I — KylePDM
– KylePDM, Commented Jan 11, 2012 at 1:59
I'm also thinking of not even looping over td and a and just making 2 temp td values when I loop through tr. — KylePDM
– KylePDM, Commented Jan 11, 2012 at 2:01
The text "Suite 042 ..." is not contained in an <a></a>, so why do you expect your code to print it? — Francis Avila
– Francis Avila, Commented Jan 11, 2012 at 2:11

ekhumoro · Accepted Answer · 2012-01-11 04:11:47Z

1

This should do most of what you want:

import os
from BeautifulSoup import BeautifulSoup

soup = BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)

with open('output.txt', 'wb') as stream:
    for tr in soup.findAll('tr')[1:]: # [1:] skips the header
        columns = tr.findAll('td')
        line = [columns[0].a.string.strip()]
        for item in (item.strip() for item in columns[1].findAll(text=True)):
            if (item and not item.startswith('Phone:')
                and not item.startswith('Fax:')):
                line.append(item)
        stream.write(' '.join(line).encode('utf-8'))
        stream.write(os.linesep)

UPDATE

Added some code to show how to write the names and addresses to file.

Also changed the output so that names and addresses are written on one line, and phone and fax numbers are omitted.

edited Jan 11, 2012 at 4:11

answered Jan 11, 2012 at 2:46

ekhumoro

122k23 gold badges272 silver badges400 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

KylePDM Over a year ago

When i changed the print methods to .write, I get a UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0'

KylePDM Over a year ago

Still didn't work.

http://www.crummy.com/software/BeautifulSoup/documentation.html#Why can't Beautiful Soup print out the non-ASCII characters I gave it?

^ says that it could be my Python installation or something with BeautifulSoup itself. According to that, it's BeautifulSoup >>;

ekhumoro Over a year ago

@KylePDM. It's very unlikely that there's anything wrong with either Python or BeautifulSoup. What does your code for reading the html and writing the output look like?

KylePDM Over a year ago

Thanks for looking at my problem! I'll just try to remove phone/fax and put each doctor's information on oneline

KylePDM Over a year ago

If I only had the reputation to give Vote up your solution I would. For some reason nothing is being written to the output file. But I think I can tinker and fix it :D

Derek Litz · Accepted Answer · 2012-01-11 03:24:46Z

1

In [243]: soup.getText(' ').replace('&nbsp;', ' ').strip()
Out[243]: u'Smith, Jane (#72374)  Suite 042 21 Jump St Toronto ON  M4C 5T2 Phone: (555) 555-5555 Fax: (555) 555-555'

To get exactly what you want:

In [246]: address = soup.getText(' ').replace('&nbsp;', ' ').strip()
In [247]: import re
In [248]: address = re.sub(r' Phone.*$', '', address)
In [249]: address = address.replace('  ', ' ')
In [250]: address = re.sub(r' \(.*?\)', '', address)
In [251]: print address
Smith, Jane Suite 042 21 Jump St Toronto ON M4C 5T2

edited Jan 11, 2012 at 3:24

answered Jan 11, 2012 at 2:41

Derek Litz

10.9k8 gold badges45 silver badges53 bronze badges

Comments

soulcheck · Accepted Answer · 2012-01-11 03:25:19Z

1

I'd try something like this:

from BeautifulSoup import BeautifulSoup
soup = BeautifulSoup(your_html, 
               convertEntities=BeautifulSoup.HTML_ENTITIES)

for tr in soup.findAll('tr'):
   td = tr.findAll('td')

   target.write(td[0].a.string)
   target.write(' ')

   target.write(' '.join(text.strip() for text in td[1].findAll(text = True)[:-2]))) #finds all text subnodes, except 2 last ones (phone number), and joins them with ' ' separator
   target.write("\n")

edited Jan 11, 2012 at 3:25

answered Jan 11, 2012 at 2:47

soulcheck

36.9k6 gold badges95 silver badges91 bronze badges

1 Comment

KylePDM Over a year ago

This just gives me the names separated by a new line

Collectives™ on Stack Overflow

Parsing html with Python

3 Answers 3

5 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related