Extracting HTML data fields with Python

Question

Please forgive me for my lack of knowledge, but given HTML in the following format, what is the best way to extract the individual data fields? Please keep in mind that more often than not some, or all, of them will be NULL in which case we'll keep them at NULL.

<div class="profile-section" id="a-bit-more-about">
                            <dl>
            <dt>Name:</dt>
            <dd><span class="given-name">Clem</span> <span class="family-name">Kadiddlehopper</span></dd>
        </dl>
        <!-- <span class="RealName">/ <span class="fn n"><span class="given-name">Clem</span> <span class="family-name">Kadiddlehopper</span></span></span> -->
                        <dl>
        <dt>Joined:</dt>
        <dd>September 1910</dd>
    </dl>
    <div class="sep"></div>
    <dl>
        <dt>Hometown:</dt>
        <dd>Quiet Rest Maximum Security Twilight Home</dd>
    </dl>
    <dl>
        <dt>Currently:</dt>
        <dd><span class="adr"><span class="locality">They won't tell me</span>, <span class="country-name">Zimbobwe</span></span></dd>
    </dl>
    <div class="sep"></div>

WeaselFox · Accepted Answer · 2013-03-21 06:48:28Z

3

You want an HTML parser. I recommend beautiful soup or lxml.

answered Mar 21, 2013 at 6:48

WeaselFox

7,3989 gold badges53 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Deina Underhill Over a year ago

Yes, I have both, I just haven't sorted out how to use them properly. :(

zhangyangyu · Accepted Answer · 2013-03-21 09:43:25Z

2

Use third-party modules beautiful soup, lxml or built-in module html.parser. For example:

from bs4 import BeautifulSoup
soup = BeautifulSoup('<html><body><a>bbb</a></body></html')
soup.find('a')

Or if like, you can use regex for small target.

answered Mar 21, 2013 at 9:43

zhangyangyu

8,6103 gold badges35 silver badges43 bronze badges

1 Comment

Deina Underhill Over a year ago

I finally got regex to do what I needed; at least for the most part. Thank you! :)

Collectives™ on Stack Overflow

Extracting HTML data fields with Python

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related