Converting HTML list to nested Python list

Question

If I have a nested html (unordered) list that looks like this:

<ul>
    <li><a href="Page1_Level1.html">Page1_Level1</a> 
    <ul>
        <li><a href="Page1_Level2.html">Page1_Level2</a> 
            <ul>
                <li><a href="Page1_Level3.html">Page1_Level3</a></li>
            </ul>
            <ul>
                <li><a href="Page2_Level3.html">Page2_Level3</a></li>
            </ul>
            <ul>
                <li><a href="Page3_Level3.html">Page3_Level3</a></li>
            </ul>
        </li>
    </ul>
    </li>
    <li><a href="Page2_Level1.html">Page2_Level1</a> 
    <ul>
        <li><a href="Page2_Level2.html">Page2_Level2</a></li>
    </ul>
    </li>
</ul>

How do I form a nested list out of it in Python? For example:

["Page1_Level1.html", ["Page1_Level2.html", ["Page1_Leve3.html", "Page2_Level3.html", "Page3_Level3.html"]], "Page2_Level1.html", ["Page2_Level2.html"]]

I presume libraries like Beautiful Soup and HTML Parser have facilities to do this, but I haven't been able it figure it out. Thanks for any help / pointers!

Community · Accepted Answer · 2017-05-23 11:51:39Z

4

You can take a recursive approach:

from pprint import pprint
from bs4 import BeautifulSoup

text = """your html goes here"""

def find_li(element):
    return [{li.a['href']: find_li(li)}
            for ul in element('ul', recursive=False)
            for li in ul('li', recursive=False)]


soup = BeautifulSoup(text, 'html.parser')
data = find_li(soup)
pprint(data)

Prints:

[{u'Page1_Level1.html': [{u'Page1_Level2.html': [{u'Page1_Level3.html': []},
                                                 {u'Page2_Level3.html': []},
                                                 {u'Page3_Level3.html': []}]}]},
 {u'Page2_Level1.html': [{u'Page2_Level2.html': []}]}]

FYI, here is why I had to use html.parser here:

Don't put html, head and body tags automatically, beautifulsoup

edited May 23, 2017 at 11:51

CommunityBot

11 silver badge

answered Jun 14, 2014 at 3:45

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Anthony Kong · Accepted Answer · 2014-06-14 03:32:33Z

1

It is an overview of a possible solution

# variable 'markup' contains the html string
from bs4 import BeautifulSoup
soup = BeautifulSoup(markup)
for a in soup.descendants:
   # construct a nested list when going thru the descendants
   print id(a), id(a.parent) if a.parent else None, a

answered Jun 14, 2014 at 3:32

Anthony Kong

41.4k52 gold badges192 silver badges325 bronze badges

Collectives™ on Stack Overflow

Converting HTML list to nested Python list

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related