Process all text from nested divs inside ul tags using selenium in python

Question

I have some HTML which looks like this one:

<div class="class1">
    <div class="some multiple classes here">
        <div class="some multiple classes here">
            <ul class="other classes">
                <li>
                    <div class="random">some text1</div>
                    <div class="random1">some text2</div>
                    <div class="random2">some text3</div>
                </li>
                <li>
                    <div class="random">some text4</div>
                    <div class="random1">some text5</div>
                    <div class="random2">some text6</div>
                </li>
                <li>
                    <div class="random">some text7</div>
                    <div class="random1">some text8</div>
                    <div class="random2">some text9</div>
                </li>
                <!-- here can appear more <li></li> elements -->
            </ul>
        </div>
    </div>
</div>

The situation is a little bit complex:

first, there are only a few <li></li> tags with the same <div> classes inside but different text. More, <li></li> (with the same divs) appear as time passes.

I struggled to find a solution of retrieving all some textX elements in a single loop (not necessary if not possible) using xpath and selenium but I couldn't find a away. More, when I finish all the <li></li> tags, I'd like to just wait for others to appear and take some actions.

The pseudo-code would be the following:

for mydriver.find_element_by_xpath('xpath of ul'):
    # here get all the texts
    # process them

Obsidian · Accepted Answer · 2016-02-11 13:55:13Z

1

You need all the divs that are direct children of li tags?

divs = driver.find_elements_by_xpath('//li/div')

Or by css selectors

divs = driver.find_elements_by_css_selector('li > div')

answered Feb 11, 2016 at 13:55

Obsidian

5153 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Guy · Accepted Answer · 2016-02-11 14:06:59Z

0

You can find all the <li> tags using the <ul> and all the <div> tags using the <li> tags

divs = [] # will hold the divs
ul = driver.find_element_by_class_name('classes')
lis = ul.find_elements_by_tag_name('li') # list of the <li> tags
for li in lis:
    divs.append(li.find_elements_by_tag_name('div')) # add all the <div> tags to divs list

for div in divs:
    text = div.text

Another solution will be using contains on the "random" in the class

divs = driver.find_elements_by_css_selector('[class*="random"]')

for div in divs:
    text = div.text

edited Feb 11, 2016 at 14:06

answered Feb 11, 2016 at 13:44

Guy

51.2k10 gold badges49 silver badges96 bronze badges

7 Comments

Cajuu' Over a year ago

For the first example I get an error: AttributeError: 'NoneType' object has no attribute 'text'

Guy Over a year ago

@Alexander it should be lis = ul.find_elements_by_tag_name('li'), I updated my answer

Cajuu' Over a year ago

Well, I knew this and I modified it before but I still get the same error. Any ideas why ?

Guy Over a year ago

@Alexander maybe the list declaration is the problem, try divs = []

Cajuu' Over a year ago

That's not the problem. The problem is that I tried to print each element you wrote, but I got:

<selenium.webdriver.remote.webelement.WebElement (session="9b6579af-c027-43d6-98b4-9d14109303c0", element="{9e80dae6-dc89-4365-8306-e00aac001008}")>

|

Collectives™ on Stack Overflow

Process all text from nested divs inside ul tags using selenium in python

2 Answers 2

Comments

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related