Iterating through elements get repeating result on Selenium on Python

Question

I'm having an issue that I'm struggling to understand. I have a HTML with a ul list with a bunch of li items, and within those li items they have elements that I would like to scrape using Selenium on Python. The website makeup is as follows

<ul id="results">
    <li>
        <div class="name">John</div>
        <div class="age">23</div>
    </li>
    <li>
        <div class="name">Bob</div>
        <div class="age">39</div>
    </li>
    ..... #more li
</ul>

So this seems like a pretty simple problem where we just save the li elements as a variable, and iterate through each list to save the information. The problem is that no matter what I do, my results always gives back the first list item over and over again. It will loop through the correct number of elements, but always refer back to the first one. So if I do the following

results = driver.find_elements_by_xpath("""//*[@id="results"]/li""")
for result in results:
    name = result.find_element_by_xpath("""//*[@class="name"]""").text
    print(name)

Now if there is 10 li elements in this particular case, the name "John" will just print out 10 times rather than updating based on the iterated list.

Try doing find_element then find_elements. (note plural) There's one results list and multiple names. Also double check your xpath. — sytech
– sytech, Commented Jan 9, 2018 at 7:02

Buaban · Accepted Answer · 2018-01-09 07:10:13Z

8

Your XPath for the 2nd search is incorrect. It must begin with .. Otherwise, it will start searching from the top. That's the reason why it always find the first item. See my example below.

results = driver.find_elements_by_xpath('//*[@id="results"]/li')
for result in results:
    name = result.find_element_by_xpath('.//*[@class="name"]').text
    print(name)

edited Jan 9, 2018 at 7:10

answered Jan 9, 2018 at 7:01

Buaban

5,1271 gold badge20 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ryan N. Over a year ago

That's the one! Thank you so much for your assistance. The missing dot was all the difference.

Ryan N. Over a year ago

so to clarify the "." would start searching from the previous result or how does it work exactly?

Buaban Over a year ago

@RyanN. Let says . is the current node that is processing. Your loop changes the variable result to the new <li>, if you don't begin the expression with ., the // in the XPath expression will always start searching from the top of the DOM.

Ryan N. Over a year ago

gotcha. Once again thank you very much for the answer and explanation.

Collectives™ on Stack Overflow

Iterating through elements get repeating result on Selenium on Python

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related