Python Selenium clicking next button until the end

Question

This is a follow up question from my first question, and I am trying to scrape a website and have Selenium click on next (until it cant be clicked) and collect the results as well.

This is the html tag from the website with the button:

<div class="pager results-display">
  <span class="action-btn prev inactive" data-page="1">Prev</span>
  <span class="action-btn inactive" data-page="1">1</span>  # Currently in page 1 thus inactive
  <span class="action-btn" data-page="2">2</span>
  <span class="action-btn" data-page="3">3</span> 
  <span class="action-btn" data-page="4">4</span>
  <span class="action-btn" data-page="5">5</span>
  <span class="action-btn" data-page="6">6</span>
  <span class="action-btn" data-page="7">7</span>
  <span class="action-btn" data-page="8">8</span> 
  <span class="action-btn" data-page="9">9</span>
  <span class="action-btn" data-page="10">10</span>
  <span class="action-btn" data-page="11">11</span> 
  <span class="action-btn" data-page="12">12</span>
  <span class="action-btn" data-page="13">13</span>
  <span class="action-btn" data-page="14">14</span>
  <span class="action-btn next" data-page="2">Next</span>
</div>
<div class="no-results-display hidden" style="display: none;">
                    No companies matched your search. Try again.
                </div>

I've tried this code:

elm = driver.find_element_by_name('pager results-display')
elm.click()

And I've also checked this question out, but still wasn't able to solve it.

Any thoughts?

Guy · Accepted Answer · 2016-03-27 07:55:14Z

11

pager results-display are actually two classes that belongs to the button's parent element. They aren't name attribute, and find_element_by_name gets only one name anyway. Try

elm = driver.find_element_by_class_name('next')
elm.click()

Notice that you might have to relocate the button each time if the DOM has changed after the click.

To click in loop as long as the button is active you can check if the buuton has class inactive

while True:
    elm = driver.find_element_by_class_name('next')
    if 'inactive' in elm.get_attribute('class'):
        break;
    elm.click()

edited Mar 27, 2016 at 7:55

answered Mar 27, 2016 at 7:36

Guy

51.2k10 gold badges49 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

jake wong Over a year ago

Hello, this works. But how can I structure this into a loop that clicks on next until it ends? From what I observed, when the page is at the last page, next is still there, but the html code changes to <span class="action-btn next inactive" data-page="14">Next</span>

Guy Over a year ago

@jakewong I added example in my answer.

jake wong Over a year ago

Thank you very much. This gives me what I'm looking for :)

Collectives™ on Stack Overflow

Python Selenium clicking next button until the end

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related