Issue in navigating ASPX pagination using Selenium

Question

I would like to scrape this URL to iterate over pages and get their content. That's what I have tried:

driver.get("https://www.shoroukbookstores.com/books/publishing-house.aspx?id=429c2704-5fa3-43b0-9ad7-c2ec9062beb3")
page_count = 1
while True:
    page_count += 1
    try:
        WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='Body_AspNetPager']/a[%s]" % page_count))).click()                                                          
        print("Next page clicked")

    except NoSuchElementException:
        print("No more pages")
        break

driver.quit()

and I get 10 pages out of 102; specifically the visible page numbers in the URL. How to paginate all pages in this URL?

I can not get its content. I just get the visible pages in the navigation bar. Please have a look at this image: imgur.com/a/L96tPDO — Othman
– Othman, Commented Dec 23, 2021 at 20:45
I don't see you extracting the content in your code even. Where as the image is of pagination. — undetected Selenium
– undetected Selenium, Commented Dec 23, 2021 at 20:47
The next button solution proposed by Bilke below worked perfectly. Thank you so much for your help DebanjanB — Othman
– Othman, Commented Dec 23, 2021 at 21:37

bilke · Accepted Answer · 2021-12-23 21:44:11Z

1

Instead of clicking on every number, use the next button.

driver.implicitly_wait(5)
driver.get("https://www.shoroukbookstores.com/books/publishing-house.aspx?id=429c2704-5fa3-43b0-9ad7-c2ec9062beb3")

while True:

    try:
        navigation_links = driver.find_element_by_xpath(f"//*[@id='Body_AspNetPager']").find_elements_by_tag_name("a")
        next_page = len(navigation_links) - 1 # next page button is second from behind
        driver.find_element_by_xpath(f"//*[@id='Body_AspNetPager']/a[{next_page}]").click()
        print("Next page clicked")

    except NoSuchElementException:
        print("No more pages")
        break

driver.quit()

edited Dec 23, 2021 at 21:44

answered Dec 23, 2021 at 20:51

bilke

4153 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Issue in navigating ASPX pagination using Selenium

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related