0

I am scraping a web page with a table that has multiple pages. I have a function that finds the next page button and hits it. The function needs to return to the main table page to do that. I have the link to that main table page hardcoded in a variable.

Once I moved to e.g page number 2, how do I update that table page link to the new page link? so that once it's done going inside elements of the table, it'll go back to the 2nd-page link and move on from there, not the first page.

def nextPage(driver, desiredPage):
driver.get(desiredPage)
time.sleep(55)
button = driver.find_element_by_xpath(XPATH WRITTEN HERE)
driver.execute_script("arguments[0].click();", button)
time.sleep(45)
next_page = driver.current_url

return next_page

1 Answer 1

1

You can use a while loop, to call the method again and again. You may choose a condition on page_nb or another variable to count, to avoid infinite code to run

page_nb = 1
while page_nb != 99:
    page_nb = nextPage(driver, page_nb)
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.