Python Selenium can't click on next page button

Question

I want to click the next page until no more page, but it does not click.

returns the error:raise exception_class(message, screen, stacktrace)

StaleElementReferenceException: stale element reference: element is not attached to the page document

my codes:

Thanks in advance!

driver.get('http://www.chinamoney.com.cn/chinese/zjfxzx/?tbnm=%E6%9C%80%E6%96%B0&tc=null&isNewTab=1')
driver.implicitly_wait(10)
driver.refresh()
driver.implicitly_wait(10)
wait = WebDriverWait(driver, 5)
datefield_st = wait.until(EC.element_to_be_clickable((By.ID, "pdbp-date-1")))
datefield_st.click()
select_st = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-year"))))
select_st.select_by_visible_text("2021")
select2 = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-month"))))
select2.select_by_value("1")
day=1
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[@data-handler='selectDay']/a[text()='{}']".format(str(day))))).click()
datefield_ed = wait.until(EC.element_to_be_clickable((By.ID, "pdbp-date-2")))
datefield_ed.click()
select_ed = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-year"))))
select_ed.select_by_visible_text("2021")
select2 = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-month"))))
select2.select_by_value("1")
day=1
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[@data-handler='selectDay']/a[text()='{}']".format(str(day))))).click()
driver.find_element_by_link_text("查询").click()
while True:
    driver.implicitly_wait(10)
    links=[link.get_attribute('href') for link in driver.find_elements_by_xpath("//a[contains(@title,'同业存单') and not(contains(@title,'申购说明')) and not(contains(@title,'公告'))]")]
    titles = [title.text for title in driver.find_elements_by_xpath("//a[contains(@title,'中期票据') and not(contains(@title,'申购说明')) and not(contains(@title,'公告'))]")]
    dates = [date.text for date in driver.find_elements_by_xpath('//*[@class="san-grid-r text-date"]')]
    driver.implicitly_wait(10)
    for link, title,date in zip(links, titles,dates):
        dataframe = pd.DataFrame({'col1':date,'col2':title,'col3':link},index=[0])
        dataframe.to_csv('Chinamoney.csv',mode='a+',header=False,index=False,encoding='utf-8-sig')
        print(link,title,date)
    try:
        driver.find_element_by_xpath('//*[contains(@class, "page-next")]').click()
    except:
        print('No more pages')

JaSON · Accepted Answer · 2021-02-03 08:50:05Z

1

You passed two class names into selector while it's not allowed for search by class name. Either try

(By.CLASS_NAME, 'page-next')

or

(By.CSS_SELECTOR, '.page-btn.page-next')

Also your element and icon select the same element. So you don't need to define icon. Simply use element.click()

answered Feb 3, 2021 at 8:50

JaSON

4,8912 gold badges12 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

19 Comments

Joyce Over a year ago

thanks, but I tried driver.find_element_by_xpath('//*[@class="page-next"]').click(), still wont click

JaSON Over a year ago

@Cathy it's not correct. Correct is driver.find_element_by_xpath('//*[contains(@class, "page-next")]').click()

Joyce Over a year ago

it gives me an errorStaleElementReferenceException: stale element reference: element is not attached to the page document

JaSON Over a year ago

@Cathy it means that you might do something that refreshes the page and then trying to handle element defined earlier. It's hard to say as I can't see your complete current code

Joyce Over a year ago

Please see me edit for the code, not sure what causes that, thanks for your help! I refreshed the page once, as the page is hard to load at the first time

|

MR Mark II · Accepted Answer · 2021-02-03 09:50:14Z

0

You are using:

driver.find_element_by_xpath('//*[contains(@class, "page-next")]').click()

Try:

element = driver.find_element_by_xpath('//*[contains(@class, "page-next")]')
driver.execute_script("arguments[0].click();", element)

If this doesnt work, you can try to obtain the url/link value and store it, and later you can go to the url or do what you want without click in it.

answered Feb 3, 2021 at 9:50

MR Mark II

4533 silver badges13 bronze badges

4 Comments

Joyce Over a year ago

Thanks! but it still does not work, just return the values in first page again and again, is that I refreshed the website the reason? how should it be fixed?

MR Mark II Over a year ago

What? I dont understand, You say that you have the first url page only? Maybe the driver find more than elements, try to do a find_elements instead of find_element and check the length of the array result, or print the result and see how many elements you have.

Joyce Over a year ago

there is only one element found by class name ='page-next',so..

MR Mark II Over a year ago

If the script doesnt enter in the except, extract the href or the attribute where the element has the url, and print it, so you can see where the script is trying to access. If the url is correct, you can access "manually" to it with a get or any "browser" method.

Collectives™ on Stack Overflow

Python Selenium can't click on next page button

2 Answers 2

19 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

19 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related