Selenium Python: Unable to scroll and click on Load more stories button

Question

I am new to Selenium and would appreciate advice. I am trying to click on the load more stories button on NPR's website (https://www.npr.org/sections/news/). My code seems good, but when I run it more stories are never loaded. I don't get an error at the same time the code just doesn't behave as expected.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox(executable_path='/home/myname/projects/myproject/geckodriver')
# implicit wait for 5 seconds
driver.implicitly_wait(5)
#driver.maximize_window()
driver.get('https://www.npr.org/sections/news/')

element = driver.find_element(By.XPATH, '/html/body/main/div[2]/section/div[7]/button')
driver.execute_script("arguments[0].scrollIntoView();", element)
driver.execute_script("arguments[0].click();", element)

undetected Selenium · Accepted Answer · 2021-12-23 20:41:45Z

1

The webpage npr have a infinitescrollwrap. As a demonstration to click() thrice on Load more stories you can use the following Locator Strategies:

Code Block:

driver.get("https://www.npr.org/sections/news/")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.global-modal__dismiss"))).click()
for i in range(3):
  driver.execute_script("arguments[0].scrollIntoView();", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button.options__load-more"))))
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.options__load-more"))).click()
  print("Load more stories clicked")

Console Output:

Load more stories clicked
Load more stories clicked
Load more stories clicked

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Dec 23, 2021 at 20:41

answered Dec 23, 2021 at 20:23

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

20 Comments

generic Over a year ago

Thanks so much for this response. When I try the above code, I still get an error -- this time on the 4th line (the line with driver.execute_script()` TimeoutException: Message: Stacktrace: WebDriverError@chrome://marionette/content/error.js:181:5 NoSuchElementError@chrome://marionette/content/error.js:393:5 element.find/</<@chrome://marionette/content/element.js:306:16

undetected Selenium Over a year ago

@generic What error does it throws? I've added the required imports.

generic Over a year ago

I think I had the wrong imports and adding your imports fixed the problem. This was my error:

generic Over a year ago

TimeoutException: Message: Stacktrace: WebDriverError@chrome://marionette/content/error.js:181:5 NoSuchElementError@chrome://marionette/content/error.js:393:5 element.find/</<@chrome://marionette/content/element.js:306:16

generic Over a year ago

Actually, I am not sure what happened but I am getting that same error (above) again.

|

Collectives™ on Stack Overflow

Selenium Python: Unable to scroll and click on Load more stories button

1 Answer 1

20 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

20 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related