Click on Load more element using Selenium and Python

Question

I have a news website to scrape. But after scraping the first page, I need to click on "load more" which is a link to view other news and then scrape. Where I am encountering an issue is that I can't seem to click on "load more" because it doesn't have a "href". I have tried all possible resources, but they have all worked to no avail. Please, kindly help me out with this. Thank you. I have attached the code I used and the link to the website.

Code trials:

pip install webdriver-manager

from selenium.webdriver.common.keys import Keys
import time
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException

driver = webdriver.Chrome(ChromeDriverManager().install())
url = 'https://www.legit.ng/tag/road-accidents-in-nigeria/'
driver.get(url)
link = driver.find_element_by_link_text("Entertainment")
link.click()

undetected Selenium · Accepted Answer · 2022-01-21 23:17:34Z

1

To click on the element with text as Load more you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using XPATH and text()[contains()]:

driver.get("https://www.legit.ng/tag/road-accidents-in-nigeria/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()[contains(.,'Load more')]]"))).click()

Using XPATH and contains():

driver.get("https://www.legit.ng/tag/road-accidents-in-nigeria/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(., 'Load more')][./p[text()]]"))).click()

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Jan 21, 2022 at 23:17

answered Jan 21, 2022 at 23:08

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

zion Over a year ago

First, thank you for your comment. However, I got the following error while running the code; ElementClickInterceptedException: Message: element click intercepted: Element <div class="c-action-link__text-container">...</div> is not clickable at point (686, 458). Other element would receive the click: <svg class="c-action-link__border" width="100%" height="100%">...</svg> (Session info: chrome=97.0.4692.99)

Collectives™ on Stack Overflow

Click on Load more element using Selenium and Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related