Click on show more button; selenium scrape with python

Question

I'm trying to scrape a website with show more button; and I'm not able to click on it. The website is: https://www.wtatennis.com/rankings/singles

And my code is:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver import ActionChains
from tqdm import tqdm
import time

options = Options()
options.add_argument("--headless")

browser = webdriver.Chrome(ChromeDriverManager().install(),options=options)
browser.get('https://www.wtatennis.com/rankings/singles')

action = ActionChains(browser)
showmore = browser.find_elements_by_xpath(".//button[contains(@class, 'btn widget-footer__more-button rankings__show-more js-show-more-button')]")
action.move_to_element(showmore).perform()
showmore.click()
time.sleep(5)

Has anyone any idea? Thanks!

Hi - it’s scrape not scrap. To scrap means to throw away.

DisappointedByUnaccountableMod
– DisappointedByUnaccountableMod

2021-02-21 18:07:59 +00:00
Commented Feb 21, 2021 at 18:07 — DisappointedByUnaccountableMod
– DisappointedByUnaccountableMod, Commented Feb 21, 2021 at 18:07

PDHide · Accepted Answer · 2021-02-21 19:39:26Z

2

Don't use './/' in your locator when you are starting the search from root, as there is no current element your locator won't find any element. Also you can use any attribute to find elements uniquely. see below code:

browser = webdriver.Chrome(options=options)
browser.get('https://www.wtatennis.com/rankings/singles')


WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.XPATH,
    '//*[@data-text="Accept Cookies"]'))).click()


WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH,
    '//*[@data-text = "Show More"]'))).click()

use webdriver wait and data attributes

tu use wait import:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

To wait till all elements are loaded you have to make sure last element is not changing , if its changing keep scrolling .

browser.get('https://www.wtatennis.com/rankings/singles')


WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH,
                                                             '//*[@data-text="Accept Cookies"]'))).click()

value = "start"
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH,
                                                             '//*[@data-text = "Show More"]'))).click()


while(browser.find_element_by_xpath("(//tr[@class='rankings__row'])[last()]").text != value):
    elem = browser.find_element_by_xpath(
        '(//*[contains(text(),"Loading")])[2]')
    value = browser.find_element_by_xpath(
        "(//tr[@class='rankings__row'])[last()]").text

    browser.execute_script("arguments[0].scrollIntoView()", elem)

    WebDriverWait(browser, 10).until(EC.presence_of_all_elements_located((By.XPATH,
                                                                          "//tr[@class='rankings__row']")))

    try:
       WebDriverWait(browser, 10).until_not(EC.text_to_be_present_in_element((By.XPATH,
                                                                           "(//tr[@class='rankings__row'])[last()]"), value))
    except:
        None

edited Feb 21, 2021 at 19:39

answered Feb 21, 2021 at 17:51

PDHide

20.1k2 gold badges43 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

owce Over a year ago

Thank you for the help! This is working, but clicking only once. Is it possible to click until there is no button anymore?

PDHide Over a year ago

@owce added the answer see , the updated part please accept the asnwer and upvote by ckicking the tick sign if it helped

owce Over a year ago

Thank you @PDHide, this is perfect!

Collectives™ on Stack Overflow

Click on show more button; selenium scrape with python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related