selenium python load more button not working

Question

I am trying to click load more button infinite time until load all products. But my load more button not working. here is my code:

url = 'https://www.pursemall.ru/vuitton/damier-azur'
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)


try:
    for i in range(1000):
     load_more_button = driver.find_element_by_xpath("//span[text()='Load More Products']")
     load_more_button.click()
except:
    pass
    print("task load more button completed")

why my load more button not working? where I am doing mistake?

Try to inspect the element and copy its CSS selector or XPath — crissal
– crissal, Commented Sep 24, 2021 at 21:50

balderman · Accepted Answer · 2021-09-24 22:14:05Z

1

What you can do is the below. Each call inside the loop will fetch a set of products. You can scrape each response and collect the data.

In the browser do F12 -> Network -> XHR and see how the browser does the HTTP call every time you hit Load More Products

import requests


for page in range(1,4):
    url = f'https://www.pursemall.ru/vuitton/damier-azur?page={page}'
    print(url)
    r = requests.get(url)
    print(r.status_code)

answered Sep 24, 2021 at 22:14

balderman

24k8 gold badges39 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mgermaine93 · Accepted Answer · 2021-09-24 22:34:57Z

I'm not 100% sure why your original code wasn't functioning, but I was able to get it working by copying the full XPATH of the button instead of what you have in your code above.

I also used WebDriverWait and sleep() to wait a few seconds until the button was loaded after each click, since that's good practice for web scrapers. Perhaps the issue was a combination of the two :)

url = 'https://www.pursemall.ru/vuitton/damier-azur'
driver = webdriver.Chrome(PATH) # PATH is where you have chromedriver stored on your machine
driver.maximize_window()
driver.get(url)

try:
    for i in range(1000):
        load_more_button = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
            (By.XPATH, '/html/body/div[1]/div/div[3]/div[3]/span/a/span')))
        sleep(3)
        load_more_button.click()
except:
    pass
    print("task load more button completed")

Collectives™ on Stack Overflow

selenium python load more button not working

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related