Python Selenium not scrolling down infinite page

Question

I'm having some trouble scrolling down an infinite scrolling page using Python Selenium. I'd like to scroll down this page so that I can click on to each film on the page. The code runs to the end and quits the driver but without scrolling down the page - what's wrong?

def scroll(driver):
    timeout = 5

    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # load the website
        time.sleep(5)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")

driver.get("https://www.justwatch.com/uk/provider/bfi-player/movies")
scroll(driver)
driver.quit()

I'm trying to get the page to scroll down. I can lose this line of code if needed - my priority is just to get the page to scroll down. — Eddyhut
– Eddyhut, Commented Mar 16, 2021 at 21:54

Vova · Accepted Answer · 2021-03-16 21:56:47Z

1

Try to use this:

import time
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.justwatch.com/uk/provider/bfi-player/movies")
time.sleep(3)
driver.execute_script("document.body.getElementsByTagName('ion-content')[0].scrollToBottom();")
time.sleep(3)
driver.quit()

answered Mar 16, 2021 at 21:56

Vova

3,6173 gold badges19 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Eddyhut Over a year ago

this code raises this exception: raise exception_class(message, screen, stacktrace) selenium.common.exceptions.JavascriptException: Message: javascript error: Cannot read property 'scrollToBottom' of undefined

Vova Over a year ago

@Eddyhut weirdo, mine works good, try to open browser-> move to you url->def tools-> console and run that command: document.body.getElementsByTagName('ion-content')[0].scrollToBottom();

PDHide · Accepted Answer · 2021-03-16 22:10:46Z

driver = webdriver.Chrome()


def scroll(driver):
    timeout = 5
    time.sleep(5)
    # Get scroll height
    scrollelement = driver.execute_script(
        "return document.querySelector('.tabs-inner [tab=\"popular\"] ion-content').shadowRoot.querySelector('[class=\"inner-scroll scroll-y\"]')")
    last_height = driver.execute_script(
        "return arguments[0].scrollHeight", scrollelement)
    new_height=0
    print(new_height, last_height)
    while True:
        # Scroll down to bottom
       
        driver.execute_script(
            "arguments[0].scrollTo(0, arguments[0].scrollHeight);", scrollelement)

        # load the website
        time.sleep(5)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script(
            "return arguments[0].scrollHeight", scrollelement)
        print(new_height,last_height)


driver.get("https://www.justwatch.com/uk/provider/bfi-player/movies")
scroll(driver)
driver.quit()

You should do the scroll on the scrollelement as the scrollbar is also an element in the page. But the scrollbar is in side shadowRoot so use execute_script

Collectives™ on Stack Overflow

Python Selenium not scrolling down infinite page

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related