Selenium: Scroll to end of page in dynamically loading webpage

Question

I have a webpage that keeps loading new items when scrolling down the page until every item is loaded.

I'm working with Selenium in Java, and need to scroll down to the bottom of the page in order to load everything.

I have tried several different options, like scrolling to an element of the bottom of the page:

WebElement copyrightAtEndOfPage = webDriver.findElement(By.xpath("//a[@href='/utils/copyright.html']"));
((JavascriptExecutor) webDriver).executeScript("arguments[0].scrollIntoView();", copyrightAtEndOfPage);

This only scrolls down once though, and then the webpage keeps loading.

I also tried this approach, which also only scrolls down once, because it only takes the browser height into consideration.

Any help is highly appreciated.

Can we also do similar thing using JS?

Ninja
– Ninja

2020-08-05 13:42:53 +00:00
Commented Aug 5, 2020 at 13:42 — Ninja
– Ninja, Commented Aug 5, 2020 at 13:42

Ratmir Asanov · Accepted Answer · 2019-01-22 10:27:56Z

63

I will provide you code in Python for this. I think it's easy to translate to Java:

def scroll_down(self):
    """A method for scrolling the page."""

    # Get scroll height.
    last_height = self.driver.execute_script("return document.body.scrollHeight")

    while True:

        # Scroll down to the bottom.
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load the page.
        time.sleep(2)

        # Calculate new scroll height and compare with last scroll height.
        new_height = self.driver.execute_script("return document.body.scrollHeight")

        if new_height == last_height:

            break

        last_height = new_height

Hope it helps you!

edited Jan 22, 2019 at 10:27

answered Feb 18, 2018 at 11:33

Ratmir Asanov

6,4795 gold badges30 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Johannes Mols Over a year ago

That works just perfectly, thank you so much! I will also post an answer with the code that I translated into Java.

Johannes Mols Over a year ago

Did it. Thanks again!

Shruti Joshi Over a year ago

hey.. thanks for the answer.. this should help me out as well! Just a query, when I use this code, my browser does scroll to the end but the new elements are not being captured still. I am a bit new to this. I am using driver.get(url) followed by the above code.. could you help?

Ratmir Asanov Over a year ago

@ShrutiJoshi, this code just will scroll down to the bottom of the visible page (on the screen), when you call this function.

abdou_dev Over a year ago

@my previous comment. It looks like this website is using a different selector. so instead of using "driver.execute_script("return document.body.scrollHeight")" it uses "driver.execute_script("return document.documentElement.scrollHeight")". This has solved my problem

|

Johannes Mols · Accepted Answer · 2018-02-18 11:42:25Z

13

Thanks to Ratmir Asanov (see the approved answer above), I translated the Python code into Java to make it easier to implement for other people.

try {
    long lastHeight = (long) ((JavascriptExecutor) webDriver).executeScript("return document.body.scrollHeight");

    while (true) {
        ((JavascriptExecutor) webDriver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
        Thread.sleep(2000);

        long newHeight = (long) ((JavascriptExecutor) webDriver).executeScript("return document.body.scrollHeight");
        if (newHeight == lastHeight) {
            break;
        }
        lastHeight = newHeight;
    }
} catch (InterruptedException e) {
    e.printStackTrace();
}

answered Feb 18, 2018 at 11:42

Johannes Mols

1,0421 gold badge18 silver badges39 bronze badges

Comments

Prabhat Arya · Accepted Answer · 2019-06-20 07:07:40Z

3

Updated Johannes code a bit to make it functional.

JavascriptExecutor js = (JavascriptExecutor) driver;
try {
    long lastHeight=((Number)js.executeScript("return document.body.scrollHeight")).longValue();
    while (true) {
        ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
        Thread.sleep(2000);

        long newHeight = ((Number)js.executeScript("return document.body.scrollHeight")).longValue();
        if (newHeight == lastHeight) {
            break;
        }
        lastHeight = newHeight;
    }
} catch (InterruptedException e) {
    e.printStackTrace();
}

answered Jun 20, 2019 at 7:07

Prabhat Arya

392 bronze badges

2 Comments

Prabhat Arya Over a year ago

typecasting was the issue, It was throwing compilation error.

Johannes Mols Over a year ago

What was the error? Because I never had any issues with that code and I used it 1:1 in my project like that.

ukod · Accepted Answer · 2022-07-06 11:17:41Z

3

C# version of the Ratmir Asanov's answer:

var lastHeight =  driver.ExecuteScript("returndocument.body.scrollHeight");
while (true)
{
    driver.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");
    await Task.Delay(500);

    var newHeight = driver.ExecuteScript("return document.body.scrollHeight");
    Console.WriteLine(lastHeight + " - " + newHeight);
    if (newHeight.Equals(lastHeight))
        break;

    lastHeight = newHeight;
}

answered Jul 6, 2022 at 11:17

ukod

1592 silver badges9 bronze badges

Comments

buggy_code312 · Accepted Answer · 2021-02-02 23:41:15Z

1

I found another solution to the dynamically loading page.

Count the elements that are displayed every scroll before and after the scroll and compare them to determine if you've scrolled to the bottom.

var reachedEnd = false;
oldCount = driver.FindElements(By.CssSelector(".searchDataContainer.table-row.raw")).Count;

while (!reachedEnd)
{
    driver.FindElement(By.CssSelector("body")).SendKeys(Keys.End);
    Thread.Sleep(500);
    oldCount = driver.FindElements(By.CssSelector(".searchDataContainer.table-row.raw")).Count;

    if (newCount == oldCount)
    {
        reachedEnd = true;
    }
    else
    {
        newCount = oldCount;
    }
}

answered Feb 2, 2021 at 23:41

buggy_code312

215 bronze badges

Comments

Nayanjyoti Rabha · Accepted Answer · 2022-07-05 06:13:10Z

Making a slight correction to the above stated answers. The variable 'start' of type long keeps changing after every scroll and the value becomes same after it reaches the end of the webpage. And as it an infinite loop with will keep returning the same value again and again. So, I just took the 'temp' variable and checked two consecutive values are same or not as the values remain same after the end is reached. As soon at it finds the same it exits the loop.

    try {
            long temp = 0;
            while (true) {
                ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
                Thread.sleep(2500);
                long start = (Long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                if (start == temp) {
                    break;
                }
                temp = start;
            }
            System.out.println("completed the scroll");
        } catch (Exception e) {
            e.printStackTrace();
        }

Zenab Gorach · Accepted Answer · 2019-09-28 16:36:02Z

0

Updating the above solution by Prabhat further as it was still giving me compilation error.

    try {
        Object lastHeight = ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");

        while (true) {
            ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
            Thread.sleep(2000);

            Object newHeight = ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
            if (newHeight.equals(lastHeight)) {
                break;
            }
            lastHeight = newHeight;
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

edited Sep 28, 2019 at 16:36

answered Sep 28, 2019 at 16:26

Zenab Gorach

413 bronze badges

Comments

MdaCosta · Accepted Answer · 2021-10-12 12:11:52Z

0

Updated code that worked for me:

try {
                    long lastHeight = (long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                    int cont=1000;
                    while (true) {
                        ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, "+cont+");");
                        Thread.sleep(2000);

                        long newHeight = (long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                        if (newHeight <= cont) {
                            break;
                        }
//                      lastHeight = newHeight;
                        cont+=500;
                    }
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

answered Oct 12, 2021 at 12:11

MdaCosta

491 silver badge10 bronze badges

1 Comment

Community Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Japhei · Accepted Answer · 2022-07-10 22:17:15Z

0

You can add the following code to keep pressing page Down button:

new Actions(driver).sendKeys(Keys.PAGE_DOWN).perform();

edited Jul 10, 2022 at 22:17

Japhei

6454 silver badges19 bronze badges

answered Jul 5, 2022 at 18:59

indayush

672 silver badges10 bronze badges

Comments

Mahmud Oly · Accepted Answer · 2022-12-12 09:15:13Z

0

to scroll and wait to load more property till end

                lenOfPage = driver.instance.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")

                match = False
                while not match:
                    lastCount = lenOfPage
                    time.sleep(2)
                    lenOfPage = driver.instance.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")

                    if lastCount == lenOfPage:
                        match = True

answered Dec 12, 2022 at 9:15

Mahmud Oly

1

Comments

user2427306 · Accepted Answer · 2024-04-09 02:30:16Z

modal_elem = browser.find_element(By.CLASS_NAME,'yourClassName')`
def scroll_down(self):
"""A method for scrolling the page."""

    # Get scroll height.
    last_height = self.execute_script("return arguments[0].scrollHeight", modal_elem)
    print('last_height')
    print(last_height)
 while True:

    # Scroll down to the bottom.
    self.execute_script("arguments[0].scrollTo(0, arguments[0].scrollHeight);", modal_elem)
    
    # Wait to load the page.
    time.sleep(2)
    
    # Calculate new scroll height and compare with last scroll height.
    new_height = self.execute_script("return arguments[0].scrollHeight", modal_elem)
    print('new_height')
    print(new_height)
    if new_height == last_height:

        break

    last_height = new_height
    
 scroll_down(browser)

Collectives™ on Stack Overflow

Selenium: Scroll to end of page in dynamically loading webpage

11 Answers 11

7 Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

to scroll and wait to load more property till end

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

11 Answers 11

7 Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

to scroll and wait to load more property till end

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related