Selenium: Scroll to the bottom of a dynamically loading div

Question

I'm trying to use selenium to scrape a webpage, and one of my elements dynamically loads a finite amount of content (IE not like twitter where you can just keep scrolling) when I scroll to the bottom of it.

Now I could simply do something like

var scrollbar = document.getElementsByClassName("dataTables_scrollBody")[0]
var last = -1;
var curr = scrollbar.scrollHeight;
while (curr != last) {
    last = curr;
    scrollbar.scrollTop = curr;
    curr = scrollbar.scrollHeight;
}

And in fact, that's what I'm doing. However, the actual load operation on the data table takes so long that this script exits by the time the load finishes, meaning I only ever get halfway down the scrollbar.

What's a good way to make sure that I've scrolled all the way to the bottom of the scroll bar?

This question is different from Scroll to bottom of div? because I don't have access to the raw HTML, and it's different from Scroll Automatically to the Bottom of the Page because my page is dynamically loading content.

Perhaps I could force an ajax load of all the content in the panel, but I don't know how to do that, and I don't want to wade through miles of minified code to figure out how the page owner does it.

Is there anyway to insert a "pause" in your while loop so you can slow it down? — GetSet
– GetSet, Commented Mar 7, 2022 at 20:10
No, javascript only seems to have asynchronous waits/pauses. — Jakob Lovern
– Jakob Lovern, Commented Mar 7, 2022 at 20:12
What about a nested loop that does nothing but increment a number as a makeshift pause? — GetSet
– GetSet, Commented Mar 7, 2022 at 20:12
Perhaps. I might be able to use a ResizeObserver on the scrollable element instead and just get it to scroll every time it resizes. — Jakob Lovern
– Jakob Lovern, Commented Mar 7, 2022 at 20:14
You can make a function for sleep. Hope this link will help you stackoverflow.com/questions/951021/… — KunduK
– KunduK, Commented Mar 7, 2022 at 20:15

Jakob Lovern · Accepted Answer · 2022-11-28 20:33:40Z

1

Here's an ugly solution:

scroll_elem = document.getElementsByClassName("dataTables_scrollBody")[0];
table_elem = document.getElementsByClassName("dataTable")[0];
function scrollToBottom() {
    scroll_elem.scrollTop = scroll_elem.scrollHeight;
};
new ResizeObserver(scrollToBottom).observe(table_elem)

Basically how this works is we have an object watching the element we want to scroll to the bottom of. If and when it resizes, it instantly scrolls to the bottom of the new window. This works well enough, then you could use the sleep function KunduK recommended in comments to wait for the process to complete

edited Nov 28, 2022 at 20:33

answered Mar 7, 2022 at 20:25

Jakob Lovern

1,3518 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Selenium: Scroll to the bottom of a dynamically loading div

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related