Selenium Python: How to Scroll Down in a Pop Up window

Question

I am working on a Linkedin web scraping project. I am trying to get the list of companies that interest someone (notice I am not using the API). It is a dynamic website, so I would need to scroll down while scraping the names of the companies. I know how to do this in the MAIN window, but since Interest are a pop-up window this solution to scroll does not work. My code so far was:

from selenium.webdriver.common.keys import Keys
bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
bar.send_keys(Keys.END)

Since it didn't work, I also tried:

bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', bar)

The problem is that I am not acting on the pop up window but in the main one so it has not the desired effect.

enter image description here

Find the "scroll down" arrow css and click it until you find the element? That's how a human would do it. — BoboDarph
– BoboDarph, Commented Aug 30, 2017 at 13:15
Yes, I agree with JeffC. I added my code so far to make it more complete. — G. Macia
– G. Macia, Commented Aug 30, 2017 at 21:06

Andersson · Accepted Answer · 2017-08-30 13:22:44Z

9

You can try to find element inside popup (the one that can be focused), for example some anchor:

element_inside_popup = driver.find_element_by_xpath('//div[@class="entity-list-wrapper ember-view"]//a')

and then use below code to scroll popup down:

from selenium.webdriver.common.keys import Keys

element_inside_popup.send_keys(Keys.END)

answered Aug 30, 2017 at 13:22

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

G. Macia Over a year ago

Just a clarification: what does '//a' exactly do? I realized that without it, the code cannot focus the element. How would the driver know when the element is over so that it does not need to scroll down any more?

Andersson Over a year ago

//a means to return you the any first link on popup. Do you want to scroll to the very bottom of popup or to specific element?

G. Macia Over a year ago

Thanks! I understand the '//a' magic now. I want to get the list of all companies and I realized this is only displayed fully if I get to the end of the element.

Andersson Over a year ago

Oh. I see. You need to make XHR while complete list not received? Try element_inside_popup.send_keys(Keys.END) in a while loop. You should check the count of companies and if on some iteration there is no new companies- break the loop

Tintinabulator Zea Over a year ago

I get this error selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <a class="class-small "> stale; either the element is no longer attached to the DOM or the document has been refreshed . any way to get arround it without time.sleep a bunch?

|

Dhifaoui Bilel · Accepted Answer · 2022-02-22 15:42:27Z

2

first I tried with this script but it doesn't work:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    scrollable_popup.send_keys(Keys.END)
    time.sleep(2)

and then I used this script and it works fine with me:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_popup)
    time.sleep(2)

answered Feb 22, 2022 at 15:42

Dhifaoui Bilel

512 bronze badges

2 Comments

Aalexander Over a year ago

Both is not working

Dhifaoui Bilel Over a year ago

The second one is working fine for me .. verify the XPATH of the popup

PaulMuadDib · Accepted Answer · 2024-08-17 19:53:09Z

I know this question is quite a few years old at this point. It does still come up in search on Google. I tried similar methods before similar to the answers in this thread. In my experience using Selenium, the JavaScript scroll method is great for getting dynamic content to load into a page. But it can be janky at times when trying to get specific elements to scroll into view. That could be from me not using JavaScript that often.

I ended up referring to the documentation Selenium provides and it looks like there is an element property that I didn't know about called location_once_scrolled_into_view which scrolls elements into view. It even works on those overlay popup windows. https://www.selenium.dev/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webelement.html#selenium.webdriver.remote.webelement.WebElement.location_once_scrolled_into_view

Collectives™ on Stack Overflow

Selenium Python: How to Scroll Down in a Pop Up window

3 Answers 3

7 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related