Page won't automatically load when scrolled down with selenium Python

Question

I am using headless Chrome with the selenium package. When I manually visit the website and scroll it down, it loads more itens and the list "nomes" updates in the while loop shown below. When I do it with selenium and a headed browser it also works. Why the page isn't loading with headless? Maybe it is irrelevant, but I also changed the userAgent from ua.random to ua['Chrome'].

import fake_useragent
import selenium

chrome_options = Options()
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--incognito')
chrome_options.add_argument('--headless')
chrome_options.add_argument("--window-size=1920,1080")
userAgent = ua['Chrome']
chrome_options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(executable_path=chromedriver_path, options=chrome_options)
driver.get('https://www.website.com/')
nomes = driver.find_elements_by_css_selector('my.css')
iteracao = 0
        if nomes:
            while iteracao < 3:
                iteracao += 1
                nomes = driver.find_elements_by_css_selector('my.css')
                driver.execute_script("arguments[0].scrollIntoView();", nomes[-1])
                time.sleep(1)
                wait(driver, 10).until(
                    wait_for_more_than_n_elements((By.CSS_SELECTOR, 'my.css'), len(nomes)))

where, which I got from here,

class wait_for_more_than_n_elements(object):
    def __init__(self, locator, count):
        self.locator = locator
        self.count = count

    def __call__(self, driver):
        try:
            count = len(EC._find_elements(driver, self.locator))
            return count >= self.count
        except selexcept.StaleElementReferenceException:
            return False

C. Peck · Accepted Answer · 2021-05-12 17:21:19Z

1

I had the same issue a while back! Try adding these to your chrome options, that's what worked for me.

chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-dev-shm-usage")

Worth noting, I guess, that I've heard users say having the --no-sandbox arg defined first is important, but it never seemed to matter for me.

answered May 12, 2021 at 17:21

C. Peck

3,7193 gold badges21 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Pedro Kaneto Suzuki · Accepted Answer · 2021-05-13 13:37:55Z

0

This worked for my website.

loc = nomes[-1].location['y']
driver.execute_script(f"window.scrollTo(0, {loc} + 200);")
driver.execute_script(f"window.scrollTo(0, {loc} + 210);")

answered May 13, 2021 at 13:37

Pedro Kaneto Suzuki

1968 bronze badges

Collectives™ on Stack Overflow

Page won't automatically load when scrolled down with selenium Python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related