Why is my code for scrolling down this webpage not working selenium python?

Question

The code is supposed to scroll to the bottom of the webpage, but it seems to go at first then oldHeight and newHeight do not change values properly.

I have tried to return the window innerheight and the Yoffset methods but neither worked as well as just "return document.body.scrollHeight" they all either return the same value no matter what position im at on the screen or don't change the value when im at a different section of the screen. I have also tried the window.scrollTo that just doesnt work all together by not scrolling down the screen. What i've settled on scrolls once but the value of the screen only changes once and i've tried sending down the screen with page down multiple times before changing the value, but that also doesnt work. here's the code

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("https://www.google.com/search?client=firefox-b-1-d&q=NFL+game+dates#sie=lg;"
           "/g/11k4hhjvj6;6;/m/059yj;mt;fp;1;;;2023-08-04T00:00:00Z/robots.txt")
driver.maximize_window()

body = driver.find_element(By.CSS_SELECTOR, "body")
oldHeight = driver.execute_script("return document.body.scrollHeight")
while True:
    print(oldHeight)
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(2)
    newHeight = driver.execute_script("return document.body.scrollHeight")
    print(newHeight)
    if newHeight == oldHeight:
        break
    else:
        oldHeight = newHeight

driver.close()

I'm sorry if this is a really noob question

Edit: Added an answer that works for me

Mak3rT0n · Accepted Answer · 2023-07-15 16:34:34Z

this is what worked for me, and thank you @ckhan for telling me I can answer it down here

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("https://www.google.com/search?client=firefox-b-1-d&q=NFL+game+dates#sie=lg;"
           "/g/11k4hhjvj6;6;/m/059yj;mt;fp;1;;;2023-08-04T00:00:00Z/robots.txt")
driver.maximize_window()

body = driver.find_element(By.CSS_SELECTOR, "body")
wait = WebDriverWait(driver, 10)
bottom_Element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".AA8jgb")))
counter = 0
while True:
    counter += 1
    print(counter)
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(2)
    if bottom_Element.is_displayed() and counter == 23:
        print("it was")
        break
    driver.close()

Hopefully this helps someone!

Collectives™ on Stack Overflow

Why is my code for scrolling down this webpage not working selenium python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related