How to scroll down google job page using selenium python

Question

I am trying to scroll down the job posts using below lines, but it will give sometime correct results to scroll down to the end and sometimes it won't.

html = driver.find_element_by_tag_name('html')
time.sleep(5)
html.send_keys(Keys.END)

Can anyone suggest me how to scroll down to the end, please find the link and screenshot below.

https://www.google.com/search?q=upsc+jobs+in+india&rlz=1C1CHBF_enIN869IN869&oq=upsc+jo&aqs=chrome.1.69i57j0i433i512j0i131i433i512j0i512l3j0i131i433i512l2j0i512j0i433i512&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&sqi=2&ved=2ahUKEwjR27GN_qPzAhX4ppUCHb_0B_QQkd0GegQIORAB#fpstate=tldetail&sxsrf=AOaemvIxuJXh3if0tw7ezZfjkXRe5DSxsA:1632911697417&htivrt=jobs&htidocid=N1_BNfzt8n8auXjGAAAAAA%3D%3D

cruisepandey · Accepted Answer · 2021-09-29 11:14:49Z

2

The more you scroll the more data you get, basically it's a dynamic web site. I have hardcoded 50 as a dummy number, you can have 100 or any other number for this matter.

You can use the sample code :

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.google.com/search?q=upsc+jobs+in+india&rlz=1C1CHBF_enIN869IN869&oq=upsc+jo&aqs=chrome.1.69i57j0i433i512j0i131i433i512j0i512l3j0i131i433i512l2j0i512j0i433i512&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&sqi=2&ved=2ahUKEwjR27GN_qPzAhX4ppUCHb_0B_QQkd0GegQIORAB#fpstate=tldetail&sxsrf=AOaemvIxuJXh3if0tw7ezZfjkXRe5DSxsA:1632911697417&htivrt=jobs&htidocid=N1_BNfzt8n8auXjGAAAAAA%3D%3D")
j = 1
for i in range(50):
    element = driver.find_element(By.XPATH, f"(//div[@role='heading'])[{j}]")
    driver.execute_script("arguments[0].scrollIntoView(true);", element)
    j =  j + 1

answered Sep 29, 2021 at 11:14

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user19408629 Over a year ago

hello i have a same code to scroll down in a site and add new items to a list i am new to python the xpath is the xpath of that scroll bar ?

pmadhu · Accepted Answer · 2021-09-29 11:48:28Z

1

You can also try this to scroll till the end.

driver.get("https://www.google.com/search?q=upsc+jobs+in+india&rlz=1C1CHBF_enIN869IN869&oq=upsc+jo&aqs=chrome.1.69i57j0i433i512j0i131i433i512j0i512l3j0i131i433i512l2j0i512j0i433i512&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&sqi=2&ved=2ahUKEwjR27GN_qPzAhX4ppUCHb_0B_QQkd0GegQIORAB#fpstate=tldetail&sxsrf=AOaemvIxuJXh3if0tw7ezZfjkXRe5DSxsA:1632911697417&htivrt=jobs&htidocid=N1_BNfzt8n8auXjGAAAAAA%3D%3D")
i = 0
try:
    while True:
        options = driver.find_elements_by_xpath("//div[@role='treeitem']")
        driver.execute_script("arguments[0].scrollIntoView(true);",options[i])
        i+=1
        time.sleep(.5)
except:
    pass

answered Sep 29, 2021 at 11:48

pmadhu

3,4332 gold badges14 silver badges24 bronze badges

2 Comments

cruisepandey Over a year ago

Is there really an end ? I tried manually and data kept on coming..

pmadhu Over a year ago

@cruisepandey - Well I saw the end, so suggested this code.

Collectives™ on Stack Overflow

How to scroll down google job page using selenium python

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related