Webscraping Click Button Selenium

Question

I am trying to webscrape indeed.com to search for jobs using python, with selenium and beautifulsoup. I want to click next page but cant seem to figure out how to do this. Looked at many threads but it is unclear to me which element I am supposed to perform on. Here is the web page html and the code marked with grey comes up when I inspect the next button.

Also just to mention I tried first to follow what happens to the url when mousedown is executed. After reading the addppurlparam function and adding the strings in the function and using that url I just get thrown back to page one.

Here is my code for the class with selenium meant to click on the button:

   from selenium import webdriver
from selenium.webdriver import ActionChains

driver = webdriver.Chrome("C:/Users/alleballe/Downloads/chromedriver.exe")
driver.get("https://se.indeed.com/Internship-jobb")
print(driver.title)
#assert "Python" in driver.title
elem = driver.find_element_by_class_name("pagination-list")
elem = elem.find_element_by_xpath("//li/a[@aria-label='Nästa']")
print(elem)
assert "No results found." not in driver.page_source
assert elem

action = ActionChains(driver).click(elem)
action.perform()
print(elem)

driver.close()

GAP2002 · Accepted Answer · 2020-11-14 19:18:02Z

1

The indeed site is formatted so that it shows 10 per page.

Your photo shows the wrong section of HTML instead you can see the links contain start=0 for the first page, start=10 for the second, start=20 for the third,...

You could use this knowledge to do a code like this:

while True:
    i = 0
    driver.get(f'https://se.indeed.com/jobs?q=Internship&start={i}')
    # code here
    i = i + 10

But, to directly answer to your question you should do:

next_page_link = driver.find_element_by_xpath('/html/head/link[6]')
driver.get(next_page_link)

This will find the link and then get it.

answered Nov 14, 2020 at 19:18

GAP2002

9798 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

brovatten Over a year ago

Thank you so much I should definitly have noticed this. My thoughts was that the addppurlparam-function changed the href to the new url destination (next page), is that not true? Because I did try manually to compute the outcome which returned me to page 1.

GAP2002 Over a year ago

@brovatten the addppurlparam-function does do that for sure, but it's not necessary to use that URL. That URL adds another parameter (probably to track request sources or something similar) but not having this parameter does not affect the contents of the page.

Samsul Islam · Accepted Answer · 2020-11-14 19:23:43Z

1

its work. paginated to next page.

driver.find_element_by_class_name("pagination-list").find_element_by_tag_name('a').click()

answered Nov 14, 2020 at 19:23

Samsul Islam

2,6192 gold badges20 silver badges24 bronze badges

Collectives™ on Stack Overflow

Webscraping Click Button Selenium

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related