How to scrape Next button on Linkedin with Selenium using Python?

Question

I am trying to scrape LinkedIn website using Selenium. I can't parse Next button. It resists as much as it can. I've spent a half of a day to adress this, but all in vain.

I tried absolutely various options, with text and so on. Only work with start ID but scrape other button.

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@aria-label='Далее']"}

This is quite common for this site:

//*[starts-with(@id,'e')]

My code:

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from time import sleep



chrome_driver_path = Service("E:\programming\chromedriver_win32\chromedriver.exe")
driver = webdriver.Chrome(service=chrome_driver_path)
url = "https://www.linkedin.com/feed/"
driver.get(url)
SEARCH_QUERY = "python developer"
LOGIN = "EMAIL"
PASSWORD = "PASSWORD"
sleep(10)

sign_in_link = driver.find_element(By.XPATH, '/html/body/div[1]/main/p[1]/a')
sign_in_link.click()

login_input = driver.find_element(By.XPATH, '//*[@id="username"]')
login_input.send_keys(LOGIN)
sleep(1)
password_input = driver.find_element(By.XPATH, '//*[@id="password"]')
password_input.send_keys(PASSWORD)
sleep(1)
enter_button = driver.find_element(By.XPATH, '//*[@id="organic-div"]/form/div[3]/button')
enter_button.click()
sleep(25)

lens_button = driver.find_element(By.XPATH, '//*[@id="global-nav-search"]/div/button')
lens_button.click()
sleep(5)

search_input = driver.find_element(By.XPATH, '//*[@id="global-nav-typeahead"]/input')
search_input.send_keys(SEARCH_QUERY)
search_input.send_keys(Keys.ENTER)
sleep(5)

people_button = driver.find_element(By.XPATH, '//*[@id="search-reusables__filters-bar"]/ul/li[1]/button')
people_button.click()
sleep(5)

page_button = driver.find_element(By.XPATH, "//button[@aria-label='Далее']")
page_button.click()

sleep(60)

Chrome inspection of button Next Button

Please share ALL your selenium code

Prophet
– Prophet

2022-11-30 16:30:42 +00:00
Commented Nov 30, 2022 at 16:30 — Prophet
– Prophet, Commented Nov 30, 2022 at 16:30
Added my code .

Sergey
– Sergey

2022-11-30 16:37:44 +00:00
Commented Nov 30, 2022 at 16:37 — Sergey
– Sergey, Commented Nov 30, 2022 at 16:37
I asked for ALL relevant code.

Prophet
– Prophet

2022-11-30 16:40:07 +00:00
Commented Nov 30, 2022 at 16:40 — Prophet
– Prophet, Commented Nov 30, 2022 at 16:40
shared all relevant code

Sergey
– Sergey

2022-11-30 17:30:57 +00:00
Commented Nov 30, 2022 at 17:30 — Sergey
– Sergey, Commented Nov 30, 2022 at 17:30

Prophet · Accepted Answer · 2022-11-30 18:27:34Z

3

OK, there are several issues here:

The main problem why your code not worked is because the "next" pagination is initially even not created on the page until you scrolling the page, so I added the mechanism, to scroll the page until that button can be clicked.
it's not good to create locators based on local language texts.
You should use WebDriverWait expected_conditions explicit waits, not hardcoded pauses.

I used mixed locators types to show that sometimes it's better to use By.ID and sometimes By.XPATH etc.
the following code works:

import time

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)

url = "https://www.linkedin.com/feed/"
driver.get(url)

wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(@href,'login')]"))).click()
wait.until(EC.element_to_be_clickable((By.ID, "username"))).send_keys(my_email)
wait.until(EC.element_to_be_clickable((By.ID, "password"))).send_keys(my_password)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[type='submit']"))).click()
search_input = wait.until(EC.element_to_be_clickable((By.XPATH, "//input[contains(@class,'search-global')]")))
search_input.click()
search_input.send_keys("python developer" + Keys.ENTER)
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="search-reusables__filters-bar"]/ul/li[1]/button'))).click()
wait = WebDriverWait(driver, 4)
while True:
    try:
        next_btn = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "button.artdeco-pagination__button.artdeco-pagination__button--next")))
        next_btn.location_once_scrolled_into_view
        time.sleep(0.2)
        next_btn.click()
        break
    except:
        driver.execute_script("window.scrollBy(0, arguments[0]);", 600)

answered Nov 30, 2022 at 18:27

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sergey Over a year ago

You've hit the nail on the head! I didn't even think about scrolling. And thanks for the recommendations! I'll take your answer as a sample of good code.

Prophet Over a year ago

You are welcome! I'm always happy to share my professional knowledge and what you see here are a very basic things.

Sidath Rashen Over a year ago

That's amazing. I also didn't even think about scrolling. So the code was unable to detect the pagination button. Thank you!

Collectives™ on Stack Overflow

How to scrape Next button on Linkedin with Selenium using Python?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related