Why by scraping LinkedIn it cannot load the requested url? Python

Question

I am trying to scrape LinkedIn, the script was working for 3 months but yesterday it crashed.

I use selenium webdriver, Firefox with fake useragent.

The URL is https://www.linkedin.com/company/my_company/

def init_driver():
    """Initiates selenium webdriver.
    :return: Firefox browser instance
    """
    try:
        #  use random UserAgent to avoid captcha
        fp = webdriver.FirefoxProfile()
        fp.set_preference("general.useragent.override", UserAgent().random)
        fp.update_preferences()
        # initiate driver
        options = FirefoxOptions()
        #options.add_argument("--headless")
        return webdriver.Firefox(firefox_options=options)
    except Exception as e:
        logging.error('Exception occurred initiating webdriver', exc_info=True)

And then just open a page driver.get(url)

at this moment it opens it but cannot load

the same situation happens without fake agent and by using chrome.

Has anyone encountered something like this? When I open the link myself everything os ok.

https://www.linkedin.com/authwall?trk=gf&trkInfo=AQFvPeNP8NQIxwAAAXLqc-uI5rnQe1ZIysPcZOgjZCzbrBHZj7q6gd68fPG9NzbX00Rlre_yC0tITChjMDEXSNnD8tZRaMXqcRG-z_3QUMlCvQPR4uVGBQYoSOl3ycoO2E6Jl9w=&originalReferer=&sessionRedirect=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2my_company%2F

Other URLs are opened without problems by the function

Mostafa · Accepted Answer · 2020-06-25 08:45:24Z

This is how you should modify your code.

I modified your code and your code was executed correctly.

from selenium import webdriver
from fake_useragent import UserAgent
import logging

def init_driver():
    """Initiates selenium webdriver.
    :return: Firefox browser instance
    """

    path = r"your firefox driver path"

    try:
        #  use random UserAgent to avoid captcha
        fp = webdriver.FirefoxProfile()
        fp.set_preference("general.useragent.override", UserAgent().random)
        fp.update_preferences()
        # initiate driver
        options = webdriver.FirefoxOptions()
        # options.add_argument("--headless")
        return webdriver.Firefox(firefox_options=options, executable_path=path)
    except Exception:
        logging.error('Exception occurred initiating webdriver', exc_info=True)




url = "your url"

driver = init_driver()


driver.get(url)

Collectives™ on Stack Overflow

Why by scraping LinkedIn it cannot load the requested url? Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related