How to use proxy in selenium to avoid IP restriction while scraping data?

Question

As we use user-agent or proxy-pool while scraping with scrapy, what tool should be used in case of selenium? And also want to know how to use. Can anyone help me with this issue?

user12541086 · Accepted Answer · 2020-07-17 15:10:00Z

1

When running Selenium with FireFox you can specify the proxy settings for the driver. The following is Python specific code for setting FireFox proxy settings.

from selenium import webdriver

PROXY = "<HOST:PORT>"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
    "httpProxy": PROXY,
    "ftpProxy": PROXY,
    "sslProxy": PROXY,
    "proxyType": "MANUAL",

}

with webdriver.Firefox() as driver:
    # Open URL
    driver.get("https://selenium.dev")

Check out selenium https proxy documentation for other languages.

For Chrome you can do something similar and pass in options for the browser:

from selenium import webdriver

PROXY = "<HOST:PORT>"
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=%s' % PROXY)
driver = webdriver.Chrome(chrome_options=options')
driver.get("https://selenium.dev")

answered Jul 17, 2020 at 15:10

user12541086

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use proxy in selenium to avoid IP restriction while scraping data?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related