Running selenium behind a proxy server

Question

I have been using selenium for automatic browser simulations and web scraping in python and it has worked well for me. But now, I have to run it behind a proxy server. So now selenium open up the window but could not open the requested page because of proxy settings not set on the opened browser. Current code is as follows (sample):

from selenium import webdriver

sel = webdriver.Firefox()
sel.get('http://www.google.com')
sel.title
sel.quit()

How do I change the above code to work with proxy server now as well?

you can try this stackoverflow.com/a/38168865/5409601

Rafayet Ullah
– Rafayet Ullah

2016-07-03 10:24:37 +00:00
Commented Jul 3, 2016 at 10:24 — Rafayet Ullah
– Rafayet Ullah, Commented Jul 3, 2016 at 10:24

Community · Accepted Answer · 2017-05-23 11:54:05Z

23

You need to set desired capabilities or browser profile, like this:

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", "proxy.server.address")
profile.set_preference("network.proxy.http_port", "port_number")
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile)

Also see related threads:

edited May 23, 2017 at 11:54

CommunityBot

11 silver badge

answered Aug 1, 2013 at 8:35

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Aryabhatt Over a year ago

I tried what you suggested but still not able to get past the proxy server

Aryabhatt Over a year ago

I checked the browser settings opened by selenium after doing the preference update. Actually issue is that, it is not setting http_port correctly (and leaving it out to be 0), because of which it is not connecting. Are there any issues in port setting ?

alecxe Over a year ago

Hm, can you try setting it as a number (not string)?

Aryabhatt Over a year ago

I was doing a stupid error. Basically, you dont need to put the port number under braces. It works well now.

proprius Over a year ago

how to do the same thing for phantomjs. Basically,instead of ` profile = webdriver.FirefoxProfile() profile.set_proxy(proxy.selenium_proxy())`,i want to use phantomjs for same proxy server.

kntonas · Accepted Answer · 2013-11-04 21:16:21Z

8

The official Selenium documentation (http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#using-a-proxy) provides clear and helpful guidelines about using a proxy. For Firefox (which is the browser of choice in your sample code) you should do the following:

from selenium import webdriver
from selenium.webdriver.common.proxy import *

myProxy = "host:8080"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy,
    'ftpProxy': myProxy,
    'sslProxy': myProxy,
    'noProxy': '' # set this value as desired
    })

driver = webdriver.Firefox(proxy=proxy)

answered Nov 4, 2013 at 21:16

kntonas

811 silver badge1 bronze badge

Comments

Pedro Lobito · Accepted Answer · 2015-04-01 07:41:57Z

3

This will do the job:

import selenium
from selenium.webdriver.common.proxy import *

proxyHost = "my.proxy.host or IP"
proxyPort = "55555"

fp = webdriver.FirefoxProfile()
fp.set_preference("network.proxy.type", 1)
#fp.set_preference("network.proxy.http", proxyHost) #HTTP PROXY
#fp.set_preference("network.proxy.http_port", int(proxyPort))
#fp.set_preference("network.proxy.ssl", proxyHost) #SSL  PROXY
#fp.set_preference("network.proxy.ssl_port", int(proxyPort))
fp.set_preference('network.proxy.socks', proxyHost) #SOCKS PROXY
fp.set_preference('network.proxy.socks_port', int(proxyPort))
fp.update_preferences()

driver = webdriver.Firefox(firefox_profile=fp)

driver.get("http://www.whatismyip.com/")

answered Apr 1, 2015 at 7:41

Pedro Lobito

99.8k36 gold badges274 silver badges278 bronze badges

3 Comments

Umair Ayub Over a year ago

how to add user:pass

salvob Over a year ago

Important to note: if proxyHost is a hostname should not contain the "http://" prefix

Mpizos Dimitris Over a year ago

@UmairAyub have you found a solution for adding user:pass?

Alex · Accepted Answer · 2023-09-07 09:51:14Z

2

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.firefox.options import Options

proxy = "xxx.xx.xx.xxxx"
port = 8080

options = Options()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", proxy)
options.set_preference("network.proxy.http_port", port)
options.set_preference("network.proxy.https", proxy)
options.set_preference("network.proxy.https_port", port)

ser = Service(r"C:/Users/geckodriver.exe")
driver = webdriver.Firefox(service=ser, options=options)

answered Sep 7, 2023 at 9:51

Alex

558 bronze badges

Comments

chowmean · Accepted Answer · 2018-03-06 13:26:23Z

def install_proxy(PROXY_HOST,PROXY_PORT):
    fp = webdriver.FirefoxProfile()
    print PROXY_PORT
    print PROXY_HOST
    fp.set_preference("network.proxy.type", 1)
    fp.set_preference("network.proxy.http",PROXY_HOST)
    fp.set_preference("network.proxy.http_port",int(PROXY_PORT))
    fp.set_preference("network.proxy.https",PROXY_HOST)
    fp.set_preference("network.proxy.https_port",int(PROXY_PORT))
    fp.set_preference("network.proxy.ssl",PROXY_HOST)
    fp.set_preference("network.proxy.ssl_port",int(PROXY_PORT))  
    fp.set_preference("network.proxy.ftp",PROXY_HOST)
    fp.set_preference("network.proxy.ftp_port",int(PROXY_PORT))   
    fp.set_preference("network.proxy.socks",PROXY_HOST)
    fp.set_preference("network.proxy.socks_port",int(PROXY_PORT))   
    fp.set_preference("general.useragent.override","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A")
    fp.update_preferences()
    return webdriver.Firefox(firefox_profile=fp)

Collectives™ on Stack Overflow

Running selenium behind a proxy server

5 Answers 5

5 Comments

Comments

3 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

5 Comments

Comments

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related