Proxy server for Selenium and Python

Question

I'm trying to use a proxy server to launch the Selenium Chrome driver. The only solution so far that I found was to use a sort of plugin for Chrome to authenticate, but it's not very reliable, so I was wondering if there is any other option.

Here is what I use now

    manifest_json = """
    {
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
    "proxy",
    "tabs",
    "unlimitedStorage",
    "storage",
    "<all_urls>",
    "webRequest",
    "webRequestBlocking"
    ],
    "background": {
    "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
    }
    """

background_js = """
    var config = {
    mode: "fixed_servers",
    rules: {
    singleProxy: {
    scheme: "http",
    host: "",
    port: parseInt(6060)
    },
    bypassList: ["foobar.com"]
    }
    };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
    return {
    authCredentials: {
    username: "",
    password: ""
    }
    };
    }

    chrome.webRequest.onAuthRequired.addListener(
    callbackFn,
    {urls: ["<all_urls>"]},
    ['blocking']
    );
    """


pluginfile = 'proxy_auth_plugin.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)

co = Options()
co.add_argument("--start-maximized")
co.add_extension(pluginfile)


driver = webdriver.Chrome(executable_path='/usr/local/bin/chromedriver')

Community · Accepted Answer · 2017-05-23 12:15:26Z

0

I don't think plugins are a very good solution, certainly not when there are pure HTTP solutions that work the same across all browsers, versions, and operating systems.

You could use this Python wrapper for Browsermob, which is a very well-known standalone or embedded proxy server written in Java.

The code shows how the headers() Python method can be used to add headers, without the need to POST to the REST API. The tests have some examples of it in use, e.g.:

self.client = Client("localhost:9090")
self.client.headers({'User-Agent': 'rubber ducks floating in a row'})
self.client.headers({'Authorization': 'Basic dXNlcjpwYXNz'})  # User 'user', password 'pass'

Update:

Just to clarify how the WebDriver and the proxy fit together:

Start your proxy server first, and wait until it's ready. You can do that externally and pass host:port to WebDriver, or start it embedded in your application then pass WebDriver a proxy object.
This example demonstrates the second approach, using Firefox profiles and Chrome options.
Alternatively, start the proxy embedded, use it to obtain a Proxy object, which represents a Selenium proxy server, then add it to your DesiredCapabilities object, and create your driver that way, as per this example.

From that point on, the proxy-listening is automatic, and you can start to create your HAR files.

Alternatively, you could see this answer for a custom Python proxy server using Twisted.

I have a longer answer about Selenium proxies in Python here.

edited May 23, 2017 at 12:15

CommunityBot

11 silver badge

answered Mar 14, 2016 at 23:59

Andrew Regan

5,1237 gold badges41 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Andrew Regan Over a year ago

Sorry, I missed out the link. Fixed in the answer now. github.com/AutomatedTester/browsermob-proxy-py

Christian Over a year ago

thx again. So this is the code for Chrome for example from

browsermobproxy import Server server = Server("path/to/browsermob-proxy") server.start() proxy = server.create_proxy()  chrome_options = webdriver.ChromeOptions() chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy)) browser = webdriver.Chrome(chrome_options = chrome_options)

Christian Over a year ago

does it also let you use a username and password? Thanks

Andrew Regan Over a year ago

Only time for a quick answer, but if you mean using a user/pass for basic authentication dialogs, yes. I cover that here: stackoverflow.com/q/35004026/954442

Christian Over a year ago

thanks. Trying to figure out now how to implement that POST call in Python.

|

Collectives™ on Stack Overflow

Proxy server for Selenium and Python

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related