Configuring proxies on Selenium + Python3 + Chrome WebDriver + Docker

Question

I can't connect using proxies using Selenium Chrome WebDriver on Python 3.7

1 - When using a proxy (adding -proxy-server=%s on chrome_options)

def selenium_connect():
    PROXY = "66.97.38.58:80"
    url = "http://whatsmyip.org"
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('--headless')
    chrome_options.add_argument('--no-sandbox')
    chrome_options.add_argument('--disable-dev-shm-usage')
    chrome_options.add_argument('--proxy-server=%s' % PROXY)
    driver = webdriver.Chrome(chrome_options=chrome_options)
    driver.get(url)

In this case I get an empty result.

2 - I have tried this alternative: How do you run headless chrome and a proxy using selenium in python?

def selenium_connect():
    PROXY = "66.97.38.58:80"
    url = "http://whatsmyip.org"
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('--headless')
    chrome_options.add_argument('--no-sandbox')
    chrome_options.add_argument('--disable-dev-shm-usage')

    desired_caps = chrome_options.to_capabilities()
    prox = Proxy()
    prox.proxy_type = ProxyType.MANUAL
    prox.http_proxy = PROXY
    prox.add_to_capabilities(desired_caps)

    driver = webdriver.Chrome(chrome_options=chrome_options, desired_capabilities=desired_caps)
    driver.get(url)

I this case, the connection is routed via my local ip and not the proxy's ip. I've uploaded the function in docker here:

https://github.com/gerimo/challenge

I'm using a standard ubuntu Docker environment, selenium-3.141.0, chromium-chromedriver 68.0

The proxy complains that the remote ended the connection. What log do you have on that remote ? Have you tried with other targets than http://whatsmyip.org? — Thomasleveil
– Thomasleveil, Commented Mar 17, 2019 at 16:13
yes, I've tried different urls :( If I comment this line: chrome_options.add_argument('--proxy-server=%s' % PROXY) it connects OK. — Rimo
– Rimo, Commented Mar 17, 2019 at 16:18
Yes, I've tried with luminati and a bunch different proxies from this resource: free-proxy-list.net — Rimo
– Rimo, Commented Mar 17, 2019 at 16:32

Amundeep Singh Dhaliwal · Accepted Answer · 2020-12-02 19:38:49Z

This is not the shortest of solutions, but I thought it might be beneficial for someone. I managed to use a remote webdriver connected to the selenium hub with a luminati proxy in a docker container:

from selenium import webdriver
from zipfile import ZipFile

PROXY_HOST = 'servercountry-gb.zproxy.luminati.io'
PROXY_PORT = 22225
PROXY_USER = 'lum-customer-####-zone-####-country-gb-dns-remote'
PROXY_PASS = '#####'

chrome_options = webdriver.ChromeOptions()

manifest_json = """
    {
        "version": "1.0.0",
        "manifest_version": 2,
        "name": "Chrome Proxy",
        "permissions": [
            "proxy",
            "tabs",
            "unlimitedStorage",
            "storage",
            "<all_urls>",
            "webRequest",
            "webRequestBlocking"
        ],
        "background": {
            "scripts": ["background.js"]
        },
        "minimum_chrome_version":"50.0.0"
    }
    """
background_js = """
    var config = {
            mode: "fixed_servers",
            rules: {
            singleProxy: {
                scheme: "http",
                host: "%s",
                port: parseInt(%s)
            },
            bypassList: ["localhost"]
            }
        };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
                username: "%s",
                password: "%s"
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
                callbackFn,
                {urls: ["<all_urls>"]},
                ['blocking']
    );
    """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)

pluginfile = 'proxy_auth_plugin.zip'

with ZipFile(pluginfile, 'w') as zp:
    zp.writestr('manifest.json', manifest_json)
    zp.writestr('background.js', background_js)
chrome_options.add_extension(pluginfile)

Once we have authorised ourselves we could then use this somewhere in our code:

with webdriver.Remote(options= chrome_options, command_executor = 'http://selenium-hub:4444/wd/hub') as driver:
    driver.get('http://ipconfig.me')
    # whatever you need to do

I have encountered a strange issue where I cannot use headless chrome but non-headless chrome seemed to be fine. The remote webdriver was headless in the docker container. (python 3.7.4)

Collectives™ on Stack Overflow

Configuring proxies on Selenium + Python3 + Chrome WebDriver + Docker

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related