Selenium proxy with authentication

Question

I have to use selenium and proxy with authentication. I have a few constraints

I can't use selenium-wire (only pure selenium allowed)
I have to use headless mode (e.g. chrome_options.add_argument("--headless"))

I read this answer Python proxy authentication through Selenium chromedriver but it doesn't work for headless mode.

Is it possible to use proxy with authentication in my case? (browser(Chrome, Firefox) is not important)

I need python function to create selenium webdriver object authenticate to proxy

Does this answer your question? how to set proxy with authentication in selenium chromedriver python?. Answer link - stackoverflow.com/a/55582859/5372079 — Abhishek Dhoundiyal
– Abhishek Dhoundiyal, Commented Jul 30, 2021 at 5:22
No, because this does not work in headless mode, see comments!!! — swor
– swor, Commented Jul 30, 2021 at 16:33
yes, my bad because headless mode doesn't support extension. Can you try this solution stackoverflow.com/a/56276796/5372079 — Abhishek Dhoundiyal
– Abhishek Dhoundiyal, Commented Jul 31, 2021 at 11:50

CatChMeIfUCan · Accepted Answer · 2021-08-01 12:32:46Z

2

+100

you can't because you need a GUI to handle it with selenium in your case so I would recommend using a virtual display like Xvfb display server

You can use PyVirtualDisplay (a Python wrapper for Xvfb) to run headless.

for Linux

sudo apt-get install firefox xvfb

install virtual display for python

pip install pyvirtualdisplay

then

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

# now Firefox will run in a virtual display. 
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

in this case, you do not need to add options.add_argument("--headless") argument and you can follow the answers commented above as a solution or doing it your way but I think this is the best solution for using pure selenium for proxy

answered Aug 1, 2021 at 12:32

CatChMeIfUCan

5891 gold badge7 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

swor Over a year ago

Why do i need GUI to handle work with proxy? (for example i can use proxy without authentication in HEADLESS mode)

CatChMeIfUCan Over a year ago

because there is an alert/prompt that you should interact with...

CatChMeIfUCan Over a year ago

the easiest way is virtual display but there is another solution which is mitmproxy which is way harder and complicated that's why I recommended Xvfb

CatChMeIfUCan Over a year ago

in the mitmproxy case, you have to run a separate proxy without authentication and redirect it to the target proxy with authentication, as I said the Xvfb is easier and more logical, by the way, headless selenium itself does not support authentication, See Issue 718235 and Issue 706008

Joseph Bolton · Accepted Answer · 2024-09-19 09:41:23Z

It's the future, and there is a simpler solution now! (I'm using Chrome here)

pip install seleniumbase # tested on seleniumbase==4.30.8

import re
from types import SimpleNamespace

from seleniumbase import Driver

proxy_args = SimpleNamespace(
    username="your-proxy-server-username",
    password="your-proxy-server-password",
    port=6969,
    endpoint="your.proxy.server.domain",  # e.g. bright data's is "brd.superproxy.io",
)
proxy_args.auth_url = f"{proxy_args.username}:{proxy_args.password}@{proxy_args.endpoint}:{proxy_args.port}"

for _ in range(3):
    selenium_driver = Driver(uc=True, headless2=True, proxy=proxy_args.auth_url)
    selenium_driver.uc_open_with_reconnect("https://ipinfo.io/ip", 3)
    print(
        "your IP address is ",
        re.search(r"(\d+\.){3}\d+", selenium_driver.page_source).group(),
    )
    selenium_driver.close()

Collectives™ on Stack Overflow

Selenium proxy with authentication

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related