Run multiple web browsers simultaneously Selenium Python 3

Question

I'm attempting to open multiple chrome drivers at once and have it run as fast as possible. It opens the first page, and it has to load completely before the second function executes with "browser_2".
Is there a way to make these functions load at the same time?

notice - I'm hiding "Proxy_list" from my post to protect those ips for this post.

browser_1 = 0
browser_2 = 1
browser_3 = 2
browser_4 = 3
browser_5 = 4
browser_6 = 5
browser_7 = 6
browser_8 = 7
browser_9 = 8
browser_10 = 9
Link_1 = "https://www.google.com"

session_list = [browser_1, browser_2, browser_3, browser_4, browser_5, browser_6, browser_7, browser_8, browser_9, browser_10]



def create_browser(browser):
    options = webdriver.ChromeOptions()
    options.add_argument('--proxy-server=%s' % (Proxy_list[browser]))
    options.add_argument("start-maximized")
    options.add_argument("disable-infobars")
    options.add_argument("--disable-extensions")
    print("opening session #" + str(browser+ 1))
    # Defines Browser
    browser = webdriver.Chrome(options=options)
    browser.get(Link_1)
    return browser


create_browser(browser_1)
create_browser(browser_2)
create_browser(browser_3)
create_browser(browser_4)
create_browser(browser_5)

would you mind showing me how to implement that or a simple example? I'm a python noob sorry :) — William Rogers
– William Rogers, Commented Feb 6, 2020 at 17:54

Yun · Accepted Answer · 2020-02-10 07:41:26Z

2

Here is the sample code.

I have tried and it would open multiple browsers at same time.

from selenium import webdriver
from multiprocessing import Process, Pipe, Pool

def create_browser(num):
    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_argument("disable-infobars")
    options.add_argument("--disable-extensions")
    # Defines Browser
    browser = webdriver.Chrome(options=options)
    browser.get('https://mail.google.com/')

    return browser


pool = Pool(processes=10)  # Maximum number of browsers opened at same time

for i in range(0, 5):  # Five browsers will be created
    async_result = pool.apply_async(create_browser, args=(i))

pool.close()
pool.join()

Updated:

You can pass parameter like below.

def test_function(x, y, z=0):
    # do something

...

async_result = pool.apply_async(test_function, args=(1, 2), kwds={'z':3})  # x=1, y=2, z=3

In your case:

session_list = [browser_1, browser_2, browser_3, browser_4, browser_5, browser_6, browser_7, browser_8, browser_9, browser_10]

pool = Pool(processes=10)  # Maximum number of browsers opened at same time

for i in range(0, len(session_list)):  # Ten browsers will be created
    async_result = pool.apply_async(create_browser, args=(session_list[i]))

pool.close()
pool.join()

edited Feb 10, 2020 at 7:41

answered Feb 7, 2020 at 6:12

Yun

1,0528 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Austin Reed Over a year ago

But how would this apply to something in a for loop? Like if you have a huge for loop of links and you need each browser to complete the task without doing double work.

William Rogers Over a year ago

appreciate the response, how do I make it input the arguments for browser_1, browser_2 etc?

Yun Over a year ago

@WilliamRogers I have updated my answer. Hope it can help you :)

Collectives™ on Stack Overflow

Run multiple web browsers simultaneously Selenium Python 3

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related