Python Selenium multiple threads wont work

Question

I worked with selenium in python. I wanted to make the program multithreaded (4-10 threads) but it wont work. I don't know why.

When I start: Output: Thread started!

What should happen: Output: Thread started! Thread started! Thread started! Thread started! Thread started! ...

Here is what I tried:

from selenium import webdriver
import string
import random
import time
from threading import Thread


def bot():
    while True:
        options = webdriver.ChromeOptions()
        options.add_experimental_option('excludeSwitches', ['enable-logging'])  # Disables logging
        options.add_argument('--disable-extensions')
        options.add_argument('--profile-directory=Default')
        options.add_argument("--incognito")
        options.add_argument("--disable-plugins-discovery");
        options.add_argument("--headless") # Turns Chrome into headless browser
        options.add_argument("--mute-audio")
        driver = webdriver.Chrome(options=options)
        driver.get("https://example.com/")
        driver.close()


def main():
    for _ in range(10):
        print("Thread started!")
        worker = Thread(target=bot())
        worker.setDaemon(True)
        worker.start()


if __name__ == '__main__':
    main()

Booboo · Accepted Answer · 2021-01-08 21:14:23Z

You have three problems. First, you said:

worker = Thread(target=bot())

It should be:

worker = Thread(target=bot)

The second problem is the following line:

worker.setDaemon(True)

Can you say why you did this? A daemon thread is one that will automatically terminate as soon as all the non-daemon threads terminate. You have one non-daemon thread, namely the main thread that is launching all these daemon threads in function main(). As soon as main returns, which is immediately after starting all these dameon threads, it terminates! And thus all the daemon threads terminate with it. This is what you need:

def main():
    threads = [Thread(target=bot) for _ in range(10)]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join() # wait for this thread to terminate

The third problem is in function bot:

driver.close()

should be:

driver.quit()

To not only close the window but to also terminate the process and allow the thread to terminate.

sboomi · Accepted Answer · 2021-01-08 21:03:36Z

1

Have you tried to remove the while True in your bot() function? Threading means the main program executes your secondary function multiple times.

answered Jan 8, 2021 at 21:03

sboomi

111 silver badge3 bronze badges

4 Comments

Yariya Over a year ago

Hello, thanks for the answer. Yes but I need it in loop. Is there any way to get this work?

Yariya Over a year ago

And yes I tried it without loop also. But it's waiting until the process is finished. After process is finished the next thread gets started. I need to start all 10 threads at the same time.

sboomi Over a year ago

You have to define a certain number of loops or a break condition to make it work or else the first thread won't end. If you want to launch them in parallel, you can try multiprocessing in Python instead.docs.python.org/3/library/concurrent.futures.html

undetected Selenium Over a year ago

@sboomi ...If you want to launch them in parallel, you can try multiprocessing... great observation.

Collectives™ on Stack Overflow

Python Selenium multiple threads wont work

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related