Multithreading and Arrays in Python

Question

I'm currently doing some changes in an existing script, but the thing is, I have no experience with multithreading and the more examples that I see, the more confused I get.

Here is a little insight on what I'm trying to achieve.

The script gets 2 inputs, the service and the action, the service can use * to look for all of the services running, which causes the service to become a list stored into an array.

Now for the good part, there is a function that based on the service and the action, it performs something, i.e.

perform_service_action(service1, stop)

This will cause the service1 to stop and so on.

Unfortunately this process was configured to run sequentially and the script takes too long to process, so what I'm wondering is if any of you guys have any tips on how I should approach this, here is a small part of the code:

def main(argv):
    if len(argv) == 1:
        if argv[0] == 'list':
            list_of_serv = get_list(None)
            for service in list_of_serv:
                if not service == "":
                    print service
        else:
            help()
    elif len(argv) == 2:
        service_arg = argv[0]
        action = argv[1]
        list_of_serv = get_list(service_arg)
        for service in list_of_serv:
                perform_service_action(service, action)
        print_service_output()
    else:
        help()

I've tried a few things here and there but the more I try, the more I get confused with the threading topic. Is this a matter of just changing the main part of the code to run in threads the same function with different values?, or is it a matter of changing the functions from scratch, meaning the function that gets the list and the one that performs the action on the services.

I hope anyone is able to give some pointers on this.

Thanks in advance.

What happens inside those functions? Is it I/O heavy or CPU heavy? — Pruthvi Kumar
– Pruthvi Kumar, Commented Jul 24, 2018 at 2:10
Array and list have very specific meanings in Python, and I'm not sure if you're using them correctly throughout. — Mad Physicist
– Mad Physicist, Commented Jul 24, 2018 at 2:38
Basically it's just a list of names of the services residing in the server, it's just the first value that gets fed to the function perform_service_action. — Isael Macias
– Isael Macias, Commented Jul 24, 2018 at 18:56
Also it is I/O intensive, it basically starts, stops or restarts services, reads out the pid and the uptime, writes it down in a defined format and prints the result. i.e. feeding service1 stop will give a stop request to service1, read its current pid and uptime, print the status stopped after the service stops and show something like this: service pid status uptime service1 1111 stopped 00:00:00 If we feed * to the script, that will get a list of the existing services in the server and feed it to list_of_serv — Isael Macias
– Isael Macias, Commented Jul 24, 2018 at 19:05

skarchmit · Accepted Answer · 2018-07-24 02:30:30Z

3

You can change your code to be instead of

 for service in list_of_serv:

    perform_service_action(service, action)

To be :

 from multiprocessing import Pool
 from multiprocessing.dummy import Pool as ThreadPool

 pool.map(perform_service_action, zip( [service],[action]*len(service) ))

 pool.close()
 pool.join()

What’s happening is that you can create a pool or processes that are mapped to the function

  perform_service_action

And then passing the iterable (the lists) for the function to process

If you need the result from the function you can also

result = pool.map(perform_service_action, zip( [service],[action]*len(service) ))

print (result )

which gives you a list of all the results.

That should help with speed, but that will also depend on having all of user input or just data.

edited Jul 24, 2018 at 2:30

answered Jul 24, 2018 at 2:24

skarchmit

4576 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Isael Macias Over a year ago

So, the thing about this solution is that the only array here is the service, which can be passed down to the function using the zip, but to be sending the exact same value in action it should not be in the zip, right? ... I'm asking because I'm totally new to Python, still not used to the syntax and each time i try the code, it throws an error mentioning that I'm only passing down 1 argument instead of the 2 it takes.

skarchmit Over a year ago

@IsaelMacias I had to make a list of the same string because it was part of an argument that you had to put in your function. If the argument would always be the same, you can put the variable elsewhere, like a global. In that case, if you were only passing one variable, it would only have to be an iterable (a list is one of them), which you already have, so no zip would be required.

Pruthvi Kumar · Accepted Answer · 2018-07-24 02:19:17Z

You are better choosing multithreading for I/O heavy operations and multiProcessing for CPU heavy operations. So, depending on what perform_service_action does, choose one over other.

Since your question does not provide clarity on type of operation, i will assume its I/O heavy. Inside Python gevents is my goto library for concurrency.

In your main(), change this:

for service in list_of_serv:
            perform_service_action(service, action)

to

jobs = [gevent.spawn(perform_service_action, params) for service in list_of_serv]

This will spawn multiple threads (precisely greenlets). To execute them concurrently,

gevent.joinall(jobs, timeout=2)

You can access the results for each job using:

[job.value for job in jobs]

** NOTE that your arguments during spawning greenlets will need unpacking within perform_service_action method. (you could use *args or *kwargs depending on how you want to implement it)

Isael Macias · Accepted Answer · 2018-07-24 20:28:26Z

So, at the end i went ahead and did the change with multithreading instead of multiprocessing, the solution went as follows:

import threading



# For parallel execution
thread_limiter = threading.BoundedSemaphore(5)
thread_lock = threading.Lock()
thread_list = []

def main(argv):
if len(argv) == 1:
    if argv[0] == 'list':
        list_of_serv = get_list(None)
        for service in list_of_serv:
            if not service == "":
                print service
    else:
        help()
elif len(argv) == 2:
    service_arg = argv[0]
    action = argv[1]
    list_of_serv = get_list(service_arg)
    for service in list_of_serv:
        thread_limiter.acquire()
        thread = threading.Thread(target=perform_service_action, args=(service, action))
        thread.start()
        thread_list.append(thread)
        thread_limiter.release()
    [thread.join() for thread in thread_list]
    print_service_output()
else:
    help()

To be honest i complicated things more than i should have, but for anyone out there having a similar issue, please feel free to use this as an example.

Thanks and regards to everyone who helped me in this post.

Collectives™ on Stack Overflow

Multithreading and Arrays in Python

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related