Multi threading with shared variable

Question

I've got this program:

def optimal(t0, tf, frequences, delay, ratio = 0):

    First = True              
    for s in delay:
        delay = 0                           
        timelines = list()

        for i in range(len(frequences)):
            timelines.append(time_builder(frequences[i], t0+delay, tf))
            delay += s

        It takes in input an initial time t0, a final time tf, a list of 5 frequencies, and this `

If the value s meets the requirements, i.e valid = True and optim_param_1 < optimal[3], then optimal is replaced with optimal = (s, _overlap, __trio_overlap, optim_param_1).

The problem is that there are many possibilities for s to be tested and that the computation within is quite long. I would like to use several threads to speed up the process. I've never actually done it and I've been read there are limitations and algorithm design choices to make to avoid false results.

First question, is it possible to modify this algorithm into a multithread one? Second question, is from threading import Thread the right solution? Third question, since optimal is shared between threads (comparison to find out if the solution is better than the previous best one), should I create one optimal per thread and then compare which one of the optimal is the best, or is there another way?

Thanks for the tips, I really don't know where to start with multithreading....

EDIT: Some s value computation might be fast, and some quite long. There is not 2 with the same duration.

Sven Rusch · Accepted Answer · 2018-02-21 11:42:43Z

1

I currently do not have time to create an extended answer, but still would like to give you some hints.

Since your computations do not directly depend on each other, it is possible to parallelize the problem.
Using threading.Thread is not the correct solution, since it runs all threads in the same process. Use multiprocessing.Process instead (https://docs.python.org/3.6/library/multiprocessing.html). The interface is similar to that of threading.Thread.
There are at least two solutions I can imagine:
- Use one optimal variable per process to store the optimum values for that process. Your current program stops when optim_param_1 < 0.3 is satisfied. So you could have a shared multiprocessing.Event that is set when one process finds a solution with optim_param_1 < 0.3. All other threads can then stop themselves when the event is set.
- Share a multiprocessing.Queue between the worker processes and the main process. Whenever a worker process calculated a new result, put it in the queue. The main process should check for new results in the queue in a loop and compare the results to its optimal variable. When it found a solution that satisfies optim_param_1 < 0.3 it can stop all worker processes.

Personally I would choose the second solution. It separates the calculation of the results (in the worker process) from the valuation of the results (in the main process). You can easily scale and manage the whole thing using the multiprocessing.pool.Pool class.

The documentation on the multiprocessing module has some nice examples: https://docs.python.org/3.6/library/multiprocessing.html#examples

answered Feb 21, 2018 at 11:42

Sven Rusch

1,37211 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mathieu Over a year ago

Thanks. I'm gonna look into it as soon as I can. I understand the 2 solutions, although I have no idea of how implementing a loop checking when a variable is updated (the queue).

Mathieu Over a year ago

If you have time, could you have a look to the modification I brought ?stackoverflow.com/questions/48927506/…

Collectives™ on Stack Overflow

Multi threading with shared variable

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related