How to use multithreading in a function in python

Question

I have a loop that takes 21 minutes to run because of for loop. It is not complicated loop but I need to iterate all that many times.

My code looks like this

results=[]
for i in range(len(files)) ### around 5000 files with 96 rows each that evaluates
    results_f= function(arg1[i], arg2)
    results= results.append(results_f)

So how can I make that with multi threading?

I have tried something like

with concurrent.futures.ThreadPoolExecutor() as executor:    
    for i in range(len(files)):
        results = executor.map(function, [arg1[i],arg2])

that I saw working somewhere but it is not working at all

There's a syntax error in your code which makes this a really bad example. Also, you're indexing arg1 with an index valid for files. Further, "not working at all" is not a problem description. Please try out any actual example code that works and then adapt it to your case. If that causes problems, be specific! At the moment, your Q could be summarized "please teach me multithreading", which is not a valid topic here. — Ulrich Eckhardt
– Ulrich Eckhardt, Commented Mar 17, 2022 at 8:12

Albert Badretdinov · Accepted Answer · 2022-03-17 05:46:27Z

1

Not sure that it will help you, but that is what I came up with by myself.

First of all executor.map(funct, *iterable, ...) takes an iterables as input. It means you need to pass to lists with the same len() as arguments, since it splits this iterables into chunks and passes them to function as arguments. First iterable will be the first argument on each call function, second will be the second =).

Secondly, since it splits inputs into chunks you can get rid of the for loop.

And finally, it will return the generator. What to do with it decide by yourself.

with ThreadPoolExecutor(max_workers=4) as executor:
     results = executor.map(function_name, arg1, [arg2 for i in range(len(arg1)))

list(results)

Link to the documentation.

answered Mar 17, 2022 at 5:46

Albert Badretdinov

1461 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alexjandro Manjon Over a year ago

Thanks for your answer but results is not the result from the loop I wrote in the question, it is as you said a generator but I don't know how to turn that into my numerical result

Albert Badretdinov Over a year ago

list(results) in my example forces it to run. Also, you may want to use ProcessPoolExecutor to really have parallelism

Collectives™ on Stack Overflow

How to use multithreading in a function in python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related