How to handle out-of-memory error while using threads in python

Question

I have memory leak, but I can't find a way to solve it. I think the reason is for that because I use threads and don't stop/kill it in a right way.

I have following method:

import threading
def worker():
     if nextJobActive() and number_of_active_threads<5:
         t = threading.Thread(target=startThread, args=(my_list, my_item))
         t.start() 

def startThread(): 
    #do something here, which takes ~15 Min.

I run the worker() method in while(true) loop. I always have to start new threads in my case. But I never stop a thread. I also don't know how to do this. Is there anyway to safely stop a thread in my case?

VRComp · Accepted Answer · 2021-06-08 22:33:17Z

1

As you know already, you are creating an endless amount of threads without properly stopping the previous one. To wait for a thread to terminate there is a .join() method. Here is the documentation for the Thread module: docs.

import threading
def worker():
     if nextJobActive() and number_of_active_threads<5:
         t = threading.Thread(target=startThread, args=(my_list, my_item))
         t.start()
         t.join() 

def startThread(): 
    #do something here, which takes ~15 Min.

answered Jun 8, 2021 at 22:33

VRComp

1311 gold badge1 silver badge12 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Cenk Ten Over a year ago

This means, new threads will always be started and the old ones will be terminated when their job is done?

Cenk Ten Over a year ago

I just tested it; without join it runs always 5 threads parallel, but with join it runs at first 5 but then only 1 thread. Any thoughts why?

VRComp Over a year ago

The one thread is the main thread that is running. The first five threads execute then they each waits to be terminated then the main program continues running as the main thread and only thread. If you plan to achieve multithreading then you can use lock in the threading module or concurrent.futures module (docs.python.org/3/library/concurrent.futures.html)

Cenk Ten Over a year ago

Thank you for the answer. In my case I have to use threading, replacing it with concurrent will cause a big overhead. What if I kill the thread when its job is done manually? Or thoughts if it will work for me?

VRComp Over a year ago

Your main program is your main thread. Since your main program never stops calling return won't help. If you can handle all exceptions and know which threads finish their work then killing a thread should work.

|

Collectives™ on Stack Overflow

How to handle out-of-memory error while using threads in python

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related