I have a requirement which requires me to run 480 threads and 16 processes( each process will have 30 threads, so it's 480). I have heard that due to GIL, python does not have a good support for multi threading! Is there any way to execute the application with above requirements efficiently in python? Thanks in advance.
-
Why do you need so many threads in the first place?Cristian Ciupitu– Cristian Ciupitu2015-08-08 11:40:33 +00:00Commented Aug 8, 2015 at 11:40
-
Its complicated... I am setting up a POS(point of sale) simulator.... The code is written in C (server) I have to invoke the functions from python(client side) via sockets! So...abhay kanade– abhay kanade2015-08-08 11:44:14 +00:00Commented Aug 8, 2015 at 11:44
-
There's a good set of slides showing how the GIL interferes with threads.meuh– meuh2015-08-08 13:44:05 +00:00Commented Aug 8, 2015 at 13:44
2 Answers
It depends on your code and the type of the problem you are trying to solve. Python GIL applies to CPU-bound threads, i.e. threads that want to do CPU-intensive tasks*.
However, if your threads are I/O-bound, i.e. spend most of the time waiting for input/output, the GIL would not be a problem because while waiting for the I/O operation to complete, threads can do not need a lock on the Python interpreter. An example would be waiting for a network operation to complete (downloading a file, for example). You can easily fire multiple threads and download the files simultaneously.
*footnote: even for CPU-bound tasks, the GIL only applies to Python code. If a thread uses a C-extension (i.e. library written in C), it won't need the GIL while executing, because it won't be running Python instructions.
Edit: Reading your comment saying that you will be invoking the POS simulator functions via sockets... that's an I/O operation. While the server will be executing function calls, Python (the client) will simply wait doing nothing, and the threads will not need to hold GIL most of the time.
Comments
This should give you an idea of how it's done.
import multiprocessing
import threading
def thread():
pass
def process():
threads = []
for i in range(30):
t = threading.Thread(target=thread)
threads.append(t)
t.start()
for t in threads:
t.join()
if __name__ == '__main__':
processes = []
for i in range(16):
p = multiprocessing.Process(target=process)
processes.append(p)
p.start()
for p in processes:
p.join()