0

I have a requirement which requires me to run 480 threads and 16 processes( each process will have 30 threads, so it's 480). I have heard that due to GIL, python does not have a good support for multi threading! Is there any way to execute the application with above requirements efficiently in python? Thanks in advance.

3
  • Why do you need so many threads in the first place? Commented Aug 8, 2015 at 11:40
  • Its complicated... I am setting up a POS(point of sale) simulator.... The code is written in C (server) I have to invoke the functions from python(client side) via sockets! So... Commented Aug 8, 2015 at 11:44
  • There's a good set of slides showing how the GIL interferes with threads. Commented Aug 8, 2015 at 13:44

2 Answers 2

2

It depends on your code and the type of the problem you are trying to solve. Python GIL applies to CPU-bound threads, i.e. threads that want to do CPU-intensive tasks*.

However, if your threads are I/O-bound, i.e. spend most of the time waiting for input/output, the GIL would not be a problem because while waiting for the I/O operation to complete, threads can do not need a lock on the Python interpreter. An example would be waiting for a network operation to complete (downloading a file, for example). You can easily fire multiple threads and download the files simultaneously.

*footnote: even for CPU-bound tasks, the GIL only applies to Python code. If a thread uses a C-extension (i.e. library written in C), it won't need the GIL while executing, because it won't be running Python instructions.

Edit: Reading your comment saying that you will be invoking the POS simulator functions via sockets... that's an I/O operation. While the server will be executing function calls, Python (the client) will simply wait doing nothing, and the threads will not need to hold GIL most of the time.

Sign up to request clarification or add additional context in comments.

Comments

0

This should give you an idea of how it's done.

import multiprocessing
import threading

def thread():
    pass


def process():
    threads = []
    for i in range(30):
        t = threading.Thread(target=thread)
        threads.append(t)
        t.start()
    for t in threads:
        t.join()

if __name__ == '__main__':
    processes = []
    for i in range(16):
        p = multiprocessing.Process(target=process)
        processes.append(p)
        p.start()
    for p in processes:
        p.join()

5 Comments

Unfortunately the operations are more CPU intensive than I/O and the code is in pure python! Is it good to go with python or choose some other language?
If you just need multithreading b/c of sockets, read this: docs.python.org/2/library/asyncore.html
Didn't you say the POS simulator is written in C and Python will only send requests to it via sockets? I sounds like it's the server that will do the actual heavy lifting? Please elaborate, thanks.
Yes. It is written in C (I.e. server)....the python code has to validate each received data against a predefined template file and it has to keep track of statistics, logging....so is it OK to go with python?
It might be doable, yes, but it depends on how computationally expensive the validation is. Also, depending on the requirements, statistics, logging, etc. do not have to be performed in the same thread that receives the data. The thread can perhaps just dump/log the collected data somewhere (again an I/O operation) and then some other process can periodically collect this data and calculate the statistics... ||| Just thinking of various possibilities, hope it gives you some ideas that will help you make your system better.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.