1

I want to stop my program when the user presses ctrl-C. The following answer suggests catching the KeyboardInterrupt exception.

python: how to terminate a thread when main program ends

Sometimes it works. But in the following example, it stops working after I increase the number of threads from 25 to 30.

import threading, sys, signal, os

stderr_lock = threading.Lock()

def Log(module, msg):
    with stderr_lock:
        sys.stderr.write("%s: %s\n" % (module, msg))

class My_Thread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        Log("Init", "Initing.")
        self.start()
    def run(self):
        try:
            while True:
                Log("Run", "Running.")
        except KeyboardInterrupt:
            os._exit(0)

for i in range(30):
    My_Thread()

# trap ctrl-C in main thread
try:
    while True:
        pass
except KeyboardInterrupt:
    os._exit(0)

This has a very suspiciously similar feel to the following question:

Thread-Safe Signal API in Python 2.7

In that case, I was unable to catch signals after increasing the number of threads beyond 87.

Improve this question
4
  • This works fine for me. What OS are you using? Commented Sep 11, 2017 at 20:34
  • the KeyboardInterrupt in the threads is not needed Commented Sep 11, 2017 at 21:40
  • @JohanL Ubuntu14. Out of curiosity, does it still work if you increase the range(30) to range(1000)? Commented Sep 12, 2017 at 16:21
  • At 1000 threads I had the same problem as you. See my answer to why and what you could do about it. Commented Sep 13, 2017 at 18:23

2 Answers 2

Reset to default
1

There are actually two different issues with your code that gives this behavior. The first is that your threads should be made into daemon threads, so that they automatically stops when the main thread exits, the second is that your try block does not encapsulate the thread creation and start-up.

When you create a number of threads, the thread creation won't be finished for quite a while (since it is continuously interrupted by the created threads and the GIL prevents them to run in parallel). Therefore, you send your KeyboardInterrupt before being set up to be handled. However, the KeyboardInterrupt will still kill the main thread (with a Traceback), but not the child threads.

Thus, your code works if you modify it as:

import threading, sys, signal, os

stderr_lock = threading.Lock()

def Log(module, msg):
    with stderr_lock:
        sys.stderr.write("%s: %s\n" % (module, msg))

class My_Thread(threading.Thread):
    def __init__(self, value):
        threading.Thread.__init__(self)
        self.value = value
        Log("Init", "Initing %d." % self.value)
        self.daemon = True
        self.start()
    def run(self):
        while True:
            Log("Run", "Running %d." % self.value)

# trap ctrl-C in main thread
try:
    for i in range(1000):
        My_Thread(i)

    while True:
        pass
except KeyboardInterrupt:
    os._exit(0)

Note, that making the threads into daemons is not strictly necessary in the current example, but I would consider that to be good practice for threads that are supposed to end when the main program ends.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

"When you create a number of threads, the thread creation won't be finished for quite a while (since it is continuously interrupted by the created threads and the GIL prevents them to run in parallel)." Bingo! I never would have thought of that. Wow, in all my experience in other languages, I have never run into one with such a bad scheduler (even on single-processor systems). Great answer. Yes, now it works. Thank you.
Or, if I had just been patient enough to let all the threads get created, my original example would have worked.... but, as I said, I never would have thought of that.
The daemon=True is even better. Somehow I thought that wasn't supported in pre-3.0 Python. I was wrong! Thank you for pointing this out. Actually I wasn't the first one to be confused about pre-3.0 daemon threads: github.com/dask/distributed/issues/18. So apparently Python 3.0 added daemon as an argument to Thread and then people started writing code that wasn't doing it the backward-compatible way.
The daemon=True has been available since Python 2.6 and before that you could use setDaemon instead, so I think the page you are linking to, had another type of issue, but I don't know.
As far as the GIL is concerned, it is an annoying implementation detail, but it is not only Python that has it. E.g. Ruby has a similar architecture. Threads works for code that releases the GIL (typically numpy) and to simplify code that is mostly waiting (typically sockets). For processor bound code that needs to be run faster, consider to use multiprocessing instead. As long as the processes don't communicate or share state, that is rather straight-forward. If you need to share data, it is still possible but requires a bit more of work, typically using queues.
|
0

You may want to read https://stackoverflow.com/a/35430500/1656850, namely:

There are 3 exit functions, in addition to raising SystemExit.

The underlying one is os._exit, which requires 1 int argument, and exits immediately with no cleanup. It's unlikely you'll ever want to touch this one, but it is there.

sys.exit is defined in sysmodule.c and just runs PyErr_SetObject(PyExc_SystemExit, exit_code);, which is effectively the same as directly raising SystemExit. In fine detail, raising SystemExit is probably faster, since sys.exit requires an LOAD_ATTR and CALL_FUNCTION vs RAISE_VARARGS opcalls. Also, raise SystemExit produces slightly smaller bytecode (4bytes less), (1 byte extra if you use from sys import exit since sys.exit is expected to return None, so includes an extra POP_TOP).

The last exit function is defined in site.py, and aliased to exit or quit in the REPL. It's actually an instance of the Quitter class (so it can have a custom repr, so is probably the slowest running. Also, it closes sys.stdin prior to raising SystemExit, so it's recommended for use only in the REPL.

As for how SystemExit is handled, it eventually causes the VM to call os._exit, but before that, it does some cleanup. It also runs atexit._run_exitfuncs() which runs any callbacks registered via the atexit module. Calling os._exit directly bypasses the atexit step.

so, raise SystemExit may be the preferable way to exit when the exception is caught.

Improve this answer

3 Comments

I tried it with raise SystemExit instead of os._exit(0). That made it much worse. With raise SystemExit then the program only works with 1 thread (the main thread only). At least with os._exit(0) I can get 25 threads.
Can you add an edit to your question showing what you did when you replaced os._exit(0) with raise SystemExit? It seems inconceivable that a 1-to-1 exchange of them would change the number of allowable threads, but if you say it does - then it'd be interesting to test your code in my environment to verify this strange phenomenon.
Why is it strange for raise SystemExit to behave any differently from raise KeyboardInterrupt? Given that the latter is broken (otherwise this whole question wouldn't exist) I am not surprised about the former being broken too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.