Stop multithreaded Python script on Windows

Question

I have troubles with a simple multithreaded Python looping program. It should loop infinitely and stop with Ctrl+C. Here is an implementation using threading:

from threading import Thread, Event
from time import sleep

stop = Event()

def loop():
    while not stop.is_set():
        print("looping")
        sleep(2)

try:
    thread = Thread(target=loop)
    thread.start()
    thread.join()

except KeyboardInterrupt:
    print("stopping")
    stop.set()

This MWE is extracted from a more complex code (obviously, I do not need multithreading to create an infinite loop).

It works as expected on Linux, but not on Windows: the Ctrl+C event is not intercepted and the loop continues infinitely. According to the Python Dev mailing list, the different behaviors are due to the way Ctrl+C is handled by the two OSs.

So, it appears that one cannot simply rely on Ctrl+C with threading on Windows. My question is: what are the other ways to stop a multithreaded Python script on this OS with Ctrl+C?

Well, obviously you can kill the task with taskkill (or a Python script using os or the third-party psutil, or Task Manager, Process Explorer, etc.). Is that not acceptable here? — abarnert
– abarnert, Commented Aug 21, 2018 at 18:29
This is correct, but I want to stop it with Ctrl+C (I updated the question). — Neraste
– Neraste, Commented Aug 21, 2018 at 18:32

abarnert · Accepted Answer · 2018-08-21 18:57:14Z

1

As explained by Nathaniel J. Smith in the link from your question, at least as of CPython 3.7, Ctrl-C cannot wake your main thread on Windows:

The end result is that on Windows, control-C almost never works to wake up a blocked Python process, with a few special exceptions where someone did the work to implement this. On Python 2 the only functions that have this implemented are time.sleep() and multiprocessing.Semaphore.acquire; on Python 3 there are a few more (you can grep the source for _PyOS_SigintEvent to find them), but Thread.join isn't one of them.

So, what can you do?

One option is to just not use Ctrl-C to kill your program, and instead use something that calls, e.g., TerminateProcess, such as the builtin taskkill tool, or a Python script using the os module. But you don't want that.

And obviously, waiting until they come up with a fix in Python 3.8 or 3.9 or never before you can Ctrl-C your program is not acceptable.

So, the only thing you can do is not block the main thread on Thread.join, or anything else non-interruptable.

The quick&dirty solution is to just poll join with a timeout:

while thread.is_alive():
    thread.join(0.2)

Now, your program is briefly interruptable while it's doing the while loop and calling is_alive, before going back to an uninterruptable sleep for another 200ms. Any Ctrl-C that comes in during that 200ms will just wait for you to process it, so that isn't a problem.

Except that 200ms is already long enough to be noticeable and maybe annoying.

And it may be too short as well as too long. Sure, it's not wasting much CPU to wake up every 200ms and execute a handful of Python bytecodes, but it's not nothing, and it's still getting a timeslice in the scheduler, and that may be enough to, e.g., keep a laptop from going into one of its long-term low-power modes.

The clean solution is to find another function to block on. As Nathaniel J. Smith says:

you can grep the source for _PyOS_SigintEvent to find them

But there may not be anything that fits very well. It's hard to imagine how you'd design your program to block on multiprocessing.Semaphore.acquire in a way that wouldn't be horribly confusing to the reader…

In that case, you might want to drag in the Win32 API directly, whether via PyWin32 or ctypes. Look at how functions like time.sleep and multiprocessing.Semaphore.acquire manage to be interruptible, block on whatever they're using, and have your thread signal whatever it is you're blocking on at exit.

If you're willing to use undocumented internals of CPython, it looks like, at least in 3.7, the hidden _winapi module has a wrapper function around WaitForMultipleObjects that appends the magic _PyOSSigintEvent for you when you're doing a wait-first rather than wait-all.

One of the things you can pass to WaitForMultipleObjects is a Win32 thread handle, which has the same effect as a join, although I'm not sure if there's an easy way to get the thread handle out of a Python thread.

Alternatively, you can manually create some kind of kernel sync object (I don't know the _winapi module very well, and I don't have a Windows system, so you'll probably have to read the source yourself, or at least help it in the interactive interpreter, to see what wrappers it offers), WaitForMultipleObjects on that, and have the thread signal it.

edited Aug 21, 2018 at 18:57

answered Aug 21, 2018 at 18:37

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

14 Comments

user4396006 Over a year ago

Isn't your while loop equivalent to not setting a timeout in join?

user4396006 Over a year ago

I mean, I don't get why is that it is interruptable between each time it does the loop

abarnert Over a year ago

@J.C.Rocamonde After 0.2 seconds, it gives up trying to join and returns None. Then we're just executing the bytecodes to return to the top of the loop, and calling thread.is_alive, so we're not blocking on anything, and we can be interrupted by the normal CPython code that checks for signals between bytecodes.

Neraste Over a year ago

The quick&dirty way is acceptable for me, as the program will always run on foreground. It is not perfect, I get it, but it is good enough. Digging in Python's Windows specific source code or using undocumented internals is too much for my taste. I am a bit puzzled by the fact that the Ctrl+C event will be held until the join timeout has expired.

Eryk Sun Over a year ago

As long as the thread is shut down before Python exits, the interpreter shouldn't try to join it. Whether or not there's a race condition here depends on the overall way the program orchestrates shutdown when Ctrl+C is handled. Like I said, I haven't actually experimented with scenarios that try to make Python hang in this case.

|

Collectives™ on Stack Overflow

Stop multithreaded Python script on Windows

1 Answer 1

14 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

14 Comments

Your Answer

Sign up or log in

Post as a guest

Related