Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Signals and Threads in Python 3.5 or so

50 views
Skip to first unread message

Dan Stromberg

unread,
Oct 9, 2016, 7:52:25 PM10/9/16
to
I have a program http://stromberg.dnsalias.org/~dstromberg/looper/
that I use and maintain.

It's like GNU parallel or similar - yet another "run n processes, m at
a time" implementation. Interestingly, I've only used/tested it on
Linux, but it's under a Microsoft copyright because Microsoft acquired
a Linux company I was working for.

Anyway, it seems to work well, except one annoying bug.

That bug is: if you control-C the top-level process, all the
subprocesses are left running.

I've been thinking about making it catch SIGINT, SIGTERM and SIGHUP,
and having it SIGKILL its active subprocesses upon receiving one of
these signals.

However, it's multithreaded, and I've heard that in CPython, threads
and signals don't mix well.

Is this still an issue in CPython 3.5? If yes, how can I work around it?

Thanks!

Chris Angelico

unread,
Oct 9, 2016, 8:45:39 PM10/9/16
to
On Mon, Oct 10, 2016 at 10:52 AM, Dan Stromberg <drsa...@gmail.com> wrote:
> That bug is: if you control-C the top-level process, all the
> subprocesses are left running.
>
> I've been thinking about making it catch SIGINT, SIGTERM and SIGHUP,
> and having it SIGKILL its active subprocesses upon receiving one of
> these signals.
>
> However, it's multithreaded, and I've heard that in CPython, threads
> and signals don't mix well.

Generally, expect SIGINT to be handled by the main thread, and code
accordingly. But I've never used the low-level thread and _thread
modules, so I'd be inclined to follow through with the TODO and make
it use threading instead.

This simple example appears to work fine on my system (Debian Stretch,
Linux 4.6, amd64), but I can't make any hard-and-fast guarantees.

import threading
import signal
import random
import time

halt = False

def thread():
global halt
try:
print(threading.current_thread(), "-- start")
for i in range(random.randrange(10, 20)):
time.sleep(1)
if halt: break
print(threading.current_thread(), "--", i)
except KeyboardInterrupt:
print(threading.current_thread(), "-- SIGINT")
halt = True
print(threading.current_thread(), "-- end")

for i in range(5):
threading.Thread(target=thread).start()
thread()


Hit Ctrl-C and all threads will quickly terminate.

ChrisA

Paul Rubin

unread,
Oct 9, 2016, 8:57:16 PM10/9/16
to
Dan Stromberg <drsa...@gmail.com> writes:
> That bug is: if you control-C the top-level process, all the
> subprocesses are left running.

Are you setting the daemon flag?

Marko Rauhamaa

unread,
Oct 10, 2016, 2:27:47 AM10/10/16
to
Dan Stromberg <drsa...@gmail.com>:
> That bug is: if you control-C the top-level process, all the
> subprocesses are left running.

Sorry, don't have a solution for your particular Python situation.

> I've been thinking about making it catch SIGINT, SIGTERM and SIGHUP,
> and having it SIGKILL its active subprocesses upon receiving one of
> these signals.

SIGKILL is probably a bad idea. It doesn't give the subprocess any
opportunity for a graceful exit. For example, Python's try/finally
blocks will fail to work as advertised. Better use a catchable signal.

> However, it's multithreaded, and I've heard that in CPython, threads
> and signals don't mix well.

Python does confuse matters, but both threads and signals are
problematic entities under Linux. You need to be very well versed in the
semantics of both operating system concepts (man 7 pthreads, man 7
signal).


Marko

dieter

unread,
Oct 10, 2016, 3:06:02 AM10/10/16
to
Dan Stromberg <drsa...@gmail.com> writes:
> I have a program http://stromberg.dnsalias.org/~dstromberg/looper/
> that I use and maintain.
>
> It's like GNU parallel or similar - yet another "run n processes, m at
> a time" implementation. Interestingly, I've only used/tested it on
> Linux, but it's under a Microsoft copyright because Microsoft acquired
> a Linux company I was working for.
>
> Anyway, it seems to work well, except one annoying bug.
>
> That bug is: if you control-C the top-level process, all the
> subprocesses are left running.
>
> I've been thinking about making it catch SIGINT, SIGTERM and SIGHUP,
> and having it SIGKILL its active subprocesses upon receiving one of
> these signals.
>
> However, it's multithreaded, and I've heard that in CPython, threads
> and signals don't mix well.

I would not state it this way.

Signals are delivered to processes (not process threads).
That leads to the question which thread (in a multi thread application)
will handle a signal when it arrives.
In Python, the design decision has been not to use the currently
running thread (which may not be a Python thread at all or
may have executed Python code but at the moment runs C code outside
of Python) but let the signal be handled deterministically by
the main thread (once it starts again to execute Python code).
This looks like a sane decision to me.


Your problem description indicates that the problem is not
with signal handling by threads but with signal delivery by
the operating system.

In Linux, SIGINT signals in response to a "CTRL-C" are delivered
to the process group of the "control terminal".
Thus, with appropriate Linux calls, you can control whether or not
such a SIGINT is delivered to a subprocess.

Other operating systems may not provide such a control or it may
need different (OS dependent) calls.

Dan Stromberg

unread,
Oct 24, 2016, 4:40:36 PM10/24/16
to
Thank you for the clear explanation.
0 new messages