A user has reported a bug in my posix_ipc module. When a Python app is  
waiting to acquire an IPC semaphore and the user hits Ctrl-C, my code  
should return a custom error indicating that the semaphore wait was  
interrupted by a signal.

However, the caller never sees the error I set. Instead they get a  
KeyboardInterrupt that refuses to be caught by try/except. Here's a  
sample program that demonstrates the problem when run from the command  
line:

# -----------------------------------
import posix_ipc

sem = posix_ipc.Semaphore(None, posix_ipc.O_CREX)

try:
    sem.acquire()   # User hits Ctrl + C while this is waiting
except:
    print "********* I caught it!"

sem.close()
sem.unlink()
# -----------------------------------

I expected that code to raise a posix_ipc.Error with the text, "The  
wait was interrupted by a signal" which would then be trapped by the  
except statement which would print the "I caught it!" message.

Instead a KeyboardInterrupt error is propagated up to the interpreter  
and the process is killed as if the try/except wasn't there at all.

I have verified that the C function sem_wait() returns -1 (failure),  
that errno is set to EINTR and that my detects that properly. So far,  
so good. PyErr_Occurred() returns NULL at that point. So my code calls  
PyErr_SetString() to set a custom error for the caller and returns  
NULL. It's apparently at some point after that that the  
KeyboardInterrupt error is being set.

If I substitute my sysv_ipc module for posix_ipc (very similar to  
posix_ipc but uses Sys V semaphores instead of POSIX), I get the same  
behavior.

I see this w/Python 2.5 under OS X and also w/Python 2.5 under Ubuntu  
8.0.4.

If anyone wants to look at my C code, the relevant case statement is  
on line 555 of posix_ipc_module.c.

http://semanchuk.com/philip/posix_ipc/
http://semanchuk.com/philip/sysv_ipc/

Any suggestions would be appreciated.

So there can be a short delay between the Ctrl-C signal
being received and KeyboardInterrupt being raised, and it
seems that this delay results in it happening after the
try-except has exited.

You could try using signal.signal() to install a handler
for Ctrl-C that does nothing in a section around the
sem.acquire call(). That should prevent the KeyboardInterrupt
flag from being set, but the signal will still be occurring
at the Unix level, so the system call will get interrupted.

There are actually two exceptions occurring: One is the Ctrl-C, which as
you correctly say, will probably delayed until there is a `check' in the
interpreter (see also David Beazley's wonderful presentation on the
GIL). The other one is the exception that is generated in the IPC code
by returning a NULL. This one should caught by the except clause.

As the call to sem.acquire releases and reacquires the GIL, I think
signal processing will be done immediately. This causes the
KeyboardInterrupt exception to occur immediately i.e. to interrupt the
handling of the other exception.

import posix_ipc
import signal

sem = posix_ipc.Semaphore(None, posix_ipc.O_CREX)
signal.signal(signal.SIGINT, lambda sig, frame: None)
status = []
try:
    status.append("Trying")
    sem.acquire()   # User hits Ctrl + C while this is waiting
    status.append("Acquired")
except:
    status.append("I caught it!")
print status

sem.close()
sem.unlink()

prints: ['Trying', 'I caught it!']

I also tried some other variants, catching the KeyboardInterrupt at
various places:

This one prints: ['Trying', 'Keyboard Interrupt']
This suggests to me that the first exception handling is aborted by the
Ctrl-C handling.

import posix_ipc

sem = posix_ipc.Semaphore(None, posix_ipc.O_CREX)

status = []
try:
    try:
        status.append("Trying")
        sem.acquire()   # User hits Ctrl + C while this is waiting
        status.append("Acquired")
    except:
        status.append("I caught it!")
except KeyboardInterrupt:
    status.append("Keyboard Interrupt")
print status

sem.close()
sem.unlink()

And this one prints: ['Trying', 'I caught it!']

import posix_ipc

sem = posix_ipc.Semaphore(None, posix_ipc.O_CREX)

status = []
try:
    status.append("Trying")
    try:
        sem.acquire()   # User hits Ctrl + C while this is waiting
        status.append("Acquired")
    except KeyboardInterrupt:
        status.append("Interrupt")
except:
    status.append("I caught it!")
print status

sem.close()
sem.unlink()

I was actually a bit surprised that the addition of the try/except
KeyboardInterrupt helps solve the problem but that apparently the
exception handler is not executed.

Folding the two try's into one with two except clauses will not help as
there are indeed two exceptions to be handled.

I also added traceback printout in the outer exception handler and it
points to the sem.acquire line.

My conclusion is that if there are two exceptions at the same time, the
inner exception handler is interrupted by the other exception even
before the except clause can be entered. And only the outer one is
really executed. This explains the behaviour that the OP described.

So I looked in the source code how they did it:    
The code is in Parser/myreadline.c.

On Jun 20, 2009, at 7:41 AM, Piet van Oostrum wrote:

Piet, your research into raw_input() gave me a solution.

In my C code, I added a call to PyErr_CheckSignals() as the first line  
inside the "case EINTR". Apparently calling PyErr_CheckSignals() sets  
the Python error indicator -- PyErr_Occurred() returns true and the  
error is PyExc_KeyboardInterrupt. I'm able to clear that error and  
return my own.

Prior to adding the call to PyErr_CheckSignals(), PyErr_Occurred()  
returned false, so my code had no error to clear.

Best of all, PyErr_CheckSignals() doesn't interfere with a Python-
level signal handler if one is set.

This worked under OS X and Linux.

I'll have to think some more about exactly how I want my module to  
report the Ctrl-C, but at least now I have the tools to control the  
situation.

Thanks again for your invaluable assistance. If I'm ever in NL or NZ  
I'll buy you a beer. =)

For C code, the solution you give sounds like a
good one.

My only misgiving is that the user might expect to
get a KeyboardInterrupt in response to Ctrl-C, so
it might be better to just let it propagate instead
of turning it into a different exception.

On Jun 20, 2009, at 10:21 PM, greg wrote:

When my C code sees EINTR, I will probably raise KeyboardError when I  
see that and add a specific posix_ipc.SignalError to raise in other  
EINTR circumstances.