SIGINT vs mandatory cleanup actions

[Andreas Kloeckner]

Hi all,

I generally like working with cffi, but I've found it to be less robust
than (say) Boost.Python in one peculiar way: Cleanup in the case of
interruptions.

Suppose I'm calling a C function that assumes ownership of a pointer I
pass to it. After that function returns successfully, I must release any
ownership that my Python-side handle objects may still have of those
objects. If I don't do this and the Python-side handle eventually gets
freed, then I've got a double-free situation and a crash.

However, through a number of ways ranging from SIGINT to threading,
execution can "escape" before these cleanup actions are complete. This
makes the cffi wrappers I've written pretty crashy if ^C is
pressed. I've gotten pretty inventive in hacking around this:

https://github.com/inducer/islpy/blob/master/gen_wrap.py#L434

but none of this is really robust.

Boost.Python (again, for example) does not have this problem because any
handle juggling can be done in C where it's (mostly) uninterruptible.

What's the recommended approach to this?

Andreas

[Ryan Gonzalez]

Didn't have time to read the entire post, but in most cases you can do this:

try:

    # Some random code

    my_c_function(the_pointer)

finally:

    # Release ownership of pointer.

 

Andreas

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to the Google Groups "python-cffi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ryan

[ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong.

http://kirbyfan64.github.io/

 

[Andreas Kloeckner]

Fair point. Turns out I misremembered the main issue. Sorry about that!
The actual issue (tm) is the following. Suppose I ^C while the C
function being wrapped executes. After a while, it returns, and
KeyboardInterrupt hits before the assignment of the return value
to _result takes place:

try:
_result = None
_result = lib.isl_pw_qpolynomial_fold_drop_dims(_copy_self._release(), type, first, n)
finally:
if _result is None:
# This should never happen.
sys.stderr.write("*** islpy was interrupted while collecting "
"a result. "
"System state is inconsistent as a result, aborting.\n")
sys.stderr.flush()
import os
os._exit(-1)
_result = None if (_result == ffi.NULL or _result is None) else PwQPolynomialFold(_data=_result)
pass
if _result is None:
raise Error("call to isl_pw_qpolynomial_fold_drop_dims failed")

Now I've lost an object.

Andreas

[Armin Rigo]

Hi,

On 27 February 2016 at 04:08, Andreas Kloeckner <li...@informa.tiker.net> wrote:
> Now I've lost an object.

There are several ways to prevent that, but none is very clean. I'd
recommend moving slightly more code to C, where it will run
uninterrupted. This assumes that you're using the API mode, not the
ABI mode. Here is a minimal example:


in file foo_build.py:
ffi=FFI()
ffi.cdef("void my_do_stuff(int arg, void **result);")
ffi.set_source("_foo_cffi", """
#include <somelib.h>
void my_do_stuff(int arg, void **result)
{
*result = somelib_doing_stuff(arg);
}
""")
ffi.compile()

Then you call lib.my_do_stuff() instead of directly
somelib_doing_stuff() from Python:

from _foo_cffi import ffi, lib
p = ffi.new("void **") # initialized to NULL
try:
lib.my_do_stuff(arg, p)
# do things with p[0]
finally:
# free p[0] if it is not NULL

The difference with your version is that you can read 'p[0]' in all
cases in the "finally" block, and you get either NULL if the C
function did not run yet, or a non-NULL value returned by
somelib_doing_stuff() if it ran. If the C function ran, then the
result is in 'p[0]' and should never be lost.

Maybe cffi could grow support to do something like that more
transparently (and a solution should also work in the ABI mode). We
could automate the "call a function and store the result here", even
though it looks like a very ad-hoc solution:

p = ffi.new("void **")
ffi.call_and_store_result(p, lib.somelib_doing_stuff, arg)

A different approach is to combine the function call with ffi.gc() on
the result. Something to do the equivalent of:

p = ffi.gc(lib.somelib_doing_stuff(arg), destructor)

but in a single call that is not interruptible, e.g.:

p = ffi.gc_call(destructor, lib.somelib_doing_stuff, arg)

or maybe with hints on ``lib.somelib_doing_stuff`` that say "you
should call ffi.gc(result, destructor)" (maybe with a #pragma in the
cdef?).

Finally, yet another solution would be to add a way to temporarily
disable or enable signal processing. In PyPy such a solution already
exists, though it's not really public: you can say
"__pypy__.thread._signals_exit()" to stop processing signals, and
later "__pypy__.thread._signals_enter()" to re-enable them.


A bientôt,

Armin.

[Armin Rigo]

Hi again,

On 29 February 2016 at 12:16, Armin Rigo <ar...@tunes.org> wrote:
> p = ffi.new("void **")
> ffi.call_and_store_result(p, lib.somelib_doing_stuff, arg)

...or the same without an extra method, but with a new keyword argument instead:

p = ffi.new("void **")

lib.somelib_doing_stuff(arg, result=p)


A bientôt,

Armin.

[Andreas Kloeckner]

Thanks for your reply! I think that what's ultimately npeeded is more a
portable way of having an uninterrupted section rather than a band-aid
for the result-loss issue, because there's a symmetric concern at the
other end (which just occurred to me). You might have noticed the
_copy_self.release() in my code. This relinquishes ownership of an
object. Now if that code is interrupted before the C function is called,
then the released object is never freed. Similarly, I can imagine other
(non-memory-related) actions that would always need to happen together.

Andreas

[Armin Rigo]

Hi Andreas,

On 29 February 2016 at 18:25, Andreas Kloeckner <li...@informa.tiker.net> wrote:
> Thanks for your reply! I think that what's ultimately npeeded is more a
> portable way of having an uninterrupted section rather than a band-aid
> for the result-loss issue

Needs more thoughts. It needs hacking into CPython's internals. This
is certainly possible from cffi, but the general problem is that it
sacrifices some portability. PyPy is easy, but I'm always thinking
about what a Jython or IronPython version of cffi would need to do.

One point of view is that cffi what makes more obvious is actually an
existing, latent problem in Python. In many non-small programs there
are points between bytecodes where if an interrupt occurs *exactly
here* then some resource is not freed. The problem is more obvious
with cffi because "immediately after a C function call" is a point
that is both likely to suffer from the problem, and likely to be where
a signal is recognized (any signal that arrived during the execution
of the C function will be handled there).

Similarly, it is rare to see Python code that resists receiving two
Ctrl-C in very quick succession (and in general for any code there is
an ``n`` such that receiving ``n`` Ctrl-C in very quick succession
will break that code).

For now, the solution I recommend (see the first solution on my first
reply on this thread) should work. I used it myself in various
situations whenever I need a bit more than just a single external C
function to be considered atomic and uninterruptible.


A bientôt,

Armin.