Execution not stopping after python exception raised from callback

[mdriu]

Hi all,

I'm wrapping a C++ class from cython. The class constructor takes a callback as an argument, and the callback can be later called via a call() method. The call() method is declared in cython as except*. I'm passing a cython function as a callback, which internally calls a python function. The cython function passed as a callback is also declared as except.

-------------- caller.hpp -------------

typedef void (*CallBack)(void);

class Caller
{
  public:
    Caller(CallBack cb) : m_cb(cb), m_count(0) {};
    void call(void) { m_cb(); m_count++; }
    CallBack m_cb;
    unsigned m_count;
};
--------------

-------------- caller.pyx -------------

# distutils: language = c++

cdef extern from "caller.hpp":

    ctypedef void (*CallBack)() except*;

    cdef cppclass Caller:
        Caller(CallBack cb)
        void call() except*
        unsigned m_count

_py_callback = None

cdef void c_callback() except*:
    if _py_callback:
        _py_callback()

cdef class PyCaller():

    cdef Caller *p

    def __cinit__(self, py_callback):
        global _py_callback
        _py_callback = py_callback
        self.p = new Caller(c_callback)

    def __dealloc__(self):
        del self.p

    def call(self):
        self.p.call()

    def get_count(self):
        return self.p.m_count

--------------

If the python callback raises an exception, I'm expecting the C++ code execution to stop, i.e., m_count++ not to be executed. However, testing with:

-------------- test_caller.py -------------

from pycaller import PyCaller

def py_callback():
    assert 0

obj = PyCaller(py_callback)

try:
    obj.call()
except AssertionError as e:
    print("Yay, exception caught..")
    try:
        assert obj.get_count() == 0
    except AssertionError as e:
        print("Doh, execution didn't stop..")
    else:
        print("Yay, execution did stop..")
else:
    print("Doh, exception not caught..")

--------------

I get as a result:

>python test_caller.py
Yay, exception caught..
Doh, execution didn't stop..

meaning that execution didn't stop. Am I missing something? Is my expectation of code stopping after python exception raised correct?

Cheers,

Marco

[da-woods]

What `except *` says is "after any call to this function the Python exception state might be set so the caller is responsible for checking it". When Cython calls an `except *` function it inserts a `PyErr_Occurred()` check to do this. If your C++ function wants this behaviour then it must do so itself.

C++ has no concept of "Python exceptions" (which are really just a global pointer somewhere) so won't respond to them.

I have a vague plan to add a PyObject->std::function wrapper to Cython, which might be able to translate Python exceptions to C++ exceptions for example. However that doesn't exist yet.

David

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/b3c11cad-5d60-489d-ad97-6489a3492298n%40googlegroups.com.

[mdriu]

Ok, I see, thank you.

Then I guess the only way to achieve what I want (i.e. stop code execution if the callback rises an exception) is to raise a C++ exception from the callback, and qualify the callback type and the call() method with except+ so that the C++ exception is then translated to a Python exception. Something like:

-------------- caller.hpp -------------

#include <stdexcept>

typedef void (*CallBack)(void);

class Caller
{
  public:
    Caller(CallBack cb) : m_cb(cb), m_count(0) {};
    void call(void) { m_cb(); m_count++; }
    CallBack m_cb;
    unsigned m_count;
};

void raise_exception(void)
{
  throw std::runtime_error("You shall not pass!");
}

---------------------------

-------------- caller.pyx -------------

# distutils: language = c++

cdef extern from "caller.hpp":

    ctypedef void (*CallBack)() except+;

    cdef cppclass Caller:
        Caller(CallBack cb)

        void call() except+
        unsigned m_count

    void raise_exception() except+

cdef class PyCaller():

    cdef Caller *p

    def __cinit__(self):
        self.p = new Caller(raise_exception)

    def __dealloc__(self):
        del self.p

    def call(self):
        self.p.call()

    def get_count(self):
        return self.p.m_count

---------------------------

[da-woods]

No that doesn't do what you want. Cython catches C++ exceptions at the first opportunity and converts them to Python exceptions so it should be caught immediately after raise_exception. Even if that fails, it isn't safe to propagate C++ exceptions through Cython code - it'll mess up the reference counting).

If you're able to modify the C++ code then you could consider one of:

1.     void call(void) { m_cb(); if (!PyErr_Occurred()) m_count++; }

2. change the type of the callback to have a return value indicating an error.

If you can't do one of these, then your C++ code just doesn't have the concept that the callback can fail and that's that.

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/92645c0f-1abb-42db-a971-792f5fde71e4n%40googlegroups.com.

[mdriu]

Unfortunately modifying the C++ code isn't really an option. I'll try to step out from the MWE and explain a bit more what I'm trying to do, perhaps that would make more sense.

I have a C++ library which has an assert component. In case of condition not met, the assert component does some logging and calls stdlib.h exit() by default. The assert component also gives the option of calling a registered callback rather than exit() in case of condition not met. When I'm wrapping the C++ library using cython to make that available in python, I don't want exit() to be called (e.g. this is annoying when running tests with pytest, as an exit() would cause the whole test suite to stop). Hence I'd rather prefer a python exception to be raised, and I thought I could do that via the callback mechanism. However, in most of the C++ library assertions are used to protect the code from misconfigured parameters, so stopping at condition not met is critical.

You say "Cython catches C++ exceptions at the first opportunity and converts them to Python exceptions so it should be caught immediately after raise_exception" and this feels like exactly what I'd like to achieve.. So why this would fail?

[da-woods]

> You say "Cython catches C++ exceptions at the first opportunity and
> converts them to Python exceptions so it should be caught immediately
> after raise_exception" and this feels like exactly what I'd like to
> achieve.. So why this would fail?
>

The call to raise_exception is converted into something like:

try {
  raise_exception()
} catch (...) {
  convert_cpp_exception_to_py_exception();
  return_with_py_exception_flag_set;
}

It's essentially a very complicated way of writing `raise
RuntimeError()` in your Cython code. Since you don't handle the Python
exception in C++ it gets you nothing.

Essentially there are roughly three ways of signalling that a function
has encountered an error in C++:

1. throw a C++ exception
2. put the error code in the return value
3. Set a global flag indicating the error.

In this context, raising a Python exception is #3 - you must check the
global flag with `PyErr_Occurred()`. Since your C++ code
can't/won't/doesn't do any of these three things I do not believe there
is any way for your function to indicate failure.

[mdriu]

I understand what you mean, but this doesn't match with what I see.

If I test the "raise_exception()" variant with:

-------------- test_caller.py -------------

from pycaller import PyCaller

obj = PyCaller()

try:
    obj.call()
except RuntimeError as e:

    print("Yay, exception caught..")
    try:
        assert obj.get_count() == 0
    except AssertionError as e:
        print("Doh, execution didn't stop..")
    else:
        print("Yay, execution did stop..")
else:
    print("Doh, exception not caught..")

---------------------------

I get:

>python test_caller.py
Yay, exception caught..

Yay, execution did stop..

[da-woods]

On 21/09/2022 11:41, mdriu wrote:
> I understand what you mean, but this doesn't match with what I see.
> If I test the "raise_exception()" variant with:
>

Sorry - you're right. I misread what you were trying to do. You pass
"raise_exception" directly as the callback so Cython doesn't interfere
with how it works. I somehow thought that you were calling
"raise_exception" from within some Cython function and passing the
Cython function as the callback. My mistake!

[mdriu]

All clear now, thanks for your help David!

Cheers,

Marco