Polymorphic Inheritance across Cython and Python - aka Polymorphic Bridge

[Keith Halligan]

Hi,

I have a unique issue with Cython where I need to wrap a number of classes that are inherited from each other, which using the Cython extension types is perfectly fine.  I then need to be able to extend the most derived extension type in Python for users to use. 

 - The base class (Base) has a C++ pure virtual function (call_me())

 - The middle class (Derived) has overridden the base classes pure virtual function (call_me()) and also another pure virtual function (op1()).  

 - The Python class then extends Derived, and overrides op1(), we then want to pass an  instance of this Python class, to another Cython extension type that can invoke the call_me() function.

The issue as I see it, is that the instantiation of the Python class (SampleImpl), never properly instantiates the extension types correctly.  There doesn't seem to be a way to construct the wrapped C++ pointer types inside these extension types. 

For a time we were using SWIG for wrapping our C++ code, and SWIG provided such a feature called "Directors" to allow a wrapped C++ class to be extended in Python.  Unfortunately SWIG didn't work out for us, for other reasons, and Cython feels like the right fit for us, but this issue is a show-stopper for us.

Does such a feature exist in Cython?  If not, then would it be a major undertaking to implement such a feature.

Environment:

 - Windows 10

 - Python 3.10 (x64)

 - Cython 0.29.24

Output:

> py setup.py build_ext -i

> py test_app.py

[Tester::call_call_me()]: error Base pointer is NULL

Any help would be graciously appreciated.

Thanks,

Keith

====

// my_app.hpp
#ifndef _MY_APP_HPP__
#define _MY_APP_HPP__

#include <iostream>

class Base {
public:
    virtual void call_me() = 0;
};

class Derived {
public:

    virtual void op1() = 0;

    virtual void call_me() {
        std::cout << "In Derived::call_me()" << std::endl;
    }
};

class Tester {
public:
    void call_call_me(Base* base) {
        if (!base) {
            std::cerr << "[Tester::call_call_me()]: error Base pointer is NULL" << std::endl;
        }
        else {
            base->call_me();
        }
    }
};
#endif // _MY_APP_HPP__

-----------

// my_app.cpp
#include "my_app.hpp"

-----------

# sample_mod.pxd
cdef extern from "my_app.hpp":
    cdef cppclass Base:
        pass
    cdef cppclass Derived(Base):
        void op1()
        void call_me()
    cdef cppclass Tester:
        void call_call_me(Base* base)

-----------

# sample_mod.pyx
cimport sample_mod

cdef class PyBase:
    cdef sample_mod.Base* _base

    def __cinit__(self):
        self._base = new sample_mod.Base()

    cdef sample_mod.Base* get_ptr(self):
        return self._base

cdef class PyDerived(PyBase):
    cdef sample_mod.Derived* _derived

    def __cinit__(self):
        self._derived = new sample_mod.Derived()

    @staticmethod
    cdef create_py_derived(sample_mod.Derived* der):
        ret = PyDerived()
        ret._derived = der
        return ret

    def call_me(self):
        self._derived.call_me()

cdef class PyTester:
    cdef sample_mod.Tester* _tester

    def __cinit__(self):
        self._tester = new sample_mod.Tester()

    def call_call_me(self, base_ptr : PyBase):
        self._tester.call_call_me(base_ptr.get_ptr())

-----------

#!/usr/bin/env python3
# setup.py

from setuptools import setup, Extension
from Cython.Build import cythonize

test_module = Extension(
    'sample_mod',
    sources=[
        'sample_mod.pyx',
        'my_app.cpp'
    ],
    language = 'c++'
)

setup (name = 'My Module',
       version = '0.1',
       author      = "",
       description = "",
       ext_modules = cythonize(
           [test_module],
           compiler_directives = {'language_level' : '3'},
       )
)

-----------

#!/usr/bin/python3
# test_app.py

import sample_mod

class SampleImpl(sample_mod.PyDerived):
    def op1(self):
        print("*** in op1")

tester = sample_mod.PyTester()
sampl = SampleImpl()
tester.call_call_me(sampl)

[Robert Bradshaw]

What you have are two separate class hierarchies. A C++ one with
classes Derived : Base, and a Python one with classes SimpleImpl :
PyDerived : PyBase. The Python ones point to the C++ ones, but not the
other way around (which seems to be what you want).

You can create

cdef cppclass MyCppClassWrappingPythonClasss(Derived):
cdef PyObject* py_derived
void op1():
(this.py_derived).op1()
...

You can then set them up to point to each other.

I agree it'd be nice if this was set up automatically, but Cython is
primarily a tool for invoking C/C++, not automatically generating
wrappers.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/ec14a3b4-64f2-4388-9744-f3ba909a2a61n%40googlegroups.com.

[Keith Halligan]

Thanks for the reply Robert.

> You can create
>
> cdef cppclass MyCppClassWrappingPythonClasss(Derived):
>   cdef PyObject* py_derived
>   void op1():
>     (this.py_derived).op1()
>   ...
>
> You can then set them up to point to each other.
>
> I agree it'd be nice if this was set up automatically, but Cython is
> primarily a tool for invoking C/C++, not automatically generating
> wrappers.
>

I think I managed to get some variant of what you were suggesting going.

Here's what I did, in case anyone on the mailing list ever finds themselves wanting to achieve this.

Created a new concrete C++ class that inherits from Derived, let's call it "CppDerivedWrapper", in which we store a PyObject*, and we override the pure virtual method (op1()), from the Derived class.  In the implementation of op1(), I made a simple CPython API call to our op1() in the Python class, I used a PyObject_CallObject(..., "op1", "") method.

I then created a wrapped Extension type (PyDerivedWrapperOfCpp) for this C++ class, and registered the Python class's self object (passed in from the constructor) with it and passed it into the CppDerivedWrapper instance that's being wrapped.

cdef class  PyDerivedWrapperOfCpp (PyDerived):
    cdef sample_mod.CppDerivedWrapper* _derived_wrapper
    cdef object py_obj

    def __cinit__(self):
        self._derived_wrapper = self._intermed = self._base = new sample_mod.CppDerivedWrapper()

    def __init__(self, py_self):
        self.py_obj = py_self
        self._derived_wrapper.set_py_obj(py_self)

The python class extends from the PyDerivedWrapperOfCpp extension type, and it's constructor needs a "super().__init(self)", to start the ball rolling.

class SampleImpl(sample_mod. PyDerivedWrapperOfCpp):
    def __init__(self):
        super().__init__(self)
    def op1(self):
        print("[SampleImpl (Python) op1!!!!]")

tester = sample_mod.PyTester()
sampl = SampleImpl()

tester.call_op1(sampl)

--

Output:

[Tester::call_op1()]
[DerivedWrapper::op1()]
[SampleImpl (Python) op1!!!!]

> You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/iW6So0nFVPo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/CADiQ%2BQBQCuwCsWccOr48Q6c8h_794Ow8Sxoem7_q1hwmmAeT%2BA%40mail.gmail.com.

[Stefan Behnel]

Keith Halligan schrieb am 15.12.21 um 12:14:

> Thanks for the reply Robert.
>
>> You can create
>>
>> cdef cppclass MyCppClassWrappingPythonClasss(Derived):
>> cdef PyObject* py_derived
>> void op1():
>> (this.py_derived).op1()
>> ...
>>
>> You can then set them up to point to each other.
>>
>> I agree it'd be nice if this was set up automatically, but Cython is
>> primarily a tool for invoking C/C++, not automatically generating
>> wrappers.
>>
>
> I think I managed to get some variant of what you were suggesting going.
>
> Here's what I did, in case anyone on the mailing list ever finds themselves
> wanting to achieve this.
>
> Created a new concrete C++ class that inherits from Derived, let's call it
> "CppDerivedWrapper", in which we store a PyObject*, and we override the
> pure virtual method (op1()), from the Derived class. In the implementation
> of op1(), I made a simple CPython API call to our op1() in the Python
> class, I used a PyObject_CallObject(..., "op1", "") method.

I assume you implemented this class in C++, not in Cython?

If you implement it in Cython, you can do the same in regular Python
syntax, i.e. something like

cdef cppclass ...:
int op1():
return (<object>this._pyobject_ptr).op1()

Might be a little faster than a generic C-API call.

Stefan

[Keith Halligan]

> I assume you implemented this class in C++, not in Cython?
>
> If you implement it in Cython, you can do the same in regular Python
> syntax, i.e. something like
>
> cdef cppclass ...:
> int op1():
> return (<object>this._pyobject_ptr).op1()
>
> Might be a little faster than a generic C-API call.
>
> Stefan
>

Thanks for the reply, and yes I did implement the previous solution using a C++ class.  I've used the "cdef cppclass " approach, and it does indeed work as well.

pxd:

cdef cppclass DerivedWrapper(Intermediate):
    object py_obj
    inline __init__(object py_obj):
        this.py_obj = py_obj
    void op1()

pyx:

cdef cppclass DerivedWrapper(Derived):
    void op1():
        (<object>this.py_obj).op1()

cdef class PyDerivedWrapper(PyDerived):
    cdef sample_mod.DerivedWrapper* _derived_wrapper
    def __init__(self, py_self):
        self._derived_wrapper = self._derived = self._base = new sample_mod.DerivedWrapper(py_self)

I've a few questions regarding it:

 1. Is this a fully supported construct?  I ask this because the Cython docs do not seem to refer to using the "cppclass" keyword in this way.  All references I've seen come from wrapping an existing C++ class, using the "cdef extern from "<header>"" construct.

 2. Is there any limitations with using this construct over the C++ class?

Thanks,

Keith