How to use Cython to wrap a C++ template class for use in Python?

[Paul Leopardi]

Hi all,

I have been using Cython with great success to wrap my GluCat template library for use with Python, all except for one problem. There does not seem to be any way to wrap the template classes themselves for use with Python, only specific instantiations. I am using the two-level method as currently documented, with a *.pxd file wrapping the template class directly, and a *.pyx file defining a Python class, containing a pointer to a specific instance of the template class. I even tried using fused types with the current Head via Github, but I could not successfully use fused types as template parameters. The only method that I know will work would be to completely duplicate all of the Cython code to create a separate Python class for each different typename that I want to use as a template parameter in the template class.

Example: I want to wrap glucat::framed_multi<Scalar_T> where Scalar_T is a type, eg. float, double. With the current Head, I can successfully create a pxd file containing e.g. framed_multi[Scalar_T], but I can only create a Python type wrapping wrap framed_multi[double] or framed_multi[float]. I cannot create a Python type wrapping framed_multi[floating], where floating is a fused type, and if I want to wrap both  framed_multi[double] and framed_multi[float], I have to duplicate the whole class definition with all of the member functions, doctests tec. for each.

Is this correct? Is there a better wy to do this, now or planned?

[mark florisson]

I think in the future we'll support fused cdef class attributes, e.g.

cdef class MyClass:
cdef framed_multi[cython.floating] obj

which might be specialized through constructor arguments or indexing the class.

For now you might be stuck with methods, attributed with faked types
and casting. I wrote a little snippet that lets you specialize by
indexing the class using slightly hacky code. It basically uses a
metaclass that when indexed returns a subclass where all the methods
are specialized. You will still have to cast self.vector to the right
specialized type and you have to use a dummy argument for each method,
but at least it should be somewhat more comfortable to deal with.

Of course, the ideal thing to do would be to fix Cython :) If you want
to fix this issue in Cython, we'd be happy to give you pointers and
help. This will be supported, but the developers are busy with their
lives and don't really desperately need it, so it's hard to tell when.

################## test.pyx
cimport cython

import inspect
import functools

cdef extern from "multivector.h":
cdef cppclass multivector[T]:
int method()

ctypedef multivector[cython.floating] multivector_floating_t

def wrap(func):
@functools.wraps(func)
def wrapper(self, *a, **kw):
return func(self, 0.0, *a, **kw,)

return wrapper

cdef class _MultiVector:
cdef multivector[float] *vector

def method(self, cython.floating dummy, *args, **kwargs):
cdef multivector_floating_t *actual_vector =
<multivector_floating_t *> self.vector
print actual_vector.method()
print args, kwargs

class Meta(type):
_cache = {}

def __getitem__(cls, specialization):
if specialization in cls._cache:
return cls._cache[specialization]

print "getitem"

name = specialization.capitalize() + cls.__name__
newcls = type(name, (cls,), {})
for name, attribute in inspect.getmembers(cls):
if hasattr(attribute, '__signatures__'):
# function has a fused dummy argument
setattr(newcls, name, wrap(attribute[specialization]))

cls._cache[specialization] = newcls
return newcls

class MultiVector(_MultiVector):
__metaclass__ = Meta

obj = MultiVector["float"]()
obj.method(10, kw=20)

obj = MultiVector["double"]()
obj.method(10, kw=20)

################# multivector.h
template <class T>
class multivector {
T attrib;
public:
int method() {
return sizeof(attrib);
}
};

[Paul Leopardi]

Thanks for the hint. I'll examine your code in more detail, a little later.
All the best, Paul

[Paul Leopardi]

Now that Cython 0.16 is out and Cython 0.17 is planned, is this a good time to revisit my question? GluCat 0.6.0 uses Cython to implement the PyClical Python extension module for calculations in Clifford algebras, using the Python class clifford, based on the GluCat template class glucat::matrix_multi<Scalar_T,LO,HI>. I would like to be able to expose the Scalar_T template parameter to the Python user, in some way other than having to write multiple class definitions, e.g. clifford_float, clifford_double, clifford_dd_real, clifford_qd_real, and multiple versions of all the function. Is there any way to do this, using fused types, or otherwise, that does not involve lost of code duplication o lots of delving into Cython internals?

[Chris Barker]

Well, the third option is to use your own home-made templatesto
auto-generate cython code for each type. That's actually not as
painful as it sounds -- there are lot's of templating systems for
python -- Cheetah, for instance is designed for "any" use, not just
html (though I'm never used it for Cython).

http://www.cheetahtemplate.org/

You might want to look at the "bottleneck" project -- they did
something like this -- not for calling a C++ templates, but the
principle is the same.

http://berkeleyanalytics.com/bottleneck/

-Chris









--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Paul Leopardi]

On Wednesday, 30 May 2012 02:40:47 UTC+10, Chris Barker wrote:

Well, the third option is to use your own home-made templatesto
auto-generate cython code for each type. That's actually not as
painful as it sounds -- there are lot's of templating systems for
python -- Cheetah, for instance is designed for "any" use, not just
html (though I'm never used it for Cython).

http://www.cheetahtemplate.org/

You might want to look at the "bottleneck" project -- they did
something like this -- not for calling a C++ templates, but the
principle is the same.

http://berkeleyanalytics.com/bottleneck/

Chris, thanks for your suggestions. I will take a look at Cheetah, but I'm afraid it would add extra complication to an already very complicated build process for PyClical. What I would like is a way for Cython to make instantiations of my C++ template classes visible to Python. Cython supports C++ templates, and Cython now has fused types, but as far as I can tell, these two template concepts within Cython do not work with each other in any meaningful, documented way, let alone support what I am proposing. Maybe I am just too late to propose another Google Summer of Code project, or perhaps the whole idea is all too hard? Surely there must be a wider use-case for this idea than just me and my one library? Should I repost to the core developers list?

[Paul Leopardi]

See also this thread in cython-devel for an idea of other, very similar, use-cases, and where the current discussion is leading.

http://grokbase.com/t/python/cython-devel/124wtfc8ee/cython-wacky-idea-proper-macros

[Robert Bradshaw]

I think there are several issues with why Cython does not (yet?) have
these capabilities, primarily:

(1) Metaprogramming done Right can be very nice, but done wrong is disastrous,
(2) The AST of Cython is really not that nice to work with, and
(3) If it's just about specifying types, template preprocessing and,
eventually, using JITs may be sufficient, more flexible, and certainly
better than a half-baked solution.

The thread you linked to also has some good discussion.

We actually had a lot of discussion about this issue at the Cython
Days workshop last year, and never hit upon a metaprogramming
framework that seemed Right. Consensus was that until a clean proposal
was put forward, we would focus on making it easy to use your
templating engine of choice and implement fused types which would
cover 85% or more of the need for metaproramming (especially tight
loops over numeric types, as opposed to more generic cases where
Python objects and vtables are often good enough).

The fact that fused types don't work with C++ specializations is
certainly a bug, though this still limits us to a fixed number of
instantiations (and separate Python classes) in Python space due to
the nature of C++ templates.

- Robert

[mark florisson]

Definitely, fused types are far from ideal, even if cdef class fused
attributes would be supported.

> The fact that fused types don't work with C++ specializations is
> certainly a bug, though this still limits us to a fixed number of
> instantiations (and separate Python classes) in Python space due to
> the nature of C++ templates.
>
> - Robert

If there really is a bug, then please report the use case (anyone who
can find one). C++ and fused types *do* work together (unless there
really is a bug), but the thing that is not supported (for anything)
is fused types for cdef class attributes. One, entirely terrible but
working, way to get around from Python space is the metaclass hack I
posted earlier.