How to expose a function returning a C++ object to Python using Cython?

[Julien Ricard]

I'm building C++<->Python bindings using Cython, and I cannot find how to return a C++ object from a Python method.

More specifically, when compiling peak_detection_.pyx, shown below, I get

peak_detection_.pyx:35:36: Cannot convert 'vector[Peak]' to Python object

for the last lines

def getPeaks(self,data):
    return self.thisptr.getPeaks(data)

I understand the error, but I would not mind some help/pointers about how to fix it!

Thank you.

peak_detection.hpp

#ifndef PEAKDETECTION_H
#define PEAKDETECTION_H

#include <string>
#include <map>
#include <vector>

#include "peak.hpp"


class PeakDetection
{
    public:
        PeakDetection(std::map<std::string, std::string> config);
        std::vector<Peak> getPeaks(std::vector<float> &data);

    private:
        float _threshold;               
};

#endif

peak_detection.cpp

#include <iostream>
#include <string>

#include "peak.hpp"
#include "peak_detection.hpp"


using namespace std;


PeakDetection::PeakDetection(map<string, string> config)
{   
    _threshold = stof(config["_threshold"]);
}

vector<Peak> PeakDetection::getPeaks(vector<float> &data){

    Peak peak1 = Peak(10,1);
    Peak peak2 = Peak(20,2);

    vector<Peak> test;
    test.push_back(peak1);
    test.push_back(peak2);

    return test;
}

peak.hpp

#ifndef PEAK_H
#define PEAK_H

class Peak {
    public:
        float freq;
        float mag;

        Peak() : freq(), mag() {}
        Peak(float f, float m) : freq(f), mag(m) {}
};

#endif

peak_detection_.pyx

# distutils: language = c++
# distutils: sources = peak_detection.cpp

from libcpp.vector cimport vector
from libcpp.map cimport map
from libcpp.string cimport string

cdef extern from "peak.hpp":
    cdef cppclass Peak:
        Peak()

cdef class PyPeak:
    cdef Peak *thisptr
    def __cinit__(self):
        self.thisptr = new Peak()
    def __dealloc__(self):
        del self.thisptr

cdef extern from "peak_detection.hpp":
    cdef cppclass PeakDetection:
        PeakDetection(map[string,string])
        vector[Peak] getPeaks(vector[float])

cdef class PyPeakDetection:
    cdef PeakDetection *thisptr
    def __cinit__(self, map[string,string] config):
        self.thisptr = new PeakDetection(config)
    def __dealloc__(self):
        del self.thisptr
    def getPeaks(self, data):
        return self.thisptr.getPeaks(data)

[houbaastef .]

Your getPeaks is a pure python method declared with a 'def' statement.
You can't return C++ types for such methods (the C++ object would be
visible to python space, how that be ?)

You can:

- build a python PyPeak object from the C++ Peak object and return it
in a normal def method. (For that you need to adapt the PyPeak class
and make a factory staticmethod to build PyPeak from Peak.)

- or return directly a C++ object if you use a "cdef Peak
getPeaks(self, data)" method. But this method will not be visible from
python space.

Regards,
Stephane

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Julien Ricard]

I already define a PyPeak object in my code. How can I get the method getPeaks to return a PyPeak object?

[houbaastef .]

from libcpp.vector cimport vector
from libcpp.map cimport map
from libcpp.string cimport string

from libcpp.utility cimport pair

cdef extern from "peak.hpp":
    cdef cppclass Peak:
        Peak()


cdef class PyPeak:
    cdef Peak *thisptr

    def __cinit__(self, _raw=False):
        if _raw is False:

            self.thisptr = new Peak()

    def __dealloc__(self):

        if self.thisptr is not NULL:
            del self.thisptr
            self.thisptr = NULL

    @staticmethod
    cdef factory(Peak cpp_obj):
        py_obj = PyPeak.__new__(PyPeak, _raw=True)
        (<PyPeak> py_obj).thisptr = new Peak(cpp_obj)     # supposed that Peak had copy constructor
        return py_obj

cdef class PyPeakDetection:
    cdef PeakDetection *thisptr

    def __cinit__(self, config):
        assert(isinstance(config, dict))
        cdef map[string,string] cpp_map
        for k, v in config.items():
            k = bytes(k)
            v = bytes(v)
            cpp_map.insert(pair(<string> k, <string> v))

        self.thisptr = new PeakDetection(cpp_map)

    def __dealloc__(self):
        if self.thisptr is not NULL:
            del self.thisptr
            self.thisptr = NULL

    cpdef getPeaks(self, data):
        cdef vector[float] v = [float(x) for x in data]
        return Peak.factory(self.thisptr.getPeaks(v))

something like that. not tested.

[Robert Bradshaw]

And to convert a vector[Peak] into a Python object, you'd want to do
something like

cdef vector[Peak] peaks = ...
return [Peak.factory(peak) for peak in peaks]

[Julien Ricard]

Ok than you, so I have to make a copy of my C++ objects to Python? There is no way to pass a Peak object from Python to C++ without copying it?

[Julien Ricard]

For those interested, I posted the question on StackOverflow: http://stackoverflow.com/q/33677231/326849

[Robert Bradshaw]

You don't have to make a copy; it's just the simplest way to handle
ownership (and often the copy is not going to be your bottleneck).
Otherwise you have to worry about memory management and pointer
ownership, and possibly expose that to your user as well (which feels
especially foreign in Python).

If your library uses smart pointers, then the answer is easy--e.g.
move direct pointers into your newly allocated object, or assign the
shared pointer.

If your library returns pointers to which you're supposed to take over
ownership, that too is straightforward--just assign the pointer in
PyPeak to the obtained pointer.

If your library returns unowned pointers, there things get tricky
because you don't want the lifetime of the returned objects to be
shorter than the lifetime of the wrapper. You'll also need to store a
boolean in the wrapper of whether its pointer is owned (i.e. whether
__dealloc__ should call del on the pointer, or someone else will do
that).

The complexity also depends on how much control you have over the API
of the C++ library itself. E.g. if it were to take a vector to
populate, rather than return a vector, that'd make things much easier.