Calling Cython defined function from C

[Phil Elson]

Hi all,

My C is fairly weak, so please excuse any obvious blunders.

I'm trying to call a function defined in Cython from C. My motivation is to slowly build up enough knowledge to create a custom numpy dtype using Cython, but I'm still making baby steps...

I've produced a minimal example, where my cdef Cython function simply returns an integer with success. However, I get a Seg fault as soon as I try to do anything with the Python C api (like raise an error).

I've put my code in a branch at https://github.com/pelson/calling_publicly_declared_cython_from_c/tree/a64437630e2eb5b3dd086ac63b83d156869609da, and for longevity of the list, I include it here too:

$ cat setup.py

 

from distutils.core import setup, Extension

from Cython.Build import cythonize

ext_modules = []

ext = Extension('foo',

                sources=['foo/thing.c',

                         'foo/pyx_funcs.pyx',

                         ],

                language="c")

ext_modules.extend(cythonize([ext]))

setup(name='foo',

      version='0.1',

      packages=['foo',

                ],

      ext_modules=ext_modules)

 

$ cat foo/thing.c 

#include <Python.h>

int val;

static PyObject* bar(PyObject *self, PyObject *args) {

    val = the_answer();

    return Py_BuildValue("i", val);

}

static PyMethodDef foo_methods[] = {

    {

        "bar", bar, METH_VARARGS,

    },

    {NULL, NULL, 0, NULL}

};

static struct PyModuleDef foo_definition = {

    PyModuleDef_HEAD_INIT,

    "foo",

    "",

    -1,

    foo_methods

};

PyMODINIT_FUNC PyInit_foo(void) {

    Py_Initialize();

    return PyModule_Create(&foo_definition);

} 

$ cat foo/pyx_funcs.pyx 

cdef extern int the_answer():

    raise ValueError("Testing an error")

    return 42

I've gone through a number of iterations, including annotating my definitions as nogil and using the gil context manager. I've also been looking for an initfoo function that perhaps I should be calling etc.. I've also written up an example from StackOverflow at https://github.com/pelson/calling_publicly_declared_cython_from_c that relates to this space (but doesn't solve my issue).

The relevant docs (http://docs.cython.org/en/latest/src/userguide/extension_types.html#public-and-external-extension-types) are a little sparse, and I can't see any examples in https://github.com/cython/cython/tree/master/Demos.

Any pointers greatly appreciated...

Phil

[Jeroen Demeyer]

On 2017-09-08 12:36, Phil Elson wrote:
> I'm trying to call a function defined in Cython from C.

See
http://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations

I don't understand why you are trying to implement a Python module in C
(see foo/thing.c). The whole point of using Cython is that you do *not*
need to do that.

[Chris Barker]

I'll agree here. Though I suspect that the end game here is that you want to add code to a module that is written in C, so you are testing our how that works.

But maybe that is adding a level of complexity that you don't need now, so you could try:

writing some  "pure C" code i.e. without a module init, that can't be imported directly inot python

call that C code from a Cython module.

Once that works, have that C code call other Cython-generated code.

Also -- I'd reach out to the numpy folks here -- has anyone done this yet? made a dytpe with Cython? or, for that matter, extended numpy in any other way with Cython?

And maybe more to the point -- using Cython to write what needs to be "pure C" code may not be worth it -- it can be done, but you need to understand C enough that it's really just different syntax, so what's the point?

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Phil Elson]

Thanks Chris and Jeroen,

Chris - your suggestion is a good way of getting around the need to write a C extension type, I had initially decided against doing it that way as it would add another layer of complexity. Given my C knowledge, perhaps it would make sense to revisit that idea.

Jeroen - thanks. I'd missed the external_c_code content, and was mostly referencing the extension_types docs, so that was a helpful pointer.

I've been looking at the output from the public / extern / api cythonisation, and it would suggest that the docs are a little out of date wrt. Python 3. In particular, I believe I need to call a function called "init{funcname}" in Python 2, and "PyInit_{funcname}" in Python 3. Do you know if this is actually true?

I'm keen to document any successful findings, and will happily submit a PR to the docs once I have a (minimal) working prototype.

Cheers,

[Stefan Behnel]

Phil Elson schrieb am 12.09.2017 um 08:22:
>> On Fri, Sep 8, 2017 at 9:34 AM, Jeroen Demeyer wrote:
>>> See
>>> http://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations

>
> Jeroen - thanks. I'd missed the external_c_code content, and was mostly
> referencing the extension_types docs, so that was a helpful pointer.
>
> I've been looking at the output from the public / extern / api
> cythonisation, and it would suggest that the docs are a little out of date
> wrt. Python 3. In particular, I believe I need to call a function called
> "init{funcname}" in Python 2, and "PyInit_{funcname}" in Python 3.

That part is outdated, yes. The module init function was renamed and now
returns the module reference. Thus, I would suggest that users should not
call it directly at all, but instead use Cython's "embed" option. It
generates a C main() function from the template file Embed.c. The example
in the docs is more description than recommendation.

I doubt that you would have to do that in your case, though. Just because
you're exporting something to C doesn't mean that your code controls the
startup of the CPython runtime. That's unrelated [1]. You would probably
just need to implement a normal Cython extension module and provide some C
interface to it. Whether you need "public" declarations for that or "api",
I can't say. That depends on the way you want to integrate it with the
external C code.

BTW, are you even sure that you need that external C code? What is going to
use and call your code? NumPy? If so, then that would suggest to me that
you probably don't need any C (or public/api exports) at all, and can do
everything from Cython directly.

Maybe you should try to further explain your use case here, and the API
that you are programming against. That way, it might become clearer to
yourself what you actually want and need, and easier for us to push you
into the right direction.

> I'm keen to document any successful findings, and will happily submit a PR
> to the docs once I have a (minimal) working prototype.

Please do.

Stefan


[1] The CPython documentation forces you to decide early whether
"embedding" or "extending" is your goal, but the only difference between
the two is really what starts up the CPython runtime. Once you are beyond
that, both are exactly the same.