Initializing module from C++ after Py_Initialize

[Ray Speth]

Hi all,

I’m working on an application where I want to embed a Cython module in C++, but the Python interpreter has already been initialized, so I can’t use the approach described here that relies on using PyImport_AppendInittab.

I found a relevant question and answer on StackOverflow (here), but when trying the approach described, I just get a segfault due to the __pyx_builtin_print being a null pointer.

This seems to have something to do with PEP489 (Multi-phase extension module initialization). If I edit the Cython-generated .cpp file to set the CYTHON_PEP489_MULTI_PHASE_INIT macro to 0, then the module appears to work fine. Is there some other function that needs to be called if a module uses this initialization structure?

I’m using Python 3.10 and Cython 0.29.32. Further, trying this with Cython 3.0.0a11 issues a compiler warning:

warning: '__PYX_WARN_IF_PyInit_spam_INIT_CALLED' is deprecated:
Use PyImport_AppendInittab("spam", PyInit_spam) instead of calling PyInit_spam directly.

which I don’t believe is a viable solution once Py_Initialize() has been called.

Included below is a minimal example demonstrating the behavior. Any guidance on how to correctly load a Cython module so I can call it’s public cdef members in this situation would be greatly appreciated.

Regards,
Ray

spam.pyx:

#cython: language_level=3
#distutils: language=c++

cdef public int add_ints(int x, int y):
    print("Greetings from add_ints...")
    return x + y

test_embed.cpp:

#include "Python.h"
#include "spam.h"

#include <iostream>

int main(int argc, char** argv)
{
    Py_Initialize();
    PyImport_AddModule("spam");
    PyObject* pyModule = PyInit_spam();
    if (!pyModule) {
        PyErr_Print();
        exit(1);
    }
    PyObject* sys_modules = PyImport_GetModuleDict();
    if (!sys_modules) {
        PyErr_Print();
        exit(1);
    }
    PyDict_SetItemString(sys_modules, "spam", pyModule);
    int x = add_ints(3, 4);
    std::cout << "x = " << x << std::endl;
    return 0;

}

​

[da-woods]

This seems to have something to do with PEP489 (Multi-phase extension module initialization). If I edit the Cython-generated .cpp file to set the CYTHON_PEP489_MULTI_PHASE_INIT macro to 0, then the module appears to work fine. Is there some other function that needs to be called if a module uses this initialization structure?

What you;re doing is right - you should be able to just call the init function if CYTHON_PEP489_MULTI_PHASE_INIT is set to 0. (Note that you can also set compiler flags on the command line so probably don't need to edit the file)

I’m using Python 3.10 and Cython 0.29.32. Further, trying this with Cython 3.0.0a11 issues a compiler warning:

warning: '__PYX_WARN_IF_PyInit_spam_INIT_CALLED' is deprecated:
Use PyImport_AppendInittab("spam", PyInit_spam) instead of calling PyInit_spam directly.

which I don’t believe is a viable solution once Py_Initialize() has been called.

If you've set CYTHON_PEP489_MULTI_PHASE_INIT to 0 then you can ignore the compiler warning.

The other option might be to use the low-level module creation functions in the C API (https://docs.python.org/3/c-api/module.html#low-level-module-creation-functions). Essentially you'd call PyInit_modname, confirm that it's returned a PyModuleDef. Then you'd call PyModule_FromDefAndSpec and PyModule_ExecDef. I haven't tried this myself though, so there may be pitfalls I haven't noticed.