Properly unloading an embedded cython-based extension

[shakfu]

Hi,

As I have mentioned previously my current project consists of a python3 'external' object in Max/MSP, which can run arbitrary python code in a patch, and features cython-based access to the Max c-api. The project is open-source and available at https://github.com/shakfu/py

The design consists of a `py` object (as a max external) which is a struct instance essentally with a PyObject* globals dict. Each object is initialized with PyInitialize when created. There can be more than one `py` object alive and there is a global py_object_count to a keep a count. When the last object is closed or deleted, Py_FinalizeEx() is called. The key functions as described are as below:

void py_init(t_py* x)

{

    /* Add the cythonized 'api' built-in module, before Py_Initialize */

    if (PyImport_AppendInittab("api", PyInit_api) == -1) {

        py_error(x, "could not add api to builtin modules table");

    }

 

    Py_Initialize();

    // python init

    PyObject* main_mod = PyImport_AddModule(x->p_name->s_name); // borrowed

    x->p_globals = PyModule_GetDict(main_mod); // borrowed reference

    py_init_builtins(x);

    // register the object

    object_register(CLASS_BOX, x->p_name, x);

    // increment global object counter

    py_global_obj_count++;

    if (py_global_obj_count == 1) {

        // if first py object create the py_global_registry;

        py_global_registry = (t_hashtab*)hashtab_new(0);

        hashtab_flags(py_global_registry, OBJ_FLAG_REF);

    }

}

void py_free(t_py* x)

{

    // code editor cleanup

    object_free(x->p_code_editor);

    if (x->p_code)

        sysmem_freehandle(x->p_code);

    Py_XDECREF(x->p_globals);

    // python objects cleanup

    py_global_obj_count--;

    if (py_global_obj_count == 0) {

        hashtab_chuck(py_global_registry);

        post("last py obj freed -> finalizing py mem / interpreter.");

        Py_FinalizeEx();

    }

}

note: the cython extension is the "api" module above initialized by PyInit_api

One of the remaining issues with my implementation is that even after calling Py_FinalizeEx() I have to restart the Max application to get the embedded cython extension to reload properly.

If I free the last object (calling PyFinalizeEx()), and I open up a new patch (which calls PyInitialize), I am able to access the python interpreter without any issues, however, if I import 'api' in this case, I get a None. As mentioned, if I restart the (Max) application and open a new patch I can import 'api' without issues.

Ultimately, this is not the biggest problem: asking your users to restart the app is a small ask, however, I am curious if this has a solution.

I have searched the forum for a proper way to unload an embedded (cython-based) c-extension and I found this post from 2012 (

https://groups.google.com/forum/#!searchin/cython-users/unloading%7Csort:date/cython-users/AMpbGPE_x0M/DSbDk0wOhS4J)

Stefan's advice then was to look at at pep-3121 (https://www.python.org/dev/peps/pep-3121/) and 'register an "atexit" function that does

the cleanup when the interpreter terminates' (https://docs.python.org/3/library/atexit.html)

Given my issue as described above, would an 'atexit' function which deletes the module sound about right or is there something else to be done?

Any advice on this or the general task of freeing embedded extensions would be very much appreciated.

S

[shakfu]

shakfu wrote:

Any advice on this or the general task of freeing embedded extensions would be very much appreciated.

I tried to remove the cython module 'api' with the following code, but it does not unload properly:

    PyObject* api = PyDict_GetItemString(x->p_globals, "api");
    if (api != NULL) {
        PyModuleDef* api_def = PyModule_GetDef(api);
        if (PyState_RemoveModule(api_def) == 0) {
            py_log(x, "removed api module");
        }
        if (PyDict_DelItemString(x->p_globals, "api") == 0) {
            py_log(x, "removed ref to api module in globals");
        }
    }

Any help would be much appreciated.

S

[Stefan Behnel]

shakfu schrieb am 20.06.20 um 07:48:

> One of the remaining issues with my implementation is that even after
> calling Py_FinalizeEx() I have to restart the Max application to get the
> embedded cython extension to reload properly.

See https://bugs.python.org/issue34309

AFAIK, work is being done in Python 3.10 to improve the situation, and in
Cython 3.0 to make better use of PEP-489.

Stefan

[shakfu]

 Stefan Behnel wrote:

See https://bugs.python.org/issue34309

AFAIK, work is being done in Python 3.10 to improve the situation, and in
Cython 3.0 to make better use of PEP-489.

Thanks for that, Stefan.

I read the bug report, it's exactly my case. Well at least I can't do anything about it right now except to recommend users to restart the application if they want to use the cython-based embedded extension.

I'll keep an eye out for PEP 489 and progress on cython 3.0 then. Thanks very much for your help!

S

[shakfu]

This issue is seemingly general to python extensions which give unpredictable behavior on reload: cython extensions thankfully return a harmless None on reload, but Numpy crashes the whole application which could be a deal breaker for users who will lose unsaved data as a result!

Since this seems to be a long-standing general problem for embedded python applications, is there any advice on how to programmatically prevent a reload unless the application is restarted?

Thanks in advance.

S

[shakfu]

shakfu wrote:

This issue is seemingly general to python extensions which give unpredictable behavior on reload: cython extensions thankfully return a harmless None on reload, but Numpy crashes the whole application which could be a deal breaker for users who will lose unsaved data as a result!

Another interesting observation: if I restrict Numpy imports exclusively to the cython module, Numpy also returns a harmless None on reload. Whereas Numpy will reliably crash the python interpreter on reload if imported directly into non-extension code.

Cython seem to be doing something right here (-:

S