Segmentation fault when importing a module with embedded python3

[Adam]

I am attempting to use Cython to:
1. Embed the Python interpreter
and then
2. Import and call some custom Python functions.

I can successfully do this when using Python 2.7. However, my custom
Python modules require Python 3. When I (minimally) adapt my Cython
code for Python 3, the end result is a segmentation fault during
import of my Python module. Attached is a minimal example. It
segfaults on the 'import sys' line. Compile by running something like:

> python3 setup.py build_ext
> gcc crashtest.c build/lib.linux-i686-3.2/crash.cpython-32mu.so -I /usr/include/python3.2 -lpython3.2mu

Note that before running gcc, you must edit crash.h and remove the
DL_IMPORT. (See http://codespeak.net/pipermail/cython-dev/2010-April/008888.html.)

Here is the backtrace.

#0 0x001b9f95 in PyType_IsSubtype () from /usr/lib/libpython3.2mu.so.
1.0
#1 0x001a79b3 in PyModule_GetDict () from /usr/lib/libpython3.2mu.so.
1.0
#2 0x00132736 in __Pyx_Import (name=0xb7caf220, from_list=<optimized
out>, level=<optimized out>) at crash.c:768
#3 initializePython () at crash.c:521
#4 0x080484ff in main ()

Cython version: 0.15
Python version: 3.2.2
OS: Ubuntu 11.10

Any idea what is going on? Let me know if any other information would
be helpful.

Adam


----- crash.pyx -------------------
# cython: language_level=3

cdef extern from "stdio.h":
int printf(char *format, ...)

cdef extern from "Python.h":

# embedding functions
void Py_Initialize()
int Py_IsInitialized()

cdef extern object PyInit_crash()
#cdef extern void initcrash()

cdef public void initializePython():
if not Py_IsInitialized():
printf("initializing Python\n")
Py_Initialize()
printf("initializing crash Cython module\n")
PyInit_crash()
#initcrash()

print('importing sys')
import sys
print('imported sys!')


------- setup.py -----------
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
name = 'python3 embed crash example',
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("crash", ["crash.pyx"],)]
)


--------- crashtest.c ----------------
#include "Python.h"
#include "crash.h"

int main(void * argv, int argc) {
initializePython();
return 0;
}

[Adam]

I figured out the problem.

During initialization (PyInit_crash()), Cython creates a Python module
named 'crash', and sets a global pointer. This global pointer is
*also* the return value of the initialization function. If the cython
file (crash.pyx) ignores the return value, then Cython accounts for
this by decrementing the reference count of the initialization return
value. The reference count of the module is now zero, and Python
destroys the newly created module. The global pointer now points to
garbage.

There are two ways to get around this.

1. Fix Cython so that it increments the reference count of the module
to 2 (one for the global reference, one for the returned reference)
during initialization.
2. Store the return value of the initialization as a global variable
in the cython module (crash.pyx) itself. (See example below.)

Option 1 seems best, but I don't know the Cython code, so maybe
someone else can prepare a patch.

Adam


---------- crash.pyx (workaround) --------------------

# cython: language_level=3

cdef extern from "stdio.h":
int printf(char *format, ...)

cdef extern from "Python.h":

# embedding functions
void Py_Initialize()
int Py_IsInitialized()

cdef extern object PyInit_crash()

# global reference to this module
cdef object crashModule

cdef public void initializePython():
global crashModule

if not Py_IsInitialized():
printf("initializing Python\n")
Py_Initialize()
printf("initializing crash Cython module\n")

# Important! Keep a global reference
# for the lifetime of the program.
crashModule = PyInit_crash()

print('importing sys')
import sys
print('imported sys!')

------------------------------------------

On Nov 3, 2:43 pm, Adam <adampaetzn...@gmail.com> wrote:
> I am attempting to use Cython to:
> 1. Embed the Python interpreter
> and then
> 2. Import and call some custom Python functions.
>
> I can successfully do this when using Python 2.7.  However, my custom
> Python modules require Python 3.  When I (minimally) adapt my Cython
> code for Python 3, the end result is a segmentation fault during
> import of my Python module.  Attached is a minimal example.  It
> segfaults on the 'import sys' line. Compile by running something like:
>
> > python3 setup.py build_ext
> > gcc crashtest.c build/lib.linux-i686-3.2/crash.cpython-32mu.so -I /usr/include/python3.2 -lpython3.2mu
>
> Note that before running gcc, you must edit crash.h and remove the

> DL_IMPORT.  (Seehttp://codespeak.net/pipermail/cython-dev/2010-April/008888.html.)

[Stefan Behnel]

Adam, 04.11.2011 16:54:

> I figured out the problem.
>
> During initialization (PyInit_crash()), Cython creates a Python module
> named 'crash', and sets a global pointer. This global pointer is
> *also* the return value of the initialization function. If the cython
> file (crash.pyx) ignores the return value, then Cython accounts for
> this by decrementing the reference count of the initialization return
> value. The reference count of the module is now zero, and Python
> destroys the newly created module. The global pointer now points to
> garbage.
>
> There are two ways to get around this.
>
> 1. Fix Cython so that it increments the reference count of the module
> to 2 (one for the global reference, one for the returned reference)
> during initialization.
> 2. Store the return value of the initialization as a global variable
> in the cython module (crash.pyx) itself. (See example below.)

3. properly register the module with CPython.

If you want to do this manually, you should use the CPython import C-API to
get the module stored in sys.modules.

Stefan