cython 0.28.1 and sagemath

Martin R

unread,

Apr 23, 2018, 11:41:30 PM4/23/18

to sage-devel

Hi there,

Since 8.2.rc0 (the update to cython 0.28.1) I'm having trouble using pyximport. Here is a minimal non working example:

######## minimal.py ############

import pyximport; pyximport.install();

from minimal_pyx import minimal_fun

######## end minimal.py ############

######## minimal_pyx.pyx ############

from sage.misc.cachefunc import cached_method

def minimal_fun():

print("hi")

######## end minimal_pyx.pyx ############

Running this with "sage minimal.py" fails as reproduced below.

It seems that I should somehow set the compile_time_env variable 'PY_VERSION_HEX', and set it to sys.hexversion. I don't know how to do that (especially not when using pyximport), and I'm puzzled because I would have thought that dict_del_by_value.pyx is already compiled...

Thanks for any hints or workarounds!

Martin

martin@Martin-Laptop:~/test$ sage minimal.py

Error compiling Cython file:

------------------------------------------------------------

...

cdef del_dictitem_by_exact_value(PyDictObject *mp, PyObject *value, Py_hash_t hash)

#This is for compatibility: this routine is available in Py3 and we

#implement it ourselves in Py2.

IF PY_VERSION_HEX<=0x02ffffff:

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/cpython/dict_del_by_value.pxd:9:3: Compile-time name 'PY_VERSION_HEX' not defined

Error compiling Cython file:

------------------------------------------------------------

...

#in the underlying dictionary, we have stored a KeyedRef r

#under a key k. The attribute r.key is the hash of k.

if D._guard_level:

D._pending_removals.append(r)

else:

del_dictitem_by_exact_value(<PyDictObject *>D, <PyObject *>r, r.key)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:213:12: undeclared name not builtin: del_dictitem_by_exact_value

Error compiling Cython file:

------------------------------------------------------------

...

#in the underlying dictionary, we have stored a KeyedRef r

#under a key k. The attribute r.key is the hash of k.

if D._guard_level:

D._pending_removals.append(r)

else:

del_dictitem_by_exact_value(<PyDictObject *>D, <PyObject *>r, r.key)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:213:41: 'PyDictObject' is not a type identifier

Error compiling Cython file:

------------------------------------------------------------

...

#in the underlying dictionary, we have stored a KeyedRef r

#under a key k. The attribute r.key is the hash of k.

if D._guard_level:

D._pending_removals.append(r)

else:

del_dictitem_by_exact_value(<PyDictObject *>D, <PyObject *>r, r.key)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:213:59: Cannot convert 'PyObject *' to Python object

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:466:28: undeclared name not builtin: PyDict_GetItemWithError

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:466:51: Cannot convert Python object to 'PyObject *'

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:466:13: Storing unsafe C derivative of temporary Python reference

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:578:51: Cannot convert Python object to 'PyObject *'

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:578:13: Storing unsafe C derivative of temporary Python reference

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject * wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:654:52: Cannot convert Python object to 'PyObject *'

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject * wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:654:13: Storing unsafe C derivative of temporary Python reference

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:694:51: Cannot convert Python object to 'PyObject *'

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:694:13: Storing unsafe C derivative of temporary Python reference

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:738:51: Cannot convert Python object to 'PyObject *'

Error compiling Cython file:

------------------------------------------------------------

...

Traceback (most recent call last):

...

TypeError: mutable matrices are unhashable

"""

cdef PyObject* wr = PyDict_GetItemWithError(self, k)

^

------------------------------------------------------------

/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/misc/weak_dict.pyx:738:13: Storing unsafe C derivative of temporary Python reference

/home/martin/.pyxbld/temp.linux-x86_64-2.7/pyrex/sage/misc/weak_dict.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.

#error Do not use this file, it is the result of a failed Cython compilation.

^~~~~

Traceback (most recent call last):

File "minimal.py", line 2, in <module>

from minimal_pyx import minimal_fun

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 458, in load_module

language_level=self.language_level)

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 233, in load_module

exec("raise exc, None, tb", {'exc': exc, 'tb': tb})

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 216, in load_module

mod = imp.load_dynamic(name, so_path)

File "minimal_pyx.pyx", line 1, in init minimal_pyx

from sage.misc.cachefunc import cached_method

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 458, in load_module

language_level=self.language_level)

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 233, in load_module

exec("raise exc, None, tb", {'exc': exc, 'tb': tb})

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 216, in load_module

mod = imp.load_dynamic(name, so_path)

File "weak_dict.pxd", line 1, in init sage.misc.cachefunc

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 458, in load_module

language_level=self.language_level)

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 233, in load_module

exec("raise exc, None, tb", {'exc': exc, 'tb': tb})

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 215, in load_module

inplace=build_inplace, language_level=language_level)

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 191, in build_module

reload_support=pyxargs.reload_support)

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll

dist.run_commands()

File "/home/martin/sage-develop/local/lib/python2.7/distutils/dist.py", line 953, in run_commands

self.run_command(cmd)

File "/home/martin/sage-develop/local/lib/python2.7/distutils/dist.py", line 972, in run_command

cmd_obj.run()

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run

_build_ext.build_ext.run(self)

File "/home/martin/sage-develop/local/lib/python2.7/distutils/command/build_ext.py", line 340, in run

self.build_extensions()

File "/home/martin/sage-develop/local/lib/python2.7/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions

self.build_extension(ext)

File "/home/martin/sage-develop/local/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension

depends=ext.depends)

File "/home/martin/sage-develop/local/lib/python2.7/distutils/ccompiler.py", line 574, in compile

self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)

File "/home/martin/sage-develop/local/lib/python2.7/distutils/unixccompiler.py", line 126, in _compile

raise CompileError, msg

ImportError: Building module minimal_pyx failed: ['ImportError: Building module sage.misc.cachefunc failed: [\'ImportError: Building module sage.misc.weak_dict failed: ["CompileError: command \\\'gcc\\\' failed with exit status 1\\\\n"]\\n\']\n']

Nils Bruin

unread,

Apr 24, 2018, 1:13:24 AM4/24/18

to sage-devel

On Monday, April 23, 2018 at 8:41:30 PM UTC-7, Martin R wrote:

Hi there,

Since 8.2.rc0 (the update to cython 0.28.1) I'm having trouble using pyximport. Here is a minimal non working example:

######## minimal.py ############
import pyximport; pyximport.install();
from minimal_pyx import minimal_fun
######## end minimal.py ############

######## minimal_pyx.pyx ############
from sage.misc.cachefunc import cached_method

def minimal_fun():
print("hi")
######## end minimal_pyx.pyx ############

Running this with "sage minimal.py" fails as reproduced below.

It seems that I should somehow set the compile_time_env variable 'PY_VERSION_HEX', and set it to sys.hexversion. I don't know how to do that (especially not when using pyximport), and I'm puzzled because I would have thought that dict_del_by_value.pyx is already compiled...

Thanks for any hints or workarounds!

Martin

The problem seems to be that pyximport gets its hooks a little too deeply into the import system (see the documentation: they warn that pyximport only works in simple cases).

In this case, it seems that "import sage.misc.cachefunc" tries to do a little too much when the pyximport hook is active. A workaround is to make sure it's imported before the hook is activated, i.e.,

import sage.misc.cachefunc

import pyximport; pyximport.install();
from minimal_pyx import minimal_fun

seems to do the trick.

To cover all cases, perhaps try "import sage.all" before activating the hook. That should get you into a state that's comparable to what you have at the sage prompt. Quoting from the cython manual:

This allows you to automatically run Cython on every .pyx that Python is trying to import. You should use this for simple Cython builds only where no extra C libraries and no special building setup is needed.

ostensibly, sage doesn't qualify; so I think it's not a bug that you need a workaround.

Martin R

unread,

Apr 24, 2018, 4:43:46 AM4/24/18

to sage-devel

Thank you! This gets me past the first problem. Unfortunately, I now get further compilation errors, like

Building module sage.graphs.base.sparse_graph failed: ["CompileError: command 'gcc' failed with exit status 1\n"]

and

./.pyxbld_local/temp.linux-x86_64-2.7/pyrex/sage/modules/vector_real_double_dense.c:567:10: fatal error: cython_metaclass.h: No such file or directory
#include "cython_metaclass.h"
^~~~~~~~~~~~~~~~~~~~
compilation terminated.

but it will take some time to create a minimal non-working example :-(

Thanks again,

Martin

Martin R

unread,

Apr 24, 2018, 5:20:39 AM4/24/18

to sage-devel

Here goes the new minimal non working example.

(I would use setup.py, if I knew how and if I knew that it would cure this problem)

Thanks for any help!

Martin

######## minimal.py ############
import sage.all

import pyximport; pyximport.install();
from minimal_pyx import minimal_fun

print(minimal_fun())

######## end minimal.py ############

######## minimal_pyx.pyx ############

from sage.all import *

def minimal_fun():
    A = AlternatingSignMatrix([ [0,1,0], [1,-1,1],[0,1,0]])
    return str(map(list, A.to_matrix()))

######## end minimal_pyx.pyx ############

martin@convex63:~/tmp$ sage minimal.py
/home/martin/.pyxbld/temp.linux-x86_64-2.7/pyrex/sage/modules/vector_real_double_dense.c:567:10: fatal error: cython_metaclass.h: No such file or directory

#include "cython_metaclass.h"
^~~~~~~~~~~~~~~~~~~~
compilation terminated.

Traceback (most recent call last):

File "minimal.py", line 7, in <module>
    print(minimal_fun())
File "minimal_pyx.pyx", line 6, in minimal_pyx.minimal_fun
    return str(map(list, A.to_matrix()))
File "/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/matrix/matrix_misc.py", line 30, in row_iterator
    yield A.row(i)
File "sage/matrix/matrix1.pyx", line 1180, in sage.matrix.matrix1.Matrix.row (build/cythonized/sage/matrix/matrix1.c:12016)
File "sage/structure/factory.pyx", line 368, in sage.structure.factory.UniqueFactory.__call__ (build/cythonized/sage/structure/factory.c:2014)
File "sage/structure/factory.pyx", line 411, in sage.structure.factory.UniqueFactory.get_object (build/cythonized/sage/structure/factory.c:2390)
File "/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/modules/free_module.py", line 379, in create_object
    return FreeModule_ambient_pid(base_ring, rank, sparse=sparse)
File "/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/modules/free_module.py", line 5416, in __init__
    rank=rank, sparse=sparse, coordinate_ring=coordinate_ring)
File "/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/modules/free_module.py", line 5225, in __init__
    rank, sparse, coordinate_ring)
File "/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/modules/free_module.py", line 4593, in __init__
    degree=rank, sparse=sparse, coordinate_ring=coordinate_ring)
File "/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/modules/free_module.py", line 738, in __init__
    self.Element = element_class(coordinate_ring, sparse)
File "/home/martin/sage-develop/local/lib/python2.7/site-packages/sage/modules/free_module.py", line 7243, in element_class
    import sage.modules.vector_real_double_dense

ImportError: Building module sage.modules.vector_real_double_dense failed: ["CompileError: command 'gcc' failed with exit status 1\n"]

Nils Bruin

unread,

Apr 24, 2018, 11:48:21 AM4/24/18

to sage-devel

On Tuesday, April 24, 2018 at 2:20:39 AM UTC-7, Martin R wrote:

Here goes the new minimal non working example.

(I would use setup.py, if I knew how and if I knew that it would cure this problem)

Thanks for any help!

Martin

######## minimal.py ############
import sage.all
import pyximport; pyximport.install();
from minimal_pyx import minimal_fun

print(minimal_fun())
######## end minimal.py ############

######## minimal_pyx.pyx ############
from sage.all import *

def minimal_fun():
    A = AlternatingSignMatrix([ [0,1,0], [1,-1,1],[0,1,0]])
    return str(map(list, A.to_matrix()))

######## end minimal_pyx.pyx ############

ImportError: Building module sage.modules.vector_real_double_dense failed: ["CompileError: command 'gcc' failed with exit status 1\n"]

The joys of run time imports. I don't think this is the game of whack-a-mole you want to play. Obviously, sage imports are generally *NOT* suitable to be executed with the pyximport hook active. So, you have to make sure all the imports you need are done *before* you install the hook. "sage.all" takes care of a lot, but there are plenty of imports in sage that are done only when they are needed (to save start-up time in general). I wonder how bad things crash when you start using symbolics.

In this case, you can just work your way down the failures. It looks to me like

###

import sage.all
import sage.modules.vector_real_double_dense
import sage.modules.vector_complex_double_dense

import pyximport; pyximport.install();
from minimal_pyx import minimal_fun
print(minimal_fun())
###

works, but I don't see a way of predicting beforehand what imports you need.

It may still be that a mild redesign of the sage build process might make it suitable for pyximport, or alternatively that with some extra customization pyximport can be made to work in a sage environment. However, it looks to me it's trying to compile *way* too much.

Martin R

unread,

Apr 24, 2018, 1:25:23 PM4/24/18

to sage-devel

Would a setup.py be an alternative to pyximport? If so, how would I go about it (making the example at hand work would probably be sufficient.)

Thanks in any case!

Martin

Martin R

unread,

Apr 24, 2018, 2:13:29 PM4/24/18

to sage-devel

It just ocurred to me: why does all this work with sage 8.1? Is it only the cython upgrade?

Martin

Nils Bruin

unread,

Apr 24, 2018, 3:42:43 PM4/24/18

to sage-devel

On Tuesday, April 24, 2018 at 11:13:29 AM UTC-7, Martin R wrote:

It just ocurred to me: why does all this work with sage 8.1? Is it only the cython upgrade?

Martin

I don't think the things you're running into have changed much on the sage side, so I would suspect the cython upgrade. It looks like the pyximport hook has become much more invasive. Perhaps it's worth mentioning on the cython list. Perhaps there's something that can be done to make it less so. Perhaps configuration flags to making it more opt-in.

In any case, setup.py should really work.

Alternatively, for your example

sage -sh -c 'cythonize -i -a minimal_pyx.pyx'

seems to work as well, so if you're willing to run that prior to doing `import minimal_pyx` you should be OK. That doesn't seem so bad?

This wisdom is coming from http://cython.readthedocs.io/en/latest/src/reference/compilation.html

Martin R

unread,

Apr 24, 2018, 4:22:11 PM4/24/18

to sage-devel

In any case, setup.py should really work.

indeed, it was easier than I thought! I simply copied the first example from http://cython.readthedocs.io/en/latest/src/reference/compilation.html