Is it possible to cdef a metaclass?

[Simon King]

Hi!

Sage uses some metaclasses, such as
sage.misc.nested_class.NestedClassMetaclass, and also
sage.misc.classcall_metaclass.ClasscallMetaclass, which is derived
from the former.

Some people try to cythonise the metaclasses, aiming at a faster class
creation (cf. sage trac #12808).

What works is to rename the ..._metaclass.py files into .pyx files,
but keep the classes non-cdef.

It also works to create a cdef class ClasscallType (implementing the
methods of ClasscallMetaclass in Cython), and then have a non-cdef
class ClasscallMetaclass(ClasscallType, NestedClass):
pass

I tried to go further, and have
cdef class NestedClassMetaclass(type):
...
cdef class (ClasscallMetaclass(NestedClassMetaclass):
...

However, that led to segfaults. Therefore my question: Is it possible
to have a metaclass cdef'ed? Or is there likely to be a different
error?

Best regards,
Simon

[Vitja Makarov]

2012/4/24 Simon King <simon...@uni-jena.de>:

No, cdef'ed metaclasses aren't supported. Can you provide a minimal
example that segfaults that must be a bug?

--
vitja.

[Simon King]

Hi Vitja,

On 24 Apr., 11:19, Vitja Makarov <vitja.maka...@gmail.com> wrote:
> 2012/4/24 Simon King <simon.k...@uni-jena.de>:

> No, cdef'ed metaclasses aren't supported.

Thank you! So, it will be the slightly more complicated approach with
a metaclass being a non-cdef class that inherits from a cdef class (it
is tested that it works).

> Can you provide a minimal
> example that segfaults that must be a bug?

Well, is it a bug if it is not supported? But I can try to cook
something up.

Best regards,
Simon

[Simon King]

Hi all!

On 24 Apr., 11:57, Simon King <simon.k...@uni-jena.de> wrote:
> > Can you provide a minimal
> > example that segfaults that must be a bug?
>
> Well, is it a bug if it is not supported? But I can try to cook
> something up.

Here is a small example.

Put the following into a file meta.pyx:
cdef class CyMetaclass(type):
def __init__(cls, *args):
print cls.__name__

class PyMetaclass(type):
def __init__(cls, *args):
print
cls.__name__

Then, start Sage and do:

sage: attach meta.pyx
Compiling ./meta.pyx...
sage: class A(object):
....: __metaclass__ = PyMetaclass
....:
A
sage: a = A()
sage: a
<__main__.A object at 0x450bd10>
sage: class B(object):
....: __metaclass__ = CyMetaclass
....:
B
*** glibc detected *** python: corrupted double-linked list:
0x0000000004611870 ***

Afterwards, it is not possible to do anything, as neither the sage
prompt nor a shell prompt appears - I have to close the terminal.

I hope that the example helps - in fact, Sage's NestedClassMetaclass
provides an __init__ method, and so the example probably demonstrates
what goes wrong in the "real" application (including the fact that one
does not return to the shell).

Best regards,
Simon

[Simon King]

PS:

On 24 Apr., 12:07, Simon King <simon.k...@uni-jena.de> wrote:
> Put the following into a file meta.pyx:
> cdef class CyMetaclass(type):
>     def __init__(cls, *args):
>         print cls.__name__
>
> class PyMetaclass(type):
>     def __init__(cls, *args):
>         print
> cls.__name__

If one rename __init__ into a different name (say, "blubb"), one gets
a different segfault:

sage: attach meta.pyx
Compiling ./meta.pyx...
sage: class A(object):

....: __metaclass__ = CyMetaclass
....:
/home/simon/SAGE/sage-5.0.beta7/local/lib/libcsage.so(print_backtrace
+0x31)[0x7f73b5217966]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libcsage.so(sigdie+0x14)
[0x7f73b5217998]
/home/simon/SAGE/sage-5.0.beta7/local/lib/
libcsage.so(sage_signal_handler+0x20c)[0x7f73b52175e6]
/lib64/libpthread.so.0(+0xfd00)[0x7f73ba500d00]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.1.0(+0xa6ff7)
[0x7f73ba7b4ff7]
/home/simon/.sage//temp/linux_sqwp.site/3264//spyx//
_home_simon_SAGE_tests_meta_pyx/
_home_simon_SAGE_tests_meta_pyx_0.so(+0x1fe8)[0x7f738c1ddfe8]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.1.0(+0xaaba8)
[0x7f73ba7b8ba8]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyObject_Call+0x53)[0x7f73ba75b823]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyObject_CallFunctionObjArgs+0xb0)[0x7f73ba75c110]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x1dcb)[0x7f73ba7fbb9b]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f73ba800d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCode+0x32)[0x7f73ba800e82]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x57bf)[0x7f73ba7ff58f]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f73ba800d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f73ba7fefc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f73ba800d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f73ba7fefc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x5dc1)[0x7f73ba7ffb91]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f73ba800d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f73ba7fefc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f73ba800d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f73ba7fefc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f73ba800d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f73ba7fefc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f73ba800d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCode+0x32)[0x7f73ba800e82]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyRun_FileExFlags+0xb0)[0x7f73ba8233c0]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyRun_SimpleFileExFlags+0xdf)[0x7f73ba823e5f]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.1.0(Py_Main
+0xb6b)[0x7f73ba836a6b]
/lib64/libc.so.6(__libc_start_main+0xed)[0x7f73b9b2423d]
python[0x4006c1]

------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred in Sage.
This probably occurred because a *compiled* component of Sage has a
bug
in it and is not properly wrapped with sig_on(), sig_off(). You might
want to run Sage under gdb with 'sage -gdb' to debug this.
Sage will now terminate.
------------------------------------------------------------------------
/home/simon/SAGE/sage-5.0.beta7/spkg/bin/sage: Zeile 308: 3264
Speicherzugriffsfehler sage-ipython "$@" -i

This time, one does return to the shell prompt.

Cheers,
Simon

[Simon King]

PPS:

Florent Hivert's solution at #12808 is roughly as follows:

In meta.pyx:
cdef class CyMetaclass:
def __call__(cls, *args, **opts):
print cls.__name__
return type.__call__(cls, *args, **opts)

class PyMetaclass(type):
def __init__(cls, *args):
print cls.__name__

class PyCyMetaclass(CyMetaclass, PyMetaclass): pass

And then in a Sage session:

sage: attach meta.pyx
Compiling ./meta.pyx...
sage: class A(object):

....: __metaclass__ = PyCyMetaclass

....:
A
sage: a = A()

A
sage: a
<__main__.A object at 0x450bf50>

Hence, the metaclass is non-cdef, in particular the init method
belongs to a non-cdef class, but the __call__ method is inherited from
a cdef class.

Best regards,
Simon

[Stefan Behnel]

Simon King, 24.04.2012 12:29:

> In meta.pyx:
> cdef class CyMetaclass:
> def __call__(cls, *args, **opts):
> print cls.__name__
> return type.__call__(cls, *args, **opts)

Did you notice that you didn't call the supertype's __init__() in your
first example? Not sure if it matters - just occurred to me.

Stefan

[Simon King]

Hi Stefan,

On 24 Apr., 12:39, Stefan Behnel <stefan...@behnel.de> wrote:
> Did you notice that you didn't call the supertype's __init__() in your
> first example? Not sure if it matters - just occurred to me.

Yes, I noticed, and no, calling it does not help.

Actually in the current code in sage.misc.nested_class.py,
type.__init__ it is not called either.

Let's try in meta.pyx:
cdef class CyMetaclass(type):
cdef class CyMetaclass(type):
def __init__(cls, *args, **opts):
print cls.__name__
type.__init__(cls, *args, **opts)

class PyMetaclass(type):
def __init__(cls, *args):
print cls.__name__

In a Sage session:

sage: attach meta.pyx
Compiling ./meta.pyx...
sage: class A(object):

....: __metaclass__ = CyMetaclass
....:
/home/simon/SAGE/sage-5.0.beta7/local/lib/libcsage.so(print_backtrace
+0x31)[0x7f2b4b8ff966]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libcsage.so(sigdie+0x14)
[0x7f2b4b8ff998]
/home/simon/SAGE/sage-5.0.beta7/local/lib/
libcsage.so(sage_signal_handler+0x20c)[0x7f2b4b8ff5e6]
/lib64/libpthread.so.0(+0xfd00)[0x7f2b50be8d00]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.1.0(+0xa6ff7)
[0x7f2b50e9cff7]
/home/simon/.sage//temp/linux_sqwp.site/4043//spyx//
_home_simon_SAGE_tests_meta_pyx/
_home_simon_SAGE_tests_meta_pyx_0.so(+0x23c8)[0x7f2b228c53c8]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.1.0(+0xaaba8)
[0x7f2b50ea0ba8]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyObject_Call+0x53)[0x7f2b50e43823]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyObject_CallFunctionObjArgs+0xb0)[0x7f2b50e44110]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x1dcb)[0x7f2b50ee3b9b]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f2b50ee8d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCode+0x32)[0x7f2b50ee8e82]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x57bf)[0x7f2b50ee758f]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f2b50ee8d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f2b50ee6fc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f2b50ee8d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f2b50ee6fc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x5dc1)[0x7f2b50ee7b91]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f2b50ee8d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f2b50ee6fc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f2b50ee8d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f2b50ee6fc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f2b50ee8d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalFrameEx+0x51f1)[0x7f2b50ee6fc1]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCodeEx+0x855)[0x7f2b50ee8d45]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyEval_EvalCode+0x32)[0x7f2b50ee8e82]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyRun_FileExFlags+0xb0)[0x7f2b50f0b3c0]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.
1.0(PyRun_SimpleFileExFlags+0xdf)[0x7f2b50f0be5f]
/home/simon/SAGE/sage-5.0.beta7/local/lib/libpython2.7.so.1.0(Py_Main
+0xb6b)[0x7f2b50f1ea6b]
/lib64/libc.so.6(__libc_start_main+0xed)[0x7f2b5020c23d]

python[0x4006c1]

------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred in Sage.
This probably occurred because a *compiled* component of Sage has a
bug
in it and is not properly wrapped with sig_on(), sig_off(). You might
want to run Sage under gdb with 'sage -gdb' to debug this.
Sage will now terminate.
------------------------------------------------------------------------

/home/simon/SAGE/sage-5.0.beta7/spkg/bin/sage: Zeile 308: 4043
Speicherzugriffsfehler sage-ipython "$@" -i

Best regards,
Simon

[Vitja Makarov]

2012/4/24 Simon King <simon...@uni-jena.de>:

Hmm, it seems that Cython doesn't support type objects:

cdef class CyMetaclass(type):
def __init__(cls, *args):
print 'haha'
print cls.__name__

CyMetaclass('foo', (object,), {})

print cls.__name__ causes segfault here and glibc's SIGABRT if removed.

--
vitja.

[Stefan Behnel]

Vitja Makarov, 24.04.2012 14:52:

> Hmm, it seems that Cython doesn't support type objects:
>
> cdef class CyMetaclass(type):
> def __init__(cls, *args):
> print 'haha'
> print cls.__name__
>
> CyMetaclass('foo', (object,), {})
>
> print cls.__name__ causes segfault here and glibc's SIGABRT if removed.

Maybe be due to this:

http://trac.cython.org/cython_trac/ticket/152

Stefan

[Florent Hivert]

Hi Simon,

> On 24 Apr., 12:39, Stefan Behnel <stefan...@behnel.de> wrote:
> > Did you notice that you didn't call the supertype's __init__() in your
> > first example? Not sure if it matters - just occurred to me.
>
> Yes, I noticed, and no, calling it does not help.
>
> Actually in the current code in sage.misc.nested_class.py,
> type.__init__ it is not called either.
>
> Let's try in meta.pyx:
> cdef class CyMetaclass(type):
> cdef class CyMetaclass(type):
> def __init__(cls, *args, **opts):
> print cls.__name__
> type.__init__(cls, *args, **opts)
>
> class PyMetaclass(type):
> def __init__(cls, *args):
> print cls.__name__

I think you should add the following three line at the beginning of you file:

cdef extern from "object.h":
ctypedef class __builtin__.type [object PyHeapTypeObject]:
pass

I'm not exactly sure what it does if you don't do that but it solve the
segfault for me.

Cheers,

Florent

Meta2.pyx::
============
cdef extern from "object.h":
ctypedef class __builtin__.type [object PyHeapTypeObject]:
pass

cdef class CyMetaclass(type):
def __init__(cls, *args, **opts):
print cls.__name__
type.__init__(cls, *args, **opts)

============
sage: attach Meta2.pyx
Compiling ./Meta2.pyx...

sage: class A(object):
....: __metaclass__ = CyMetaclass
....:

A

[Simon King]

Hi Florent,

On 24 Apr., 15:02, Florent Hivert <Florent.Hiv...@lri.fr> wrote:
> cdef extern from "object.h":
>     ctypedef class __builtin__.type [object PyHeapTypeObject]:
>         pass

That's very cool! It even works in the "real" application.

I will see (at #12808) whether it has an effect on the time spent for
creating classes. I think the speedup could be considerable: After
all, NestedClassMetaclass occurs even more often in Sage than
ClasscallMetaclass.

Best regards,
Simon