In 0.22, overwriting base cdef method with except * and then calling base directly makes gcc fail

[Valcortez]

Hi,

This is an issue which was not a problem in 0.21 with identical code. If the base class (MyClassBase) has a cdef method (apply) which then gets overwritten and except * is added to the method in the derived class, then when calling the method on the base class directly (MyClassBase.apply(self)) then gcc will fail. Following is example code showing the issue:

a.pxd

cdef class MyClassBase(object):
    cdef void apply(self)

cdef class MyClassNext(MyClassBase):
    cdef void apply(self) except *

a.pyx

cdef class MyClassBase(object):
    cdef void apply(self):
        return

cdef class MyClassNext(MyClassBase):
    cdef void apply(self) except *:
        return

b.pxd

from a cimport MyClassNext

cdef class MyClass(MyClassNext):
    cdef void apply(self) except *

b.pyx

from a cimport MyClassNext

cdef class MyClass(MyClassNext):
    cdef void apply(self) except *:
        MyClassNext.apply(self)

setup.py

from distutils.core import setup
from distutils.extension import Extension
import Cython.Compiler.Options
#Cython.Compiler.Options.annotate = True
from Cython.Distutils import build_ext
from os.path import join, sep, dirname, abspath

def expand(*args):
    return abspath(join(dirname(__file__), 'src', *args))

def get_extensions_from_sources():
    ext_modules = []
    pyx = expand('b.pyx')
    ext_modules.append(Extension('src.b', [pyx]))
    pyx = expand('a.pyx')
    ext_modules.append(Extension('src.a', [pyx]))
    return ext_modules

ext_modules = get_extensions_from_sources()

setup(
    name='Playground',
    version='.1',
    author='Matthew Einhorn',
    ext_modules=ext_modules,
    cmdclass={'build_ext': build_ext},
    packages=['src']
      )

output then is

python setup.py build_ext --inplace -f
running build_ext
cythoning G:\Python\libs\Playground\src\cypp_playground.pyx to G:\Python\libs\Pl
ayground\src\cypp_playground.c
building 'src.cypp_playground' extension
e:\MinGW\bin\gcc.exe -mdll -O -Wall -Ie:\Python27\include -Ie:\Python27\PC -c G:
\Python\libs\Playground\src\cypp_playground.c -o g:\python\libs\playground\src\c
ypp_playground.o
G:\Python\libs\Playground\src\cypp_playground.c: In function '__pyx_f_3src_15cyp
p_playground_7MyClass_apply':
G:\Python\libs\Playground\src\cypp_playground.c:664:3: warning: implicit declara
tion of function '__pyx_f_12cyplayground_11MyClassNext_apply' [-Wimplicit-functi
on-declaration]
   __pyx_f_12cyplayground_11MyClassNext_apply(((struct __pyx_obj_12cyplayground_
MyClassNext *)__pyx_v_self)); if (unlikely(PyErr_Occurred())) {__pyx_filename =
__pyx_f[0]; __pyx_lineno = 6; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
   ^
writing g:\python\libs\playground\src\cypp_playground.def
e:\MinGW\bin\gcc.exe -shared -s g:\python\libs\playground\src\cypp_playground.o
g:\python\libs\playground\src\cypp_playground.def -Le:\Python27\libs -Le:\Python
27\PCbuild -lpython27 -lmsvcr90 -o G:\Python\libs\Playground\src\cypp_playground
.pyd
g:\python\libs\playground\src\cypp_playground.o:cypp_playground.c:(.text+0x3fe):
 undefined reference to `__pyx_f_12cyplayground_11MyClassNext_apply'
collect2.exe: error: ld returned 1 exit status
error: command 'e:\\MinGW\\bin\\gcc.exe' failed with exit status 1
make: *** [force] Error 1

Note that for the issue to arise the inheritance needs to be at least two levels deep, like in the example. Also, removing the except * make the issue go away.

[Valcortez]

I've done a git bisect, and it says that e6f48d7e08d13c2a32d0f90b785351ff655b8ead (also compare "except" declaration when comparing C function signatures) is responsible for this issue.

[Ryan Pessa]

Is anything happening with this bug? It would be nice if it was fixed...

[Thomas Hisch]

This bug also affects bempp - see https://github.com/bempp/bempp/issues/206

[Tennessee Leeuwenburg]

I ran into this in the wild today. I was trying to use macports to install kivy, and it failed due to this particular cython issue. I'm sad about it, but it's way beyond be to fix. Will be grateful to see a fix to this.

[Stefan Behnel]

I don't think this is legitimate code. If a subtype adds an exception
declaration, then it depends on what Cython thinks you're calling in a
given case whether an exception will be handled or not. That sounds very
brittle.

In latest 0.22.x/master, I get an error for your code:

"""
Error compiling Cython file:
------------------------------------------------------------
...

cdef class MyClassBase(object):
cdef void apply(self):
return

cdef class MyClassNext(MyClassBase):
cdef void apply(self) except *:

^
------------------------------------------------------------

a.pyx:7:9: 'apply' already defined
"""

Maybe not the best error message ever, but it prevents the code from
compiling, rather than failing later in C.

Stefan

[Stefan Behnel]

Thomas Hisch schrieb am 25.03.2015 um 19:48:
> This bug also affects bempp - see https://github.com/bempp/bempp/issues/206

That's actually a different case. Here, the .pxd file deliberately lies
about the actual declaration in the code, meaning that anything that calls
the functions/methods from within the module will behave different from
code that calls them from other modules. I hope this wasn't intended. It's
a very bad idea.

Stefan

[Stefan Behnel]

Tennessee Leeuwenburg schrieb am 16.04.2015 um 09:06:
> I ran into this in the wild today. I was trying to use macports to install
> kivy, and it failed due to this particular cython issue. I'm sad about it,
> but it's way beyond be to fix. Will be grateful to see a fix to this.

What's the problem you're seeing there? How does the failing code look like?

And, why do you need Cython to install kivy?

Stefan

[Matthew Einhorn]

I abstracted the example code I posted above based on a similar issue we were having in Kivy where gcc failed. Kivy has Cython code, hence the issue of installing Kivy with 0.22.

Regarding the issue, we decided to fix it in Kivy by making them all except -1. But I'm wondering now, if the parent class declares with `except -1`, would a derived class be able to overwrite it with `except *`? I'm not sure if we need that, but am just wondering about that case.

Thanks

--

---
You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/r3dQSfOKMjI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stefan Behnel]

Matthew Einhorn schrieb am 16.04.2015 um 19:23:
> I abstracted the example code I posted above based on a similar issue we
> were having in Kivy where gcc failed. Kivy has Cython code, hence the issue
> of installing Kivy with 0.22.

I understand that developers need Cython to work on kivy. I was just
wondering why users need Cython in order to *install* kivy. Usually,
shipping the generated (and tested) C sources avoids that.

> Regarding the issue, we decided to fix it in Kivy by making them all except
> -1. But I'm wondering now, if the parent class declares with `except -1`,
> would a derived class be able to overwrite it with `except *`? I'm not sure
> if we need that, but am just wondering about that case.

In this case, calling

(<BaseClass>x).method()

would propagate an exception only if method() returns -1, but

(<ChildClass>x).method()

would always check for an exception and propagate it. Now, if "x" is
actually of type ChildClass, then it could expect that calling code would
always check for an exception and return, say, -2 in that case. This is not
covered by the first case above, and the exception would fail to propagate
whenever calling code thinks it's calling the base type method.

The other direction could work, though. Meaning, if you tighten the
exception propagation exception from "except *" in the base class (always
check) to "except -1" in the child class, then code that thinks it's
calling the base class method would always check for an exception, and code
that knows that it's calling the child method would explicitly check for
the error return value and thus avoid redundant checks.

However, that being said, "except *" cannot actually be used in
non-external functions/methods (i.e. in the actual implementation), as
Cython wouldn't know what to return in the case of an error.

Stefan

[Tennessee Leeuwenburg]

I can install Kivy in as a MacOS Application using their pre-built binaries. That would be fine for a simpler application, but I really want to make use of all the other stuff in my Python ecosystem at the same time. To install it into the ecosystem, I need to install it via macports or compile my own version. Macports compiles its packages on the machine rather than using pre-built binaries.

The Kivy application on OSX can have other applications installed into its virtual environment, although I haven't personally figured that step out yet.

I don't have access to the failing code, because the Kivy developers have added a check to the makefile, and it doesn't even try to compile if version 22 is detected. You'd need a kivy developer to resolve the issue -- I'm barely literate in C.

So ... I need Cython because Macports is the Linux-developer's method-du-jour for installing packages on OSX. Some people use an alternative system called homebrew, but they're both solving the same problem, and macports has more users and more mindshare. It's analagous to apt or yum.

--

---
You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/r3dQSfOKMjI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--------------------------------------------------
Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"

[Tennessee Leeuwenburg]

Hi Matthew,

Thanks very much for diving in here.

[Matthew Einhorn]

Hi Tennessee,

I think Stefan means that if we shipped the c files with the release (perhaps generated by a older version of cython for now) then you'd not need cython on your system since you'd only need a gcc compiler. I guess we have never done that but maybe we should for releases.

Anyway Stefan, thanks for the explanations.

Matt