twisted defer.inlineCallbacks dosn't work with cython

[dynamicgl]

Hi All

I just to let you know that defer.inlineCallbacks can't work with
cython well. an example:


def await(seconds):
d = defer.Deferred()
reactor.callLater(seconds, d.callback, True)
return d

@defer.inlineCallbacks
def test():
thing = yield await(2)
print "hey"


will fail to work on cython Error info like:

TypeError: inlineCallbacks requires <built-in function test> to
produce a generator; instead got <cy3.__pyx_scope_struct__test object
at 0x02808848)

test is the function where inlineCallbacks uses.

It looks like cython changes the fucntion test to
cy3.__pyx_scope_struct__test internally so generator can't work
properly.

I know this is not a common case to use cython & twisted but my codes
use inlinecallbacks extensively. Rewriting the whole part with
traditional callback functions is an effort.

Any idea to work around?

Thanks

gelin yan

[mark florisson]

The problem is that Cython's generator is not a subclass of Python's generator, and I'm not sure it can be, as code can then make assumptions based on the type which means we need to implement a valid Python generator at the C level as well (it seems to have a frame and a code object).

It would be nice if there was some way to use the abc module here, e.g. collections.Generator or something, but there isn't. Also due to the way Cython currently works, each Cython module will have a different generator type.

I think Twisted could be adapted to define a GeneratorInterface of some sorts using zope.interface, and use that instead of instance checking. Alternatively, it could check for the iterator protocol + send/throw etc. z.i would look like something along the lines of:

# cython_module.pyx
from generator_interface import GeneratorInterface as GI
import zope.interface

def generator():
   yield

cython_generator_type = type(generator())

zope.interface.classImplements(cython_generator_type, GI)

print GI.implementedBy(cython_generator_type)

# generator_interface.py
import types
import zope.interface

class GeneratorInterface(zope.interface.Interface):
   pass

zope.interface.classImplements(types.GeneratorType, GeneratorInterface)

The easy thing to do would of course be to not use Cython for inline callbacks.

[dynamicgl]

On Apr 11, 12:39 am, mark florisson <markflorisso...@gmail.com> wrote:

Hi
Thanks for your info..You are right, I should keep my codes in .pyx
away from inlineCallbacks.
Use generatoeinterface need more work on twisted side, I am afraid
that nobody will be interested in. After all, use inlineCallbacks with
cython
is a rare case.

[Stefan Behnel]

mark florisson, 10.04.2012 18:39:

> The problem is that Cython's generator is not a subclass of Python's
> generator, and I'm not sure it can be, as code can then make assumptions
> based on the type which means we need to implement a valid Python generator
> at the C level as well (it seems to have a frame and a code object).

In fact, it must not be a subtype for exactly this reason.

> It would be nice if there was some way to use the abc module here, e.g.
> collections.Generator or something, but there isn't.

That's because there has never been a need for another generator type in
CPython.

I agree that it's Twisted that needs fixing here - assuming it's not
exactly a generator type that it really needs, including frames and all
that. It should either drop the type test or replace it by a test for what
it actually needs to assert about the decorated function.

I think it would be worth asking about this on their mailing list.

Stefan