Turning callback-based C interface into Python generator/iterator-based interface with Cython

[Mikhail Korobov]

I want to make an iterator-based interface for the C function that takes a callback as an argument, but don't know how :)

 

Based on callback example from cython github repo I was able to make this: https://github.com/kmike/datrie/commit/137b269246bd952ceef0727adb8c7d6a58d4cf2b - but I can't wrap my head around at how to yield a value from a callback. Are there any pointers? 

[Stefan Behnel]

Mikhail Korobov, 09.07.2012 01:36:

Yes, these things are rather tricky. Basically, the C code uses a visitor
pattern ("push") and you want to map it to iteration ("pull").

There are ways to do that, but I'd go two steps back and ask some questions
first. Is there really no C interface to the data structure that resembles
iteration more closely? Do you really need to expose it as iterator, or
would a visitor pattern work in Python space as well?

However, in this case, I agree that the visitor pattern looks like a rather
clumsy choice (even in C, which lacks closures after all).

Stefan

[Mikhail Korobov]

Thanks for your answer, it was helpful.

The library provides 'walk' interface with explicit state passing (it can be used to walk specific paths), but this interface is unfortunately not appliable to 'da_enumerate' method (that walks all paths) because there is no interface for obtaining all possible paths from a given node so I can't walk all paths using 'walk' interface. 

It should be possible to rewrite library's 'da_enumerate' from the current (recursive) implementation to the implementation with explicit state passing though; this function it is quite small and well-written. But the library lacks tests and my C skills are not fresh (I stopped writing C many years ago) so if this is the only way or other ways are harder I'd better leave visitor pattern as-is for now and (maybe) come back to it later if this decision bite me. 

What do you think, what are the other ways?

понедельник, 9 июля 2012 г., 13:35:49 UTC+6 пользователь Stefan Behnel написал:

[Dag Sverre Seljebotn]

On 07/09/2012 10:48 AM, Mikhail Korobov wrote:
> Thanks for your answer, it was helpful.
>
> The library provides 'walk' interface with explicit state passing (it
> can be used to walk specific paths), but this interface is unfortunately
> not appliable to 'da_enumerate' method (that walks all paths) because
> there is no interface for obtaining all possible paths from a given node
> so I can't walk all paths using 'walk' interface.
>
> It should be possible to rewrite library's 'da_enumerate' from the
> current (recursive) implementation to the implementation with explicit
> state passing though; this function it is quite small and well-written.
> But the library lacks tests and my C skills are not fresh (I stopped
> writing C many years ago) so if this is the only way or other ways are
> harder I'd better leave visitor pattern as-is for now and (maybe) come
> back to it later if this decision bite me.
>
> What do you think, what are the other ways?

I think the way to solve it is to use two threads, one that pulls from
the queue and one that pushes to the queue. It is possible to program
things so that you have a 1-length queue; i.e. the thread that pushes to
the queue blocks until the iterator reads.

Myself I'd use ZeroMQ (pyzmq) to prototype that.

Dag