turning callback into generator

[Wai Yip Tung]

I'm attempting to turn some process than uses callback to return result
into a more user friendly generator. I'm hitting some road block so any
pointer would be appriciated.

Let say there is an existing method producer(x, cb). It calls the supplied
cb() every time when there is data available. The definititon of cb should
be:

def cb(data)

To turn it into generator, I tried:

def product_generator(x):
producer(x,cb)

def cb(data):
yield data

But that's not really the way Python works. 'yield' is not only a
imperative statement but it also the keyword to signify the containing
function as a generator function. So in this case product_generator is an
oridinary function and cb is a generator.

Note that producer() is an existing method that I cannot change. Otherwise
I might strip the cb and yield directly from the producer().

If python would define another keyword 'generator' to explicit distinguish
normal function from generator and decouple it from the 'yield' keyword,
it could perhaps be done that way:

generator product_generator(x):
producer(x,cb)

def cb(data):
yield data

Is this just an syntactic issue or am I hitting on some generator's
implementation issue? Is there a proper way to do this in Python 2.3?

Thanks for any pointer.

[Peter Otten]

Wai Yip Tung wrote:

> I'm attempting to turn some process than uses callback to return result
> into a more user friendly generator. I'm hitting some road block so any
> pointer would be appriciated.

I asked the same question a while back and there were no satisfying
suggestions.

http://mail.python.org/pipermail/python-list/2003-December/197726.html

At the very least you need threads, and when the generator is not fully
exhausted it's easy to end up with a thread waiting forever.

I finally dropped the idea, but if you are really determined and want to
hack something together, the following might (or might not, this is really
a shot in the dark) serve as a starting point:

http://mail.python.org/pipermail/python-list/2003-July/173872.html

Peter

[Bengt Richter]

Not that familiar with python threads and queue, but here's a start:

>>> import threading
>>> import Queue
>>> q = Queue.Queue(3) # ridiculously short
>>>
>>> def producer(n, cb):
... for i in xrange(n): cb(i)
...
>>> def product_generator(p, *a, **kw):
... tpq = threading.Thread(target=p, args=a, kwargs=kw)
... tpq.start()
... try:
... while True: yield q.get(True,5)
... except Queue.Empty:
... print 'No data for 5 seconds'
...
>>> for item in product_generator(producer, 8, q.put): print item
...
0
1
2
3
4
5
6
7
No data for 5 seconds
>>>

Not tested beyond what you see ;-)

Regards,
Bengt Richter

[Peter Otten]

Bengt Richter wrote:

> Not that familiar with python threads and queue, but here's a start:
>
> >>> import threading
> >>> import Queue
> >>> q = Queue.Queue(3) # ridiculously short

But already too long if the items yielded are somehow interdependant.

> >>>
> >>> def producer(n, cb):
> ... for i in xrange(n): cb(i)
> ...
> >>> def product_generator(p, *a, **kw):
> ... tpq = threading.Thread(target=p, args=a, kwargs=kw)
> ... tpq.start()
> ... try:
> ... while True: yield q.get(True,5)
> ... except Queue.Empty:
> ... print 'No data for 5 seconds'
> ...
> >>> for item in product_generator(producer, 8, q.put): print item
> ...
> 0
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> No data for 5 seconds
> >>>
>
> Not tested beyond what you see ;-)

The problem is _not_ to signal that there are no more items to be expected -
you can easily do that by wrapping the producer to put a special object
into the queue when it's done and test for that in the product_generator().
This would avoid the 5-second jet lag. It's a bit harder to avoid a
dangling thread when you have code like

for item in product_generator(...):
if pred(item): break

You could attack that with a long timeout for Queue.put(), too, but I'd
rather signal the generator that we're done (via a second Queue, maybe).
Unfortunately this imposes changes on the client code, thus defeating the
original goal of making generators and visitors/callbacks freely
interchangeable.

Peter

[Wai Yip Tung]

Thanks for the pointers. Actually I'm looking for a solution that do not
use thread or to buffer all result in memory since one of the great thing
about generator is that it can achieve the process without using those
resource intensive techniques.

About the dangling thread, perhaps the destructor method __del__() can
help (http://www.python.org/dev/doc/devel/ref/customization.html). It is
really nasty to end up with dangling threads.

Wai Yip