Simplify code working on top of gen.engin

[Serge S. Koval]

Hi,

 I was following asynchronous discussions in python mailing list and stumbled upon Guido idea of callback handling: raise StopIteration if you want to return something. It is pretty neat idea and not sure why I haven't thought of it before :-)

 As an example, if you use gen.engine now, your function might look like this:

@gen.engine

def handle(param1, param2, callback):

  if param1 == '1':

    callback(param1)

  elif param2 == '2':

    callback(param2)

  else:

    callback(None)

As Ben mentioned, with current approach, you have to fall through the method after you called callback function to prevent rest of the function from the execution.

This won't work as expected:

@gen.engine

def handle(param1, param2, callback):

  if param1 == '1':

    callback(param1)

  if param2 == '2':

    callback(param2)

  else:

    callback(None)

Instead, you can write something like this, which does not look very pretty

@gen.engine

def handle(param1, param2, callback):

  if param1 == '1':

    callback(param1)

    raise StopIteration()

  if param2 == '2':

    callback(param2)

  else:

    callback(None)

Luckily enough, you can stop iterator by raising StopIteration exception and you can use exception instance to pass values back from generator.

For example, previous function can be modified to look like this:

@gen.async

def handle(param1, param2):

  if param1 == '1':

    gen.ret(param1)

  

  if param2 == '2':

    gen.ret(param2)

 

Comments:

1. gen.async is just modified version of gen.engine with custom runner;

2. gen.ret will raise StopIteration() with return value(s);

3. Callback won't be exposed to the function and will be called by the runner. No need to keep queue of "not called" callbacks for safety reasons - there's guarantee that callback will be called;

4. Default return value is None, same as in normal Python functions. If caller does not expect None, that's their problem;

5. Only problem that I see - if you have exception handler in the function, you have to care about StopIteration exceptions:

@gen.async

def handle(param1, param2):

  try:

    if param1 == '1':

      gen.ret(param1)

  

    if param2 == '2':

      gen.ret(param2)

  except StopIteration:

    raise

  except Exception:

    # Do something

    pass

 

I can make gist to illustrate approach along with few examples. 

Any thoughts or comments?

Serge.

[Serge S. Koval]

3. Callback won't be exposed to the function and will be called by the runner. No need to keep queue of "not called" callbacks for safety reasons - there's guarantee that callback will be called;

Minor correction - with current tornado.gen implementation it is possible to skip calling callback function and it won't trigger the error:

Consider following:

@gen.engine

def handle(a, callback):

  b = yield gen.Task(fetch, a)

  if b == '1':

    callback('hi')

@gen.engine

def test():

  yield gen.Task(handle, 'http://www.google.com/')

  print 'hi'

In this example, 'hi' will be never printed if fetch does not return 1. Sure, it is programmers concern to make sure that all terminating branches should run callback, but if that's decorator responsibility to run callback function, this type of error should not happen.

Serge.

[A. Jesse Jiryu Davis]

I like this idea. So with your example:

@gen.async

def handle(param1, param2):
    ....

The decorated function handle gets an optional callback kwarg, perhaps?:

handle(param1, param2, callback=None)

This way I can either choose to block on it:

yield gen.Task(handle, 'a', 'b')
# This line runs after handle completes

or not:

handle('a', 'b')
# This line runs immediately without waiting for handle

[Serge S. Koval]

I see where're you're going, but my idea was simpler.

There's no callback parameter in the handle function. Whenever function is decorated with @gen.async, it will be converted to asynchronous function with implicit callback parameter (handled by decorator). If function returns decorator, it is treated as yield point. If it finishes with StopIteration, use its value as callback parameter and execute callback with the values. So, it will be possible to run example above with:

handle('a', 'b', callback=result)

Not sure it is good idea to mix async/non-async functions in one place, but maybe there's way to implement it properly. For now, I thought of easier way to return values from gen.engine decorated functions.

Serge.

[Boris Kizeshteyn]

Excuse my brevity, I'm on the run
G
On Oct 14, frui2012, at 5:12 PM, "Serge S. Koval" <serge...@gmail.com> wrote:

> parameterjm

[A Jesse Jiryu Davis]

Oh! So the thing returned by gen.async(function) is a subclass of
gen.YieldPoint?

[Serge S. Koval]

No, gen.async is just customised version of gen.engine, which uses customised gen.engine.Runner class.

I will provide gist tomorrow along with few usage samples.

Serge.

[Daniel Dotsenko]

What's wrong with plain returning?

Generators (those using yield) don't need to finish up with StopIteration() they just need to run out of code. return is a way to make it run out of code.

The only reason StopIteration() works magically (as in does not cause Exception, but stops iteration) is that it's so per PEP. In fact, any version of running out of code, including raising something will stop iteration, just in case of return or StopIteration() that running out of code does not produce noise.

Here is a real, working example utilizing present (as in v2.4) tornado.gen on a modified StaticHTTPServer:

    @asynchronous

    @gen.engine

    def get(self, path):

        answer = yield gen.Task(self._get_file_stats, path)

        if 'redirected' in answer:

            self.finish()

            return

        elif 'error' in answer:

            raise answer['error']

        with open(abspath, "rb") as fp:

            while size:

                not_important, chunk = yield [

                    gen.Task(self.flush)

                    , gen.Task(self._read_from_file, fp)

                ]

                size -= len(chunk)

                self.write(chunk)

            self.finish()

Note, all ways of running out of code are exemplified, except for StopIteration():

- explicit return (return None)

- raising non StopIteration (intentionally bubbling up)

- implicit return at the end (return None)

I'd say current tornado.gen is so nice and pleasant to work with, it's "fricken magical" category.

Full code here:

https://github.com/dvdotsenko/tornado/blob/stable/tornado/web.py

Daniel.

[Serge S. Koval]

In your example, there's no callback parameter, so you don't have to worry about return control to the caller.

Imagine you're writing application on top of Tornado framework and need to make asynchronous database queries along with some logic in this method. So, "normal" application, not just web request handler.

With @gen.engine it is slightly simpler: you're attempting to write asynchronous code in sort of synchronous fashion, but still have to make sure you don't forget to call callback function (or you can just raise exception to indicate error):

@gen.engine

def get_user(user_id, callback):

  user = yield gen.Task(..)

  if not user:

    callback(None)

    return

  data = yield.gen.Task(..)

  if not data:

    callback(None)

    return

  user.set_data(data)

  callback(user)

Sure, this works, but when you have more complex logic in your function and you return different stuff based on different conditions, it becomes a little messed up - you will have pairs of callback(arg); return everywhere in your application code.

Also, I really forgot that you can use return in generators without parameters. In one of the newer python 3 peps, there's mention that 'return with parameter' in generator will be supported and it will work as raise StopIteration(parameter). So, code that I provided in one of the previous messages will look like:

@gen.async

def get_user(user_id):

  user = yield gen.Task(...)

  if not user:

    return None

  data = yield gen.Task(...)

  if not data:

    return None

  user.set_data(data)

  return user

And it looks like ordinary "synchronous" python function, doing asynchronous stuff without help of greenlets.

Serge.

[Andrew Grigorev]

15.10.2012 08:40, Daniel Dotsenko пишет:

> What's wrong with plain returning?
>
> Generators (those using yield) don't need to finish up with
> StopIteration() they just need to run out of code. return is a way to
> make it run out of code.

'return' statement in the generators is equal to 'raise StopIteration()'
expression... but looks nicer, yea

--
Andrew

[Serge S. Koval]

Here's gist to illustrate approach: https://gist.github.com/3891601

Check test_genmod.py to see usage samples.

Few notes:

1. It is possible to use genmod.async decorator without provided callback. In this case it decides that there's no callback should be called and just exits.

2. If callback is passed, it will be called with parameters passed to genmod.ret

3. If gen.ret is not called at all and there's callback, callback will be called with None as single parameter. Together with rest of the gen module methods, this will return None as well, see line 59 of test_genmod.py example.

If you're using Python 3.3 by any chance, you can use PEP-0380: http://www.python.org/dev/peps/pep-0380/, which states: return expr in a generator causes StopIteration(expr) to be raised upon exit from the generator.

So, it means, instead of using gen.ret(), you can just 'return' with parameter to pass control to callback chain. Isn't this great? :-)

def async_check(a):
    yield gen.Task(io_loop.add_timeout, time() + 0.1)
    if a == 10:
        return 15

    return 20

gen.async code should be adapted to handle non-generator functions properly, but that's surely doable.

Serge.

[A. Jesse Jiryu Davis]

Thanks for the code, now I finally understand what you're proposing. =) Looks cool to me.