closing event loops

[Martin Teichmann]

Hi List,

I use asyncio to start several tasks with asyncio.async. Several
of them need to close some things before exiting, so I use
a try...finally construction to do that. Unfortunately, the
finalizers are never called.

When running the event loop, I am using the same code as in
several examples in the docs:

    try:
        loop.run_forever()
    finally:
        loop.close()

The documentation for BaseEventLoop.close says that it "clears the queues".
Going through the code I realized that it is doing exactly just that.

I do think it should actually do more, namely it should close the running
tasks (i.e. raise a CancelledError in the coroutines). Otherwise there is not
much sense to have this close method at all, the garbage collector is good
enough to deal with clearing queues.

Now one might argue that the behavior shouldn't change anymore, but
maybe we can add a new method, e.g. BaseEventLoop.cancel(),
which cancels all running coroutines.

Greetings

Martin

[Guido van Rossum]

I can think of a variety of reasons where you don't want to bother with the tasks (possibly because they might resist being cancelled) but you still want to close the loop. In your finally clause you should be able to write something like this:

for t in asyncio.Task.all_tasks(loop):

    t.cancel()

loop.run_until_complete(asyncio.sleep(0.1))  # Give them a little time to recover

loop.close()

Exactly how long you want to give the tasks to handle their cancellation is one of the design decisions that make it difficult to do this automatically in loop.close() -- but if you don't run the loop at all there is not much of a point in cancelling the tasks (as the cancelled task won't run to process the exception thrown into it until the loop schedules it).

--
--Guido van Rossum (python.org/~guido)

[Martin Teichmann]

Hi Guido, Hi List,

for t in asyncio.Task.all_tasks(loop):

    t.cancel()

loop.run_until_complete(asyncio.sleep(0.1))  # Give them a little time to recover

loop.close()

That solves my problem. Couln't we write def cancel(self): on top of it and put it into
BaseEventLoop? In this case we should even be able to replace
the run_until_complete by a mere self._run_once(), then we don't need to wait
an extra 0.1.

Sure, if tasks start to resist being cancelled, that's a big mess. But I do think there
is quite a number of programmers out there who withstood the temptation to let
tasks resist their cancellation, but who would like to have their finalizers called in
an orderly fashion.

Greetings

Martin

[Guido van Rossum]

On Tue, Sep 9, 2014 at 12:53 AM, Martin Teichmann <martin.t...@gmail.com> wrote:

Hi Guido, Hi List,

for t in asyncio.Task.all_tasks(loop):

    t.cancel()

loop.run_until_complete(asyncio.sleep(0.1))  # Give them a little time to recover

loop.close()

That solves my problem. Couln't we write def cancel(self): on top of it and put it into
BaseEventLoop? In this case we should even be able to replace
the run_until_complete by a mere self._run_once(), then we don't need to wait
an extra 0.1.

I suppose we could do that, but then we'd have to explain the limitations and repercussions  as well. (Also, I don't think such a helper should include the close() call.)

 

Sure, if tasks start to resist being cancelled, that's a big mess. But I do think there
is quite a number of programmers out there who withstood the temptation to let
tasks resist their cancellation, but who would like to have their finalizers called in
an orderly fashion.

And yet having a try/finally around a yield-from is an easy recipe for resisting cancellation -- it is all too convenient to put another yield-from in the finally clause, and then you are requiring multiple trips through the event loop.

Where are the finalizers you need called and what do they do? And why do you need them called when you're shutting down the process?

For servers a convenient idiom is to call close() in the Server object(s) returned by create_server() and then wait a certain time for all requests to be handled. Which reminds me, cancelling *all* tasks can easily cause a problem if there are tasks waiting for other tasks -- dealing with multiple cancellations simultaneously is nearly impossible. It's almost always better to bite down and do the right thing, which to keep track of the higher-level activities you have going and come up with a way to shut those down in an orderly fashion, rather than just shooting all tasks.

[Martin Teichmann]

Hi Guido, Hi List,

And yet having a try/finally around a yield-from is an easy recipe for resisting cancellation -- it is all too convenient to put another yield-from in the finally clause, and then you are requiring multiple trips through the event loop.

This is why I would love to have a well-documented function in the standard

library, where someone thought of all those corner-cases, and the function

either handles those, raises an exception or there is something in the

documentation telling the programmer DON'T DO THAT!

Where are the finalizers you need called and what do they do? And why do you need them called when you're shutting down the process?

I am writing an RPC-like system, along the lines of

@coroutine

def server(self):

    open_database()

    try:

        yield from getattr(self, yield from read_command())()

    finally:

        close_database()

now, there is a command to shut down the server, and I would like

the database to be properly closed for every server. First time I

discovered all of this was actually when I was pressing Ctrl-C

and realized it didn't close the database, something I always

kept for granted in a finalizer...

I am sure all this can be achieved differently, but nevertheless I think

a well thought out cancel method would still be a very nice thing to

have.

Greetings

Martin

[Guido van Rossum]

On Tue, Sep 9, 2014 at 11:12 AM, Martin Teichmann <martin.t...@gmail.com> wrote:

Hi Guido, Hi List,

And yet having a try/finally around a yield-from is an easy recipe for resisting cancellation -- it is all too convenient to put another yield-from in the finally clause, and then you are requiring multiple trips through the event loop.

This is why I would love to have a well-documented function in the standard

library, where someone thought of all those corner-cases, and the function

either handles those, raises an exception or there is something in the

documentation telling the programmer DON'T DO THAT!

 understand where you're coming from, but I want to repeat, with emphasis, that not every pattern deserves to become a library function, and education can take other forms than documentation of existing methods.

 

Where are the finalizers you need called and what do they do? And why do you need them called when you're shutting down the process?

I am writing an RPC-like system, along the lines of

@coroutine

def server(self):

    open_database()

    try:

        yield from getattr(self, yield from read_command())()

    finally:

        close_database()

now, there is a command to shut down the server, and I would like

the database to be properly closed for every server. First time I

discovered all of this was actually when I was pressing Ctrl-C

and realized it didn't close the database, something I always

kept for granted in a finalizer...

You realize that if the process crashes (e.g. with a SEGV) or is killed by some other signal, the finalizers aren't called either? I would assume that especially database connections are good at implicitly rolling back any uncommitted transaction regardless of whether any finalization code is run.

 

I am sure all this can be achieved differently, but nevertheless I think

a well thought out cancel method would still be a very nice thing to

have.

Perhaps, but clearly we're not even scratching the surface of how it should behave. I haven't heard from enough people with other use cases that would actually be helped by the same helper to warrant adding the new API you are proposing.

API design generally ought to follow from use cases encountered frequently, otherwise you get a complex hodge-podge that is hard to learn.

[Martin Teichmann]

Hi again,

thanks for all the great input!

I still think there should still a warning in the documentation for BaseEventLoop.close
that it does not cancel the tasks still running so that no finalizers are called. It is the
last chance to do that in a controlled fashion, as the event loop cannot be run after
close anymore. We should then discourage programmers to use try:..finally: or
with: context managers in tasks since, well, there is no guarantee they're ever called.

To me, the name close implies strongly "clean up after you and then finish the thing
off". The argument that we cannot cancel the tasks because they might resist cancellation
is a bit as if file.close wouldn't flush its buffers because the disk might be full.

Greetings

Martin

[Victor Stinner]

Please, help me to enhance the documentation. Get the documentation source at:
http://hg.python.org/cpython/file/5bc23c111de1/Doc/library/asyncio-eventloop.rst

And send a patch.

Victor