Best practice for managing loop: don't close it (really!?)

[Luciano Ramalho]

Like Victor Stinner in this bug report [1], I was craving for a
context-manager enabled loop so I did not forget to close() it.

But reading Guido's last message rejecting that bug, it seems callign
loop.close() is only recommended "if you own the loop" -- while most
asyncio users probably don't "own" it.

[1] http://bugs.python.org/issue19860#msg205062

Can I conclude that in practice, close() should not be called at all
unless your own code actually created the loop instead of merely
fetching it with asyncio.get_event_loop()?

Is that a sensible recommendation?

Cheers,

Luciano



--
Luciano Ramalho
Twitter: @ramalhoorg

Professor em: http://python.pro.br
Twitter: @pythonprobr

[Guido van Rossum]

Yes. It's a bit like memory allocation -- if you don't own it, don't free it.

--

--Guido van Rossum (python.org/~guido)

[Luciano Ramalho]

Ok, that makes sense, thanks!

[Victor Stinner]

Le lundi 19 janvier 2015, Luciano Ramalho <luc...@ramalho.org> a écrit :

Can I conclude that in practice, close() should not be called at all
unless your own code actually created the loop instead of merely
fetching it with asyncio.get_event_loop()?

 

Event loops must be closed. The first call to get_event_loop() creates an event loop which must be closed when you are done.

If you run python with -Wd, you should see a warning because a pair of sockets are not closed. Sockets are owned by the event loop.

I opened a issue to emit a ResourceWarning if an event loop (and transports) are not explicitly closed:

http://bugs.python.org/issue23243

Victor

[Luciano Ramalho]

On Mon, Jan 19, 2015 at 4:56 AM, Victor Stinner
<victor....@gmail.com> wrote:
> Event loops must be closed. The first call to get_event_loop() creates an
> event loop which must be closed when you are done.

In a non-trivial program, how do I know mine was the first call to
get_event_loop?

Even a trivial program running in iPython Notebook under Qt will get
an event loop that already existed, and that it should close.

> If you run python with -Wd, you should see a warning because a pair of
> sockets are not closed. Sockets are owned by the event loop.
>
> I opened a issue to emit a ResourceWarning if an event loop (and transports)
> are not explicitly closed:
> http://bugs.python.org/issue23243

OK, but if I see such a warning and dont't have the information to do
the right thing I will be frustrated.

To determine whether I must close the event loop, I need to know
whether I started the event loop when I called get_event_loop(), or
whether I got a preexisting event loop from the runtime environment.
Is there an API that tells me that?

Even better: the BaseEventLoop could have another method, perhaps
named release(), which lets the user say he won't use the event loop
any more, delegating the decision to actually close() the loop to the
same infrastructure level that makes the decision to create a new loop
or give me an existing one when I call get_event_loop.

What do you think?

Best,

[Luciano Ramalho]

In my previous message the crucial word "NOT" was missing from this sentence:

"""
Even a trivial program running in iPython Notebook under Qt will get

an event loop that already existed, and that it should NOT close.
"""

Sorry...

Best,

Luciano

[Victor Stinner]

2015-01-19 12:52 GMT+01:00 Luciano Ramalho <luc...@ramalho.org>:
> In a non-trivial program, how do I know mine was the first call to
> get_event_loop?

It doesn't matter who called it first. You may even call close() more
than once ;-)

Just try to be kind, don't close the event loop if something will need it later.

In an application, I hope that you know when the application is going to exit!

Victor

[Luciano Ramalho]

On Mon, Jan 19, 2015 at 12:02 PM, Victor Stinner
<victor....@gmail.com> wrote:
> 2015-01-19 12:52 GMT+01:00 Luciano Ramalho <luc...@ramalho.org>:
>> In a non-trivial program, how do I know mine was the first call to
>> get_event_loop?
>
> It doesn't matter who called it first. You may even call close() more
> than once ;-)

I am sorry, but I can't reconcile that with this quote from Guido's
last comment closing your bug report [1]. What am I missing?

"""
Well, calling close() is best practice *if you own the loop*. But the
thing is that you may not own what get_event_loop() returns, and in
fact in most cases where it is called you don't own it.
"""

[1] http://bugs.python.org/msg205062

> In an application, I hope that you know when the application is going to exit!

In a GUI application you may know it's about to exit but you don't own
the event loop.

Victor, I think your bug report is right on: the event loop should be
useable as context manager, what is missing is a method in the
AbstractEventLoopPolicy interface to make it possible to implement a
suitable __exit__.

If AbstractEventLoopPolicy.get_event_loop() exists to accommodate
environments providing event loops in their different ways, then the
AbstractEventLoopPolicy interface should have a function that could be
safely called by anyone to notify the environment you don't need the
loop anymore. Such a function -- maybe release_event_loop() -- could
be called by a BaseEventLoop.release() instance method and by a
BaseEventLoop.__exit__ method.

I'd appreciate some feedback to this idea, or clarifications if I am
getting something wrong.

Cheers,

Luciano

[Victor Stinner]

2015-01-19 15:55 GMT+01:00 Luciano Ramalho <luc...@ramalho.org>:
> I am sorry, but I can't reconcile that with this quote from Guido's
> last comment closing your bug report [1]. What am I missing?

Event loops are more "global" than other resources like files,
subprocesses, locks, IMAP or FTP connections (classes implementing
__exit__). asyncio.get_event_loop() doesn't tell you if the event loop
already existed or not.

I disagree with Guido on this issue (I would like to support the
context manager protocol), so I cannot explain you his point of view
:-)

> In a GUI application you may know it's about to exit but you don't own
> the event loop.

Who created the loop and who run the blocking run_forever() function?
If it's a library, the library should be responsible to call
loop.close(), or provide a function which calls loop.close().

All asyncio examples have this pattern in the "main" function:

loop = asyncio.get_event_loop()
# ... use the event loop ...
loop.close()

> If AbstractEventLoopPolicy.get_event_loop() exists to accommodate
> environments providing event loops in their different ways, then the
> AbstractEventLoopPolicy interface should have a function that could be
> safely called by anyone to notify the environment you don't need the
> loop anymore. Such a function -- maybe release_event_loop() -- could
> be called by a BaseEventLoop.release() instance method and by a
> BaseEventLoop.__exit__ method.

I don't understand your proposition. It looks like a reference
counter. Almost all functions in asyncio may call get_event_loop().
Having to call a "release()" method would make the code much more
verbose. It's tricky to ensure that the method is called even if an
exception was raised, and decide when we are done with the event loop.

Victor

[Leonardo Rochael Almeida]

On Monday, 19 January 2015 13:56:37 UTC-2, Victor Stinner wrote:

2015-01-19 15:55 GMT+01:00 Luciano Ramalho <luc...@ramalho.org>:

[...]

> If AbstractEventLoopPolicy.get_event_loop() exists to accommodate
> environments providing event loops in their different ways, then the
> AbstractEventLoopPolicy interface should have a function that could be
> safely called by anyone to notify the environment you don't need the
> loop anymore. Such a function -- maybe release_event_loop() -- could
> be called by a BaseEventLoop.release() instance method and by a
> BaseEventLoop.__exit__ method.

I don't understand your proposition. It looks like a reference
counter. Almost all functions in asyncio may call get_event_loop().
Having to call a "release()" method would make the code much more
verbose.

I believe a reference counter is similar to what Luciano had in mind, but it wouldn't increment the reference on `get_event_loop()`. Rather, it would increment the reference on `get_event_loop().__enter__()`

If you call `get_event_loop()` but not inside a context manager, you're explicitly saying that you very well expect the event loop to survive your scope. On the other hand, if you call it as part of a `with` blok, you're saying that, if nobody else is expecting the event loop to survive, then the system would be free to close it, as far as you're concerned.

 

It's tricky to ensure that the method is called even if an
exception was raised, and decide when we are done with the event loop.

That's where a `with` block would be most helpful.

Cheers,

Leo