Which other futures my come out of asyncio.as_completed?

[Luciano Ramalho]

Hello, in the docs for the asyncio.as_completed function [1] there's a
note that says "The futures f are not necessarily members of fs."

[1] https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed

Which other futures apart from those in the fs argument may be yielded
by as_completed?

Thanks!

--
Luciano Ramalho
Twitter: @ramalhoorg

Professor em: http://python.pro.br
Twitter: @pythonprobr

[Guido van Rossum]

Because of the way as_completed() works, it sometimes makes up new futures on the spot. The new futures will have the result/error from the original future but the mapping from original future to new future is indeterminate until a result is ready.

--

--Guido van Rossum (python.org/~guido)

[Luciano Ramalho]

On Fri, Feb 27, 2015 at 5:52 PM, Guido van Rossum <gu...@python.org> wrote:
> Because of the way as_completed() works, it sometimes makes up new futures
> on the spot. The new futures will have the result/error from the original
> future but the mapping from original future to new future is indeterminate
> until a result is ready.

Excellent, thank you for that!

Best,

Luciano

[Luciano Ramalho]

On Fri, Feb 27, 2015 at 5:52 PM, Guido van Rossum <gu...@python.org> wrote:

> Because of the way as_completed() works, it sometimes makes up new futures
> on the spot. The new futures will have the result/error from the original
> future but the mapping from original future to new future is indeterminate
> until a result is ready.

Using Guido's explanation above, we could improve that note in
asyncio.as_completed [1] adding a second sentence:

"""
Note: The futures f are not necessarily members of fs. A given future
may be wrapped in another future by as_completed; when that happens,
the result/error of the new future will be the same as the original
future.
"""

[1] https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed

Best,

Luciano

[Victor Stinner]

Hi,

2015-03-01 13:43 GMT+01:00 Luciano Ramalho <luc...@ramalho.org>:
> """
> Note: The futures f are not necessarily members of fs. A given future
> may be wrapped in another future by as_completed; when that happens,
> the result/error of the new future will be the same as the original
> future.
> """

I read the source code of as_completed() and I don't see where a
future can be wrapped in another future. I only saw that async() is
called on each item of the fs parameter, so coroutine objects are
wrapped into future. So I propose:

"""
Note: The futures f are not necessarily members of fs. Coroutine
objects of fs are
wrapped in futures.
"""

Victor

[Guido van Rossum]

It's exceedingly subtle -- that's why the docstring contains an example of how to use it.

Note the final two lines:

    for _ in range(len(todo)):
        yield _wait_for_one()

This is yield, not yield-from. as_completed() is not a coroutine -- it is an iterator and the caller must loop over it. Each time the caller gets one thing from the loop, what is that thing? It's _wait_for_one() -- i.e. it's a coroutine! Then the caller has to use "yield from" on that coroutine, which will then wait until a future emerges from the queue, and then _wait_for_one() either returns that future's result or raises its exception. This is also where timeouts are processed (they result in a dummy None emerging from the queue).

Hope this helps. The input futures never appear in the output, so the warning in the docstring is an understatement (but don't remove it -- a future implementation might use an optimized path if some of the futures are *already* done.

--Guido

[Luciano Ramalho]

On Tue, Mar 3, 2015 at 1:49 AM, Guido van Rossum <gu...@python.org> wrote:
> It's exceedingly subtle -- that's why the docstring contains an example of
> how to use it.
>
> Note the final two lines:
>
> for _ in range(len(todo)):
> yield _wait_for_one()
>
> This is yield, not yield-from. as_completed() is not a coroutine -- it is an
> iterator and the caller must loop over it. Each time the caller gets one
> thing from the loop, what is that thing? It's _wait_for_one() -- i.e. it's a
> coroutine! Then the caller has to use "yield from" on that coroutine, which
> will then wait until a future emerges from the queue, and then
> _wait_for_one() either returns that future's result or raises its exception.
> This is also where timeouts are processed (they result in a dummy None
> emerging from the queue).

Thank you for this, Guido! I can now appreciate the subtle wording of
"Return an iterator whose values, when waited for, are Future
instances."

Is there a name for these _wait_for_one generators that yield a
Future? Giving those objects a good name could make it easier to
explain.

As it stands, the note is misleading: "The futures f are not
necessarily members of fs." The f you get from as_completed is not a
Future, but that thing-you-yield-from-to-get-a-Future. Without giving
that thing a name, the note could say "The future yielded from each f
is not necessarily a member of fs."

> Hope this helps. The input futures never appear in the output, so the
> warning in the docstring is an understatement (but don't remove it -- a
> future implementation might use an optimized path if some of the futures are
> *already* done.

Do you mean it's possible some day as_completed() may return either
things-you-yield-from-to-get-a-Future or actual Futures, i.e.
instances of asyncio.Future?

Best,

Luciano

>
> --Guido
>
> On Mon, Mar 2, 2015 at 1:48 AM, Victor Stinner <victor....@gmail.com>
> wrote:
>>
>> Hi,
>>
>> 2015-03-01 13:43 GMT+01:00 Luciano Ramalho <luc...@ramalho.org>:
>> > """
>> > Note: The futures f are not necessarily members of fs. A given future
>> > may be wrapped in another future by as_completed; when that happens,
>> > the result/error of the new future will be the same as the original
>> > future.
>> > """
>>
>> I read the source code of as_completed() and I don't see where a
>> future can be wrapped in another future. I only saw that async() is
>> called on each item of the fs parameter, so coroutine objects are
>> wrapped into future. So I propose:
>>
>> """
>> Note: The futures f are not necessarily members of fs. Coroutine
>> objects of fs are
>> wrapped in futures.
>> """
>>
>> Victor
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)

[Guido van Rossum]

On Tue, Mar 3, 2015 at 2:06 PM, Luciano Ramalho <luc...@ramalho.org> wrote:

On Tue, Mar 3, 2015 at 1:49 AM, Guido van Rossum <gu...@python.org> wrote:
> It's exceedingly subtle -- that's why the docstring contains an example of
> how to use it.
>
> Note the final two lines:
>
>     for _ in range(len(todo)):
>         yield _wait_for_one()
>
> This is yield, not yield-from. as_completed() is not a coroutine -- it is an
> iterator and the caller must loop over it. Each time the caller gets one
> thing from the loop, what is that thing? It's _wait_for_one() -- i.e. it's a
> coroutine! Then the caller has to use "yield from" on that coroutine, which
> will then wait until a future emerges from the queue, and then
> _wait_for_one() either returns that future's result or raises its exception.
> This is also where timeouts are processed (they result in a dummy None
> emerging from the queue).

Thank you for this, Guido! I can now appreciate the subtle wording of
"Return an iterator whose values, when waited for, are Future
instances."

Is there a name for these _wait_for_one generators that yield a
Future? Giving those objects a good name could make it easier to
explain.

Well, as_completed is a generator; generators yield a stream of results, and in this case each result is itself a coroutine (to be precise, a coroutine object, not a coroutine function -- _wait_for_one is the latter, _wait_for_one() is the former).

 

As it stands, the note is misleading: "The futures f are not
necessarily members of fs." The f you get from as_completed is not a
Future, but that thing-you-yield-from-to-get-a-Future. Without giving
that thing a name, the note could say "The future yielded from each f
is not necessarily a member of fs."

Yeah, this is a general concept, "either a coroutine or a future" and it doesn't really have a name. It's also known informally as "the kind of thing you pass to yield from", which isn't so helpful either. :-)

 

> Hope this helps. The input futures never appear in the output, so the
> warning in the docstring is an understatement (but don't remove it -- a
> future implementation might use an optimized path if some of the futures are
> *already* done.

Do you mean it's possible some day as_completed() may return either
things-you-yield-from-to-get-a-Future or actual Futures, i.e.
instances of asyncio.Future?

Yes, that's totally possible. And those Futures could either be amongst the input Futures or new Futures.

A trivial optimization which would return some Futures amongst the input would be to add something like this to the top:

readies = {f for f in todo if f.done()}

if readies:

    todo -= readies

    for f in readies: yield f

I think at some point I had that, but I think the use case is marginal (you almost never call this when there's much of a chance that any of the futures are already done), and I wasn't sure about timeouts and other guarantees.

Also, since the input can also be coroutines-or-Futures, this loop might still yield some Futures that aren't in the input (because they were made up by the async() call: todo = {async(f, loop=loop) for f in set(fs)}. Also note that fs may be an iterator -- it's iterated over exactly once (by the set(fs) call here).