Relation between asyncio and concurrent.futures?

[Victor Stinner]

Hi,

Many times, I was confused by the link between asyncio and
concurrent.futures modules.

For example, on CPython FreeBSD 6.2 buildbot, concurrent.futures
module is missing, and so asyncio module cannot be imported. In my
opinion, this is a bug: you should be able to use callbacks and
coroutines without a pool of threads or processes. Another example is
my backport of asyncio on Python 2.x, I didn't understand why I had to
backport also concurrent.futures (by the way, there is a backport on
PyPI, I will reuse it).

I found some references to concurrent.futures in asyncio source code.

asyncio/futures.py:
---
# TODO: Do we really want to depend on concurrent.futures internals?
Error = concurrent.futures._base.Error
CancelledError = concurrent.futures.CancelledError
TimeoutError = concurrent.futures.TimeoutError
---

asyncio/tasks.py:
---
FIRST_COMPLETED = concurrent.futures.FIRST_COMPLETED
FIRST_EXCEPTION = concurrent.futures.FIRST_EXCEPTION
ALL_COMPLETED = concurrent.futures.ALL_COMPLETED
---

And BaseEventLoop.run_in_executor() creates a default executor (a pool
of threads) if none was defined.

Is there a reason to reuse concurrent.futures exceptions and states
(string constants), except confusing me? :-)

Victor

[Guido van Rossum]

I like that the exceptions and constants are shared between the two
modules; it expresses that the APIs are intended to look alike and the
similarity isn't just a coincidence.

However, I don't like the dependency either (hence the TODO comment
:-). When I introduced it I didn't realize there were platforms where
concurrent.futures can't be imported.

I would be fine with conditionally importing concurrent.futures and
using the constants from it, and when it can't be imported, define the
constants and exception classes locally. This could be put in a new
submodule just for the purpose (to avoid having the conditional import
logic in both futures.py and tasks.py). There is probably something
you have to do to wrap_future() as well.

However, that still leaves the use of a thread pool in
run_in_executor() -- how else are you going to call getaddrinfo() in a
separate thread? (Did you solve this in Trollius yet?) I would be okay
with a private backport of a minimal executor for this purpose.

Finally, on the one platform where concurrent.futures can't be
imported, isn't it because it doesn't have threads? So even a private
thread pool wouldn't solve the problem on that platform (though you
could get a lot more tests to pass). I guess nothing would solve the
problem on that platform except just calling getaddrinfo() in-line --
I suppose you could have a dummy thread pool that uses the
dummy_threading stdlib module to minimize the conditional stuff you
have to do in the Tulip code.

--
--Guido van Rossum (python.org/~guido)

[Victor Stinner]

2014/1/3 Guido van Rossum <gu...@python.org>:

> However, I don't like the dependency either (hence the TODO comment
> :-). When I introduced it I didn't realize there were platforms where
> concurrent.futures can't be imported.
>
> I would be fine with conditionally importing concurrent.futures and
> using the constants from it, and when it can't be imported, define the
> constants and exception classes locally.

Yes, it's possible to do that. I implemented this in Trollius 0.1 to
try to mimick Tulip as much as possible.

> However, that still leaves the use of a thread pool in
> run_in_executor() -- how else are you going to call getaddrinfo() in a
> separate thread? (Did you solve this in Trollius yet?) I would be okay
> with a private backport of a minimal executor for this purpose.

If concurrent.futures is missing, it's maybe possible to write a
synchronous fallback used as a fallback? It's maybe better than
raising an ImportError.

Or concurrent.futures dependency can be mandatory.

> Finally, on the one platform where concurrent.futures can't be
> imported, isn't it because it doesn't have threads?

concurrent.futures cannot be loaded on old FreeBSD 6 setup because of
a limit on POSIX semaphores, not because thread support is disabled. I
didn't try asyncio on Python compiled with thread support disabled.

> So even a private
> thread pool wouldn't solve the problem on that platform (though you
> could get a lot more tests to pass). I guess nothing would solve the
> problem on that platform except just calling getaddrinfo() in-line --
> I suppose you could have a dummy thread pool that uses the
> dummy_threading stdlib module to minimize the conditional stuff you
> have to do in the Tulip code.

I'm not sure that it's worth to reimplement a pseudo
concurrent.futures with something more complex than a synchronous
executor.

concurrent.futures is always present in Python 3.2+ and a backport is
available for older Python versions.

Victor

[Guido van Rossum]

On Fri, Jan 3, 2014 at 4:22 PM, Victor Stinner <victor....@gmail.com> wrote:
> 2014/1/3 Guido van Rossum <gu...@python.org>:
>> However, I don't like the dependency either (hence the TODO comment
>> :-). When I introduced it I didn't realize there were platforms where
>> concurrent.futures can't be imported.
>>
>> I would be fine with conditionally importing concurrent.futures and
>> using the constants from it, and when it can't be imported, define the
>> constants and exception classes locally.
>
> Yes, it's possible to do that. I implemented this in Trollius 0.1 to
> try to mimick Tulip as much as possible.

I'd be fine if this was backported to Tulip and CPython, given that at
least one platform indeed does not have a functioning
concurrent.futures.

(For CPython 3.4 and beyond, another refactoring might be possible
where the shared constants and exception classes are defined in a new
module that can always be imported, even if concurrent.futures can't
be.)

>> However, that still leaves the use of a thread pool in
>> run_in_executor() -- how else are you going to call getaddrinfo() in a
>> separate thread? (Did you solve this in Trollius yet?) I would be okay
>> with a private backport of a minimal executor for this purpose.
>
> If concurrent.futures is missing, it's maybe possible to write a
> synchronous fallback used as a fallback? It's maybe better than
> raising an ImportError.

Yes, a synchronous fallback would be fine.

> Or concurrent.futures dependency can be mandatory.
>
>> Finally, on the one platform where concurrent.futures can't be
>> imported, isn't it because it doesn't have threads?
>
> concurrent.futures cannot be loaded on old FreeBSD 6 setup because of
> a limit on POSIX semaphores, not because thread support is disabled. I
> didn't try asyncio on Python compiled with thread support disabled.

I think I once tried it (it's not hard to sabotage the threading
import) and quickly gave up. But it's possible that it was mainly the
executor needed for getaddrinfo(), so the synchronous fallback would
solve that.

>> So even a private
>> thread pool wouldn't solve the problem on that platform (though you
>> could get a lot more tests to pass). I guess nothing would solve the
>> problem on that platform except just calling getaddrinfo() in-line --
>> I suppose you could have a dummy thread pool that uses the
>> dummy_threading stdlib module to minimize the conditional stuff you
>> have to do in the Tulip code.
>
> I'm not sure that it's worth to reimplement a pseudo
> concurrent.futures with something more complex than a synchronous
> executor.

That's fine.

> concurrent.futures is always present in Python 3.2+ and a backport is
> available for older Python versions.

If you're fine with that backport as a dependency for Trollius I am
also okay with that. :-)

Do we have any action items for Tulip left here?

[Victor Stinner]

Hi,

2014/1/5 Guido van Rossum <gu...@python.org>:

> Yes, a synchronous fallback would be fine.

> (...)

> If you're fine with that backport as a dependency for Trollius I am
> also okay with that. :-)

I opened the following issue to track this idea:
http://bugs.python.org/issue20130

> Do we have any action items for Tulip left here?

I may forward-port some minor changes I made in Trollius to limit
differences between Trolius and Tulip.

Victor