Curio

[Imran Geriskovan]

As I noted in my previous posts in this group,
I mostly try keeping async code with parity of
blocking code. (Reasons: Ease of streams based
development, Easy migration to compiled langs,
threads, etc, etc..)

For couple of months I've been playing with
Curio, to which now I'm a total convert.

For an async code based on Curio, you can almost
drop all "await/async" keywords with some
minor manupulations; and bang: You get working
blocking version. Or you can go from blocking
version to async.

Even better, you can mix async and blocking code
in hybrid combinations to get best of both worlds.
(Ex: when using blocking libraries, performance
critical routines, etc..)

With enough effort these can also be done with
asyncio. However, with Curio, this approach
is directly supported out of the box. And it's
a compact one without any bells and whistles.
It has nothing more than necessary to get
the job done.

Performance of pure async code is about 2x with
respect to asyncio. And your millage may be
extended with hybrid blocking combinations.

I'd be happy to see Curio in std library..

Regards,
Imran

[Paul Sokolovsky]

Hello,

On Sun, 23 Oct 2016 15:28:20 +0300
Imran Geriskovan <imran.ge...@gmail.com> wrote:

> As I noted in my previous posts in this group,
> I mostly try keeping async code with parity of
> blocking code. (Reasons: Ease of streams based
> development, Easy migration to compiled langs,
> threads, etc, etc..)
>
> For couple of months I've been playing with
> Curio, to which now I'm a total convert.
>
> For an async code based on Curio, you can almost
> drop all "await/async" keywords with some
> minor manupulations; and bang: You get working
> blocking version. Or you can go from blocking
> version to async.

Thanks for sharing your experiences.

Quoiting from Curio's README:

"Existing I/O libraries are mainly built on event-loops, callback
functions, futures, and various abstractions that predate Python's
proper support for coroutines. As a result, they are either overly
complicated or dependent on esoteric magic involving C extensions,
monkeypatching, or reimplementing half of the TCP flow-control
protocol."

So, it's the same motivation as for MicroPython's uasyncio which was
presented on this list previously.

Looking at an example:

"""
await client.sendall(data)
"""

So, Curio consistently uses asynchronous reads and write for sockets,
unlike asyncio, which suddenly has synchronous write. And being asked
why, the author of the library says "but because reads and writes are
very different!" Well, now we have an alternative explanation - because
of "reimplementing half of the TCP flow-control protocol".



--
Best regards,
Paul mailto:pmi...@gmail.com

[Imran Geriskovan]

>> As I noted in my previous posts in this group,
>> I mostly try keeping async code with parity of
>> blocking code. (Reasons: Ease of streams based
>> development, Easy migration to compiled langs,
>> threads, etc, etc..)
>>

>> For an async code based on Curio, you can almost
>> drop all "await/async" keywords with some
>> minor manupulations; and bang: You get working
>> blocking version. Or you can go from blocking
>> version to async.

> Thanks for sharing your experiences.

You're welcome.

> So, it's the same motivation as for MicroPython's uasyncio which was
> presented on this list previously.

I think, it is good to see such a class of async implementations.
I can even release my own version of hacks for running asyncio
in the same spirit. However, I found Curio much more elegant,
so dumped most of it.

After some settlement about the direction, with such async
engines around, I'd expect Python standard library
to support a similar approach.

It is much more than async/sync write().

Regards,
Imran

[Guido van Rossum]

Dave has repeatedly stated that he's not interested in maintaining
Curio long term or keeping the API stable. That means it's not going
to happen. It might make more sense to propose carefully designed
additions to asyncio that aim to fill in the gaps you've found by
using curio. This should focus on API functionality; the performance
is being worked on separately, and there's also uvloop.

--
--Guido van Rossum (python.org/~guido)

[Imran Geriskovan]

>> I'd be happy to see Curio in std library..

> Dave has repeatedly stated that he's not interested in maintaining
> Curio long term or keeping the API stable. That means it's not going
> to happen. It might make more sense to propose carefully designed
> additions to asyncio that aim to fill in the gaps you've found by
> using curio. This should focus on API functionality; the performance
> is being worked on separately, and there's also uvloop.

Providing direct async versions of blocking operations is the
key. That's it. Curio just does that. At the surface its that simple.
Is providing this on asyncio possible?

I've carried some hacks for asyncio streams. However
covering async versions of all blocking use cases at some
point required some kind of dirty hack involving going into async
internals. Frankly I've given up. I dont think a carefully designed
patch can solve it.

I've ended up with this:
Async programming is good, as long as it is the mirror
image of blocking one.

Inverting the control and then trying to get it back is not
a good design. As I've tried it once..

[Guido van Rossum]

Can you provide something more concrete, like a design for an API? So
far it seems you're just complaining. Nobody likes that.

[Imran Geriskovan]

On 10/23/16, Guido van Rossum <gu...@python.org> wrote:
> Can you provide something more concrete, like a design for an API?

I wont. But there is already a very good example.

> So far it seems you're just complaining. Nobody likes that.

There is no complaining. I'm already on my way.
Here, I'll respectfully aggree with anyone objecting any discussion
about any python async engine other than asyncio. That's
understandable.

[Yury Selivanov]

Imran,

I don’t think this is a productive discussion. FWIW it's possible to implement all of Curio’s APIs on top of current asyncio: there are no fundamental blockers to that. If you think that Curio is better in some regards, please feel free to post concrete proposals to github.com/python/asyncio, let’s discuss. We’ve implemented 10s of proposals that added new APIs in the past few years.

Yury

[Xavier Combelle]

> Dave has repeatedly stated that he's not interested in maintaining
> Curio long term or keeping the API stable. That means it's not going
> to happen. It might make more sense to propose carefully designed
> additions to asyncio that aim to fill in the gaps you've found by
> using curio. This should focus on API functionality; the performance
> is being worked on separately, and there's also uvloop.
>

I did not did really experienced asyncio by finding it too much complicated
(and that I did not had real use case).
But when I read curio documentation I found it wonderful to the point
I want to experiment with it
I would love to see something similar in python standard library as
(from my point of view)
it looks like more understandable than current asyncio.

[Andrew Svetlov]

I believe it is documentation issue, not internal implementation problem.

--

Thanks,

Andrew Svetlov

[Xavier Combelle]

I might be wrong, but from my point of view it is about the API proposed.

The asyncio API is far more rich of new concept than curio one
there are at least protocols, ioloop
and to my knowledge you can't do write similar code of the tutorial
without using protocol and ioloop
at the opposite, the only new concept bring by the main of curio tutorial are
- the run function
- the await, async keyword
The other api are copy paste to the synchrone ones

[Yury Selivanov]

> On Oct 24, 2016, at 12:24 PM, Xavier Combelle <xavier....@gmail.com> wrote:
>
> - the run function
> - the await, async keyword
> The other api are copy paste to the synchrone ones

Yes, this is something that has to be fixed in asyncio docs. Protocols/transports are advanced low level concepts, you don’t need to even know about them to use asyncio programs. I’ll try to find some time to work on the docs before 3.6 is released...

Yury

[Xavier Combelle]

Yes, this is something that has to be fixed in asyncio docs.  Protocols/transports are advanced low level concepts, you don’t need to even know about them to use asyncio programs.  I’ll try to find some time to work on the docs before 3.6 is released...

Yury

If I don't mistake, that is the high level api https://docs.python.org/3.6/library/asyncio-stream.html#tcp-echo-client-using-streams
looks like the api is different than sync ones (with notably one reader and one writer) and that the loop is needed everywhere

[Yury Selivanov]

Right, passing the “loop” explicitly is another point to fix in the docs. Short answer - don’t do that in your programs. You might want to pass the loop explicitly when you’re working on a reusable library like aiohttp, but that’s about it.

We also can have a third-party package that would implement curio-like stream object and socket object if a lot of people think it’s easier to work with. This is something that can be easily addressed even with the current asyncio.

Yury

[Paul Sokolovsky]

Hello,

On Mon, 24 Oct 2016 18:38:20 +0200
Xavier Combelle <xavier....@gmail.com> wrote:

>
> > Yes, this is something that has to be fixed in asyncio docs.
> > Protocols/transports are advanced low level concepts, you don’t
> > need to even know about them to use asyncio programs. I’ll try to
> > find some time to work on the docs before 3.6 is released...
> >
> > Yury
>
> If I don't mistake, that is the high level

> apihttps://docs.python.org/3.6/library/asyncio-stream.html#tcp-echo-client-using-streams

> looks like the api is different than sync ones (with notably one
> reader and one writer) and that the loop is needed everywhere

.read() and .write() are generic methods of Python's stream interface
("protocol"). Sadly, Python's sockets don't implement stream interface
natively, but require converting to a stream with .makefile() method.
So, you can consider that in the example above,
asyncio.open_connection() does the same as socket().makefile().
Otherwise, it's the same as "sync ones". Except:

> writer.write(message.encode())

In asyncio, asynchronous library, .write() is somehow synchronous.

> data = yield from reader.read(100)

.read() is expectedly asynchronous.

> writer.close()

.close() is somehow synchronous again.

Pretty inconsistent.


It was proposed to add an asynchronous .write method to asyncio
(which would be called differently of course), but...

[Yury Selivanov]

> On Oct 24, 2016, at 12:49 PM, Paul Sokolovsky <pmi...@gmail.com> wrote:
>
> .read() and .write() are generic methods of Python's stream interface
> ("protocol"). Sadly, Python's sockets don't implement stream interface
> natively, but require converting to a stream with .makefile() method.
> So, you can consider that in the example above,
> asyncio.open_connection() does the same as socket().makefile().
> Otherwise, it's the same as "sync ones". Except:
>
>> writer.write(message.encode())
>
> In asyncio, asynchronous library, .write() is somehow synchronous.
>
>> data = yield from reader.read(100)
>
> .read() is expectedly asynchronous.

First, both “read” and “write” ops are asynchronous in asyncio. What you probably are trying to say is that “read” is a coroutine, whereas “write” is a regular method. There is a good reason for that.

“read” is a coroutine because you have to wait until the data has arrived. The arrival of the data is obvious, and you know with 100% certainty that you have the data. So you can resume the coroutine one you have it.

“write” is a regular method because, contrary to “read”, you have no idea where the data that you have written is. In curio, “socket.write” coroutine resumes when the corresponding system call succeeds, but it would be wrong to assume that the data you’re trying to send was received by anyone! The only way of knowing that your “write" operation has really succeeded, is when you receive a confirmation from the client (how exactly that confirmation is carried is defined in the protocol you are implementing).

Network “write” is fundamentally different from file/memory “write". asyncio handles read/write discrepancy in its streaming API, by letting users to write the data when they need, and the framework then takes care of the rest (and it will, in general, be more efficient this way).

When you work with the non-blocking low-level socket API in asyncio, you have ‘loop.sock_sendall’ coroutine, that essentially does what you want - resumes after the write syscall. It’s the correct API for low-level socket programming, but not for high-level protocol programming.

Yury