Using AsyncSSH with Python 3.5's async and await

[Nicholas Chammas]

Hi Ron,

Python 3.5 introduced new syntax to better support asynchronous programming.

Do you have any thoughts on how this new syntax can be used with AsyncSSH? Is it mostly a matter of making simple replacements like putting in await instead of yield from? Are there any potential benefits to using the new syntax apart from improved legibility?

Nick

​

[Ron Frederick]

Hi Nick,

On Nov 24, 2015, at 3:52 PM, Nicholas Chammas <nicholas...@gmail.com> wrote:

Python 3.5 introduced new syntax to better support asynchronous programming.

Do you have any thoughts on how this new syntax can be used with AsyncSSH? Is it mostly a matter of making simple replacements like putting in await instead of yield from? Are there any potential benefits to using the new syntax apart from improved legibility?

Yeah, I read about the new async syntax in Python 3.5 while it was still in beta, and am quite eager to eventually convert AsyncSSH over to it. However, since 3.5 is still so new, I thought it would be best to wait a while before requiring it. It should provide some significant additional compile-time checking that coroutines are properly declared, though, and that calls to wait on them properly use “await”. I can’t count the number of times when writing code I forget to add the “yield from” and then wonder for a bit why my coroutine is never executed.

As far as using the new syntax in Python 3.5 with the existing AsyncSSH code, I did a quick test here where I converted my simple_client.py to use the new “async” and “await” syntax and it seemed to run just fine, so apparently it is possible to mix the old and new syntax if you’re willing to write an application which requires 3.5. Here’s a sample:

import asyncio, asyncssh, sys

class MySSHClientSession(asyncssh.SSHClientSession):

    def data_received(self, data, datatype):

        print(data, end='')

    def connection_lost(self, exc):

        if exc:

            print('SSH session error: ' + str(exc), file=sys.stderr)

async def run_client():

    with (await asyncssh.connect('localhost')) as conn:

        chan, session = await conn.create_session(MySSHClientSession, 'ls abc')

        await chan.wait_closed()

try:

    asyncio.get_event_loop().run_until_complete(run_client())

except (OSError, asyncssh.Error) as exc:

    sys.exit('SSH connection failed: ' + str(exc))

Unfortunately, there are some deprecations in 3.5 which affect the AsyncSSH code, but I can’t switch to the new preferred syntax and still support version 3.4.0. One of the deprecations happened in 3.4.4 for instance, and only 3.4.4 and later have the replacement function (asyncio.ensure_future instead of asyncio.async). It would get messy to try and support both, and I’m not quite sure what to do about that.

-- 

Ron Frederick

ro...@timeheart.net

[Nicholas Chammas]

Good to see that using the new syntax with AsyncSSH as it stands today is straightforward!

Unfortunately, there are some deprecations in 3.5 which affect the AsyncSSH code, but I can’t switch to the new preferred syntax and still support version 3.4.0. One of the deprecations happened in 3.4.4 for instance, and only 3.4.4 and later have the replacement function (asyncio.ensure_future instead of asyncio.async). It would get messy to try and support both, and I’m not quite sure what to do about that.

Yeah, I noticed that deprecation too. Funny thing is 3.4.4 hasn’t even been released yet.

Once 3.5 becomes a bit more widely used, it might make sense to declare some version of AsyncSSH as the last version that supports Python 3.4 and then require users to start using 3.5 if they want newer versions of AsyncSSH.

The upgrade path from 3.4 to 3.5 is probably quite easy, but I’m sure it will nonetheless be a nuisance to users still on 3.4. In the long run, it might still pay off to force that upgrade if it means writing and maintaining AsyncSSH becomes easier more enjoyable for you and for other contributors.

I understand the tension though. Writing a library is tough business.

Nick

[Ron Frederick]

On Nov 24, 2015, at 9:44 PM, Nicholas Chammas <nicholas...@gmail.com> wrote:

Good to see that using the new syntax with AsyncSSH as it stands today is straightforward!

Unfortunately, there are some deprecations in 3.5 which affect the AsyncSSH code, but I can’t switch to the new preferred syntax and still support version 3.4.0. One of the deprecations happened in 3.4.4 for instance, and only 3.4.4 and later have the replacement function (asyncio.ensure_future instead of asyncio.async). It would get messy to try and support both, and I’m not quite sure what to do about that.

Yeah, I noticed that deprecation too. Funny thing is 3.4.4 hasn’t even been released yet.

Huh! I hadn’t even noticed that. :)

I took a closer look and it seems like asyncio.async() may be the only issue I have here in 3.5. I think what I really want to be using is the BaseEventLoop.create_task() function, which was added in 3.4.2, since I know the argument I’m passing in is a coroutine and not a future. If I want compatibility with 3.4.0, I’ll still need a wrapper function, but I think I’ll add my own create_task() wrapper that either calls the event loop’s create_task() or falls back to asyncio.async() for older versions. I can then pull this wrapper out later once I have a version that depends on 3.5.

Once 3.5 becomes a bit more widely used, it might make sense to declare some version of AsyncSSH as the last version that supports Python 3.4 and then require users to start using 3.5 if they want newer versions of AsyncSSH.

I was thinking here that the changes in AsyncSSH to convert over to the new syntax would be substantial enough to probably justify calling that version a “2.0” release. I can make that require 3.5, but still keep around a “1.x” branch on Github for bug fixes of the older version that supports 3.4. I may also start to experiment with some of the other 3.5 features in this release, like the ability to do type annotations. I’ll also be able to take advantage of things like async content managers there. For now, I’ve had to work around this in my SFTP implementation by having the context manager initiate a close of the remote file when it exits but not wait for the response.

The upgrade path from 3.4 to 3.5 is probably quite easy, but I’m sure it will nonetheless be a nuisance to users still on 3.4. In the long run, it might still pay off to force that upgrade if it means writing and maintaining AsyncSSH becomes easier more enjoyable for you and for other contributors.

Yeah. I think the benefits of 3.5 in terms of code readability and maintainability will make it worth it, and keeping the old branch around in case someone really needs a change back-ported should hopefully ease the pain somewhat for those stuck on 3.4.

The main reason I haven’t gone ahead with this is that I’m still busy adding unit tests. I’ve got about 80% of the modules (about 60% of the code) fully covered now. There’s a fair amount of work left, but I’m making progress.

-- 

Ron Frederick

ro...@timeheart.net

[Nicholas Chammas]

Sounds like a plan. I’ve seen other projects maintain maintenance branches that they just backport critical fixes to as a courtesy to their users.

I may also start to experiment with some of the other 3.5 features in this release, like the ability to do type annotations.

By the way, you can do this in Python 3.0+. It’s just that in 3.5 they standardized the annotations under the typing module to make life easier for static type checkers like mypy.

Nick

​

[Nicholas Chammas]

Hey Ron,

Regarding this part of your example:

async def run_client():
    with (await asyncssh.connect('localhost')) as conn:
        chan, session = await conn.create_session(MySSHClientSession, 'ls abc')
        await chan.wait_closed()

Would this be better expressed as follows?

async def run_client():
    async with asyncssh.connect('localhost') as conn:
        chan, session = await conn.create_session(MySSHClientSession, 'ls abc')
        await chan.wait_closed()

One of the new pieces of syntax that got introduced alongside async and await is async with.

To support this, the connect() context manager would need to implement __aenter__() and __aexit__() methods, both of which return awaitables. I think these can be added in a backwards compatible fashion by using yield from, which is an awaitable.

Nick

[Ron Frederick]

Hi Nick,

Yeah - going forward I think it will make sense to migrate from my existing synchronous context handlers to the new asynchronous ones. In doing so, I would be able to not only perform the close() call but also include the wait_closed() call in __aexit__(). It might make sense to add something similar for channels, but it wouldn't be as easy to do something like that with the streams API where readers & writers are returned.

One other place that could use this is the SFTP client. Right now, I have to do the a non-blocking close call in the SFTPFile exit handler, so the file is not fully closed when the context handler exits.