async client for implementing revere SSH

[Rajeshwari Srinivasan]

Hi, I'm new to asyncSSH and I'm trying to implement reverse SSH here. 

Ref: https://medium.com/@ismailakkila/black-hat-python-reverse-shells-using-ssh-and-paramiko-8489acfc8fbf

Reusing the server code, I'm trying to write async client for this. 

class MySSHClientSession(asyncssh.SSHClientSession):
    def data_received(self, data, datatype):
        print(data, end='')
    def connection_made(self, conn):
        print("connection made")
    def connection_lost(self, exc):
        print("connection lost")
        if exc:
            print('SSH session error: ' + str(exc), file=sys.stderr)

 

async def run_client():
    async with asyncssh.connect('localhost') as conn:
        chan, session = await conn.create_session(MySSHClientSession)
        #await chan.wait_closed()
loop = asyncio.get_event_loop()
try:
        loop.run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
        sys.exit('Error starting server: ' + str(exc))
loop.run_forever()

While the connection is successful, I notice "connection made" and "connection lost" back to back, on the client. 

On the server, I notice "SSH connection closed". Is it wrong for me to expect that the (chan, session) created will stay alive, until I close it explicitly? 

In other words, can't I reuse the (chan, session) tuple created once, for multiple commands/requests sent from the server? 

[Ron Frederick]

Hello,

You mention “reverse SSH” here, but your example below seems to be a traditional “forward” direction SSH client. AsyncSSH provides support for real reverse-direction SSH connections using connect_reverse() and listen_reverse() if you want that, though. You can see some examples of this at https://github.com/ronf/asyncssh/issues/205. Basically, you call listen_reverse() to set up a listening socket that will act as an SSH client on all inbound connections to it, and you make outbound connections to it use connect_reverse(), which acts as an SSH server once the connection is established, allowing multiple incoming SSH sessions from the remote system to be opened over that connection.

In your example code, the SSH connection will be closed when you leave the “async with” block which opens the connection, and that will close the channel/session you opened within that automatically, so that may be part of why you are seeing “SSH connection closed”.

As for reusing a session for multiple commands, it’s possible to do that if you open an interactive shell and then drive that shell by writing commands to it and parsing the output you get back, but there’s no way to tell in that case when a command has finished running unless you put something specific in the output that you can match on. Since you can open multiple SSH sessions over a single SSH connection, though, it’s much cleaner in most cases to start a new session for each command you want to run. You still only authenticate once, so it’s quite efficient. Doing this will let you actually get an end-of-file indication when the command finishes, and even get an exit status for each command. In fact, if you just want to capture the output of a command without doing any interactive input, you can use something like the run() method on the SSHClientConnection object to do all of that in a single call.

Looking at the link you sent, I see their concept of “reverse SSH” is a bit different than what I described above. It’s actually still using a forward-direction SSH connection and session, but then letting the “server” run commands on the client after the session is opened. In that implementation, it looks like the client only opens one SSH session with the server, so the server must send multiple commands over that single session. It also assumes that all of the output of the command being run is less than 1024 bytes, and that it will all be sent in a single SSH data message, which seems like a very questionable assumption. By running the actual SSH connection in reverse, you get something much more robust. The only potential issue is that now the authentication is reversed as well. So, the “server” has to authenticate to the “client”, once the client makes the outbound connection to it. In my opinion, that’s a good thing, though, as you’d want the server to prove it has the right to run commands on the client before it is allowed to do so.

-- 
Ron Frederick
ro...@timeheart.net

[Rajeshwari Srinivasan]

Thank you for a detailed answer, Ron!

You mention “reverse SSH” here, but your example below seems to be a traditional “forward” direction SSH client.

-- Yes, I'm trying to follow the method given in the shared link.

You can see some examples of this at https://github.com/ronf/asyncssh/issues/205. 

-- Would be great if you can include them in the examples folder. I'll try this out separately by having the server authenticate to the client, like you have mentioned. 

In that implementation, it looks like the client only opens one SSH session with the server, so the server must send multiple commands over that single session. It also assumes that all of the output of the command being run is less than 1024 bytes, and that it will all be sent in a single SSH data message, which seems like a very questionable assumption.

-- True, the code needs to be tweaked to perform better.

Actually, this is my requirement:

Client: Connect to server and accept reverse ssh tunnels to be opened on same connectionServer: Accepts connection from client, then opens 3 reverse ssh tunnels on the incoming connection. Each of these tunnels would run one command [“ls”, “sleep 30 && date”, “sleep 5 && cat /proc/cpuinfo”] Server program, in return, prints the received response for each of these commands (one should come back almost immediately, other after 5 and other after 30)

Do you have some advice on how this scenario should ideally be implemented using asyncssh? 

[Ron Frederick]

On Nov 21, 2019, at 8:24 AM, Rajeshwari Srinivasan <rajes...@lilac.cloud> wrote:

Thank you for a detailed answer, Ron!

You mention “reverse SSH” here, but your example below seems to be a traditional “forward” direction SSH client.

-- Yes, I'm trying to follow the method given in the shared link.

You can see some examples of this at https://github.com/ronf/asyncssh/issues/205. 

-- Would be great if you can include them in the examples folder. I'll try this out separately by having the server authenticate to the client, like you have mentioned. 

In that implementation, it looks like the client only opens one SSH session with the server, so the server must send multiple commands over that single session. It also assumes that all of the output of the command being run is less than 1024 bytes, and that it will all be sent in a single SSH data message, which seems like a very questionable assumption.

-- True, the code needs to be tweaked to perform better.

Actually, this is my requirement:

Client: Connect to server and accept reverse ssh tunnels to be opened on same connectionServer: Accepts connection from client, then opens 3 reverse ssh tunnels on the incoming connection. Each of these tunnels would run one command [“ls”, “sleep 30 && date”, “sleep 5 && cat /proc/cpuinfo”] Server program, in return, prints the received response for each of these commands (one should come back almost immediately, other after 5 and other after 30)

Do you have some advice on how this scenario should ideally be implemented using asyncssh? 

The AsyncSSH reverse connection support should be perfect for that. It would let you open multiple sessions in parallel over the connection, sending separate commands on each, and allow you to easily collect the results separately for each. Since AsyncSSH doesn’t normally fork off local processes to run commands, you’ll have to add that piece yourself, but that’s not too bad. You just need coroutines to copy stdin/stdout/stderr between the local process and the SSH session. Once you have that, the “server” code should be extremely simple. Here’s an example of the reverse client:

import asyncio

import asyncssh

from asyncio.subprocess import PIPE

async def forward_data(reader, writer):

    try:

        while not reader.at_eof():

            data = await reader.read(8192)

            writer.write(data)

        writer.write_eof()

    except BrokenPipeError:

        pass

async def handle_client(process):

    local_proc = await asyncio.create_subprocess_shell(

            process.command, stdin=PIPE, stdout=PIPE, stderr=PIPE)

    asyncio.create_task(forward_data(process.stdin, local_proc.stdin))

    

    await asyncio.gather(forward_data(local_proc.stdout, process.stdout),

                         forward_data(local_proc.stderr, process.stderr))

    

    process.exit(await local_proc.wait())

    await process.wait_closed()

async def connect_reverse():

    conn = await asyncssh.connect_reverse('localhost', 8022,

                                          server_host_keys=['ssh_host_key'],

                                          authorized_client_keys='ssh_user_ca',

                                          process_factory=handle_client,

                                          encoding=None)

    

    await conn.wait_closed()

asyncio.run(connect_reverse())

Here’s an example of the reverse server:

import asyncio

import asyncssh

async def run_client(conn):

    commands = ('ls', 'sleep 30 && date', 'sleep 5 && cat /proc/cpuinfo')

    async with conn:

        tasks = [conn.run(cmd) for cmd in commands]

        for task in asyncio.as_completed(tasks):

            result = await task

            print('Command:', result.command)

            print('Return code:', result.returncode)

            print('Stdout:')

            print(result.stdout, end='')

            print('Stderr:')

            print(result.stderr, end='')

            print(75*'-')

async def listen_reverse():

    async with asyncssh.listen_reverse(port=8022, known_hosts='known_hosts',

                                       acceptor=run_client) as server:

        await server.wait_closed()

asyncio.run(listen_reverse())

Let me know if you have any problems getting this to work.

One thing that occurred to me while writing this is that I may want to extend my SSHProcess redirection code to accept standard StreamReader/StreamWriter objects as possible arguments. Today, it accepts SSHReader/SSHWriter, but not other asyncio streams. Adding that would avoid the need for the  forward_data() data pump and instead allow use of the SSHProcess redirect() method. I’m going to experiment with that next.

After that, I’ll look at trying to add a new example for this.

-- 

Ron Frederick
ro...@timeheart.net

[Bhakta Raghavan]

Ron,

This is a good example. It should go to the examples folder in the repo.

Question:

How do we create/manage the ssh_host_key and ssh_user_ca ?

Server needs to have a way to trust the client(s) that are connecting

Client further needs a way to trust the incoming reverse ssh sessions.

Can you provide an example of generating and using the certificates between the client and server. Which files need to go on which host (client/server)?

thanks

-Bhakta

[Bhakta Raghavan]

I looked at this response: https://github.com/ronf/asyncssh/issues/87#issuecomment-298142877

Let's say I want each client to have a different user (client1_user, client2_user, etc..) For each client, I am assuming that a key pair (pvt, pub) needs to be created. The public key should be copied over to the servers authorized_keys? Is this correct? Are there other ways to do this? 

It would be good to add this to the README in examples folder (on how to guide to create/manage CA and client keys).

cheers!

On Wednesday, November 20, 2019 at 1:32:32 PM UTC+5:30, Rajeshwari Srinivasan wrote:

[Ron Frederick]

I spent some time today extending SSHProcess to support asyncio StreamReader/StreamWriter objects. This allows the reverse client code shown below to be simplified to the following if you install the “develop” branch of AsyncSSH, eliminating the need to define and schedule the forward_data() function. Instead, the local and remote processes can be plugged together using the redirect() function in SSHProcess:

import asyncio

import asyncssh

from asyncio.subprocess import PIPE

async def handle_client(process):

    local_proc = await asyncio.create_subprocess_shell(

            process.command, stdin=PIPE, stdout=PIPE, stderr=PIPE)

    await process.redirect(stdin=local_proc.stdin, stdout=local_proc.stdout,

                           stderr=local_proc.stderr)

    

    process.exit(await local_proc.wait())

    await process.wait_closed()

async def connect_reverse():

    conn = await asyncssh.connect_reverse('localhost', 8022,

                                          server_host_keys=['ssh_host_key'],

                                          authorized_client_keys='ssh_user_ca',

                                          process_factory=handle_client,

                                          encoding=None)

    

    await conn.wait_closed()

asyncio.run(connect_reverse())

This change will become part of the next release, along with a version of the sample code here (with some minor rework of variables and filenames to make it a bit clearer).

[Ron Frederick]

On Nov 23, 2019, at 12:38 AM, Bhakta Raghavan <bhak...@gmail.com> wrote:

This is a good example. It should go to the examples folder in the repo.

[Ron] Yeah - I’ll work on that. In the meantime, see the updated example I just posted which takes advantage of some new functionality I just added to SSHProcess to make I/O redirection support asyncio streams, meaning a local asyncio.subprocess.Process can be directly plugged into an SSHProcess without having to write your own data pump.

Question:

How do we create/manage the ssh_host_key and ssh_user_ca ?

Server needs to have a way to trust the client(s) that are connecting

Client further needs a way to trust the incoming reverse ssh sessions.

[Ron] Yes - mutual authentication is being done here, but since the connection is reversed, the client actually plays the role of an SSH server and vice-versa. As a result, the client in this case provides the SSH server host key via the “server_host_keys” argument and the server validates that key against known hosts via the “known_hosts” argument. For the other direction, the server provides a user public key or certificate (via the “client_keys” argument) and the client validates that against a list of authorized keys provided via the “authorized_client_keys” argument.

In the example I sent, client_keys wasn’t specified on the server, so it uses the default identity keys in the “.ssh” directory of the user running the server. However, I’ll probably make that explicit when I add this as an official example.

You have your choice about whether to use private/public key pairs here or whether to use certificates in place of the public keys. The benefit is that you can add a single entry to the authorized_client_keys and/or known_hosts and have it work for authenticating multiple clients/servers, as long as they have a certificate signed by the CA that you use to sign the certificates.

Can you provide an example of generating and using the certificates between the client and server. Which files need to go on which host (client/server)?

[Ron] The client needs server_host_keys (private keys) and authorized_client_keys (public keys or certificates associated with the username provided by the server to trust). The server needs client_keys (private keys) and known_hosts (public keys or certificates associated with the client host to trust). This is the opposite of what you’d see with a forward-direction SSH connection, since the “client” is acting as an SSH server and vice-versa.

In terms of generating the keys and/or certificates, the link you list below (issue #87) shows an example of how to generate a key pair, and to sign that key pair with a CA key to generate a certificate. From there, you’d need to copy the public keys and/or certificates to the other side, following the format of the “authorized_client_keys” or “known_hosts” files if you want to limit which users/hosts can use which keys.

I looked at this response: https://github.com/ronf/asyncssh/issues/87#issuecomment-298142877

Let's say I want each client to have a different user (client1_user, client2_user, etc..) For each client, I am assuming that a key pair (pvt, pub) needs to be created. The public key should be copied over to the servers authorized_keys? Is this correct? Are there other ways to do this? 

It would be good to add this to the README in examples folder (on how to guide to create/manage CA and client keys).

[Ron] Since the SSH connection is reversed, the client is authenticating itself to the server as a “host” in this case, not a user. As such, you’d generally specify what host keys to trust on the server for the IP address that the client connection came from. In theory, the client could specify different host keys depending on which server it was making a reverse connection to, but the common case would probably have it always providing the same set of keys.

In the other direction, the server is authenticating itself to the client as a “user”. If you have more than one server which is accepting connections, each of those servers could provide different usernames and client keys. A single server could also authenticate itself to different clients using different usernames if it could know from the client IP address which username it should use. In that case, the run_client() code on the server would need to choose which client_keys to set based on looking at the peer IP address of the incoming client connection.

-- 
Ron Frederick
ro...@timeheart.net

[Ron Frederick]

The new reverse-direction example is now available in the develop branch documentation at:

    https://asyncssh.readthedocs.io/en/develop/#reverse-direction-example

There are also two files in the “examples” directory (reverse_client.py and reverse_server.py) you can use as a starting point for using this feature.

Note that this requires the “develop” version of AsyncSSH for now to run, since it depends on the new redirect support. However, if you need to run on the released AsyncSSH, the earlier example I posted here (with the forward_data() function) should work against release 2.0.1. Both the example and the support needed for it will be part of the next official release.

[Bhakta Raghavan]

Ron,

Thanks for the real quick turn-around on these questions. Can I also request you to add a section on creating and managing keys, where you can provide examples of creating and managing host/user keys. It would be good to avoid using .ssh/ on the machine so as to keep the user separate from the system users.

-bhakta

[Bhakta Raghavan]

I should clarify this further.

I am unclear on how to create the following keys (both client and server). An example would really help. thank you!

• client_host_key

• trusted_server_keys

• server_keys

• trusted_client_host_keys

[Ron Frederick]

The new example doesn’t depend on using any of the default keys the “.ssh” directory. If you specify all four of the arguments shown (server_host_keys and authorized_client_keys on the client and client_keys and known_hosts on the server), the keys in the “.ssh" directory won’t be used on either side.

I’m hesitant to go into too much detail about generating new keys, as there really isn’t anything AsyncSSH-specific here. In fact, a lot of people just use the OpenSSH “ssh-keygen” tool to do the key generation, and that’s not part of AsyncSSH at all. If you want to programmatically generate keys with AsyncSSH, you can do so using generate_private_key() documented at https://asyncssh.readthedocs.io/en/develop/api.html#generate-private-key and functions on the returned SSHKey object documented at https://asyncssh.readthedocs.io/en/develop/api.html#asyncssh.SSHKey which can be used to output corresponding public keys or to generate certificates. However, AsyncSSH doesn’t provide any command-line tools to do this.

[Ron Frederick]

On Nov 23, 2019, at 11:43 PM, Bhakta Raghavan <bhak...@gmail.com> wrote:

I should clarify this further.

I am unclear on how to create the following keys (both client and server). An example would really help. thank you!

• client_host_key

• trusted_server_keys

• server_keys

• trusted_client_host_keys

The client_host_key and server_key files are just standard private key files. If you are using OpenSSH to generate keys, you’d just run “ssh-keygen”. An example of this would be:

    ssh-keygen -t ed25519 -f client_host_key

or

    ssh-keygen -t ed25519 -f server_key

The “trusted_server_keys” file in in “authorized_keys” format. So, if you don’t want to put any restrictions on when the key is trusted, you can just copy the data in the file “server_key.pub" into “trusted_server_keys" directly. If you want clients to trust multiple servers, you can add multiple keys in this file, each on their own line.

The “trusted_client_host_keys” file is in “known_hosts” format. So, in the simplest case you just copy in the key from “client_host_key.pub” prefixed by the IP address of each client. For example:

192.0.2.1 ssh-ed25519 CLIENT1-BASE64-DATA

192.0.2.2 ssh-ed25519 CLIENT2-BASE64-DATA

The “ssh-ed25519 CLIENT1-BASE64-DATA” would be straight out of “client_host_key.pub”, and you’d have a separate entry for each client listing their IP address and public key. You need to use IP addresses here rather than host names, as the reverse direction SSH server only has the IP address of the remote client available to it, not its hostname. There’s usually a “comment” at the end of the key data which can be used to say which client host it is associated with. It can be set using the “-C” argument on the ssh-keygen call, and you can leave this in place in this “trusted_client_host_keys" file if you like.

-- 
Ron Frederick
ro...@timeheart.net

[Bhakta Raghavan]

This is great. I will try it out. I have a question though

>> You need to use IP addresses here rather than host names, as the reverse direction SSH server only has the IP address of the remote client available to it, not its hostname

What's the best way to handle a client that is sitting across the internet (could have DHCP based addressing, could be behind one or more NATs)? Server is on a well-known public IP.

We could trust such clients based on certificates that have been pre-installed, versus any IP related information?

-bhakta

[Bhakta Raghavan]

# Installed develop version of asyncssh

# on the client-side

[~/dev] python --version

Python 3.7.4

 [~/dev] pip freeze | grep -i asy

asyncssh==2.0.1

[~/dev]

[~/dev] python c.py

<snip>

[conn=0] Trying public key auth with ssh-ed25519 key

[conn=0] Verifying request with ssh-ed25519 key

[conn=0] Auth for user bhakta succeeded

[conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536

[conn=0, chan=0] New SSH session requested

[conn=0, chan=1] Set write buffer limits: low-water=16384, high-water=65536

[conn=0, chan=1] New SSH session requested

[conn=0, chan=2] Set write buffer limits: low-water=16384, high-water=65536

[conn=0, chan=2] New SSH session requested

[conn=0, chan=0]   Command: ls

handle_request: <asyncssh.process.SSHServerProcess object at 0x106a0af90>

[conn=0, chan=1]   Command: sleep 5 && cat /proc/cpuinfo

[conn=0, chan=2]   Command: sleep 30 && date

handle_request: <asyncssh.process.SSHServerProcess object at 0x1069a1c50>

handle_request: <asyncssh.process.SSHServerProcess object at 0x106a1a110>

[conn=0, chan=0] Uncaught exception

Traceback (most recent call last):

  File "/Users/bhakta/dev/venv/lib/python3.7/site-packages/asyncssh/connection.py", line 528, in _reap_task

    task.result()

  File "c.py", line 27, in handle_request

    stderr=local_proc.stderr)

  File "/Users/bhakta/dev/venv/lib/python3.7/site-packages/asyncssh/process.py", line 1219, in redirect

    await self._create_writer(stdin, bufsize, send_eof)

  File "/Users/bhakta/dev/venv/lib/python3.7/site-packages/asyncssh/process.py", line 658, in _create_writer

    elif _is_regular_file(file):

  File "/Users/bhakta/dev/venv/lib/python3.7/site-packages/asyncssh/process.py", line 43, in _is_regular_file

    return stat.S_ISREG(os.fstat(file.fileno()).st_mode)

AttributeError: 'StreamWriter' object has no attribute 'fileno'

[conn=0, chan=0] Closing channel due to connection close

[conn=0, chan=0] Channel closed: 'StreamWriter' object has no attribute 'fileno'

[conn=0, chan=1] Closing channel due to connection close

[conn=0, chan=1] Channel closed: 'StreamWriter' object has no attribute 'fileno'

[conn=0, chan=2] Closing channel due to connection close

[conn=0, chan=2] Channel closed: 'StreamWriter' object has no attribute 'fileno'

[conn=0, chan=1] Uncaught exception

[Bhakta Raghavan]

the exception are in _is_regular_file, the code throws an AttributeError

>> asyncssh/process.py

def _is_regular_file(file):

      """Return if argument is a regular file or file-like object"""

      try:

          return stat.S_ISREG(os.fstat(file.fileno()).st_mode)

      except OSError:

          return True

I change this to a regular "except Exception" I hit another one in close. I guard this with a try: Exception Exception: and the example seems to work fine. I am guessing this has to be cleaned up correctly.

    def close(self):

          """Stop forwarding data from the file"""

          self._conn.create_task(self._file.close())

[Ron Frederick]

On Nov 24, 2019, at 7:30 PM, Bhakta Raghavan <bhak...@gmail.com> wrote:
> This is great. I will try it out. I have a question though
>
>> You need to use IP addresses here rather than host names, as the reverse direction SSH server only has the IP address of the remote client available to it, not its hostname
>
> What's the best way to handle a client that is sitting across the internet (could have DHCP based addressing, could be behind one or more NATs)? Server is on a well-known public IP.
>
> We could trust such clients based on certificates that have been pre-installed, versus any IP related information?

[Ron] You can use “*” in a known_hosts file as a wildcard if you want to trust a host key regardless of what IP address the connection came in from. Alternately, if the client is connecting through a NAT and the public IP the connection comes from is always the same even if the client’s private IP might change, you can list the public IP here to restrict the key to only be trusted when coming on a connection from that public IP.
--
Ron Frederick
ro...@timeheart.net

[Ron Frederick]

You shouldn’t be getting into the code path which calls _is_regular_file() here. From looking at the line numbers, I’m guessing that you aren’t actually running the code from the AsyncSSH “develop” branch.

The version number on the develop branch is the same as the latest real release right now, so you’ll probably need to tell “pip” to uninstall asyncssh and _then_ tell it to install the “develop” branch version if you want to switch over.

-- 
Ron Frederick
ro...@timeheart.net

[Bhakta Raghavan]

I confirm this is working fine. [I thought I uninstalled and then installed, but it appears I missed the uninstall step. Did it again and develop works fine]

thanks for the quick turn around on this!

[Ron Frederick]

On Nov 25, 2019, at 9:21 PM, Bhakta Raghavan <bhak...@gmail.com> wrote:
> I confirm this is working fine. [I thought I uninstalled and then installed, but it appears I missed the uninstall step. Did it again and develop works fine]
>
> thanks for the quick turn around on this!

My pleasure — I’m glad to hear you got it working. Thanks for helping to test out the example!
--
Ron Frederick
ro...@timeheart.net