connect_reverse throws error

[Adriano Donninelli]

Hello! Started to use asyncssh a couple of days ago, so excuse any mistake on my side.

I'm trying to start a listen_reverse + connect_reverse setup on my local machine + remote server the code looks (approximately) like this (Is pretty much copy paste from the examples https://asyncssh.readthedocs.io/en/stable/#reverse-direction-example)

```

# On localhost I run the following

async def run_reverse_client():

    """Make an outbound connection and then become an SSH server on it"""

    key = await load_agent_keys()

    conn = await asyncssh.connect(host, client_keys=key)

    conn = await asyncssh.connect_reverse(

        'localhost', port=8022, tunnel=conn, server_host_keys=key,

        process_factory=handle_request, encoding=None)

    await conn.wait_closed()

```

```

# On my remote server I run

async def start_reverse_server():

    """Accept inbound connections and then become an SSH client on them"""

    await asyncssh.listen_reverse(host='localhost', port=8022,

                                  acceptor=run_commands)

```

I get the following on my localhost running the connect_reverse

```

  File "/usr/local/lib/python3.7/site-packages/asyncssh/connection.py", line 6539, in connect_reverse

    conn_factory, 'Opening reverse SSH connection to')

  File "/usr/local/lib/python3.7/site-packages/asyncssh/connection.py", line 223, in _connect

    await conn.wait_established()

  File "/usr/local/lib/python3.7/site-packages/asyncssh/connection.py", line 2107, in wait_established

    await self._waiter

  File "/usr/local/lib/python3.7/site-packages/asyncssh/connection.py", line 915, in data_received

    while self._inpbuf and self._recv_handler():

  File "/usr/local/lib/python3.7/site-packages/asyncssh/connection.py", line 1151, in _recv_packet

    processed = handler.process_packet(pkttype, seq, packet)

  File "/usr/local/lib/python3.7/site-packages/asyncssh/packet.py", line 215, in process_packet

    self._packet_handlers[pkttype](self, pkttype, pktid, packet)

  File "/usr/local/lib/python3.7/site-packages/asyncssh/kex_dh.py", line 239, in _process_init

    self._perform_reply(host_key, host_key.public_data)

  File "/usr/local/lib/python3.7/site-packages/asyncssh/kex_dh.py", line 214, in _perform_reply

    self._send_reply(key_data, key.sign(h))

  File "/usr/local/lib/python3.7/site-packages/asyncssh/kex_dh.py", line 170, in _send_reply

    self._format_server_key(), String(sig))

  File "/usr/local/lib/python3.7/site-packages/asyncssh/packet.py", line 60, in String

    return len(value).to_bytes(4, 'big') + value

TypeError: object of type 'coroutine' has no len()

```

I have more code running ssh + sftp and it works great (on this same remote host).

Am I missing something? Thanks!

[Ron Frederick]

Hello,

I haven’t seen this problem before, but from a first glance of the code it looks like it could be related to the wrong type of key finding its way into the Diffie-Hellman key exchange code. More specifically, SSH agent keys and host “keysign” keys have asynchronous signing functions, but that key exchange code only works with keys that do signing synchronously. These async keys work for client authentication, but not for key exchange yet.

From the traceback, it appears you are getting the error on the connect_reverse() side, meaning the side making an outbound connection but then trying to act as a server. Where are you getting the server host key that is being used (“key” in your run_reverse_client function)?

--
Visit the AsyncSSH home page at http://asyncssh.timeheart.net
---
You received this message because you are subscribed to the Google Groups "asyncssh-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asyncssh-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/asyncssh-users/cfef1175-97ef-4ac6-91b0-840a8a4c13c1n%40googlegroups.com.

-- 
Ron Frederick
ro...@timeheart.net

[Adriano Donninelli]

Hello Ron,

let me start by saying these days I have looked to a lot of issues on asyncssh github and the feedback you gave were always impeccable, I appreciate your work.

I'm getting the key using a function I found online to load from ssh-agent without passphrase request.

```

async def load_agent_keys(agent_path=None):

    async def co_load_agent_keys(agent_path):

        # make sure to return an empty list when something goes wrong

        try:

            agent_client = asyncssh.SSHAgentClient(agent_path)

            keys = await agent_client.get_keys()

            agent_client.close()

            return keys

        except ValueError as exc:

            # not quite sure which exceptions to expect here

            print(f"When fetching agent keys: "

                  f"ignored exception {type(exc)} - {exc}")

            return []

    agent_path = agent_path or os.environ.get('SSH_AUTH_SOCK', None)

    if agent_path is None:

        return []

    return await co_load_agent_keys(agent_path)

 ```

I tried the same code using a function that just loads the rsa key and the error changed.

I just get `Reverse SSH connection failed: Connection lost` which seems slightly better.

The code I used to load the rsa is

```

def import_private_key(filename):

    sshkey = None

    basename = os.path.basename(filename)

    filename = os.path.expanduser(filename)

    if not os.path.exists(filename):

        print("No such key file {}".format(filename))

        return

    with open(filename) as file:

        data = file.read()

        try:

            sshkey = asyncssh.import_private_key(data)

        except asyncssh.KeyImportError:

            while True:

                passphrase = getpass.getpass("Enter passphrase for key {} : ".format(basename))

                if not passphrase:

                    print("Ignoring key {}".format(filename))

                    break

                try:

                    sshkey = asyncssh.import_private_key(data, passphrase)

                    break

                except asyncssh.KeyImportError:

                    print("Wrong passphrase")

        return sshkey

```

The call has become

```

    key = import_private_key('omitted_path/.ssh/id_rsa')

    conn = await asyncssh.connect('104.196.18.97', client_keys=key)

    conn = await asyncssh.connect_reverse(

        'localhost', port=8022, tunnel=conn, server_host_keys=key,

        process_factory=handle_request, encoding=None)

```

Could you point me in the right direction?

Is there any way to keep using the agent version so that I don't need to ask my users the passphrase at each run?

Could you point me in the right direction to solve a Connection lost problems?

Am I understanding correctly the logic doing 'localhost' on both listen/connect _reverse and doing the manual connection to the ssh server passing it as tunnel?

Thank you very much,

Adriano

[Ron Frederick]

On Dec 29, 2020, at 1:44 PM, Adriano Donninelli <realvi...@gmail.com> wrote:
> let me start by saying these days I have looked to a lot of issues on asyncssh github and the feedback you gave were always impeccable, I appreciate your work.

Thanks very much - happy to help!

> I'm getting the key using a function I found online to load from ssh-agent without passphrase request.
> ```
> async def load_agent_keys(agent_path=None):
> async def co_load_agent_keys(agent_path):
> # make sure to return an empty list when something goes wrong
> try:
> agent_client = asyncssh.SSHAgentClient(agent_path)
> keys = await agent_client.get_keys()
> agent_client.close()
> return keys
> except ValueError as exc:
> # not quite sure which exceptions to expect here
> print(f"When fetching agent keys: "
> f"ignored exception {type(exc)} - {exc}")
> return []
>
> agent_path = agent_path or os.environ.get('SSH_AUTH_SOCK', None)
> if agent_path is None:
> return []
> return await co_load_agent_keys(agent_path)
> ```

Yeah - this function will return a key that works fine for client authentication, but not yet as a server key that you can use for key exchange. Normally, a server wouldn’t be communicating with an SSH agent, so this has not really been an issue. However, when I added connect_reverse(), I can see how this might have opened the door to someone trying something like this. In theory, even a regular (non-reverse) AsyncSSH server process could attempt to use code like what you show here, even though it wouldn’t normally have any automatic communication with an SSH agent.

I can look into what it would take to allow this to work, but it’s definitely new functionality. The difficulty is that none of the existing key exchange code is currently asynchronous. It was written very early on and is completely callback-based. I did such a rewrite of the client authentication code a while back, but not key exchange.

There’s a read_private_key() function in AsyncSSH which could simplify this code. You’d still need to catch the KeyImportError exception if you want to prompt for passphrase, but you wouldn’t have to do all the OS path and file read calls yourself. That won’t affect the issue you’re seeing here, though.

> The call has become
>
> ```
> key = import_private_key('omitted_path/.ssh/id_rsa')
> conn = await asyncssh.connect('104.196.18.97', client_keys=key)
> conn = await asyncssh.connect_reverse(
> 'localhost', port=8022, tunnel=conn, server_host_keys=key,
> process_factory=handle_request, encoding=None)
> ```
>
> Could you point me in the right direction?
>
> Is there any way to keep using the agent version so that I don't need to ask my users the passphrase at each run?

Not currently, no. The SSH agent is designed not to let you retrieve the private key from it to use yourself locally, and currently the AsyncSSH server support requires local keys. It wants to do the signing for you, but that requires that the signing call be allowed to be asynchronous, so it can “block" waiting for a response from the SSH agent without interfering with any other async tasks which are running.

> Could you point me in the right direction to solve a Connection lost problems?
> Am I understanding correctly the logic doing 'localhost' on both listen/connect _reverse and doing the manual connection to the ssh server passing it as tunnel?

Generally speaking, the “Connection lost” error suggests you are seeing a crash of some kind on what you’re connecting to, or that it just didn’t like something you attempted to send it. So, you’ll probably need to turn on logging on that and see if you can get back the traceback from it to see what’s going wrong.

That said, I’m not quite understanding where tunneling is coming into play here. When using connect_reverse() and listen_reverse(), there generally isn’t any form of tunneling involved. Can you explain a bit more what you’re trying to do with the reverse connection here? Also, are you attempting to use some kind of jump host? That’s usually where you need tunneling, but that doesn’t usually involve reverse-direction connections.
--
Ron Frederick
ro...@timeheart.net

[Adriano Donninelli]

Thank you for the detailed answer.
Taking the time just to answer about what I'm trying to achieve but will come back once I have the time to read your answer throughly.

As a part of a bigger program which runs locally I need to launch a watchdog program on my remote server to notify me of file changes, to do so I would like to actually run a watchdog script I copy from local to remote, I built a simple demo by writing to a log file and reading the tail of it using ssh. While this seems to work as I started using asyncssh I thought it would be possible while connecting  using ssh to also have a reverse connection so that the server (without my key / passphrase or any direct interaction / setup on my behalf) can send me the changes intercepted by watchdog back without any intermediate log file.

That's where I found the two examples I posted, It looked to me like the correct formula and I tried to run it.

-

Adriano 

[Ron Frederick]

You may not need a reverse-direction connection to achieve making an outbound connection over SSH that runs a program on the remote system which reports back results such as a list of file changes. If you want that output to both be stored in a file on the remote system but also be delivered over SSH, you can potentially do that by running “tail” as part of your SSH command once you’ve started the process up to put content in the remote file. However, you could also just have the remote program you run output to both a file on the remote system and to stdout in parallel, with no need to get “tail” involved. To allow updates to stream back, you’d just leave open the outbound SSH connection from the local system to the remote system indefinitely, and it would continue to report back as new changes were discovered.

The main thing reverse direction connects would be good for is if you have a firewall blocking your ability to open SSH connections from the remote system to the local system. So, instead, you open an outbound TCP connection from local to remote but then have the remote system behave as an SSH client and the local system as an SSH server once the TCP connection is established.

As for using “tunnel=“, that’s mainly useful if you want to open an SSH connection from the local system to a jump/bastion host and then tunnel a second SSH connection from the local system through that jump host to something on the other side of it. Again, this would usually be triggered by firewall problems you are trying to work around.

[Adriano Donninelli]

So you're suggesting to use a conn.run('python mycode.py') and get the stdout? While that would work and seems a lot simpler I'm not sure on how I could fetch results before the conn.run() finishes it's work, is there any way to connect a callback on a conn.run stdout? Pardon my ignorance, I'm sure this is pretty obvious.

(This is what lead me to a file based version where the reading was going to be run on a separate thread, but I was not using asyncssh at the time)

Thanks,

Adriano 

[Ron Frederick]

On Dec 29, 2020, at 3:06 PM, Adriano Donninelli <realvi...@gmail.com> wrote:

So you're suggesting to use a conn.run('python mycode.py') and get the stdout? While that would work and seems a lot simpler I'm not sure on how I could fetch results before the conn.run() finishes it's work, is there any way to connect a callback on a conn.run stdout? Pardon my ignorance, I'm sure this is pretty obvious.

(This is what lead me to a file based version where the reading was going to be run on a separate thread, but I was not using asyncssh at the time)

Yes - instead of using conn.run() in this case, you’ll want to use conn.create_process(). From there, you can feed data to the remote process by writing to proc.stdin and/or read output from proc.stdout and proc.stderr. You can find an example at https://asyncssh.readthedocs.io/en/latest/#interactive-input.

If you have local data in a file you want to provide to the remote process, you can also use file redirection in create_process() to do that (possibly in conjunction with manually reading or writing other streams). See https://asyncssh.readthedocs.io/en/latest/#i-o-redirection for some examples of using redirection.

-- 
Ron Frederick
ro...@timeheart.net

[Adriano Donninelli]

Seems perfect, thank you for the help,  I will use create_process, appreciated the link to the references

Hope you have a pleasing evening

Adriano

[Adriano Donninelli]

Hey Ron, I tried to set everything up but it seems like I can't send data to stdin correctly, it seems to work fine using the 'bc' program of the example so I guess the problem is in the way I send data to python.

I also tried reading sys.stdin directly.

```

            async with asyncssh.connect(host, client_keys=key) as conn:

                async with conn.create_process('python script.py') as process:

                    print('Received: %s' % await process.stdout.readline())

                    process.stdin.write(path + '\n')

                    while True:

                        print('Received: %s' % await process.stdout.readline())

                        import time

                        time.sleep(1)

```

On the python side (the script.py) script run remotely:

```

if __name__ == '__main__':

    print('Hello')

    x = input()

    print('Received', x)

```

Thanks, and happy (very soon) new year

[Adriano Donninelli]

Sorry I might have omitted a useful detail, I receive the Hello message but I’m not able to send data.. I receive an empty line and it hangs there.

I will run further tests but wanted to share the progress

--
Visit the AsyncSSH home page at http://asyncssh.timeheart.net
---

You received this message because you are subscribed to a topic in the Google Groups "asyncssh-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/asyncssh-users/QQ0rTC7YAI4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to asyncssh-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/asyncssh-users/8963d152-e33b-47cd-aa15-02369f017c79n%40googlegroups.com.

[Adriano Donninelli]

Hey Ron, sorry for the confusion, I run more tests, this may be easier to understand:

Two examples, it seems to hang unless the python file finishes.

```

async with conn.create_process('python /tmp/iris_wd.py', input=path) as process:

                    print('Received: %s' % await process.stdout.readline())

                    while True:

                        print('Received: %s' % await process.stdout.readline())

                        print('Received: %s' % await process.stderr.readline())

```

And on remote

```

if __name__ == '__main__':

    print('Hello')

    x = input()

    print('Received', x)

    while True:

        print(0)

```

Will output (correctly):

```

Received: Hello

Received: Received /home/adryw/test_wow/

```

And hang here

While if I change the remote code to

```

if __name__ == '__main__':

    print('Hello')

    x = input()

    print('Received', x)

    for i in range(5):

        print(i)

```

I get

```

Received: Hello

Received: Received /home/adryw/test_wow/

Received:

Received: 0

Received:

Received: 1

Received:

Received: 2

Received:

Received: 3

Received:

Received: 4

Received:

Received:

```

Am I doing something wrong? Is this relative to running a python script this way?

I'm confused :)

[Ron Frederick]

Thanks for the additional detail — this last bit is the problem. You are alternating between blocking on receiving output from stdout and from stderr. If you get multiple lines of output on stdout but no lines on stderr (a common case), you’ll be blocked reading from stderr and not get a chance to read more from stdout.

If you need to read from both stdout and stderr, you’ll need to set up separate coroutines for each of those and run them in parallel with one another. Alternately, you can just tell AsyncSSH to combine stdout and stderr together and let you read them both as a single interleaved stream, if you don’t care which of the two the output is coming from (or have a way from looking at the messages themselves to know this).

To get stderr redirected to stdout, you just need to pass in “stderr=asyncssh.STDOUT” in the create_process() call.

If you need to read them separately, you’ll want to write a separate coroutine with the while True and readline() below for each of the two cases and use something like asyncio.gather() to run the two tasks in parallel. This could look something like

async def read_output(stream):

    while True:

        print(‘Received: %s’ % await stream.readline())

and then in your main function:

await asyncio.gather(read_stream(process.stdout), read_stream(process.stderr)

[Adriano Donninelli]

Thanks for the stderr feedback, I added that to the create_process constructor.

I'm not sure it was the issue though, It still hangs unless the remote script ends.

### I had written the answer already so I will send it for completeness but it seems that the python `input()` hangs unless the script ends :) Using sys.stdin seems to work.

This took a good chunk of my life so I will go to sleep, thank you for all the good feedback, will let you know if I successfully implement this tomorrow.

Have a nice evening

## the old message

Maybe I should say I'm running this on a thread where I set a new event loop for asyncio

```

loop = asyncio.new_event_loop()

asyncio.set_event_loop(loop)

(and run using loop.run_until_complete(...))

```

With the following local code

```

            async with asyncssh.connect(host, client_keys=key) as conn:

                async with conn.create_process('python /tmp/iris_wd.py', input=path, stderr=asyncssh.STDOUT) as process:

                    while True:

                        print('Received: %s' % await process.stdout.readline())

                        await asyncio.sleep(1)

```

And the remote code

```

if __name__ == '__main__':

    print('Hello')

    x = input()

    print('Received', x)

    for i in range(3):

        print(i)

        import time

        time.sleep(1)

```

The output is (not hanging and readline() returns the "expected" output

```

Received: Hello

Received: Received /home/adryw/test_wow/

Received: 0

Received: 1

Received: 2

Received:

``` 

While if the code never ends I get stuck on first hello

```

if __name__ == '__main__':

    print('Hello')

    x = input()

    print('Received', x)

    while True:

        print('ciao')

        import time

        time.sleep(3)

```

Output

```

Received: Hello

```

[Ron Frederick]

In the remote code, you’re calling input(). Are you expecting that input to come from the local process, via stdin? If so, you need to be writing that input each time you read a line of output or the remote side will end up getting blocked.

In a real-world scenario, you’re probably not going to know exactly how many lines of output are generated for a given amount of input. So, trying to alternate between writing to proc.stdin and reading from proc.stdout/proc.stderr may not be all that simple, unless you make all three of those into separate tasks. You’d meed to know exactly what to expect from the output and when you need to provide additional input.