Keep alive an idle connection / sftp

[Adriano Donninelli]

Hello! I'm here once again asking for support (hello Ron hope your 2021 started nicely!)

My project with asyncssh is working fine, but I'm failing to keep a connection open in idle for a long period of time, I found keep-alive logic on documentation but it doesn't seem to help in anyway (maybe I'm using it wrong).

I need the process to run in background, but after some time in idle I get a connection lost error.

I create the connection (which I reuse for the entire codebase) as follows:

conn = await asyncssh.connect(self.host, keepalive_interval=60, keepalive_count_max=10)

I also use sftp on this connection.

What's the standard to keep the connection open, is my server misconfigured? Is this normal? Should I setup a more complex reconnection system?

Thanks!

Adriano

[Ron Frederick]

Hi Adriano,

On Jan 28, 2021, at 11:57 AM, Adriano Donninelli <realvi...@gmail.com> wrote:
> Hello! I'm here once again asking for support (hello Ron hope your 2021 started nicely!)

Thanks - happy new year!

> My project with asyncssh is working fine, but I'm failing to keep a connection open in idle for a long period of time, I found keep-alive logic on documentation but it doesn't seem to help in anyway (maybe I'm using it wrong).
>
> I need the process to run in background, but after some time in idle I get a connection lost error.
>
> I create the connection (which I reuse for the entire codebase) as follows:
> conn = await asyncssh.connect(self.host, keepalive_interval=60, keepalive_count_max=10)
> I also use sftp on this connection.

That should be fine.

> What's the standard to keep the connection open, is my server misconfigured? Is this normal? Should I setup a more complex reconnection system?

The way you’re setting it looks reasonable to me. What you need to set isn’t necessarily tied to the server, though. In my experience SSH servers don’t generally time out inactive connections on their own, unless the server turns on keep alive itself and fails to get back keep alive responses from the client. If you are getting a “Connection lost” error, that’s most often caused by a firewall with an aggressive timeout, forgetting about connections which it hasn’t seen recent activity on. Enabling client keepalives should usually help with that, though, as long as they are sent often enough to not all the firewall to think the connection is gone. Setting it to 60 seconds SHOULD work here.

If you don’t enable client keepalives, how long can you generally leave a connection idle in your environment without seeing a “Connection lost”?

Is there any correlation between the total time the connection is open and when you see the “Connection lost”?
--
Ron Frederick
ro...@timeheart.net

[Adriano Donninelli]

Hey Ron, thanks for the quick reply.

> Enabling client keepalives should usually help with that, though, as long as they are sent often enough to not all the firewall to think the connection is gone. Setting it to 60 seconds SHOULD work here.

I enabled the Client Keep alive (on server sshd_config) as you suggested but the problem persists. (Tried both 60 seconds and 10 seconds)

> If you don’t enable client keepalives, how long can you generally leave a connection idle in your environment without seeing a “Connection lost”?

It seems to be between 5 and 10 minutes.

> Is there any correlation between the total time the connection is open and when you see the “Connection lost”?

I think there’s not, I have been using the script for hours by using it constantly, it always loses the connection once I leave it in idle for a while.

I want to add that a normal bash `ssh` command to the server (from terminal) left in idle for the same (and >) period of time does not result in any kind of connection lost / reset.
I do not have any Keep alive config on client ssh_config file.

Do you have any further suggestion to test out?

[Ron Frederick]

On Jan 29, 2021, at 11:27 AM, Adriano Donninelli <realvi...@gmail.com> wrote:
> Hey Ron, thanks for the quick reply.
>
>> Enabling client keepalives should usually help with that, though, as long as they are sent often enough to not all the firewall to think the connection is gone. Setting it to 60 seconds SHOULD work here.
>
> I enabled the Client Keep alive (on server sshd_config) as you suggested but the problem persists. (Tried both 60 seconds and 10 seconds)

Sorry, I meant enabling what OpenSSH calls ServerKeepAlive on the AsyncSSH client, which is what you were already doing. It’s interesting that it doesn’t seem to help to enable keepallives on _either_ the client or the server, though!

> If you don’t enable client keepalives, how long can you generally leave a connection idle in your environment without seeing a “Connection lost”?
> It seems to be between 5 and 10 minutes.
>
>> Is there any correlation between the total time the connection is open and when you see the “Connection lost”?

Ok - that in the right ballpark for what I’ve seen when firewalls drop idle connections, but the connection shouldn’t be considered idle for that long once keepalives are turned on, either on the client or on the server.

> I think there’s not, I have been using the script for hours by using it constantly, it always loses the connection once I leave it in idle for a while.
>
> I want to add that a normal bash `ssh` command to the server (from terminal) left in idle for the same (and >) period of time does not result in any kind of connection lost / reset. I do not have any Keep alive config on client ssh_config file.

If you have the keepalive_interval and keepalive_count_max set on the AsyncSSH outbound client connection, that should be the equivalent of specifying the ServerAliveInterval and ServerAliveCountMax in the client SSH config file. So, not having it there should be fine.

> Do you have any further suggestion to test out?

It’s very curious that you don’t see the problem from an OpenSSH client connection. With that, was the ClientAliveInterval still set on the server? Were you running that on the same machine as you were running AsyncSSH on when you saw the problem, and connecting to the same server?

One thing you mentioned was that you wanted to run the AsyncSSH client “in the background”. How exactly are you launching that? It shouldn’t be a problem, but I just want to make sure that the process isn’t getting suspended or blocking on something else that prevents it from doing I/O on the SSH connection.

If you enable debug logging on AsyncSSH, do you see log messages about the keepalives? Also, what do the messages look like around the point where you see the connection lost error?

[Adriano Donninelli]

Hey Ron, enabling debugging helped a lot!

Turns out that the keepalives where never sent.

It seems like they do not run on a special parallel dimension and you need to be in the async context of the connection for them to be sent (how crazy :p)

I have kept the keepalives enabled on the thread that runs a remote process and they work perfectly.

For the main thread where they do not get sent (for how my program is constructed) I just added a periodic check_connection (that emulates a keepalive), everything is working now.

One thing you mentioned was that you wanted to run the AsyncSSH client “in the background”. How exactly are you launching that? It shouldn’t be a problem, but I just want to make sure that the process isn’t getting suspended or blocking on something else that prevents it from doing I/O on the SSH connection.

With that I only meant to run it passively on a terminal, so exactly the same as running any other program, poorly expressed my idea.

Regarding the OpenSSH connection I have no idea why that was behaving differently, but honestly it may be a poor understanding of mine.

Thank you for your suggestions! Should have enabled debugging, with that on it was pretty obvious what was going on.

Have a lovely evening,

Adriano

-- 
Visit the AsyncSSH home page at http://asyncssh.timeheart.net
--- 
You received this message because you are subscribed to a topic in the Google Groups "asyncssh-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/asyncssh-users/wjSmLMFRit0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to asyncssh-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/asyncssh-users/DF109EEB-A16C-4C18-92AB-BC443B0A4AE3%40timeheart.net.

[Ron Frederick]

Glad you figured it out, Adriano!

You definitely need to be running the asyncio main loop for anything in AsyncSSH to function properly. This goes beyond just the keepalives. You can create other threads and run them in parallel if you need to block on something in a non-async way (and AsyncSSH itself does that in some rare cases), but you need to be very careful when communicating back and forth between the main asyncio event loop thread and any other threads. In particular, you’ll want to use things like loop.run_in_executor() to run something in another thread, and loop.call_soon_threadsafe() to schedule something in the event loop to run from another thread. See the page https://docs.python.org/3/library/asyncio-eventloop.html for more details.

[Adriano Donninelli]

Will do, thanks!

Adriano