Proxy Device "Socket is closed"

[BKeep]

I recently setup a proxy device as a test and find it works very good except for one thing. I noticed if the device sits idle for an amount of time, running commands fails with a "Socket is closed" error.

Cannot execute "get_facts" on 10.10.42.5 as admin. Reason: Socket is closed!

If I connect to the proxied device shortly after running salt-proxy --proxyid=switch -l debug then I can run salt commands without issues. I can also kill the process, restart it, and it works good again. I also notice running the salt-proxy as a daemon like this salt-proxy --proxyid=switch -d results in the same behavior.

I'm sure this is a lack of understanding something on my part but once the proxy process starts, I would expect to send commands on demand. However, if I come back an hour later, commands fail to execute. Any suggested settings to look at or change?

Regards,
Brandon

[C. R. Oldham]

Mircea Ulinic added support for proxy keepalive and it was merged into develop today.  It is designed to circumvent this issue where the salt-proxy might not be able to tell that the controlled device has "disappeared" until it tries to send a command, so we schedule a job on salt-proxy start to send keepalive packets to keep the connection up.  Proxymodules will need to have a function added to take advantage of this.

The PR is here: https://github.com/saltstack/salt/pull/38829

(I realize that I'm the PR author, but @mirceaulinic did the keepalive work.)  

--cro

C. R. Oldham

Senior Platform Engineer, Integrations Team Lead, SaltStack

c...@saltstack.com / 801-564-4673

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/3a6b7201-0dba-48dd-a907-1655974424f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mircea Ulinic]

Hi Brandon,

I reckon you are using IOS, which is a bit more tricky to automate (compared to other platforms).

Not impossible, but you’ll have a few challenges.

In addition to what @cro pointed out, I would add (some background you may be already aware of):

- IOS does not have any kind of API, so the only thing available is screen scraping. Which is as bad as it sounds like.

  Yes, unfortunately that’s all we can do on this device.

  Others, such as Juniper, Arista or Nexus are much more reliable from this point of view.

  Even IOS-XR is slightly better, but not much.

  While Kirk Byers is doing a great job here, there are quite a few problems, see for example:

  https://github.com/napalm-automation/napalm-ios/issues/74

- the "Socket is closed” error is caused by the device terminating the SSH session.

  By default IOS drops the SSH connection if idle more than 1/2/5 minutes(?)

  I don’t remember exactly the value, as I haven’t used an IOS device in a very long time

  (and I hope I will never again - but perhaps I am biased).

  Idle does not refer to the state of the SSH session itself,

  but IOS considers the connection idle when there are no commands issued over a period of time.

  Can you increase SSH idle timeout to the maximum possible?

- The pull request referenced below has been merged today, but it will be included only in the next major release,

  Nitrogen - probably in a couple of months. Meantime, I have a pretty elegant solution with a runner.

  I thought about sharing it under https://github.com/napalm-automation/napalm-salt some time ago,

  but as more and more people ask, I will provide it tomorrow & I will let you know.

Cheers,

Mircea

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CACigG6VwE-CB8vWGw%3DkXMsfBw3W3T8UascS0mDArrkFPnTdJXg%40mail.gmail.com.

[BKeep]

Thanks to both of you for your replies. Mircea, I would be interested in seeing the solution you mentioned using runners, that would be helpful. I also talked to our network team. They are willing to increase the timeout to a longer interval as well in the short term.

@cro, I will probably try to integrate the changes from the pr you pointed me to at some point but that may be slightly above my skill level. We will see if my copy/paste fu is strong. ATM we are in the beginning stages of rolling this out so I am finishing up the states to create and manage the proxy configs. Once I can implement the piece for proxy process management, I will be able to give this a try.

Regards,
Brandon

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/3a6b7201-0dba-48dd-a907-1655974424f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mircea Ulinic]

Hi Brandon,

Sorry for late reply, I’ve been away for a couple of days.

I have added the following extension module: https://github.com/napalm-automation/napalm-salt/blob/master/nitrogen/_modules/napalm_proxy.py

Which will help you keep the connection alive.

In the proxy config (e.g. /etc/salt/proxy) you can add the following lines:

schedule:
  napalm_keepalive:
    function: napalm_proxy.reconnect
    minutes: 1

Which will schedule a job called “napalm_keepalive” to be execute the “reconnect" function every x minutes (depending on your needs).

It can even be 1 minute, as it is not going to restart the connection if still alive.

In addition, there’s this SPM package: https://github.com/napalm-automation/napalm-salt/blob/master/napalm-nitrogen.spm

Which you can download and install following the instructions form https://docs.saltstack.com/en/latest/topics/spm/

or https://github.com/napalm-automation/napalm-salt#install-napalm-salt

In the SPM you’ll find few other useful modules such:

- Netconfig state: https://docs.saltstack.com/en/develop/ref/states/all/salt.states.netconfig.html

- BGP runner: https://docs.saltstack.com/en/develop/ref/runners/all/salt.runners.bgp.html#module-salt.runners.bgp

- NET find runner: https://docs.saltstack.com/en/develop/ref/runners/all/salt.runners.net.html#module-salt.runners.net

- Table outputter: https://docs.saltstack.com/en/develop/ref/output/all/salt.output.table_out.html#module-salt.output.table_out

All will be included in the next major release, Nitrogen.

Regards.

Mircea

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/7a704458-f1d9-4a9c-bc5e-75b5edbb44b3%40googlegroups.com.

[BKeep]

Awesome, Thanks!

Regards,
Brandon