frequent ssh drops due to "Connection timed out during banner exchange"

[Nick Evgeniev]

Hi,

For some reason if I'm connecting to the host using ansible connection is been dropped frequently with "Connection timed out during banner exchange" message..

Any hints? pls check output below.. again if I just 'ssh lb0014' everything is fine

GATHERING FACTS ***************************************************************

<lb0014> ESTABLISH CONNECTION FOR USER: e21170

<lb0014> REMOTE_MODULE setup

<lb0014> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/e21170/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'lb0014', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1403648080.05-169695987704675 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1403648080.05-169695987704675 && echo $HOME/.ansible/tmp/ansible-tmp-1403648080.05-169695987704675'"]

fatal: [lb0014] => SSH encountered an unknown error. The output was:

OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011

debug1: Reading configuration data /Users/e21170/.ssh/config

debug1: /Users/e21170/.ssh/config line 20: Applying options for lb0*

debug1: Reading configuration data /etc/ssh_config

debug1: /etc/ssh_config line 20: Applying options for *

debug1: auto-mux: Trying existing master

debug1: Control socket "/Users/e21170/.ansible/cp/ansible-ssh-lb0014-22-gfadm" does not exist

debug2: ssh_connect: needpriv 0

debug1: Executing proxy command: exec ssh labgw1 /usr/bin/nc lb0014 22 2> /dev/null

debug3: timeout: 10000 ms remain after connect

debug1: permanently_drop_suid: 962233211

debug3: Incorrect RSA1 identifier

debug3: Could not load "/Users/e21170/.ssh/id_dsa" as a RSA1 public key

debug1: identity file /Users/e21170/.ssh/id_dsa type 2

debug1: identity file /Users/e21170/.ssh/id_dsa-cert type -1

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_6.2

Connection timed out during banner exchange

[Michael DeHaan]

Anything particularly interesting about the setup, OSes involved (managed or managing), or network?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/b21da244-d3cb-4551-8872-0a459578fda5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick Evgeniev]

Hi,

it's a LAN with some VLANs configured (out of my control so hardly can tell more). As I don't see any problem with ssh, I guess it could be ssh library ansible is using.. (needs to be tuned or may be changed?)

ansible version is 1.6.2

controller (ansible) host is osx:

Darwin hostname 13.1.0 Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64

target host is redhat linux:

Linux lb0079 2.6.32-358.14.1.el6.x86_64 #1 SMP Mon Jun 17 15:54:20 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

ssh connection is being proxied by:

host lb0*

  user some_user

  ProxyCommand ssh proxyhost /usr/bin/nc %h %p 2> /dev/null

  identityfile /Users/e21170/.ssh/id_dsa

#Host lb0*

#  ControlPath ~/.ssh/%h.%p.%r

#  ControlMaster auto

[Paul Durivage]

How many parallel connections are you proxying through "proxyhost"?  My team encounter a similar situation where the "ssh_config" directive "MaxStartups" was too low, and the ssh connection timeout value was being exceeded connecting to the proxy server.

At any rate, check your proxy's MaxStartups, and adjust accordingly.  You can test if this is an issue by setting a fork limit to 1 -- one connection (as opposed to potentially many parallel connections) should be rather quick.

Additionally, I'd look into that proxy command.  Is there any reason why you cannot proxy using the -W argument for proxying?  I'm curious if the proxy command is wonky and causes some sort of connection problem with ansible's default SSH configuration settings.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/6fde7729-33f4-461a-851b-1353ba2fd4fb%40googlegroups.com.

[Nick Evgeniev]

Hi,

MaxStartups is a great idea, thanks! will ask sysadmins to adjust that setting. also no reason not to use -W other than 'old school' pattern. switched to it (though i doubt it's an issue)

[Nick Evgeniev]

it looks like adjusting MaxStartups doesn't help in my case.. also netstat on a proxy machine doesn't show lots of connections to 22 port... so just wondering what could it be..

just to recap ssh to target host connects quickly all the time. ansible sometimes repeatedly fails.. after waiting for approx 1 sec.

[Paul Durivage]

Have you tried increasing ConnectionTimeout to something higher than 10 seconds?

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/cd3ca5aa-09fc-437e-82d7-005fdf8bd432%40googlegroups.com.