Delay when SSH'ing into some systems in my LAN

[S.K.R. de Jong]

This started happening recently, with no apparent reason.

I have systems A, B, C, D in my LAN. When I SSH from A into B the
connection gets established immediately. However when I do it from A to C
I get the following SSH traces in A:

OpenSSH_7.4p1, OpenSSL 1.0.2u 20 Dec 2019
debug1: Reading configuration data /home/xyz/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "C" port 22

It hangs at that point for 20 seconds or so, but then the connection
succeeds. Once it is established, the performance of the session is as
expected. The same behavior is observed when I try to SSH from A into D.

Taking the "resolving "C" port 22" as a clue, I thought that I
might be having DNS problems in A. That does not seem to be the case
though:

$ dig C

; <<>> DiG 9.11.19 <<>> mimir
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63806
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0,
ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;C. IN A

;; ANSWER SECTION:
C. 0 IN A 192.168.0.9

;; Query time: 0 msec
;; SERVER: 192.168.0.7#53(192.168.0.7)
;; WHEN: Tue Jul 21 09:07:02 MDT 2020
;; MSG SIZE rcvd: 50

The resolution happens with no delay, but the SSH connection attempt to C
keeps taking 20 seconds, as above. And analogously for D.

Now once I have SSH'd into C, I can SSH from C into D with no
delay. I can even SSH from C into A (or B, or D) with no delay
whatsoever. In general, if A is not involved as a client things seem to
be normal.

So it would seem that the problem is in A alone, and not when
trying to SSH into every system - only some seem to be affected.

Any suggestions on how to diagnose this? I can't see anything
relevant in the logs of the systems involved - the only thing present is
the delay I mentioned, in the circumstances that I described. In A I have
the following IPtables rules:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

whereas in the other systems I have no IPtables rules. I stopped and
starting the network in A, to no avail. I guess that rebooting A might
solve the problem. Or not. Even if it does, I would not understand why.

[William Unruh]

On 2020-07-21, S.K.R. de Jong <SK...@nowhere.net> wrote:
> This started happening recently, with no apparent reason.
>
> I have systems A, B, C, D in my LAN. When I SSH from A into B the
> connection gets established immediately. However when I do it from A to C
> I get the following SSH traces in A:
>
> OpenSSH_7.4p1, OpenSSL 1.0.2u 20 Dec 2019
> debug1: Reading configuration data /home/xyz/.ssh/config
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug2: resolving "C" port 22

Try to ssh into C using C's IP address rather than name.
If that goes immediately
Try putting C's name into A's /etc/hosts
Since it seems to be A that is complaining, try putting C's name and
address into /etc/hosts on A.
It seem's that it is the reverse dns that is problematic.
Note that in the above A,B,C,D could be names or they could be IP
addresses. It might help to actually give us the exact log lines, rather
than the edited ones.

[S.K.R. de Jong]

On Tue, 21 Jul 2020 16:28:57 +0000, William Unruh wrote:

> On 2020-07-21, S.K.R. de Jong <SK...@nowhere.net> wrote:
>> This started happening recently, with no apparent reason.
>>
>> I have systems A, B, C, D in my LAN. When I SSH from A into B the
>> connection gets established immediately. However when I do it from A to
>> C I get the following SSH traces in A:
>>
>> OpenSSH_7.4p1, OpenSSL 1.0.2u 20 Dec 2019 debug1: Reading
>> configuration data /home/xyz/.ssh/config debug1: Reading
configuration
>> data /etc/ssh/ssh_config debug2: resolving "C" port 22
>
> Try to ssh into C using C's IP address rather than name.
> If that goes immediately Try putting C's name into A's /etc/hosts Since
> it seems to be A that is complaining, try putting C's name and address
> into /etc/hosts on A.

Thanks. When using IP addresses instead, or entering the relevant
names into /etc/hosts, the delay disappears.

> It seem's that it is the reverse dns that is problematic.

I would agree.

> Note that in the above A,B,C,D could be names or they could be IP
> addresses. It might help to actually give us the exact log lines, rather
> than the edited ones.

They were meant to be names, not IP addresses.

[S.K.R. de Jong]

On Tue, 21 Jul 2020 16:28:57 +0000, William Unruh wrote:

> On 2020-07-21, S.K.R. de Jong <SK...@nowhere.net> wrote:
>> This started happening recently, with no apparent reason.
>>
>> I have systems A, B, C, D in my LAN. When I SSH from A into B the
>> connection gets established immediately. However when I do it from A to
>> C I get the following SSH traces in A:
>>
>> OpenSSH_7.4p1, OpenSSL 1.0.2u 20 Dec 2019 debug1: Reading
>> configuration data /home/xyz/.ssh/config debug1: Reading
configuration
>> data /etc/ssh/ssh_config debug2: resolving "C" port 22
>
> Try to ssh into C using C's IP address rather than name.
> If that goes immediately Try putting C's name into A's /etc/hosts Since
> it seems to be A that is complaining, try putting C's name and address
> into /etc/hosts on A.
> It seem's that it is the reverse dns that is problematic.
> Note that in the above A,B,C,D could be names or they could be IP
> addresses. It might help to actually give us the exact log lines, rather
> than the edited ones.

Well, I found the solution, but I am not entirely sure why it was
broken in the first place.

In A, the contents of my /etc/resolv.conf have the following line
at the top:

search <my-domain> <work-domain>

where <mydomain> is a domain assigned to my LAN, and <work-domain> is the
domain of my employers, whose systems I access over a VPN.

When I do

ssh B

the connection is immediately sent to system B in my LAN, and everything
is fine - no delay at all. But when trying the same with C, the DNS
server first seems to try to resolve C.<work-domain> - which is the
origin of the delay. I don't know why it does so for C (and D) but not
for B, but the fact is that removing <work-domain> from the search line
in A's /etc/resolv.conf file solves the problem.

[William Unruh]

So you are running an internal dns server? If you have your machines on
fixed addresses, and do not have too many of them, using /etc/hosts is
probably more efficient. If they are using dhcp and the various machine
have constantly fluctuating IP addresses, or you have lots of
machines(lets say>10) then a dns server is probably best and you then
need to figure out why it is not delivering on a dns query. tcpdump on a
third machine on that network would show if the query was actually being
sent out to the dns server, and whether it is returning reasonable
answers of grabage. Also, it would seem that you have a secondary dns
server which is used after the timeout on the first one. I assume it is
also local-- so why is it returning something reasonable and the first
is not.
>