Working out a OS X 10.4 Tiger ssh implementation issue, slow logins
caca
potential solutions there are out there. I just updated to Max OS X
10.4:
ssh -V
OpenSSH_3.8.1p1, OpenSSL 0.9.7b 10 Apr 2003
There are scattered reports, and various solutions to an issue where
ssh logins take in excess of 30 seconds to instantiate a connection to
a remote host. There are various workarounds, but there is no
consensus.
What I know about the issue so far is that it is somewhat limited in
the scope to which is occurs. When it does occur, it seems to be in
routed environments.
In my case I use keys to login, but even without them, the problem
still takes upwards of 30 seconds to get to a password prompt.
ssh -vvv us...@machine.host.com
Everything is pretty normal up to this debug line:
debug3: Trying to reverse map address ip.add.re.ss.
That's the point at which the stall happens, there also seems to be
some delay as well at this point:
debug1: Next authentication method: gssapi
My connection is me sitting on a comcast line, with a linksys router
in-between me and the internet. I am allowing comcast to send me DHCP
DNS servers.
If I change to my own recursive resolver, the problem does go away.
However, this was not needed in 10.2 or 10.3 OS X. So in some ways, it
does seem to be DNS related, but I am not sure just exactly where.
I am just barely able to consider myself an admin :-) so bear with me,
I start up a ssh connection and let tcpdump take a look at what it
going on. The entire ssh transaction made a total of 139 lines in
tcpdump, almost all of them being DNS requests. It is doing a lot of
really brain dead stuff like:
100.1.168.192.in-addr.arpa
There are tons of NXDOMAIN for many of the comcast NS's
I see:
ns5.attbi.com.domain: 15486+ PTR? 68.227.148.216.in-addr.arpa
And I have no idea why it looks that up, its not the inverse of the
remote host I am trying to get at.
Basically, the resolver is running through:
Ns[1-5].attbi.com.domain and that just takes time to talk to those all
five.
The amount of times it tries to reverse map my lan address
192.168.1.xxx is just weird as well.
I don't want to use my own recusrive resolver, as I use the comcast
line as a good way to test DNS propagation and such, not to mention,
with this such a hot topic on the Apple message boards, there needs to
be a solution, not everyone is running DNS of their own. My DNS is
pretty burdened with DNSBL lookups anyway.
Some people have found success with recompiling openSSH, others have
made entries into /etc/hosts, and some have had no success with either.
Nothing about my network has changed, just the OS, so I am wondering
what Apple did to ssh in Tiger, or is 3.8.1p1 just known problematic?
Could this at all be related to the ipv6 issues OS X has, and as far as
I know, still has, as a bug somewhere in the BSD kernel?
If building new fixes this, what exactly does that entail, is it just
one binary that could be put out for the benefit of others?
If there are any tests you need from me, I am happy to work on them, I
would love to get back to being able to ssh in while on the phone and
not have the customers wait 45 seconds for me to figrue something out.
--
Scott
Richard E. Silverman
> In my case I use keys to login, but even without them, the problem
> still takes upwards of 30 seconds to get to a password prompt.
>
> ssh -vvv us...@machine.host.com
> Everything is pretty normal up to this debug line:
>
> debug3: Trying to reverse map address ip.add.re.ss.
>
> That's the point at which the stall happens,
Some background: the Apple version of OpenSSH is not just a compile or
port of the OpenSSH release; it has code added to it. In this case, it is
the ability to use Kerberos for host authentication, via various GSS key
exchange mechanisms (you can find the same code in Debian ssh-krb5). In
order to construct a ticket request for the SSH server, the client has to
find the canonical name of the server host; that's what's happening here
as it does a reverse lookup of the server's address. Unfortunately, this
can't be turned off, for two reasons. First, OpenSSH does not provide a
configuration option to set the list of key exchange mechanisms the client
will use (note that Tectia does provide this option). Second, the GSS key
exchange code unconditionally uses DNS:
--- [kexgssc.c] ----------------------------------------
kexgss_client(Kex *kex)
{
...
if (ssh_gssapi_import_name(ctxt,get_canonical_hostname(1))) {
fatal("Couldn't import hostname ");
}
...
--------------------------------------------------------
The "1" there is the culprit; it's the "use DNS" flag. It would be
useful to have a UseDNS client-side flag as well as for the server, which
in this case would have this code simply try the user-supplied hostname
for the ticket request instead of going to the DNS. But, OpenSSH doesn't
have this option either.
There are other usages of DNS as well -- for example, by default the
client will try to find the Kerberos context for the server via the DNS
(realm name & KDC's). You can prevent that part with:
--- [/etc/krb5.conf] -----------------------------------
[libdefaults]
dns_fallback = false
--------------------------------------------------------
If you recompile with GSSAPI=0, it will remove all Kerberos support and
prevent this problem.
Now, I hasten to add that these are all hacks; the real problem here is
your DNS, not OpenSSH or Kerberos. Any query to the DNS should provide an
answer (positive or negative) reasonably quickly. Alas, the world is full
of idiots and nameservers are frequently woefully misconfigured. To know
what's going on, we'd have to see all of your tcpdump output.
> there also seems to be
> some delay as well at this point:
>
> debug1: Next authentication method: gssapi
This is the use of Kerberos for user authentication, rather than server.
That *is* configurable, e.g.:
--- [~/.ssh/config ] -----------------------------------
host *
PreferredAuthentications publickey,password
--------------------------------------------------------
The default list includes gssapi(-with-mic).
> It is doing a lot of really brain dead stuff like:
> 100.1.168.192.in-addr.arpa
There is nothing "brain-dead" about this.
--
Richard Silverman
r...@qoxp.net
scott
head.
Since the main stall out seems to be at:
debug3: Trying to reverse map address ip.add.re.ss.
know each of the 10 or so machines I am connectting to, so could I not
just bypass DNS on this and put something in /etc/hosts?
If so, that is an acceptable solution to me, but what do I put in
there?
Of course, the other options is to just use my recursive resolver, but
I would rather not in this case. It seems Comcast's DNS is just being
flakey and taking the bulk of the time to answer negatively, and ssh
has to wait on those answers.
I am not 100% keen on posting the tcp dump log here, is there a way I
can get it to you off group? Would you by any chance have the time to
take a look at it and break it out for me as to what is happening?
Thanks again.
Dimitri Maziuk
> Thank you Richard for your reply, I have to say, 70% of it is over my
> head.
> Since the main stall out seems to be at:
> debug3: Trying to reverse map address ip.add.re.ss.
> What exactly is it looking for in the reverse of the IP address?
There are 2 maps: hostname to IP and IP to hostname, and then
there are aliases. E.g. my websrver's real name is "manatee",
but clients connect to it as "www". The trick is that usually
there is no alias to IP map, only alias to real (canonical)
hostname.
From Richard's description, when you ssh to "www", you have
to look up the IP address of "www" first, and then run the
reverse (IP to name) query to get the real name: "manatee".
...I
> know each of the 10 or so machines I am connectting to, so could I not
> just bypass DNS on this and put something in /etc/hosts?
>
> If so, that is an acceptable solution to me, but what do I put in
> there?
>
> Of course, the other options is to just use my recursive resolver, but
> I would rather not in this case. It seems Comcast's DNS is just being
> flakey and taking the bulk of the time to answer negatively, and ssh
> has to wait on those answers.
You could put your 10 names/addresses in /etc/hosts. I'm
not sure it's gonna work, it didn't work for me when last
10.3 update screwed up DNS resolution in kmail -- but then
kmail is a fink app, not oh sex proper.
As for DNS being flakey, it *does* take a long time to
answer negatively because it has to query 3 nameservers
(who in turn may have to query their upstream nameservers,
each) before it gives up. Richard's statement that DNS
should return an answer fairly quickly is only correct
for values of "fairly quickly" approaching 2.5 minutes
(or something -- my BIND book is at work so I can't look
up the actual figures).
Dima
--
Sufficiently advanced incompetence is indistinguishable from malice.
scott
Dimitri Maziuk wrote:
> You could put your 10 names/addresses in /etc/hosts. I'm
> not sure it's gonna work, it didn't work for me when last
> 10.3 update screwed up DNS resolution in kmail -- but then
> kmail is a fink app, not oh sex proper.
If my own personal recursive resolver can work fine, and speedy, then
there has to be something I can enter into /etc/hosts to do what my DNS
server is. Any ideas on what to enter? Or does that file not get
consulted at all in this case?
Dimitri Maziuk
You really should ask on an OS X group -- I forget what BSD uses for
nameserver switch, and whatever it is Apple sure has some other crap
hanging off the side of it (prob. NetInfo).
What do you mean by "personal recursive resolver"? Are you running
a caching nameserver?
Dima
--
I'm going to exit now since you don't want me to replace the printcap. If you
change your mind later, run -- magicfilter config script
scott
that I have set to allow me, sitting on comcast, to use it for local
lookups. So, in tcp/ip DNS settings, I enter in that DNS server, it is
under this scenario, that ssh logins are just fine.
it is when I do not use that DNS server, and use the ones that comcast
DHCP's out to me, that the connections gets stuck.
Richard E. Silverman
DM> scott sez:
>> Dimitri Maziuk wrote:
>>> You could put your 10 names/addresses in /etc/hosts. I'm not sure
>>> it's gonna work, it didn't work for me when last 10.3 update
>>> screwed up DNS resolution in kmail -- but then kmail is a fink
>>> app, not oh sex proper.
>> If my own personal recursive resolver can work fine, and speedy,
>> then there has to be something I can enter into /etc/hosts to do
>> what my DNS server is. Any ideas on what to enter? Or does that
>> file not get consulted at all in this case?
DM> You really should ask on an OS X group -- I forget what BSD uses
DM> for nameserver switch, and whatever it is Apple sure has some
DM> other crap hanging off the side of it (prob. NetInfo).
---- [/etc/lookupd/hosts] ----------------------------------------
# lookupd(8) config file for "hosts" map
# check flat files before other sources
#
LookupOrder FFAgent CacheAgent NIAgent DNSAgent NILAgent
# Make sure cache validation is on, so it respects DNS ttl's.
#
ValidateCache YES
------------------------------------------------------------------
--
Richard Silverman
r...@qoxp.net
Dimitri Maziuk
>
> # lookupd(8) config file for "hosts" map
Ah, lookupd, now I remember. Thank you Richard.
Scott, if you're running your own caching nameserver, make sure
it listens on 127.0.0.1 and then put "nameserver 127.0.0.1" in
/etc/resolv.conf as 1st of 3 nameservers listed there.
Dimitri
--
Well, lusers are technically human. -- Red Drag Diva