Kerberos behind load balancer?
Kasundra, Digant
We just bought ourselves a nifty little NetScaler load balancing router.
But we can't seem to make it work with Kerberos. I believe we're
supposed to setup the balancer to forward on the source IP and add a
loopback address (not sure how) that listens to the same virtual IP and
responds back to whoever is doing the kinit (for example). We are
having no luck making this work and can't find any documentation on
this. Does anyone know how to make this work? We're running the KDC on
a linux box (Red Hat AS 3.0).
-- DK
________________________________________________
Kerberos mailing list Kerb...@mit.edu
https://mailman.mit.edu/mailman/listinfo/kerberos
Sam Hartman
idea. It's going to introduce a lot of complexity for no real
benefit.
Jason T Hardy
Actually, a load balancer simplifies client deployment in our case (we
can't utilize DNS load balancing on our campus). We can, with a load
balancer, have all of the KDC's share one hostname. Our kadmin server
can also share that hostname.
kerberos:88 -> points to our KDC's
kerberos:749 -> point to our admin server
Further, we can bring systems up/down or add/remove new systems without
requiring modifications to the client configurations.
I've actually got this working, except that I needed to associate the
kerberos hostname with a non-arping loopback device, which required a
minor code hack.
--
Jason T Hardy
Unix Systems Administrator
Office of Information Technology
University of Texas at Arlington
Ken Raeburn
> Sam,
>
> Actually, a load balancer simplifies client deployment in our case (we
> can't utilize DNS load balancing on our campus). We can, with a load
> balancer, have all of the KDC's share one hostname. Our kadmin server
> can also share that hostname.
>
> kerberos:88 -> points to our KDC's
(I hope by "DNS load balancing" that you can't do, you're only
referring to the hacks that return a different address per query, based
on load.)
I think there are better solutions to that. (1) Create a DNS name
which points to multiple addresses; typically the nameserver will
change the order randomly, which will effect some load balancing. (2)
Use DNS SRV records to return the names of the various KDCs, with equal
priority. Granted, these approaches aren't load-sensitive, but the DNS
SRV record approach will let you do some uneven load balancing by
adjusting the weights based on the capabilities of each server.
They'll also let you spread out your KDCs to a couple of locations, if
you don't want to risk a single point of failure.
> Further, we can bring systems up/down or add/remove new systems without
> requiring modifications to the client configurations.
The DNS records pointing to the N KDC addresses can be updated without
changing client configurations.
Even if you don't update the DNS records for a while, it should only
cause a delay of a couple seconds or so (using the MIT code) per
offline KDC, and only when the clients choose those KDCs first.
Ken
Frank Cusack
> Sam,
>
> Actually, a load balancer simplifies client deployment in our case (we
> can't utilize DNS load balancing on our campus). We can, with a load
Don't need DNS load balancing (and it's broken anyway).
> balancer, have all of the KDC's share one hostname. Our kadmin server
> can also share that hostname.
>
> kerberos:88 -> points to our KDC's
> kerberos:749 -> point to our admin server
Isn't that broken? You can't load balance the admin server because
MIT isn't multi-master. For DR it's just as easy to bring up a new
server with the old server's IP.
> Further, we can bring systems up/down or add/remove new systems without
> requiring modifications to the client configurations.
True, but modern hardware can handle VERY VERY large numbers of clients.
krb5 requests are short and efficient for the most part. You shouldn't
need more than 3 IPs and you can even have them on 2 servers (reserving
the 3rd for future use if you don't want to maintain the extra HW).
If you use DNS SRV records you can also add new systems without client
config change. That's what we do.
The load balancer is simply another failure point.
/fc
Jason T Hardy
> I think there are better solutions to that. (1) Create a DNS name
> which points to multiple addresses; typically the nameserver will
> change the order randomly, which will effect some load balancing. (2)
> Use DNS SRV records to return the names of the various KDCs, with equal
> priority. Granted, these approaches aren't load-sensitive, but the DNS
> SRV record approach will let you do some uneven load balancing by
> adjusting the weights based on the capabilities of each server.
> They'll also let you spread out your KDCs to a couple of locations, if
> you don't want to risk a single point of failure.
This is precisely what I * can not * do for political reasons.
--
Jason T Hardy
Unix Systems Administrator
Office of Information Technology
University of Texas at Arlington
________________________________________________
Jason T Hardy
> > balancer, have all of the KDC's share one hostname. Our kadmin server
> > can also share that hostname.
> >
> > kerberos:88 -> points to our KDC's
> > kerberos:749 -> point to our admin server
>
> Isn't that broken? You can't load balance the admin server because
> MIT isn't multi-master. For DR it's just as easy to bring up a new
> server with the old server's IP.
No, it's not broken. The kadmin server that's active responds to the
request. If my admin server goes down I can "promote" one of the slaves.
> True, but modern hardware can handle VERY VERY large numbers of clients.
> krb5 requests are short and efficient for the most part. You shouldn't
> need more than 3 IPs and you can even have them on 2 servers (reserving
> the 3rd for future use if you don't want to maintain the extra HW).
>
> If you use DNS SRV records you can also add new systems without client
> config change. That's what we do.
I can't modify DNS.
> The load balancer is simply another failure point.
As is everything else.
> /fc
--
Jason T Hardy
Unix Systems Administrator
Office of Information Technology
University of Texas at Arlington
________________________________________________
Ken Hornstein
>> MIT isn't multi-master. For DR it's just as easy to bring up a new
>> server with the old server's IP.
>
>No, it's not broken. The kadmin server that's active responds to the
>request. If my admin server goes down I can "promote" one of the slaves.
Dumb question time: have you actually tried changing your password
with kpasswd with this setup? I'm not sure it would work.
And let me echo the comments of others: we've run our Kerberos servers on
the oldest, crappiest hardware we've had kicking around the dustbin (we
upgrade it occasionally, but it's always to the latest "crappiest" system
we've got laying around). I seriously doubt you're going to need a load
balancer. And if you don't need it, I can't see it causing you anything
but trouble in the long run.
--Ken
Tillman Hodgson
> And let me echo the comments of others: we've run our Kerberos servers on
> the oldest, crappiest hardware we've had kicking around the dustbin (we
> upgrade it occasionally, but it's always to the latest "crappiest" system
> we've got laying around). I seriously doubt you're going to need a load
> balancer. And if you don't need it, I can't see it causing you anything
> but trouble in the long run.
I can echo that sentiment as well. When I first starting looking into
Kerberos I was concerned about client load on the KDC.
This post (from 1993) put my fears to rest:
http://groups.google.ca/groups?hl=en&lr=&th=f5ea1615382bdfcc&rnum=2
I can indeed confirm that a DECStation 5000/25 (with a 25MHz MIPS R3000
CPU and a 10MBit AUI ethernet port) can handle whatever I could throw at
it, including authentication for a website (via apache mod_auth_kerb)
that did not cache tickets, without showing any real load that I could
measure. It was _idling_.
I'm now running it on a SparcStation 10, simply because I don't have the
DECStation any more and the old Sun box is the oldest crappiest hardware
I have left where I still trust the hard drive (a relatively modern
Seagate replacement, in this case).
Older RISC hardware also tends to have real serial consoles, which is
Good Thing on a KDC that doesn't allow network logins :-)
If I /was/ going to load balance a KDC in some form, I'd do it not to
shift load as in CPU-load but rather to optimize latency for wide-area
links. Anycast would be the method I'd use.
-T
--
"If you already know what recursion is, just remember the answer.
Otherwise, find someone who is standing closer to Douglas Hofstadter
than you are; then ask him or her what recursion is."
-- Andrew "Zarf" Plotkin
Kasundra, Digant
> the oldest, crappiest hardware we've had kicking around the dustbin (we
> upgrade it occasionally, but it's always to the latest "crappiest" system
> we've got laying around). I seriously doubt you're going to need a load
> balancer. And if you don't need it, I can't see it causing you anything
> but trouble in the long run.
If we could modify DNS to do DNS round-robin, we too would be okay. But we can't.
So, given the problem space where DNS round robin is not an option, a load balancer is necessary. If DNS round robin is the only way to make this work, I'm sensing a limitation that might need to be addressed. Multimaster would be nice, too. True, its easy to bring up a new master with the same IP, but I don't plan to work 24 hours a day, 7 days a week. Some standard abilities such as multi-master support and the ability to sit nicely behind a load-balancer would be quite helpful. I'm sure our campus can't be the only one that would be interested in these sort of developments in the MIT Kerb product.
Just my two cents.
-- DK
Ken Hornstein
>we can't.
This is the part I don't understand. _WHY_ do you think you need
this? I've literally run 6 years with a very simple setup: two KDCs,
each one listed in DNS and our krb5.conf. On the rare occasions we
lose a master, the backup answers requests. I either fix the master or
bring up the slave as the master temporarily (it usually takes me a
while to notice this, because everything keeps running normally). This
is pretty much what everyone I know does. We'd all like multi-master, but
it hasn't impacted operations in my experience.
--Ken
Tillman Hodgson
> I agree that the load is not an issue. But with out DNS round-robin,
> and without the load-balancer, we'd have to arbitrarily point our
> systems and services at one of the slaves. If that slave goes down,
> we'd have to scramble to see who all was pointing to it and change
> them to point to the other place.
Anycast. Kerberos auth is UDP, after all.
> DNS round-robin would be best, but some options for those of us with
> our hands-tied would be nice. I think we may have a working solution
> although we are still testing it, and it did require a code patch to
> allow listening to the loopback.
Round-robin sucks. It means that if something goes down it fails 50% of
the time.
DNS hackery to do background checks and so forth is a bit better, but
not standardized or easy to implement.
-T
--
"In the beginning, the file was without form, and void; and emptiness
was upon the face of the bits. And the Fingers of the Author moved upon
the face of the keyboard. And the Author said, Let there be words, and
there were words."
-- From Linux System Administrators' Guide
Kasundra, Digant
-----Original Message-----
From: kerberos...@mit.edu on behalf of Tillman Hodgson
Sent: Wed 10/6/2004 12:15 PM
To: kerb...@mit.edu
Subject: Re: Kerberos behind load balancer?
On Wed, Oct 06, 2004 at 12:07:23PM -0500, Kasundra, Digant wrote:
>> I agree that the load is not an issue. But with out DNS round-robin,
>> and without the load-balancer, we'd have to arbitrarily point our
>> systems and services at one of the slaves. If that slave goes down,
>> we'd have to scramble to see who all was pointing to it and change
>> them to point to the other place.
>Anycast. Kerberos auth is UDP, after all.
Kasundra, Digant
And when we tell our app programmers that is how they have to do it and they say "Why don't you have a load balancer that does that for me instead of me having to write this into all my various apps?"
We are under political pressures to get this done and bound by other politics to not do it the way everyone else is.
-- DK
-----Original Message-----
From: kerberos...@mit.edu on behalf of Ken Hornstein
Sent: Wed 10/6/2004 12:14 PM
To: kerb...@mit.edu
Subject: Re: Kerberos behind load balancer?
>If we could modify DNS to do DNS round-robin, we too would be okay. But
>we can't.
This is the part I don't understand. _WHY_ do you think you need
this? I've literally run 6 years with a very simple setup: two KDCs,
each one listed in DNS and our krb5.conf. On the rare occasions we
lose a master, the backup answers requests. I either fix the master or
bring up the slave as the master temporarily (it usually takes me a
while to notice this, because everything keeps running normally). This
is pretty much what everyone I know does. We'd all like multi-master, but
it hasn't impacted operations in my experience.
--Ken
Kasundra, Digant
-----Original Message-----
From: kerberos...@mit.edu on behalf of Tillman Hodgson
Sent: Wed 10/6/2004 12:15 PM
To: kerb...@mit.edu
Subject: Re: Kerberos behind load balancer?
On Wed, Oct 06, 2004 at 12:07:23PM -0500, Kasundra, Digant wrote:
> I agree that the load is not an issue. But with out DNS round-robin,
> and without the load-balancer, we'd have to arbitrarily point our
> systems and services at one of the slaves. If that slave goes down,
> we'd have to scramble to see who all was pointing to it and change
> them to point to the other place.
Anycast. Kerberos auth is UDP, after all.
Ken Hornstein
>(for instance) kerb1.mit.edu and kerb2.mit.edu and list both machines as
>KDCs in the krb5.conf. If so, the app then randomly picks a KDC and
>tries that and if that fails, it rolls over to the next? You then build
>that functionality into each and every app.
That's exactly it. Although, it doesn't pick one randomly. It picks
the one with the lower priority in the SRV record, or the first one in
the file. But ... you don't have to change _a single line_ of code to
do this. The Kerberos library does all this for you automagically
(this is true of the "Big Three" of Kerberos implementations: MIT,
Heimdal, and Microsoft). Speaking as someone who's written their fair
share of Kerberos code, I can say this with some confidence (there isn't
actually a supported way to say, "Send my request to KDC X" - the library
makes that decision for your).
Sam Hartman
Jason> Sam, Actually, a load balancer simplifies client deployment
Jason> in our case (we can't utilize DNS load balancing on our
Jason> campus). We can, with a load balancer, have all of the
Jason> KDC's share one hostname. Our kadmin server can also share
Jason> that hostname.
I think what I'm questioning here is the need for load balancing of
the KDC. I agree that if you need to load balance a KDC, using a load
balancer is one way to do it. If you don't actually need to load
balance access to your KDCs, you'll find you get a much simpler
deployment without the load balancer.
--Sam
Kasundra, Digant
-----Original Message-----
From: kerberos...@mit.edu on behalf of Ken Hornstein
Sent: Wed 10/6/2004 12:41 PM
To: kerb...@mit.edu
Subject: Re: Kerberos behind load balancer?
>How do you list both in DNS? Are you implying that in DNS you only have
>(for instance) kerb1.mit.edu and kerb2.mit.edu and list both machines as
>KDCs in the krb5.conf. If so, the app then randomly picks a KDC and
>tries that and if that fails, it rolls over to the next? You then build
>that functionality into each and every app.
That's exactly it. Although, it doesn't pick one randomly. It picks
the one with the lower priority in the SRV record, or the first one in
the file. But ... you don't have to change _a single line_ of code to
do this. The Kerberos library does all this for you automagically
(this is true of the "Big Three" of Kerberos implementations: MIT,
Heimdal, and Microsoft). Speaking as someone who's written their fair
share of Kerberos code, I can say this with some confidence (there isn't
actually a supported way to say, "Send my request to KDC X" - the library
makes that decision for your).
--Ken
Frank Cusack
> I can't modify DNS.
Ah, well then that's a crazy restriction (since as a sysadmin, one
with a load balancer at your disposal, you can almost certainly spoof
DNS and make it do what you want anyway. I doubt you use TSIGs even
internally). But given that restriction, yeah load balancing sounds
reasonable.
But let's be clear, it doesn't "allow" you to do things as you
described earlier in the sense that you couldn't do them anyway; MIT
krb5 out of the box allows you to do those things.
>> The load balancer is simply another failure point.
>
> As is everything else.
However load balancers are complicated devices and more prone to failure.
/fc
Gary LaVoy
>>
>> As is everything else.
>
> However load balancers are complicated devices and more prone to
> failure.
WHOA! - Yes load balancers can be complicated if you want to use all
the features, but "prone to failure"?? where do you get that from?
We have hundreds of them and they work great. We even have our 4 KDC's
behind one with not a single problem ever.
One of my load balancers has over 950 hours of continuous uptime
without so much as a hiccup. If it *did* fail, it would simply fail
over to it's standby load balancer and keep right on going (we install
them in pairs so there is never a single point of failure).
Gary LaVoy
Apple Computer, Inc.
Jason T Hardy
> >>>>> "Jason" == Jason T Hardy <jth...@uta.edu> writes:
>
> Jason> Sam, Actually, a load balancer simplifies client deployment
> Jason> in our case (we can't utilize DNS load balancing on our
> Jason> campus). We can, with a load balancer, have all of the
> Jason> KDC's share one hostname. Our kadmin server can also share
> Jason> that hostname.
>
> I think what I'm questioning here is the need for load balancing of
> the KDC. I agree that if you need to load balance a KDC, using a load
> balancer is one way to do it. If you don't actually need to load
> balance access to your KDCs, you'll find you get a much simpler
> deployment without the load balancer.
>
> --Sam
Well, the answer to this question is complex. We don't think a
load-balancer will be required for our deployment, but it would simplify
the end-user experience.
I guess the problem that everyone is having with our deployment is the
term load-balancer. We don't actually want to easy the load off of our
KDC's, we just want provide a seamless way of ensuring availability in
the event that we lose one (or more) of them. I think it's true for
everyone who's commented to this thread that their users are probably
not aware of how Kerberos works. We're trying to make deployment as easy
as possible for the end user (migrating a huge deployment from AD to MIT
Kerberos). Having one host name to remember would certainly help.
You'll say that DNS is the answer. I would agree. The problem is, we
don't have access, or support from the folks controlling our DNS, to add
the SRV records which would simplify the whole mess. Since we already
have a load balancing switch (NetScaler) around for other purposes,
we're trying to use it in place of doing things properly. :-)
--
Jason T Hardy
Unix Systems Administrator
Office of Information Technology
University of Texas at Arlington
________________________________________________
Ken Hornstein
>term load-balancer. We don't actually want to easy the load off of our
>KDC's, we just want provide a seamless way of ensuring availability in
>the event that we lose one (or more) of them. I think it's true for
>everyone who's commented to this thread that their users are probably
>not aware of how Kerberos works. We're trying to make deployment as easy
>as possible for the end user (migrating a huge deployment from AD to MIT
>Kerberos). Having one host name to remember would certainly help.
Just as an aside ... are you actually expecting end-users to create MIT
krb5.conf files? We don't do that; we just give them a krb5.conf file
to download. If you're going that route, you could simply list all of
the KDCs individually in the krb5.conf file (and put a few extra names
in there as placeholders; names that don't resolve to anything won't
cause a problem there). Given the way database propagation works with
MIT Kerberos, if you plan on doing password changes having a load
balancer in the mix could very likely screw you hard when it comes to
users changing their password (the MIT Kerberos library has a bunch of
logic that will query the "master" KDC when it has talked to a backup
KDC and the password is incorrect, which happens when a password is
changed due to propagation lag; this wouldn't work if the client
library only thought there was one KDC).
--Ken
Frank Cusack
>>>> The load balancer is simply another failure point.
>>>
>>> As is everything else.
>>
>> However load balancers are complicated devices and more prone to
>> failure.
>
> WHOA! - Yes load balancers can be complicated if you want to use all
> the features, but "prone to failure"?? where do you get that from?
Personal experience.
I have worked with every load balancer on the market, I have not seen
a single one, even in HA mode, that hasn't failed completely at one
time or another. Including netscaler, which Jason indicates they are
using. We tend to be the people that tell the LB people what features
to implement (just like large ISPs tell router vendors what they
need); specifically to Netscaler I personally have worked *very*
closely with them on their feature set. I just mention that to make
it clear that it's not a case of immature/inexperienced network
administration.
Since krb5 can work quite effectively WITHOUT a LB, it simply does not
make sense to add the extra failure point (unless you have to for
other reasons like you can't change DNS).
/fc
Frank Cusack
> I guess the problem that everyone is having with our deployment is the
> term load-balancer. We don't actually want to easy the load off of our
Good, because:
> You'll say that DNS is the answer. I would agree. The problem is, we
> don't have access, or support from the folks controlling our DNS, to add
> the SRV records which would simplify the whole mess. Since we already
> have a load balancing switch (NetScaler) around for other purposes,
> we're trying to use it in place of doing things properly. :-)
The netscaler can only do 325k UDP qps, and this is the big netscaler
(9900) without any other load on it. So putting a netscaler in front
of your KDCs actually reduces service capacity.
Availability, as mentioned earlier, is already dealt with quite well
in the libraries. (But really requires the ability to play with DNS.)
/fc
Henry B. Hotz
do its own failover without one.
Kerberos was also originally designed to require FQDN's to uniquely map
to the destination IP numbers. Violations of those assumptions
deserved to fail because they might indicate some attempted crack.
While things have changed a lot, I would not be sanguine about avoiding
all the possible side effects.
I would be concerned independent of any specific problems which have
been identified.
On Oct 6, 2004, at 5:23 PM, kerberos...@mit.edu wrote:
> Well, the answer to this question is complex. We don't think a
> load-balancer will be required for our deployment, but it would
> simplify
> the end-user experience.
I think many of us still don't understand why you say this. Let me
attempt to clarify.
My assumptions are (pretty much what Ken H suggested):
1) The user's get given a standard krb5.conf file with all the (real or
potential) slave kdc's and the master kdc listed. There is no
mentionable overhead for configuring extra kdc entries on all clients
ahead of time.
2) You're using standard Kerberos software like MIT, Heimdal, Sun, or
Microsoft.
3) Your app's use the libraries from 2).
Then all applications will try all the kdc's listed in the krb5.conf
before failing. No load balancer or DNS tricks needed.
The only place you might loose is when the master goes down. Then
admin and password change access fails. But it would fail anyway! (I
presume you weren't going to have a password change get sent to a
random kdc. That would make normal password changes usually fail.)
(I'm not trying to be insulting here, just very, very clear and basic.)
Password change and admin access do not need the same reliability that
normal authentication does. I suspect you could do without for a few
days and hardly feel it.
I get it that DNS changes and "unusual" entries are a problem. Here's
a suggestion: Don't use the load balancer for normal operations (see 1
above). If and only if the Kerberos master fails, then use the load
balancer to redirect traffic to your stand-in master. You'll probably
have to play some games with name resolution on the stand-in to make
everything work, and I recommend you test the contingency plans
carefully.
Does this help?
------------------------------------------------------------------------
----
The opinions expressed in this message are mine,
not those of Caltech, JPL, NASA, or the US Government.
Henry....@jpl.nasa.gov, or hbh...@oxy.edu