k5start -K and ticket renewals

[Russ Allbery]

Hello all,

I'm considering a mildly backwards-incompatible change to k5start and
wanted to ask here, since this is probably the best forum I have for
reaching k5start users.

Currently, when one runs k5start as a daemon with the -K option, the
argument to -K controls how frequently k5start will wake up and check
whether its ticket is expiring. However, it won't always renew the ticket
when it wakes up. It will only do so if the ticket will expire before or
within two minutes of the next time it wakes up.

This poses a couple of problems:

* It's difficult, using this approach, to guarantee a minimum ticket
lifetime at any time. In other words, if you want the cache renewed
such that the ticket will always be valid for at least an hour at any
given time, it's complex to construct the right lifetime and wakeup time
to do this.

* When using k5start in conjunction with AFS and the -t flag, new tokens
will be acquired only when new tickets are acquired. This means that,
if the AFS tokens might go away before the tickets for some reason (such
as if the AFS principal has a maximum ticket lifetime shorter than the
krbtgt principal), it may be difficult to maintain AFS tokens.

It's also sort of weird and complex, and people struggle to understand it.

I'm therefore considering changing the next release to always acquire
fresh tickets each time k5start wakes up. So if you run k5start -K 10,
then k5start will wake up every ten minutes and acquire new tickets
unconditionally, regardless of whether the current tickets are about to
expire.

I would make the similar change to krenew -K at the same time.

I think this would be more straightforward, would prevent the above
issues, and would mean that I wouldn't have to merge various patches
people have sent me to work around this or configure this in other ways.
The only drawback I can think of is that it may mean somewhat more
Kerberos KDC traffic, since I suspect a lot of people have set -K values
to be fairly short, but the minimum time is one minute anyway. An
authentication every minute isn't a huge amount, and people can adjust
their -K arguments after this release.

Does anyone think this is a bad idea? Am I missing any problem with this?

--
Russ Allbery (ea...@eyrie.org) <http://www.eyrie.org/~eagle/>

[steve]

On Wed, 2014-01-15 at 18:51 -0800, Russ Allbery wrote:

>
> It's also sort of weird and complex, and people struggle to understand it.
>
> I'm therefore considering changing the next release to always acquire
> fresh tickets each time k5start wakes up. So if you run k5start -K 10,
> then k5start will wake up every ten minutes and acquire new tickets
> unconditionally, regardless of whether the current tickets are about to
> expire.

Hi
From: The Campaign To Save -K (European branch)

Please remember that you're not always dealing with admins with mit
education;) Here, we're just simple folk who have to keep a domain
alive. Every day. We are scared to death of Kerberos. -K works an
absolute treat. We understand it. Could you please retain it? If you
must make a change then could it be an addition? We'd suggest -J. If you
already have -J then just choose another letter. Easy.

Cheers,
Steve

[Russ Allbery]

I'm fairly sure that you won't notice any difference at all except that it
will be easier to understand. I think this change will actually make -K
*better* for people who don't care about what's going on under the hood
and just need an active ticket cache.

If I'm missing something, though, please do let me know.

[steve]

Only the human aspect of it. If you are saying that our -K now will work
the same as your new -K then perfect. But you aren't. You can only give
us 'fairly sure'. My boss doesn't do 'fairly sure'. Please don't fix
something that isn't broken. If you really must do soething to k5start,
leave -K as it is and do -someOtherLetter as an additional switch.
Cheers

[Russ Allbery]

steve <st...@steve-ss.com> writes:

> Only the human aspect of it. If you are saying that our -K now will work
> the same as your new -K then perfect. But you aren't. You can only give
> us 'fairly sure'. My boss doesn't do 'fairly sure'. Please don't fix
> something that isn't broken. If you really must do soething to k5start,
> leave -K as it is and do -someOtherLetter as an additional switch.

I believe the only difference from the perspective of an existing user is
that k5start will do more Kerberos authentications. It would surprise me
a great deal if you would even notice, since Kerberos authentications are
exceptionally light-weight.

[Ken Dreyer]

On Wed, Jan 15, 2014 at 7:51 PM, Russ Allbery <ea...@eyrie.org> wrote:
> I think this would be more straightforward, would prevent the above
> issues, and would mean that I wouldn't have to merge various patches
> people have sent me to work around this or configure this in other ways.
> The only drawback I can think of is that it may mean somewhat more
> Kerberos KDC traffic, since I suspect a lot of people have set -K values
> to be fairly short, but the minimum time is one minute anyway. An
> authentication every minute isn't a huge amount, and people can adjust
> their -K arguments after this release.
>
> Does anyone think this is a bad idea? Am I missing any problem with this?

For what it's worth, I checked what we're using at work to
authenticate our Apache systems, and it's "-K 30", so I don't
anticipate that such a change would noticeably impact us.

- Ken

[Nico Williams]

Ideally the auto-renewal wake-up timer should be automatically set
from the TGT's lifetime (and libkrb5 should automatically handle any
faster expiration of non-initial tickets). Then -K shouldn't be
needed.

The hard part is how to handle transient renewal errors, particularly
when the ticket's original lifetime was short (but renew lifetime
long).

Nico
--

[Russ Allbery]

Nico Williams <ni...@cryptonector.com> writes:

> Ideally the auto-renewal wake-up timer should be automatically set from
> the TGT's lifetime (and libkrb5 should automatically handle any faster
> expiration of non-initial tickets). Then -K shouldn't be needed.

Well, -K is how one says to run as a daemon at all, so you still need the
flag, but possibly not its value.

> The hard part is how to handle transient renewal errors, particularly
> when the ticket's original lifetime was short (but renew lifetime
> long).

If acquiring credentials fails, k5start (and krenew) try every minute
until they succeed again regardless of the -K value.

[moritz....@ubs.com]

I like this much better than -K implying to constantly fetch new
tickets. On one host it may be ok to change the -K behaviour; but if you
are running k5start on thousands or ten thousands of hosts, the
multiplying factor cannot be neglected. It may also be very intentional
to only refresh the ticket once a day but check regularly that it didn't
get lost by accident.

If the behaviour is changing and k5start refresh the ticket more
regularly, then the updating of the CC must always be atomic. If I
remember correctly, this is right now only the case if -o, -g or -m are
specified.

- mo

-----Original Message-----
From: kerberos...@mit.edu [mailto:kerberos...@mit.edu] On
Behalf Of Nico Williams
Sent: 17 January 2014 00:02
To: Russ Allbery
Cc: kerb...@mit.edu
Subject: Re: k5start -K and ticket renewals

Ideally the auto-renewal wake-up timer should be automatically set
from the TGT's lifetime (and libkrb5 should automatically handle any
faster expiration of non-initial tickets). Then -K shouldn't be
needed.

The hard part is how to handle transient renewal errors, particularly
when the ticket's original lifetime was short (but renew lifetime
long).

Nico
--
________________________________________________
Kerberos mailing list Kerb...@mit.edu
https://mailman.mit.edu/mailman/listinfo/kerberos
Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mails are not encrypted and cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The sender
therefore does not accept liability for any errors or omissions in the
contents of this message which arise as a result of e-mail transmission.
If verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities
or related financial instruments.

UBS Limited is authorised by the Prudential Regulation Authority
and regulated by the Financial Conduct Authority and the Prudential
Regulation Authority.

UBS AG is a public company incorporated with limited liability in
Switzerland domiciled in the Canton of Basel-City and the Canton of
Zurich respectively registered at the Commercial Registry offices in
those Cantons with Identification No: CH-270.3.004.646-4 and having
respective head offices at Aeschenvorstadt 1, 4051 Basel and
Bahnhofstrasse 45, 8001 Zurich, Switzerland and is authorised and
regulated by the Financial Market Supervisory Authority in
Switzerland. Registered in the United Kingdom as a foreign company
with No: FC021146 and having a UK Establishment registered at
Companies House, Cardiff, with No: BR 004507. The principal office
of UK Establishment: 1 Finsbury Avenue, London EC2M 2PP. In the
United Kingdom, UBS AG is authorised by the Prudential Regulation
Authority and subject to regulation by the Financial Conduct
Authority and limited regulation by the Prudential Regulation
Authority. Details about the extent of our regulation by the
Prudential Regulation Authority are available from us on request.

UBS reserves the right to retain all messages. Messages are protected
and accessed only in legally justified cases.

[Nico Williams]

On Tue, Jan 28, 2014 at 5:10 AM, <moritz....@ubs.com> wrote:
> If the behaviour is changing and k5start refresh the ticket more
> regularly, then the updating of the CC must always be atomic. If I
> remember correctly, this is right now only the case if -o, -g or -m are
> specified.

As to atomicity... the FILE ccache currently depends on POSIX file
locking at least for additions of tickets, and this is a disaster
because POSIX file locking is a disaster (because of its drop locks on
first close semantics). But yes, *renewal* and refresh should always
result in a rename(2) into place, which should be atomic.

Nico
--

[Russ Allbery]

By that do you mean that the Kerberos libraries do that, or that I need to
do that in k5start?

I assume krenew is fine (to the extent that POSIX locking is fine).

[Nico Williams]

On Tue, Jan 28, 2014 at 12:42 PM, Russ Allbery <ea...@eyrie.org> wrote:
> Nico Williams <ni...@cryptonector.com> writes:
>> As to atomicity... the FILE ccache currently depends on POSIX file
>> locking at least for additions of tickets, and this is a disaster
>> because POSIX file locking is a disaster (because of its drop locks on
>> first close semantics). But yes, *renewal* and refresh should always
>> result in a rename(2) into place, which should be atomic.
>
> By that do you mean that the Kerberos libraries do that, or that I need to
> do that in k5start?

I mean that rename-into-place is what the krb5 libraries ought to do.
It's not what MIT does, MIT truncates to initialize. Heimdal does
rename in the krb5_cc_move() case, but only in that case, otherwise
Heimdal unlink()s then creates a new ccache to initialize.

> I assume krenew is fine (to the extent that POSIX locking is fine).

Sort of.

POSIX file locking within a single-threaded program can be sufficient,
and it is when that program is truncating and writing a new ccache
under lock. But you might be racing against threaded programs, and
since the libraries (MIT and Heimdal both) misuse POSIX file locks...

POSIX file locking is utterly useless in a threaded program unless one
also applies other synchronization within that process with great care
to never close(2) a file descriptor for a regular file that another
thread might also have independently opened and locked (see what
SQLite3 does), and even then there are situations where that's
insufficient (namely, when multiple libraries in the same process
might be accessing the same file).

In particular it's possible for a threaded app and krenew to race.
Imagine a thread A in an app acquiring a read lock to read the ccache
and another thread B blocking to get a write lock to add a service
ticket (it's already read the ccache and not found that ticket)...

The thread A finishes its search, drops the lock, then B gets the
lock, then A closes its file descriptor thus dropping B's lock so that
krenew can get its write lock and start writing. End result: B can
write over krenew's entries.

Most likely B won't write over krenew's entries because most likely B
will have found the end of the ccache farther off than where krenew
would leave it. Instead B will likely cause a hole between the end of
krenew's entries and B's, but if krenew were to obtain a *larger*
ticket than before, then B could write over krenew's writes.
Regardless, threaded ccache reader-writers can step on each other, so
krenew really plays no special role here.

We've observed ccache corruption that can only (I think) be explained
by POSIX file locking issues in threaded programs.

The only solution is to always rename into place. See my separate
posts to the list about this.

Nico
--

[Russ Allbery]

<moritz....@ubs.com> writes:

> I like this much better than -K implying to constantly fetch new
> tickets. On one host it may be ok to change the -K behaviour; but if you
> are running k5start on thousands or ten thousands of hosts, the
> multiplying factor cannot be neglected. It may also be very intentional
> to only refresh the ticket once a day but check regularly that it didn't
> get lost by accident.

Okay, I think I'm hearing enough opposition to the plan to just change -K
that I'm going to go with the previous plan of adding a new -a option that
says to renew the ticket each time it wakes up. I'm also going to support
using -H with -K, where -H is the minimum ticket lifetime and the renewal
decision will be based on maintaining at least that minimum ticket
lifetime.

> If the behaviour is changing and k5start refresh the ticket more
> regularly, then the updating of the CC must always be atomic. If I
> remember correctly, this is right now only the case if -o, -g or -m are
> specified.

I think this is true regardless, and I'm tentatively planning on changing
k5start to always obtain tickets in a new ticket cache file and then
rename it over top of the existing ticket cache in the next release.