kerberos credential cache filename with sshd causing problems for long running jobs

[Jason Keltz]

Hi.

I have a question about Kerberos credential caches with sshd and long
running compute jobs.

If a user logs into our central SSH server, a kerberos credential cache
is created in /tmp/krb5cc_<uid> as would be expected.

A user can run kinit -R before their 10 hour ticket expiry, up until the
7 day hard expiry.  It all works the way I would expect.

However, users don't run their compute jobs on the central ssh server. 
They ssh from there to a variety of compute servers. This is where the
problem occurs.

When the user SSHes to the compute server, the SSHd stores the
credential cache at /tmp/krb5cc_<uid>_<randomchars> instead of just
/tmp/krb5cc_<uid>.

As far as I can tell, there's no option to tell SSHd to just use
/tmp/krb5cc_<uid>.

This creates various problems:

If, after sshing to the compute server, the user starts their compute
job in the background, then logs out, even if their ticket would still
have been valid for 10 hours, they soon thereafter lose access to their
Kerberos NFS home directory because sshd deletes the credential cache
when they log out.

I could potentially resolve this problem by configuring SSHD to *not*
clean the credential cache on logout.  However, that creates a new
problem because now the credential cache files will stay around forever
(unless I clean them up manually).

If the user wants to say, run a cron/at job that will just run "kinit
-R" periodically on the compute server to extend their ticket up to 7
days, this won't work because kinit -R is going to renew the cache in
/tmp/krb5cc_<uid>, and not /tmp/krb5cc_<uid>_<randomchars> unless the
user explicitly specifies the cache file location (which will be
different for each ssh session they start).

In addition, if the user wants to run a really long job, and that job
will run for longer than 7 days, I can instruct the user to create a
keytab file, then pass that to kinit periodically via cron, but I'll
have the same problem -- they would have to specify the full path to
their credential cache because kinit would otherwise assume the default
location of /tmp/krb5cc_<uid>.

It all seems very complicated.

I assume that the reason that SSHd creates the sshd credential cache in
/tmp/krb5cc_<uid>_<randomchars> is so that an ssh session will not share
the same credential cache with say, a local workstation login.  I could
imagine a case where you log out of the local workstation, thereby
destroying your credential cache, but this stops a job running on the
same workstation via ssh session which is using the same credential cache.

Let's assume that the user won't be logging into the local workstation
and will only connect via SSH.  Would it be reasonable for me to
manually copy /tmp/krb5cc_<uid>_<randomchars> to /tmp/krb5cc_<uid> when
required, then change KRB5CCNAME to point to /tmp/krb5cc_<uid> instead
of /tmp/krb5cc_<uid>_<randomchars> so that things just work? This way,
sshd can delete it's cache as required on logout, and the user can
continue to easily run their compute job (albeit being careful about
local workstation login versus remote ssh login to the same machine). 
I'm surprised such an option doesn't exist in SSH to tell the daemon not
to insert the randomchars, but I may be missing the point which is why
I'm asking for clarification.

I know there are other mechanisms for credential cache.  In my case,
those won't work on my current installation.

Any thoughts? Thanks for any information you can provide...

Jason.

[James Ralston]

On Wed, Jul 7, 2021 at 8:20 PM Jason Keltz <j...@eecs.yorku.ca> wrote:

> I assume that the reason that SSHd creates the sshd credential cache
> in /tmp/krb5cc_<uid>_<randomchars> is so that an ssh session will
> not share the same credential cache with say, a local workstation
> login.

The reason why sshd creates the Kerberos file credential cache with
mkstemp() is to avoid race condition attacks, where an attacker
leverages the fact that a user's credential cache file is in a
predictable location to either execute a denial-of-service attack or
steal the victim's credentials.

This is also why there is no option to disable it.

> In addition, if the user wants to run a really long job, and that
> job will run for longer than 7 days, I can instruct the user to
> create a keytab file, then pass that to kinit periodically via cron,
> but I'll have the same problem -- they would have to specify the
> full path to their credential cache because kinit would otherwise
> assume the default location of /tmp/krb5cc_<uid>.
>

> Let's assume that the user won't be logging into the local
> workstation and will only connect via SSH. Would it be reasonable
> for me to manually copy /tmp/krb5cc_<uid>_<randomchars> to
> /tmp/krb5cc_<uid> when required, then change KRB5CCNAME to point to
> /tmp/krb5cc_<uid> instead of /tmp/krb5cc_<uid>_<randomchars> so that
> things just work? This way, sshd can delete it's cache as required
> on logout, and the user can continue to easily run their compute job
> (albeit being careful about local workstation login versus remote
> ssh login to the same machine).

Running long jobs that need Kerberos credentials is really a problem
for k5start(1). Is there a reason why you can't use it?

https://www.eyrie.org/~eagle/software/kstart/k5start.html

> I know there are other mechanisms for credential cache. In my case,
> those won't work on my current installation.

If you can't use k5start, your effort would probably be best spent in
finding a way to make other credential cache types (e.g.,
KEYRING:persistent: or KCM:) usable on your installation, as the
issues you are encountering with the FILE: cache type are inherent
limitations.