kvno X not found in keytab; ticket is likely out of date

Laura Smith

unread,

Jul 22, 2019, 6:23:03 AM7/22/19

to kerb...@mit.edu

Ok, I hold my hand up, I messed up. So the question is, how do I get myself out of this mess ?

A summary of how I got here:
• I have an NFS server and a bunch of clients connecting and auth using krb5.
• This was all working beautifully.... until today.
• Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
• Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
• keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
• that didnt' work. a reboot of the NFS server didn't work.

Summary ? I'm up a smelly creek without a paddle !

Messages on NFS server:

2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/foo.exa...@EXAMPLE.CORP kvno 3 not found in keytab; ticket is likely out of date
2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/foo.exa...@EXAMPLE.CORP kvno 3 not found in keytab; ticket is likely out of date]

John Devitofranceschi

unread,

Jul 22, 2019, 6:47:35 AM7/22/19

to Laura Smith, kerb...@mit.edu

How long has it been since this happened?

I think that the clients will be fine once their old ccaches expire. Have you tried forcing the issue by manually refreshing one of the clients?

Sent from my iPhone

> ________________________________________________
> Kerberos mailing list Kerb...@mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos

Laura Smith

unread,

Jul 22, 2019, 7:01:04 AM7/22/19

to John Devitofranceschi, kerb...@mit.edu

Maybe a couple of hours or so.

"klist -l" shows empty on a client I've tried.

When mounting, the client now shows :
mount.nfs4: access denied by server while mounting (null)
mount: mounting foo.example.com:/srv/share/foo on /mnt/foo failed: Invalid argument

Demsg on the client shows:
NFS: state manager: check lease failed on NFSv4 server foo.example.com with error 13

Rebooting the client has no effect.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Radoslav Bodó

unread,

Jul 22, 2019, 7:23:39 AM7/22/19

to kerb...@mit.edu

I'm not an expert but I'd try:

1) check if the keys for service are in sync in KDB and service keytab.
if client reboot does not help, i'd guess keys are not in proper sync

2) pull old keytab from NFS server backup and merge it with current
keytab

client with not-yet expired tickets should be able to reconnect.
service will use the old key for them, there's no communication with
kdc until it those expires

clients with new creds will use new key

depending on the state of you environment, there might be some other
issues regarding encryption types and problem that old clients cannot
connect to the service which had old enctype, but now it has very new
(due to the key regeneration)

bodik

Dne 07/22/2019 v 01:00 PM Laura Smith napsal(a):

> ________________________________________________

Charles Hedrick

unread,

Jul 22, 2019, 9:14:00 AM7/22/19

to Laura Smith, kerb...@mit.edu

Unfortunately it’s likely to take some experimentation. My starting point would be on each client, unmount the file system, maybe delete /tmp/krb5ccmachine*, restart rpc.gssd, and remount.

> On Jul 22, 2019, at 6:22 AM, Laura Smith <n5d9xq3ti2...@protonmail.ch> wrote:
>
> Ok, I hold my hand up, I messed up. So the question is, how do I get myself out of this mess ?
>
> A summary of how I got here:
> • I have an NFS server and a bunch of clients connecting and auth using krb5.
> • This was all working beautifully.... until today.
> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
> • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
> • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
> • that didnt' work. a reboot of the NFS server didn't work.
>
> Summary ? I'm up a smelly creek without a paddle !
>
> Messages on NFS server:
>
> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/foo.exa...@EXAMPLE.CORP kvno 3 not found in keytab; ticket is likely out of date
> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/foo.exa...@EXAMPLE.CORP kvno 3 not found in keytab; ticket is likely out of date]
>

Laura Smith

unread,

Jul 22, 2019, 9:26:29 AM7/22/19

to Charles Hedrick, kerb...@mit.edu

Hi Charles,

Surely the action of rebooting the client would do all of that ?

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Charles Hedrick

unread,

Jul 22, 2019, 9:34:13 AM7/22/19

to Laura Smith, kerb...@mit.edu

Probably. If I interpret your email, you recreated the key table for the server. I assume you either rebooted the server or restarted everything relevant (most critical would be rpc.svcgssd).

I agree that rebooting clients would probably do it on the client side, except that not all systems are set up to clear /tmp on reboot. I don’t think that’s critical, but I can’t guarantee it.

Radoslav Bodó

unread,

Jul 22, 2019, 10:42:25 AM7/22/19

to kerb...@mit.edu

I'm definitely not an expert on the field, but I'd guess you'd have to:

1) wait until client tickets expires and clients requests new ones for
current kvno

2) due to linux NFS credential storage burried deep in the kernel,
reboot all clients (sometimes just restarting services helps,
sometimes does not ;(

3) anyway the best would be to pull old key from backups (either from
kdc or server backup) and put it back to KDC under correct kvno

depending on your skills and other factors of your environment,
restoring whole KDC db might be easier than to mess with single entry ...

bodik

Dne 07/22/2019 v 12:22 PM Laura Smith napsal(a):

Radoslav Bodó

unread,

Jul 22, 2019, 10:42:27 AM7/22/19

to kerb...@mit.edu

> 3) anyway the best would be to pull old key from backups (either from
> kdc or server backup) and put it back to KDC under correct kvno
>
> depending on your skills and other factors of your environment,
> restoring whole KDC db might be easier than to mess with single entry ...

btw, just putting old key to the service keytab on NFS server might do
the trick most easily...

the clients still holding the not-yet expired tickes would be able to
access the service, because service would have both old and new keys
available ... there should be no need to manage the kdc i guess

b

ps: typing faster than thinking ;(