kvno X not found in keytab; ticket is likely out of date

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

kvno X not found in keytab; ticket is likely out of date

Laura Smith
Ok, I hold my hand up, I messed up.  So the question is, how do I get myself out of this mess ?

A summary of how I got here:
• I have an NFS server and a bunch of clients connecting and auth using krb5.
• This was all working beautifully.... until today.
• Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
• Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
• keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
• that didnt' work.  a reboot of the NFS server didn't work.

Summary ?  I'm up a smelly creek without a paddle !

Messages on NFS server:

2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

Radoslav Bodó
I'm definitely not an expert on the field, but I'd guess you'd have to:


1) wait until client tickets expires and clients requests new ones for
   current kvno


2) due to linux NFS credential storage burried deep in the kernel,
   reboot all clients (sometimes just restarting services helps,
   sometimes does not ;(


3) anyway the best would be to pull old key from backups (either from
kdc or server backup) and put it back to KDC under correct kvno

depending on your skills and other factors of your environment,
restoring whole KDC db might be easier than to mess with single entry ...


bodik


Dne 07/22/2019 v 12:22 PM Laura Smith napsal(a):

> Ok, I hold my hand up, I messed up.  So the question is, how do I get myself out of this mess ?
>
> A summary of how I got here:
> • I have an NFS server and a bunch of clients connecting and auth using krb5.
> • This was all working beautifully.... until today.
> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
> • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
> • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
> • that didnt' work.  a reboot of the NFS server didn't work.
>
> Summary ?  I'm up a smelly creek without a paddle !
>
> Messages on NFS server:
>
> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

John Devitofranceschi
In reply to this post by Laura Smith
How long has it been since this happened?

I think that the clients will be fine once their old ccaches expire.  Have you tried forcing the issue by manually refreshing one of the clients?

Sent from my iPhone

> On Jul 22, 2019, at 06:22, Laura Smith <[hidden email]> wrote:
>
> Ok, I hold my hand up, I messed up.  So the question is, how do I get myself out of this mess ?
>
> A summary of how I got here:
> • I have an NFS server and a bunch of clients connecting and auth using krb5.
> • This was all working beautifully.... until today.
> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
> • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
> • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
> • that didnt' work.  a reboot of the NFS server didn't work.
>
> Summary ?  I'm up a smelly creek without a paddle !
>
> Messages on NFS server:
>
> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]
>
> ________________________________________________
> Kerberos mailing list           [hidden email]
> https://mailman.mit.edu/mailman/listinfo/kerberos

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos

smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

Radoslav Bodó
In reply to this post by Radoslav Bodó
> 3) anyway the best would be to pull old key from backups (either from
> kdc or server backup) and put it back to KDC under correct kvno
>
> depending on your skills and other factors of your environment,
> restoring whole KDC db might be easier than to mess with single entry ...

btw, just putting old key to the service keytab on NFS server might do
the trick most easily...

the clients still holding the not-yet expired tickes would be able to
access the service, because service would have both old and new keys
available ... there should be no need to manage the kdc i guess


b

ps: typing faster than thinking ;(
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

Laura Smith
In reply to this post by John Devitofranceschi
Maybe a couple of hours or so.

"klist -l" shows empty on a client I've tried.

When mounting, the client now shows :
mount.nfs4: access denied by server while mounting (null)
mount: mounting foo.example.com:/srv/share/foo on /mnt/foo failed: Invalid argument

Demsg on the client shows:
NFS: state manager: check lease failed on NFSv4 server foo.example.com with error 13

Rebooting the client has no effect.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, July 22, 2019 11:47 AM, John Devitofranceschi <[hidden email]> wrote:

> How long has it been since this happened?
>
> I think that the clients will be fine once their old ccaches expire. Have you tried forcing the issue by manually refreshing one of the clients?
>
> Sent from my iPhone
>
> > On Jul 22, 2019, at 06:22, Laura Smith [hidden email] wrote:
> > Ok, I hold my hand up, I messed up. So the question is, how do I get myself out of this mess ?
> > A summary of how I got here:
> > • I have an NFS server and a bunch of clients connecting and auth using krb5.
> > • This was all working beautifully.... until today.
> > • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
> > • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
> > • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
> > • that didnt' work. a reboot of the NFS server didn't work.
> > Summary ? I'm up a smelly creek without a paddle !
> > Messages on NFS server:
> > 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
> > 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]
> >
> > Kerberos mailing list [hidden email]
> > https://mailman.mit.edu/mailman/listinfo/kerberos



________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

Radoslav Bodó
I'm not an expert but I'd try:


1) check if the keys for service are in sync in KDB and service keytab.
   if client reboot does not help, i'd guess keys are not in proper sync


2) pull old keytab from NFS server backup and merge it with current
   keytab

   client with not-yet expired tickets should be able to reconnect.
   service will use the old key for them, there's no communication with
   kdc until it those expires

   clients with new creds will use new key


depending on the state of you environment, there might be some other
issues regarding encryption types and problem that old clients cannot
connect to the service which had old enctype, but now it has very new
(due to the key regeneration)


bodik


Dne 07/22/2019 v 01:00 PM Laura Smith napsal(a):

> Maybe a couple of hours or so.
>
> "klist -l" shows empty on a client I've tried.
>
> When mounting, the client now shows :
> mount.nfs4: access denied by server while mounting (null)
> mount: mounting foo.example.com:/srv/share/foo on /mnt/foo failed: Invalid argument
>
> Demsg on the client shows:
> NFS: state manager: check lease failed on NFSv4 server foo.example.com with error 13
>
> Rebooting the client has no effect.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, July 22, 2019 11:47 AM, John Devitofranceschi <[hidden email]> wrote:
>
>> How long has it been since this happened?
>>
>> I think that the clients will be fine once their old ccaches expire. Have you tried forcing the issue by manually refreshing one of the clients?
>>
>> Sent from my iPhone
>>
>>> On Jul 22, 2019, at 06:22, Laura Smith [hidden email] wrote:
>>> Ok, I hold my hand up, I messed up. So the question is, how do I get myself out of this mess ?
>>> A summary of how I got here:
>>> • I have an NFS server and a bunch of clients connecting and auth using krb5.
>>> • This was all working beautifully.... until today.
>>> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
>>> • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
>>> • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
>>> • that didnt' work. a reboot of the NFS server didn't work.
>>> Summary ? I'm up a smelly creek without a paddle !
>>> Messages on NFS server:
>>> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
>>> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]
>>>
>>> Kerberos mailing list [hidden email]
>>> https://mailman.mit.edu/mailman/listinfo/kerberos
>
>
>
> ________________________________________________
> Kerberos mailing list           [hidden email]
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

Charles Hedrick
In reply to this post by Laura Smith
Unfortunately it’s likely to take some experimentation. My starting point would be on each client, unmount the file system, maybe delete /tmp/krb5ccmachine*, restart rpc.gssd, and remount.

> On Jul 22, 2019, at 6:22 AM, Laura Smith <[hidden email]> wrote:
>
> Ok, I hold my hand up, I messed up.  So the question is, how do I get myself out of this mess ?
>
> A summary of how I got here:
> • I have an NFS server and a bunch of clients connecting and auth using krb5.
> • This was all working beautifully.... until today.
> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
> • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
> • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
> • that didnt' work.  a reboot of the NFS server didn't work.
>
> Summary ?  I'm up a smelly creek without a paddle !
>
> Messages on NFS server:
>
> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]
>
> ________________________________________________
> Kerberos mailing list           [hidden email]
> https://mailman.mit.edu/mailman/listinfo/kerberos


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

Laura Smith
Hi Charles,

Surely the action of rebooting the client would do all of that ?


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, July 22, 2019 2:13 PM, Charles Hedrick <[hidden email]> wrote:

> Unfortunately it’s likely to take some experimentation. My starting point would be on each client, unmount the file system, maybe delete /tmp/krb5ccmachine*, restart rpc.gssd, and remount.
>
> > On Jul 22, 2019, at 6:22 AM, Laura Smith [hidden email] wrote:
> > Ok, I hold my hand up, I messed up. So the question is, how do I get myself out of this mess ?
> > A summary of how I got here:
> > • I have an NFS server and a bunch of clients connecting and auth using krb5.
> > • This was all working beautifully.... until today.
> > • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
> > • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
> > • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
> > • that didnt' work. a reboot of the NFS server didn't work.
> > Summary ? I'm up a smelly creek without a paddle !
> > Messages on NFS server:
> > 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
> > 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]
> >
> > Kerberos mailing list [hidden email]
> > https://mailman.mit.edu/mailman/listinfo/kerberos



________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kvno X not found in keytab; ticket is likely out of date

Charles Hedrick
Probably. If I interpret your email, you recreated the key table for the server. I assume you either rebooted the server or restarted everything relevant (most critical would be rpc.svcgssd).

I agree that rebooting clients would probably do it on the client side, except that not all systems are set up to clear /tmp on reboot. I don’t think that’s critical, but I can’t guarantee it.

> On Jul 22, 2019, at 9:26 AM, Laura Smith <[hidden email]> wrote:
>
> Hi Charles,
>
> Surely the action of rebooting the client would do all of that ?
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, July 22, 2019 2:13 PM, Charles Hedrick <[hidden email]> wrote:
>
>> Unfortunately it’s likely to take some experimentation. My starting point would be on each client, unmount the file system, maybe delete /tmp/krb5ccmachine*, restart rpc.gssd, and remount.
>>
>>> On Jul 22, 2019, at 6:22 AM, Laura Smith [hidden email] wrote:
>>> Ok, I hold my hand up, I messed up. So the question is, how do I get myself out of this mess ?
>>> A summary of how I got here:
>>> • I have an NFS server and a bunch of clients connecting and auth using krb5.
>>> • This was all working beautifully.... until today.
>>> • Through an act of pure fat-fingered stupidity, I ran "addprinc -randkey nfs/name.of.nfs.server" when setting up a new NFS client (i.e used server name instead of client name).
>>> • Now everything is broken (none of the NFS clients can connect to the server and I am seeing the error messages below on the NFS server).
>>> • keytab on NFS server only had credentials for NFS server, so I deleted the keytab and created a new one through ktadd
>>> • that didnt' work. a reboot of the NFS server didn't work.
>>> Summary ? I'm up a smelly creek without a paddle !
>>> Messages on NFS server:
>>> 2019-07-22T11:01:35.075247+01:00 foo rpc.svcgssd[847]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date
>>> 2019-07-22T11:01:39.460944+01:00 foo rpc.svcgssd[847]: message repeated 41 times: [ ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Request ticket server nfs/[hidden email] kvno 3 not found in keytab; ticket is likely out of date]
>>>
>>> Kerberos mailing list [hidden email]
>>> https://mailman.mit.edu/mailman/listinfo/kerberos
>
>


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos