Expired tickets not renewed

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Expired tickets not renewed

Victor Sudakov
Dear Colleauges,

I have a perfectly valid TGT and some expired service tickets (see the
klist output below). When I ssh to techno2 or fs01, the ssh client
asks me for the password. Why does it happen?

I expect that the ssh client should request a new service ticket for
the relevant host, and not just give up?


Credentials cache: FILE:/tmp/krb5cc_3001
        Principal: [hidden email]

  Issued                Expires               Principal
Aug  7 08:27:14 2017  Aug 14 08:27:14 2017  krbtgt/[hidden email]
Aug  7 08:27:16 2017  >>>Expired<<<         host/[hidden email]
[...]
Aug  7 11:01:56 2017  >>>Expired<<<         host/[hidden email]
Aug  7 11:01:56 2017  >>>Expired<<<         host/[hidden email]
Aug  7 12:58:54 2017  >>>Expired<<<         host/[hidden email]
Aug  7 12:58:54 2017  >>>Expired<<<         host/[hidden email]


--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Harald Barth-2

Against what gssapi library is your ssh linked and what does ssh -vvv
reveal why gssapi does not proceed?

Harald.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Victor Sudakov
Harald Barth wrote:
>
> Against what gssapi library is your ssh linked

Heimdal 1.5.2 from the FreeBSD 10.3 base system.

> and what does ssh -vvv
> reveal why gssapi does not proceed?

Next time a service ticket expires, I'll post it here. But don't hold
your breath, it's probably going to be something stupid like
'miscellaneous failure, see text'

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Victor Sudakov
Victor Sudakov wrote:

> > Against what gssapi library is your ssh linked
>
> Heimdal 1.5.2 from the FreeBSD 10.3 base system.
>
> > and what does ssh -vvv
> > reveal why gssapi does not proceed?
>
> Next time a service ticket expires, I'll post it here. But don't hold
> your breath, it's probably going to be something stupid like
> 'miscellaneous failure, see text'

Dear Harald,

Below is what "ssh -vvv" reveals:

debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-with-mic,keyboard-interactive
debug3: start over, passed a different list publickey,gssapi-with-mic,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1:  The context has expired
Success

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/sudakov/.ssh/id_rsa

Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
a new ticket is received and gssapi-with-mic is again successful until
the new tickets expires again.

I'm beginning to think of a cron job which would destroy hourly all
the service tickets... all except the TGT.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Harald Barth-2

> debug1: Next authentication method: gssapi-with-mic
> debug1:  The context has expired

That looks to me like a bug where the library actually should try to
get a new service ticket from the TGT. I don't know if that works in
any heimdal libkrb as most often (at least in my use case) the TGT and
the service ticket have the same lifetime. I only use this scenario
when I use GSSAPI IMAP and there the application is linked against MIT
IIRC.

> Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
> a new ticket is received and gssapi-with-mic is again successful until
> the new tickets expires again.

Yes, then you create the same situation as with a freshly aquired TGT.

> I'm beginning to think of a cron job which would destroy hourly all
> the service tickets... all except the TGT.

Does the same problem happen with kgetcred host/techno.... instead of
ssh?

Do you have time to test this in newer heimdal?

Ugly workaround: A wrapper script could destroy all expired service
tickets.

Btw, one of my ticket caches looks like this (probably MIT library):

  Issued                Expires               Principal
Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  krbtgt/[hidden email]
Aug  5 18:06:50 2017  >>>Expired<<<         imap/[hidden email]
Aug  6 18:35:34 2017  >>>Expired<<<         imap/[hidden email]
Aug  8 09:08:14 2017  >>>Expired<<<         imap/[hidden email]
Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  imap/[hidden email]

There is no reason that the expired service tickets are kept around,
but in this case, the code seems to keep them and append new tickets
at the end instead.

Harald.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Viktor Dukhovni-2
On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:

> Btw, one of my ticket caches looks like this (probably MIT library):
>
>   Issued                Expires               Principal
> Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  krbtgt/[hidden email]
> Aug  5 18:06:50 2017  >>>Expired<<<         imap/[hidden email]
> Aug  6 18:35:34 2017  >>>Expired<<<         imap/[hidden email]
> Aug  8 09:08:14 2017  >>>Expired<<<         imap/[hidden email]
> Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  imap/[hidden email]
>
> There is no reason that the expired service tickets are kept around,
> but in this case, the code seems to keep them and append new tickets
> at the end instead.

By design, the "FILE" credential cache type is *append-only*.  Much
fancier read-write locking and recovery would be needed for a cache
where entries can be deleted and replaced.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Nico Williams
On Wed, Aug 09, 2017 at 06:01:26PM +0000, Viktor Dukhovni wrote:

> On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:
>
> > Btw, one of my ticket caches looks like this (probably MIT library):
> >
> >   Issued                Expires               Principal
> > Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  krbtgt/[hidden email]
> > Aug  5 18:06:50 2017  >>>Expired<<<         imap/[hidden email]
> > Aug  6 18:35:34 2017  >>>Expired<<<         imap/[hidden email]
> > Aug  8 09:08:14 2017  >>>Expired<<<         imap/[hidden email]
> > Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  imap/[hidden email]
> >
> > There is no reason that the expired service tickets are kept around,
> > but in this case, the code seems to keep them and append new tickets
> > at the end instead.
>
> By design, the "FILE" credential cache type is *append-only*.  Much
> fancier read-write locking and recovery would be needed for a cache
> where entries can be deleted and replaced.

Actually, no, the FILE ccache does support deletion, certainly in
Heimdal 7.x.

Nico
--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Roland C. Dowdeswell-2
On Wed, Aug 09, 2017 at 01:11:07PM -0500, Nico Williams wrote:
>

> Actually, no, the FILE ccache does support deletion, certainly in
> Heimdal 7.x.

Well, we can invalidate entries but I don't think that we can re-use
the slots because of locking issues.

--
    Roland C. Dowdeswell
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Viktor Dukhovni-2
In reply to this post by Nico Williams
On Wed, Aug 09, 2017 at 01:11:07PM -0500, Nico Williams wrote:

> On Wed, Aug 09, 2017 at 06:01:26PM +0000, Viktor Dukhovni wrote:
> > On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:
> >
> > > Btw, one of my ticket caches looks like this (probably MIT library):
> > >
> > >   Issued                Expires               Principal
> > > Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  krbtgt/[hidden email]
> > > Aug  5 18:06:50 2017  >>>Expired<<<         imap/[hidden email]
> > > Aug  6 18:35:34 2017  >>>Expired<<<         imap/[hidden email]
> > > Aug  8 09:08:14 2017  >>>Expired<<<         imap/[hidden email]
> > > Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  imap/[hidden email]
> > >
> > > There is no reason that the expired service tickets are kept around,
> > > but in this case, the code seems to keep them and append new tickets
> > > at the end instead.
> >
> > By design, the "FILE" credential cache type is *append-only*.  Much
> > fancier read-write locking and recovery would be needed for a cache
> > where entries can be deleted and replaced.
>
> Actually, no, the FILE ccache does support deletion, certainly in
> Heimdal 7.x.

That's in-place invalidation, not deletion.  I was talking about
deletion.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Nico Williams
In reply to this post by Roland C. Dowdeswell-2
On Wed, Aug 09, 2017 at 02:25:11PM -0400, Roland C. Dowdeswell wrote:
> On Wed, Aug 09, 2017 at 01:11:07PM -0500, Nico Williams wrote:
> > Actually, no, the FILE ccache does support deletion, certainly in
> > Heimdal 7.x.
>
> Well, we can invalidate entries but I don't think that we can re-use
> the slots because of locking issues.

We do not try to re-use them.

We could try to, but it'd be tricky and I'd rather not.

First, if a new entry is shorter than a candidate deleted entry then
we'd have to pad it.  That's easy enough, and if we couldn't do it we
wouldn't do it and that's that.

Second, if a write is interrupted (e.g., by SIGKILL) and we get a
partial write, then we'd have a corrupted entry.  Because we no longer
lock around reading, that would be bad.  We can recover from truncated
tail entries, but not from garbage entries in the middle.

The ccache entry deletion code might seem similarly unsafe, but it turns
out to be safe anyways if any part of it completes because of what it
does and the order in which things are written.

Nico
--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Nico Williams
In reply to this post by Viktor Dukhovni-2
On Wed, Aug 09, 2017 at 06:34:27PM +0000, Viktor Dukhovni wrote:

> On Wed, Aug 09, 2017 at 01:11:07PM -0500, Nico Williams wrote:
> > On Wed, Aug 09, 2017 at 06:01:26PM +0000, Viktor Dukhovni wrote:
> > > On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:
> > >
> > > > Btw, one of my ticket caches looks like this (probably MIT library):
> > > >
> > > >   Issued                Expires               Principal
> > > > Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  krbtgt/[hidden email]
> > > > Aug  5 18:06:50 2017  >>>Expired<<<         imap/[hidden email]
> > > > Aug  6 18:35:34 2017  >>>Expired<<<         imap/[hidden email]
> > > > Aug  8 09:08:14 2017  >>>Expired<<<         imap/[hidden email]
> > > > Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  imap/[hidden email]
> > > >
> > > > There is no reason that the expired service tickets are kept around,
> > > > but in this case, the code seems to keep them and append new tickets
> > > > at the end instead.
> > >
> > > By design, the "FILE" credential cache type is *append-only*.  Much
> > > fancier read-write locking and recovery would be needed for a cache
> > > where entries can be deleted and replaced.
> >
> > Actually, no, the FILE ccache does support deletion, certainly in
> > Heimdal 7.x.
>
> That's in-place invalidation, not deletion.  I was talking about
> deletion.

Well, OK, but I'd call it logical deletion (I don't recall if we made
klist hide such entries, but if not we should).

Logical deletion is sufficient when it comes to making sure that we
fetch a new service ticket.  Actually, it's not even necessary, and I do
believe the library in 7.x will fetch a new ticket if there's only an
expired one in the ccache and an unexpired TGT is available.

Shifting subsequent entries over to *really* delete the old entry is
simply not possible with the new locking regimen.

Reusing logically-deleted slots is doable but tricky.

If the issue is ccache size, then one can always just reinitialize.  And
we should definitely consider doing that in krb5_get_credentials() and
friends to keep the ccache to a reasonable size.

We do need to re-think re-initialization in the new locking regimen --
re-init via truncation probably works well enough right now, but mostly
by accident.

Nico
--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Nico Williams
On Wed, Aug 09, 2017 at 01:44:56PM -0500, Nico Williams wrote:
> We do need to re-think re-initialization in the new locking regimen --
> re-init via truncation probably works well enough right now, but mostly
> by accident.

Ah, right, we never do that.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Jeffrey Altman-2
In reply to this post by Viktor Dukhovni-2
On 8/9/2017 2:01 PM, Viktor Dukhovni wrote:

> On Wed, Aug 09, 2017 at 07:34:15PM +0200, Harald Barth wrote:
>
>> Btw, one of my ticket caches looks like this (probably MIT library):
>>
>>   Issued                Expires               Principal
>> Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  krbtgt/[hidden email]
>> Aug  5 18:06:50 2017  >>>Expired<<<         imap/[hidden email]
>> Aug  6 18:35:34 2017  >>>Expired<<<         imap/[hidden email]
>> Aug  8 09:08:14 2017  >>>Expired<<<         imap/[hidden email]
>> Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  imap/[hidden email]
>>
>> There is no reason that the expired service tickets are kept around,
>> but in this case, the code seems to keep them and append new tickets
>> at the end instead.
>
> By design, the "FILE" credential cache type is *append-only*.  Much
> fancier read-write locking and recovery would be needed for a cache
> where entries can be deleted and replaced.
>
I hope this is an unnecessary question, but will all Kerberos libraries
that parse the file cache know to skip the expired entries and keep
searching?  Or are there implementations that will only return the first
service principal match?




smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Roland C. Dowdeswell-2
In reply to this post by Victor Sudakov
On Wed, Aug 09, 2017 at 09:58:04PM +0700, Victor Sudakov wrote:
>

> Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
> a new ticket is received and gssapi-with-mic is again successful until
> the new tickets expires again.
>
> I'm beginning to think of a cron job which would destroy hourly all
> the service tickets... all except the TGT.

To bring the conversation back a little to the original point:

It appears that Heimdal 1.5 had incorrect behaviour.  The ccache code
should skip expired credentials when finding service tickets.  This looks
like it was fixed by the following commit:

        commit 0f1ae2d10186afb654df8f50cc78663eb53f27a9
        Author: Nicolas Williams <[hidden email]>
        Date:   Fri Aug 2 18:55:36 2013 -0500

            Use KRB5_TC_MATCH_TIMES when looking for creds

So, if you upgrade, this issue will be resolved.

--
    Roland C. Dowdeswell
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Nico Williams
In reply to this post by Jeffrey Altman-2
On Wed, Aug 09, 2017 at 03:01:16PM -0400, Jeffrey Altman wrote:
> I hope this is an unnecessary question, but will all Kerberos libraries
> that parse the file cache know to skip the expired entries and keep
> searching?  Or are there implementations that will only return the first
> service principal match?

The krb5 API used, krb5_cc_retrieve_cred(), supports this going back a
long time in MIT, and probably in Heimdal, but you have to ask for this
by including KRB5_TC_MATCH_TIMES in the options flags argument.

Nico
--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Nico Williams
In reply to this post by Roland C. Dowdeswell-2
On Wed, Aug 09, 2017 at 03:06:37PM -0400, Roland C. Dowdeswell wrote:

> It appears that Heimdal 1.5 had incorrect behaviour.  The ccache code
> should skip expired credentials when finding service tickets.  This looks
> like it was fixed by the following commit:
>
> commit 0f1ae2d10186afb654df8f50cc78663eb53f27a9
> Author: Nicolas Williams <[hidden email]>
> Date:   Fri Aug 2 18:55:36 2013 -0500
>
>    Use KRB5_TC_MATCH_TIMES when looking for creds
>
> So, if you upgrade, this issue will be resolved.

Ahh, I should have recalled that I'd fixed that...  Thanks for finding
that.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Victor Sudakov
In reply to this post by Harald Barth-2
Harald Barth wrote:
>
> > debug1: Next authentication method: gssapi-with-mic
> > debug1:  The context has expired
>
> That looks to me like a bug where the library actually should try to
> get a new service ticket from the TGT. I don't know if that works in
> any heimdal libkrb as most often (at least in my use case) the TGT and
> the service ticket have the same lifetime.

How do you achieve that? My service principals have "Max ticket life"
property set to "1 week" by default, but all the krbtgt principals
have "Max ticket life=unlimited" (AFAIK it was set by "kadmin init"
and I never changed that).

>
> > Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
> > a new ticket is received and gssapi-with-mic is again successful until
> > the new tickets expires again.
>
> Yes, then you create the same situation as with a freshly aquired TGT.
>
> > I'm beginning to think of a cron job which would destroy hourly all
> > the service tickets... all except the TGT.
>
> Does the same problem happen with kgetcred host/techno.... instead of
> ssh?

That's funny. Each "kgetcred host/techno..." adds yet another
">>>Expired<<<" entry to the cache!

>
> Do you have time to test this in newer heimdal?

Well, I could install 7.4.0 from the ports collection, but my ssh and
sshd will remain linked to 1.5.2, right?

>
> Ugly workaround: A wrapper script could destroy all expired service
> tickets.

Yes, something like that in the crontab should perhaps do the job:
apply -d 'kdestroy --credential=%1' `klist | awk '/Expired/{print $6}' | sort -u`

>
> Btw, one of my ticket caches looks like this (probably MIT library):
>
>   Issued                Expires               Principal
> Aug  5 18:06:47 2017  Aug 12 18:06:45 2017  krbtgt/[hidden email]
> Aug  5 18:06:50 2017  >>>Expired<<<         imap/[hidden email]
> Aug  6 18:35:34 2017  >>>Expired<<<         imap/[hidden email]
> Aug  8 09:08:14 2017  >>>Expired<<<         imap/[hidden email]
> Aug  9 09:12:16 2017  Aug 10 09:12:16 2017  imap/[hidden email]
>
> There is no reason that the expired service tickets are kept around,
> but in this case, the code seems to keep them and append new tickets
> at the end instead.

IMHO this is fine unless the expired tickets kept in the cache don't
prevent from getting new ones.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Victor Sudakov
In reply to this post by Roland C. Dowdeswell-2
Roland C. Dowdeswell wrote:

>
> > Now if I destroy the expired ticket by "kdestroy --credential=host/techno..."
> > a new ticket is received and gssapi-with-mic is again successful until
> > the new tickets expires again.
> >
> > I'm beginning to think of a cron job which would destroy hourly all
> > the service tickets... all except the TGT.
>
> To bring the conversation back a little to the original point:
>
> It appears that Heimdal 1.5 had incorrect behaviour.  The ccache code
> should skip expired credentials when finding service tickets.  This looks
> like it was fixed by the following commit:
>
> commit 0f1ae2d10186afb654df8f50cc78663eb53f27a9
> Author: Nicolas Williams <[hidden email]>
> Date:   Fri Aug 2 18:55:36 2013 -0500
>
>    Use KRB5_TC_MATCH_TIMES when looking for creds
>
> So, if you upgrade, this issue will be resolved.

That's good news. I could install the current Heimdal 7.4.0 from the
FreeBSD ports collection, however, there were two major problems
upgrading when I tried last time:

1. The 7.x kdc did not understand the heimdal.db Kerberos database
created by 1.5.2. Are they not compatible? What should I know about
this?

2. The utilities in the FreeBSD base system will remain linked to the
base system Heimdal libs (/usr/lib/libgssapi* instead of the newer
/usr/local/lib/libgssapi*).


--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Nico Williams
On Thu, Aug 10, 2017 at 09:24:08AM +0700, Victor Sudakov wrote:
> 1. The 7.x kdc did not understand the heimdal.db Kerberos database
> created by 1.5.2. Are they not compatible? What should I know about
> this?

Looking at differences in lib/hdb/hdb.asn1... they should be compatible.
Is it possible that in FreeBSD 7.4 uses a different DB type?

Can you dump/load?  That should work.

> 2. The utilities in the FreeBSD base system will remain linked to the
> base system Heimdal libs (/usr/lib/libgssapi* instead of the newer
> /usr/local/lib/libgssapi*).

FreeBSD hasn't upgraded yet?  I thought it had.

Nico
--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Expired tickets not renewed

Benjamin Kaduk-2
On Wed, Aug 09, 2017 at 09:34:32PM -0500, Nico Williams wrote:
>
> FreeBSD hasn't upgraded yet?  I thought it had.

Nobody was even willing to think about it until there was an official
release to upgrade to.  And now one of the likely suspects to do that
work is investigating an MIT krb5 import instead of doing the upgrade...

-Ben
12
Loading...