How often does MIT krb5 request for KDC info through DNS?

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

How often does MIT krb5 request for KDC info through DNS?

Weijun Wang
Hi

KDC info can be retrieved from a DNS server but how often does MIT krb5 request for it? I grabbed some packets and it seems there are 6 rounds of requests within 3 minutes. The DNS server I am querying returns answers with TTL of 10 minutes so it looks like not honored.

I tried to read the source codes but haven't spotted a cache or something similar.

Thanks
Max


_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Nico Williams
On Mon, Aug 04, 2014 at 01:28:31PM +0800, Wang Weijun wrote:
> KDC info can be retrieved from a DNS server but how often does MIT
> krb5 request for it? I grabbed some packets and it seems there are 6
> rounds of requests within 3 minutes. The DNS server I am querying
> returns answers with TTL of 10 minutes so it looks like not honored.

The Kerberos library isn't a DNS resolver; it uses one.

Therefore the Kerberos library should ask often, possibly even every
time it does a KDC request.

You should configure your system to have a caching resolver on
127.0.0.1.

> I tried to read the source codes but haven't spotted a cache or
> something similar.

Ideally there should be no cache for DNS results in the library.

Some things should be cached, like: the local host's FQDN (it shouldn't
change, right?), default realm (if not set and it had to be determined
from context, e.g., the user's or host's realm), and so on.  But not DNS
lookups -- that's the resolver's job.  If your resolver is not a caching
resolver, then fix it :)

Nico
--
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Weijun Wang
I wonder if it's easy to set up such a service. Here we are talking
about the client side, which might be just a browser talking HTTP with
"Windows Integrated Authentication".

--Max

On 8/5/2014 1:29, Nico Williams wrote:

> On Mon, Aug 04, 2014 at 01:28:31PM +0800, Wang Weijun wrote:
>> KDC info can be retrieved from a DNS server but how often does MIT
>> krb5 request for it? I grabbed some packets and it seems there are 6
>> rounds of requests within 3 minutes. The DNS server I am querying
>> returns answers with TTL of 10 minutes so it looks like not honored.
>
> The Kerberos library isn't a DNS resolver; it uses one.
>
> Therefore the Kerberos library should ask often, possibly even every
> time it does a KDC request.
>
> You should configure your system to have a caching resolver on
> 127.0.0.1.
>
>> I tried to read the source codes but haven't spotted a cache or
>> something similar.
>
> Ideally there should be no cache for DNS results in the library.
>
> Some things should be cached, like: the local host's FQDN (it shouldn't
> change, right?), default realm (if not set and it had to be determined
> from context, e.g., the user's or host's realm), and so on.  But not DNS
> lookups -- that's the resolver's job.  If your resolver is not a caching
> resolver, then fix it :)
>
> Nico
>
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

David Woodhouse-7
In reply to this post by Nico Williams
On Mon, 2014-08-04 at 12:29 -0500, Nico Williams wrote:
>
> Some things should be cached, like: the local host's FQDN (it shouldn't
> change, right?), default realm (if not set and it had to be determined
> from context, e.g., the user's or host's realm), and so on.  But not DNS
> lookups -- that's the resolver's job.  If your resolver is not a caching
> resolver, then fix it :)

I'm not sure I agree with that.

I've watched firefox lock up for *minutes* at a time without redrawing
itself, and I've found that it's stuck in Kerberos code mostly doing the
same Legacy IP and IPv6 DNS lookups for the same set of 30-odd domain
controllers, over and over and over and over and over again.

Yes, I deployed a local caching nameserver to help with that (and
samba-winbind-krb5-locator, and now I'm playing with negative caching on
KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN...). But I shouldn't have *had* to.

Some level of "our KDC was <here> two seconds ago. Perhaps I could just
manage to talk to it again without going out on the wire to ask the DNS
server again" might be appropriate.

The latency was particularly painful in my case because the DNS lookups
were done over a VPN. Which of course made setting up the local caching
resolver relatively painful too, since it has to cope with VPN and
non-VPN mode...

Thankfully, NetworkManager under Linux copes with this fairly well these
days, but in the general case it's not a trivial thing that you're
suggesting.

--
dwmw2

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Greg Hudson
On 08/05/2014 07:12 AM, David Woodhouse wrote:
> I've watched firefox lock up for *minutes* at a time without redrawing
> itself, and I've found that it's stuck in Kerberos code mostly doing the
> same Legacy IP and IPv6 DNS lookups for the same set of 30-odd domain
> controllers, over and over and over and over and over again.
>
> Yes, I deployed a local caching nameserver to help with that (and
> samba-winbind-krb5-locator, and now I'm playing with negative caching on
> KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN...). But I shouldn't have *had* to.

System administrators shouldn't have to, but platforms should.  From a
software engineering perspective, it's much better if the platform
provides DNS caching than if every application does its own getaddrinfo
caching.  It's also better from a behavior perspective, because
applications don't have easy access to DNS TTL information, while the
platform does.

That said, if the popular platforms aren't interested in providing this
service, at some point applications have to step in and solve the
problem even if it's not optimal.  We might add some amount of DNS
caching in libkrb5 at some point (with a very low internal TTL), though
it isn't super high on the priority list.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Brandon Allbery
On Tue, 2014-08-05 at 10:19 -0400, Greg Hudson wrote:
> That said, if the popular platforms aren't interested in providing
> this
> service, at some point applications have to step in and solve the
> problem even if it's not optimal.  We might add some amount of DNS
> caching in libkrb5 at some point (with a very low internal TTL),
> though
> it isn't super high on the priority list.

Browsers do this these days. And balancing faster performance due to
local caching against correct operation took them a while. It's
something of a mess; it really does not belong in the application, as
you noted.

--
brandon s allbery kf8nh                           sine nomine associates
[hidden email]                              [hidden email]
unix openafs kerberos infrastructure xmonad        http://sinenomine.net

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Nico Williams
In reply to this post by David Woodhouse-7
On Tue, Aug 05, 2014 at 12:12:06PM +0100, David Woodhouse wrote:
> On Mon, 2014-08-04 at 12:29 -0500, Nico Williams wrote:
> > Some things should be cached, like: the local host's FQDN (it shouldn't
> > change, right?), default realm (if not set and it had to be determined
> > from context, e.g., the user's or host's realm), and so on.  But not DNS
> > lookups -- that's the resolver's job.  If your resolver is not a caching
> > resolver, then fix it :)
>
> I'm not sure I agree with that.

Once you start using DNSSEC you may really want this.

> I've watched firefox lock up for *minutes* at a time without redrawing
> itself, and I've found that it's stuck in Kerberos code mostly doing
> the same Legacy IP and IPv6 DNS lookups for the same set of 30-odd
> domain controllers, over and over and over and over and over again.

Browsers run for a long time.  Kerberos libraries often don't.  An
in-memory DNS cache often won't help (though it might not hurt
performance, but then, it will be a source of bugs).

The FILE ccache as a general-purpose cache is a performance disaster too
(because the FILE ccache is a performance disaster, full stop), so
that's out.

> Yes, I deployed a local caching nameserver to help with that (and
> samba-winbind-krb5-locator, and now I'm playing with negative caching on
> KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN...). But I shouldn't have *had* to.

No, you shouldn't have had to.  That should have been cached in the
ccache.  (I know, I'm contradicting myself, but we already use the
ccache as the cache for the successful case...)

> Some level of "our KDC was <here> two seconds ago. Perhaps I could just
> manage to talk to it again without going out on the wire to ask the DNS
> server again" might be appropriate.

Where should this be written?

> The latency was particularly painful in my case because the DNS lookups
> were done over a VPN. Which of course made setting up the local caching
> resolver relatively painful too, since it has to cope with VPN and
> non-VPN mode...

My recommendation is to run a zone/jail/VM/whatever-your-OS-calls-it for
one of the two things (VPN vs. direct Internet access), with the
security conscious running direct Internet access in a zone/jail/VM/...

Nico
--
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Simo Sorce-3
In reply to this post by Greg Hudson
On Tue, 2014-08-05 at 10:19 -0400, Greg Hudson wrote:

> On 08/05/2014 07:12 AM, David Woodhouse wrote:
> > I've watched firefox lock up for *minutes* at a time without redrawing
> > itself, and I've found that it's stuck in Kerberos code mostly doing the
> > same Legacy IP and IPv6 DNS lookups for the same set of 30-odd domain
> > controllers, over and over and over and over and over again.
> >
> > Yes, I deployed a local caching nameserver to help with that (and
> > samba-winbind-krb5-locator, and now I'm playing with negative caching on
> > KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN...). But I shouldn't have *had* to.
>
> System administrators shouldn't have to, but platforms should.  From a
> software engineering perspective, it's much better if the platform
> provides DNS caching than if every application does its own getaddrinfo
> caching.  It's also better from a behavior perspective, because
> applications don't have easy access to DNS TTL information, while the
> platform does.
>
> That said, if the popular platforms aren't interested in providing this
> service, at some point applications have to step in and solve the
> problem even if it's not optimal.  We might add some amount of DNS
> caching in libkrb5 at some point (with a very low internal TTL), though
> it isn't super high on the priority list.

In Fedora, at least, we are planning on providing a caching resolver by
default soon.

Simo.

--
Simo Sorce * Red Hat, Inc * New York

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Nico Williams
In reply to this post by Weijun Wang
On Tue, Aug 05, 2014 at 03:38:28PM +0800, Weijun Wang wrote:
> I wonder if it's easy to set up such a service. Here we are talking
> about the client side, which might be just a browser talking HTTP
> with "Windows Integrated Authentication".

Modern/decent OSes just have it, at least as an option.  You'll have to
read the docs.

As for JGSS performance, there are worse problems:

 - non-caching of some tickets

 - delegating credentials by default in the HTTP/Negotiate stack
   (forwarded tickets are generally not cached on the client side)

 - doing an HTTP request w/o authentication every time, thus getting a
   401 then trying again with Kerberos

 - servlets that don't use cookies to optimize away the GSS context
   setup per-request(!!!)

These things will kill performance worse than any lack of DNS caching.

DNS caching is most noticeable when you have connectivity or DNS server
stability issues, because the resolvers tend to have very long timeouts
and because a lot of apps do synchronous DNS lookups, so that hanging in
a DNS lookup is extremely noticeable to the user.  This isn't really
DNS' fault though, but the OS/library/app architecture's.

I'd much rather that Kerberos libraries used async DNS APIs than that
they implement a resolver cache!

Nico
--
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

David Woodhouse-7
On Tue, 2014-08-05 at 10:53 -0500, Nico Williams wrote:

>
> As for JGSS performance, there are worse problems:
>
>  - non-caching of some tickets
>
>  - delegating credentials by default in the HTTP/Negotiate stack
>    (forwarded tickets are generally not cached on the client side)
>
>  - doing an HTTP request w/o authentication every time, thus getting a
>    401 then trying again with Kerberos
>
>  - servlets that don't use cookies to optimize away the GSS context
>    setup per-request(!!!)
 - On IIS, failing to set the 'AuthPersistNonNTLM' attribute which makes
   Kerberos authentication a per-connection thing instead of per-request

http://blogs.msdn.com/b/benjaminperkins/archive/2011/10/31/kerberos-authpersistnonntlm-authentication-request-based-vs-session-based-authentication.aspx

--
dwmw2

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Nico Williams
On Tue, Aug 05, 2014 at 05:03:40PM +0100, David Woodhouse wrote:
> On Tue, 2014-08-05 at 10:53 -0500, Nico Williams wrote:
> > As for JGSS performance, there are worse problems:
> >
> >  - servlets that don't use cookies to optimize away the GSS context
> >    setup per-request(!!!)
>
>  - On IIS, failing to set the 'AuthPersistNonNTLM' attribute which makes
>    Kerberos authentication a per-connection thing instead of per-request

HTTP/1.1 is not supposed to be aware of connection state, and IIRC the
servlet interface design doesn't make it possible to make the servlet
able to cache per-connection state :(

Cookies are teh authentication state system for HTTP, for better or
worse, whether we like it or not.

Java needs an utterly trivial-to-setup session cookie system.

Nico
--
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Russ Allbery-2
In reply to this post by Brandon Allbery
Brandon Allbery <[hidden email]> writes:

> Browsers do this these days. And balancing faster performance due to
> local caching against correct operation took them a while.

In fact, it's still mostly broken, as most of that caching doesn't honor
TTLs.

--
Russ Allbery ([hidden email])              <http://www.eyrie.org/~eagle/>

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Spike_White
In reply to this post by Weijun Wang
Doesn't "name service caching"  via nscd solve this?

Setting up a caching-only DNS server is pretty simple.

Changing 1-2 lines in /etc/nscd.conf and chkconfig/starting the nscd service is even easier.

Spike

Message: 3
Date: Tue, 05 Aug 2014 12:12:06 +0100
From: David Woodhouse
Subject: Re: How often does MIT krb5 request for KDC info through DNS?
To: Nico Williams
Cc: [hidden email]
Message-ID:
Content-Type: text/plain; charset="utf-8"

On Mon, 2014-08-04 at 12:29 -0500, Nico Williams wrote:
>
> Some things should be cached, like: the local host's FQDN (it
> shouldn't change, right?), default realm (if not set and it had to be
> determined from context, e.g., the user's or host's realm), and so on.
> But not DNS lookups -- that's the resolver's job. If your resolver is
> not a caching resolver, then fix it :)

I'm not sure I agree with that.

I've watched firefox lock up for *minutes* at a time without redrawing itself, and I've found that it's stuck in Kerberos code mostly doing the same Legacy IP and IPv6 DNS lookups for the same set of 30-odd domain controllers, over and over and over and over and over again.

Yes, I deployed a local caching nameserver to help with that (and samba-winbind-krb5-locator, and now I'm playing with negative caching on KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN...). But I shouldn't have *had* to.

Some level of "our KDC was two seconds ago. Perhaps I could just manage to talk to it again without going out on the wire to ask the DNS server again" might be appropriate.

The latency was particularly painful in my case because the DNS lookups were done over a VPN. Which of course made setting up the local caching resolver relatively painful too, since it has to cope with VPN and non-VPN mode...

Thankfully, NetworkManager under Linux copes with this fairly well these days, but in the general case it's not a trivial thing that you're suggesting.

--
dwmw2
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Brandon Allbery
On Tue, 2014-08-05 at 14:50 -0500, [hidden email] wrote:
> Doesn't "name service caching"  via nscd solve this?

When it works. It doesn't work often enough that nscd has been going
away in many cases, replaced by DNS-specific caching services like
dnsmasq or unbound.

--
brandon s allbery kf8nh                           sine nomine associates
[hidden email]                              [hidden email]
unix openafs kerberos infrastructure xmonad        http://sinenomine.net

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Nico Williams
In reply to this post by Spike_White
On Tue, Aug 5, 2014 at 2:50 PM,  <[hidden email]> wrote:
> Doesn't "name service caching"  via nscd solve this?

nscd is specifically about Unix name services, lookups in the
hosts(4), passwd(4), ... DBs.

We're talking about DNS SRV RR lookups though; nscd does nothing about those.

> Setting up a caching-only DNS server is pretty simple.

Yes.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Tom Yu
Nico Williams <[hidden email]> writes:

> On Tue, Aug 5, 2014 at 2:50 PM,  <[hidden email]> wrote:
>> Doesn't "name service caching"  via nscd solve this?
>
> nscd is specifically about Unix name services, lookups in the
> hosts(4), passwd(4), ... DBs.
>
> We're talking about DNS SRV RR lookups though; nscd does nothing about those.

I thought the A and AAAA lookups for the addresses of 30+ KDCs were what
were at issue here, unless I'm misunderstanding something.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Nico Williams
On Tue, Aug 5, 2014 at 3:17 PM, Tom Yu <[hidden email]> wrote:

> Nico Williams <[hidden email]> writes:
>
>> On Tue, Aug 5, 2014 at 2:50 PM,  <[hidden email]> wrote:
>>> Doesn't "name service caching"  via nscd solve this?
>>
>> nscd is specifically about Unix name services, lookups in the
>> hosts(4), passwd(4), ... DBs.
>>
>> We're talking about DNS SRV RR lookups though; nscd does nothing about those.
>
> I thought the A and AAAA lookups for the addresses of 30+ KDCs were what
> were at issue here, unless I'm misunderstanding something.

Ah, those would get cached by nscd, yes.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Weijun Wang
In reply to this post by Nico Williams


On 8/5/2014 23:53, Nico Williams wrote:

> On Tue, Aug 05, 2014 at 03:38:28PM +0800, Weijun Wang wrote:
>> I wonder if it's easy to set up such a service. Here we are talking
>> about the client side, which might be just a browser talking HTTP
>> with "Windows Integrated Authentication".
>
> Modern/decent OSes just have it, at least as an option.  You'll have to
> read the docs.
>
> As for JGSS performance, there are worse problems:
>
>   - non-caching of some tickets

I agree.

>
>   - delegating credentials by default in the HTTP/Negotiate stack
>     (forwarded tickets are generally not cached on the client side)

Now that Java has constrained delegation, will re-consider this.

>
>   - doing an HTTP request w/o authentication every time, thus getting a
>     401 then trying again with Kerberos

I'll ask the networking team.

>
>   - servlets that don't use cookies to optimize away the GSS context
>     setup per-request(!!!)

I'll forward this to people knowing about servlets.

Thanks
Max
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: How often does MIT krb5 request for KDC info through DNS?

Weijun Wang
In reply to this post by Nico Williams


On 8/5/2014 23:53, Nico Williams wrote:
>   - doing an HTTP request w/o authentication every time, thus getting a
>     401 then trying again with Kerberos

Recalculating an auth token for each request is a little heavy. The
client is hoping the server would stop prompting for authentication
after the 1st request. I remember seeing server actually doing that.
Maybe you mean because the point below a Java server does not do that?

>
>   - servlets that don't use cookies to optimize away the GSS context
>     setup per-request(!!!)

--Max
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev