Logic behind lib/krb5/os/k5_sendto()

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Logic behind lib/krb5/os/k5_sendto()

Дилян Палаузов
Hello,

how does lib/krb5/os/k5_sendto() work in this case:

kinit [hidden email] is called and in dns the SRV records _kerberos._udp.example.org and _kerberos._tcp.example.org show
that k.example.org:88 is in charge.  But the KDC there is in fact in charge of EXAMPLE.ORG, example.org being non-local
realm.

• If k5_sendto receives an answer from a KDC, that the realm is non-local, does it retry to the other KDCs, here asking
the same process over a different transport protocol?

When is the timeout (interval) increased from 1s to to 2s?  If there is no answer within 1s, then the query is resent,
waiting for 2 more seconds?

Thanks for the clarification.
  Дилян

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Logic behind lib/krb5/os/k5_sendto()

Greg Hudson
On 4/15/19 5:48 PM, Дилян Палаузов wrote:
> kinit [hidden email] is called and in dns the SRV records _kerberos._udp.example.org and _kerberos._tcp.example.org show
> that k.example.org:88 is in charge.  But the KDC there is in fact in charge of EXAMPLE.ORG, example.org being non-local
> realm.
>
> • If k5_sendto receives an answer from a KDC, that the realm is non-local, does it retry to the other KDCs, here asking
> the same process over a different transport protocol?

If example.org issues a client referral (KDC_ERR_WRONG_REALM) to
EXAMPLE.ORG, k5_sendto() will return the error response, and the
higher-level logic will (if canonicalization is enabled) retry with
EXAMPLE.ORG, which will contact the same KDC.

> When is the timeout (interval) increased from 1s to to 2s?  If there is no answer within 1s, then the query is resent,
> waiting for 2 more seconds?

If there is a single KDC with both UDP and TCP transports, the normal
schedule is:

1. Send UDP request, wait one second
2. Begin TCP connection, wait one second
3. Wait two seconds
4. Send UDP request, wait one second
5. Wait four seconds
6. Send UDP request, wait one second
7. Wait eight seconds
8. Send UDP request, wait one second
9. Wait sixteen seconds
10. Give up

If there are multiple KDCs, the steps to send UDP requests or begin TCP
connections are iterated over each KDC, with a one-second wait after each.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Logic behind lib/krb5/os/k5_sendto()

Дилян Палаузов
Hello Greg,

thanks for your answer.  The comments in the source file say:

Restarted:
  Per UDP server, 1s
  Per TCP server, 1s

Does this mean, that the TCP connection is also retried more than once?  You wrote, that there is a single try to open a
TCP connection.

My setup is:
• There are services running (smtp, imap, webdav) where users authenticate over SASL,
• Users do not call kinit and the domain of their hosts has no relation to the realm.
• Within the server, the authentication towards the services runs over cyrus-sasl/saslauthd, which performs PAM.
• PAM means first try Kerberos using pam_krb5-2.4.12, and then LDAP.  These are two different databases and a user is in
only one of them.
• In almost all cases the answer from saslauthd is, that Kerberos does not know the server, so the authentication
continues with PAM/LDAP.

Sometimes saslauthd gets stuck.  Then some thousand service processes are waiting for saslauthd to reply, the load
average of the system increases and then of course everything gets very slow.

All saslauthd processes wait for reply in lib/krb5/os/sendto_kdc.c:k5_sendto().  

I guess krb5kdc is slow in replying.  I have removed on the meantime the logging, which should speed up the replies from
krb5kdc.

But I think resending the queries in this case to krb5kdc makes think worse, because the krb5kdc will have to deal then
with even  more (repeated) queries, and this slows everything down, when it is already slow, compared to a case, where
queries are not retried.

What do you think?

Regards
  Дилян

On Tue, 2019-04-16 at 10:39 -0400, Greg Hudson wrote:

> On 4/15/19 5:48 PM, Дилян Палаузов wrote:
> > kinit [hidden email] is called and in dns the SRV records _kerberos._udp.example.org and _kerberos._tcp.example.org show
> > that k.example.org:88 is in charge.  But the KDC there is in fact in charge of EXAMPLE.ORG, example.org being non-local
> > realm.
> >
> > • If k5_sendto receives an answer from a KDC, that the realm is non-local, does it retry to the other KDCs, here asking
> > the same process over a different transport protocol?
>
> If example.org issues a client referral (KDC_ERR_WRONG_REALM) to
> EXAMPLE.ORG, k5_sendto() will return the error response, and the
> higher-level logic will (if canonicalization is enabled) retry with
> EXAMPLE.ORG, which will contact the same KDC.
>
> > When is the timeout (interval) increased from 1s to to 2s?  If there is no answer within 1s, then the query is resent,
> > waiting for 2 more seconds?
>
> If there is a single KDC with both UDP and TCP transports, the normal
> schedule is:
>
> 1. Send UDP request, wait one second
> 2. Begin TCP connection, wait one second
> 3. Wait two seconds
> 4. Send UDP request, wait one second
> 5. Wait four seconds
> 6. Send UDP request, wait one second
> 7. Wait eight seconds
> 8. Send UDP request, wait one second
> 9. Wait sixteen seconds
> 10. Give up
>
> If there are multiple KDCs, the steps to send UDP requests or begin TCP
> connections are iterated over each KDC, with a one-second wait after each.

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Logic behind lib/krb5/os/k5_sendto()

Greg Hudson
On 4/18/19 2:25 PM, Дилян Палаузов wrote:> Does this mean, that the TCP
connection is also retried more than once?  You wrote, that there is a
single try to open a
> TCP connection.

Only a single non-blocking TCP socket is opened per KDC, but that socket
remains open for the whole duration, and the kernel will retry the TCP
connection on its own schedule.

> But I think resending the queries in this case to krb5kdc makes think worse, because the krb5kdc will have to deal then
> with even  more (repeated) queries, and this slows everything down, when it is already slow, compared to a case, where
> queries are not retried.

Possibly, but the client has no way to distinguish between a UDP packet
getting lost and the KDC being slow to respond.

The KDC does have a lookaside cache which records the responses to
recent requests, so a retransmitted request should be processed with
less effort than processing the original request.

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Logic behind lib/krb5/os/k5_sendto()

Дилян Палаузов
Hello Greg,

> If example.org issues a client referral (KDC_ERR_WRONG_REALM) to
> EXAMPLE.ORG, k5_sendto() will return the error response, and the
> higher-level logic will (if canonicalization is enabled) retry with
> EXAMPLE.ORG, which will contact the same KDC.

Does krb5kdc return KDC_ERR_WRONG_REALM?

Does canonicalizaiton only work if the host where kinit is called has the right dns-domain (so no canonicalization
happens, if host ab.cd.ef.gh calls “kinit [hidden email]”?

> The KDC does have a lookaside cache which records the responses to
> recent requests, so a retransmitted request should be processed with
> less effort than processing the original request.

Does the cache also store error answers, like answers about non existing users and answers about NON-LOCAL realms?

Regards
  Дилян

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Logic behind lib/krb5/os/k5_sendto()

Greg Hudson
On 4/18/19 5:08 PM, Дилян Палаузов wrote:
> Does krb5kdc return KDC_ERR_WRONG_REALM?

The MIT KDC only returns KDC_ERR_WRONG_REALM if it looks up the client
principal and gets a realm referral from the database.  This typically
requires a third-party database module like Samba or FreeIPA.

> Does canonicalizaiton only work if the host where kinit is called has the right dns-domain (so no canonicalization
> happens, if host ab.cd.ef.gh calls “kinit [hidden email]”?

The client hostname doesn't normally have an impact on AS requests.

> Does the cache also store error answers, like answers about non existing users and answers about NON-LOCAL realms?

Yes; it just maps request packets to reply packets, so any kind of reply
packet is cached.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Logic behind lib/krb5/os/k5_sendto()

Дилян Палаузов
Hello Greg,

thank for your answers.  On Monday I asked, if k5_sendto receives an answer from a KDC, that the realm is non-local,
does it retry to the other KDCs, here asking the same process over a different transport protocol.  

You answered, that on a client referral (KDC_ERR_WRONG_REALM) answer, k5_sendto() will return the error response, and
the higher-level logic will (if canonicalization is enabled) retry with the uppercased domain, which will contact the
same KDC.

If kdb5kdc determines that the realm is non local and no canonicalization is done or referrals are issued, does the
client / k5_sendto() retry asking other KDCs, in my case asking the same process over a different transport (UPD→TCP)?

Regards
  Дилян


On Thu, 2019-04-18 at 17:48 -0400, Greg Hudson wrote:

> On 4/18/19 5:08 PM, Дилян Палаузов wrote:
> > Does krb5kdc return KDC_ERR_WRONG_REALM?
>
> The MIT KDC only returns KDC_ERR_WRONG_REALM if it looks up the client
> principal and gets a realm referral from the database.  This typically
> requires a third-party database module like Samba or FreeIPA.
>
> > Does canonicalizaiton only work if the host where kinit is called has the right dns-domain (so no canonicalization
> > happens, if host ab.cd.ef.gh calls “kinit [hidden email]”?
>
> The client hostname doesn't normally have an impact on AS requests.
>
> > Does the cache also store error answers, like answers about non existing users and answers about NON-LOCAL realms?
>
> Yes; it just maps request packets to reply packets, so any kind of reply
> packet is cached.

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Logic behind lib/krb5/os/k5_sendto()

Greg Hudson
On 4/19/19 4:49 AM, Дилян Палаузов wrote:
> thank for your answers.  On Monday I asked, if k5_sendto receives an answer from a KDC, that the realm is non-local,
> does it retry to the other KDCs, here asking the same process over a different transport protocol.

Your question was about a specific function in the client code and you
didn't specify the KDC implementation.  I took "the realm is non-local"
to mean that the KDC had some knowledge of the correct realm as a
foreign realm.  Sorry for the miscommunication.

In the scenario where the client uses the wrong realm case (so the realm
lookup succeeds in DNS due to case-insensitivity there), with all MIT
krb5 components, a typical result is the following:

  $ kinit [hidden email]
  kinit: Client '[hidden email]' not found in Kerberos database
while getting initial credentials

Here the KDC issues a KDC_ERR_C_PRINCIPAL_UNKNOWN error (because it
looks up [hidden email] in its database and does not find it)
and the client does not retry.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev