Weird cross-realm behaviour after upgrade to Heimdal 7.3

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird cross-realm behaviour after upgrade to Heimdal 7.3

Andreas Haupt-2
Dear all,

we face a weird cross-realm-related issue after the upgrade to Heimdal 7.3
KDCs. The KDC replies with a wrong answer in case the cross-realm key does
not exist. This happens with a Heimdal 1.2.1 KDC:

[wgs03] ~ % ssh -v -o GSSAPIAuthentication=yes lxplus.cern.ch
[...]
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Server not found in Kerberos database

... and on the KDC side:

Jul  4 08:26:17 kdc-1.2 kdc[13062]: TGS-REQ <myaccount>@MYREALM from IPv4:<MY-IP> for krbtgt/CERN.CH@MYREALM [renewable, forwardable]
Jul  4 08:26:17 kdc-1.2 kdc[13062]: Server not found in database: krbtgt/CERN.CH@MYREALM: No such entry in the database
Jul  4 08:26:17 kdc-1.2 kdc[13062]: Failed building TGS-REP to IPv4:<MY-IP>

That's the correct behaviour. Now with a Heimdal 7.3 KDC:

[wgs03] ~ % ssh -v -o GSSAPIAuthentication=yes lxplus.cern.ch
[...]
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Cannot contact any KDC for realm 'MYREALM'

... and on the KDC side:

Jul  4 08:33:46 kdc-7.3 kdc[12045]: TGS-REQ <myaccount>@MYREALM from IPv4:<MY-IP> for krbtgt/CERN.CH@MYREALM [renewable, forwardable]
Jul  4 08:33:46 kdc-7.3 kdc[12045]: Server not found in database: krbtgt/CERN.CH@MYREALM: Success


This answer seems to make the client think the KDC is somehow malfunctioning
and repeats the request with any KDC combination (all KDCs it finds in
/etc/krb5.conf on ports 88 and 750 here). Of course, it causes long timeouts
before the ssh client gives up and asks for a password.

Any idea to restore the old "Heimdal-1.2-style" behaviour? Is this
considered a bug or misconfiguration?

Thanks,
Andreas
--
| Andreas Haupt            | E-Mail: [hidden email]
|  DESY Zeuthen            | WWW:    http://www-zeuthen.desy.de/~ahaupt
|  Platanenallee 6         | Phone:  +49/33762/7-7359
|  D-15738 Zeuthen         | Fax:    +49/33762/7-7216



smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Jeffrey Altman-2
On 7/4/2017 3:05 AM, Andreas Haupt wrote:

> ... and on the KDC side:
>
> Jul  4 08:33:46 kdc-7.3 kdc[12045]: TGS-REQ <myaccount>@MYREALM from IPv4:<MY-IP> for krbtgt/CERN.CH@MYREALM [renewable, forwardable]
> Jul  4 08:33:46 kdc-7.3 kdc[12045]: Server not found in database: krbtgt/CERN.CH@MYREALM: Success

I would like to see more of the log entries that follow as well as a
packet capture.  There is not enough detail here to say what is going on.

> This answer seems to make the client think the KDC is somehow malfunctioning
> and repeats the request with any KDC combination (all KDCs it finds in
> /etc/krb5.conf on ports 88 and 750 here). Of course, it causes long timeouts
> before the ssh client gives up and asks for a password.
>
> Any idea to restore the old "Heimdal-1.2-style" behaviour? Is this
> considered a bug or misconfiguration?

I can't tell you since I don't have enough information.

What is MYREALM?

What is the client?

What is the configuration of the client?

What is the configuration of the KDC?

My guess is the difference in behavior is related to Kerberos Referrals
and/or implicit hierarchical  capaths both of which are not present in 1.2.

Jeffrey Altman



smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Andreas Haupt-2
Hi Jeffrey,

On Fri, 2017-07-07 at 15:01 -0400, Jeffrey Altman wrote:
> On 7/4/2017 3:05 AM, Andreas Haupt wrote:
> I would like to see more of the log entries that follow as well as a
> packet capture.  There is not enough detail here to say what is going on.

Do you mean a tcpdump capture or something else like wireshark? From the
client or from the KDC?

No, the KDC does not log more than that in this case - looks it the client
retries 7 times:

Jul 10 10:40:07 fred-vm1 kdc[12044]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
Jul 10 10:40:07 fred-vm1 kdc[12044]: Server not found in database: krbtgt/[hidden email]: Success
Jul 10 10:40:08 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
Jul 10 10:40:08 fred-vm1 kdc[13458]: Server not found in database: krbtgt/[hidden email]: Success
Jul 10 10:40:09 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
Jul 10 10:40:09 fred-vm1 kdc[13458]: Server not found in database: krbtgt/[hidden email]: Success
Jul 10 10:40:12 fred-vm1 kdc[12044]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
Jul 10 10:40:12 fred-vm1 kdc[12044]: Server not found in database: krbtgt/[hidden email]: Success
Jul 10 10:40:13 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
Jul 10 10:40:13 fred-vm1 kdc[12047]: Server not found in database: krbtgt/[hidden email]: Success
Jul 10 10:40:18 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
Jul 10 10:40:18 fred-vm1 kdc[12047]: Server not found in database: krbtgt/[hidden email]: Success
Jul 10 10:40:19 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
Jul 10 10:40:19 fred-vm1 kdc[12047]: Server not found in database: krbtgt/[hidden email]: Success


> > This answer seems to make the client think the KDC is somehow
> > malfunctioning
> > and repeats the request with any KDC combination (all KDCs it finds in
> > /etc/krb5.conf on ports 88 and 750 here). Of course, it causes long
> > timeouts
> > before the ssh client gives up and asks for a password.
> >
> > Any idea to restore the old "Heimdal-1.2-style" behaviour? Is this
> > considered a bug or misconfiguration?
> I can't tell you since I don't have enough information.
>
> What is MYREALM?
IFH.DE

> What is the client?

lx64.zeuthen.desy.de (an SL7 system). It's the system's ssh client, so
linked to MIT kerberos version krb5-libs-1.14.1-26.el7.x86_64

> What is the configuration of the client?

Broke it down to a one-kdc setup on the client to illustrate the problem:

---
[libdefaults]
        default_realm = IFH.DE
        ticket_lifetime = 25h
        renew_lifetime = 30d
        forwardable = true
        noaddresses = true
        krb4_convert = false
        allow_weak_crypto = true

[realms]
        IFH.DE = {
                kdc = kdc1.zeuthen.desy.de
                admin_server = kdc1.zeuthen.desy.de
        }

[domain_realm]
        .ifh.de = IFH.DE
        .zeuthen.desy.de = IFH.DE
        .desy.de = DESY.DE
        .cern.ch = CERN.CH
---

> What is the configuration of the KDC?

---
[libdefaults]
        default_realm = IFH.DE
        ticket_lifetime = 25h
        renew_lifetime = 30d
        forwardable = true
        allow_weak_crypto = true

[realms]
        IFH.DE = {
                kdc = kdc1.zeuthen.desy.de
                admin_server = kdc1.zeuthen.desy.de
        }

[domain_realm]
        .ifh.de  = IFH.DE
        .zeuthen.desy.de = IFH.DE
        .desy.de = DESY.DE
        .cern.ch = CERN.CH

[kadmin]
        default_keys = v5

[kdc]
        require-preauth = true
---

> My guess is the difference in behavior is related to Kerberos Referrals
> and/or implicit hierarchical  capaths both of which are not present in
> 1.2.

OK. But I just see that kind of logs during AS-REQ. We have an integration
with AFS using a heimdal-7.3 client here.

Jul 10 10:32:42 fred-vm1 kdc[13458]: AS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email]
Jul 10 10:32:42 fred-vm1 kdc[13458]: Client sent patypes: REQ-ENC-PA-REP
Jul 10 10:32:42 fred-vm1 kdc[13458]: Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ
Jul 10 10:32:42 fred-vm1 kdc[12047]: AS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email]
Jul 10 10:32:42 fred-vm1 kdc[12047]: Client sent patypes: ENC-TS, REQ-ENC-PA-REP
Jul 10 10:32:42 fred-vm1 kdc[12047]: ENC-TS pre-authentication succeeded -- [hidden email]
Jul 10 10:32:42 fred-vm1 kdc[12047]: Client supported enctypes: aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha384-192, aes128-cts-hmac-sha256-128, des3-cbc-sha1, des3-cbc-md5, arcfour-hmac-md5, des-cbc-md5, des-cbc-md4, des-cbc-crc, using aes256-cts-hmac-sha1-96/aes256-cts-hmac-sha1-96
Jul 10 10:32:42 fred-vm1 kdc[12047]: Requested flags: renewable, forwardable
Jul 10 10:32:42 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for afs/[hidden email] [renewable, forwardable]
Jul 10 10:32:42 fred-vm1 kdc[13458]: Server not found in database: afs/[hidden email]: no such entry found in hdb
Jul 10 10:32:42 fred-vm1 kdc[13458]: Failed building TGS-REP to IPv4:141.34.15.17
Jul 10 10:32:42 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for afs/[hidden email] [canonicalize, renewable, forwardable]
Jul 10 10:32:42 fred-vm1 kdc[13458]: Searching referral for ifh.de
Jul 10 10:32:42 fred-vm1 kdc[13458]: Server not found in database: krbtgt/[hidden email]: no such entry found in hdb
Jul 10 10:32:42 fred-vm1 kdc[13458]: Failed building TGS-REP to IPv4:141.34.15.17
Jul 10 10:32:42 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for [hidden email] [renewable, forwardable]


In the end we only have a cross-realm trust with the realm DESY.DE here. So,
if possible we could disable any cross-realm stuff for the rest of the
world. We nether had to care about that before, though.

Thanks,
Andreas
-- 
| Andreas Haupt            | E-Mail: [hidden email]
|  DESY Zeuthen            | WWW:    http://www-zeuthen.desy.de/~ahaupt
|  Platanenallee 6         | Phone:  +49/33762/7-7359
|  D-15738 Zeuthen         | Fax:    +49/33762/7-7216



smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Jeffrey Altman-2
On 7/10/2017 4:49 AM, Andreas Haupt wrote:
> Hi Jeffrey,
>
> On Fri, 2017-07-07 at 15:01 -0400, Jeffrey Altman wrote:
>> On 7/4/2017 3:05 AM, Andreas Haupt wrote:
>> I would like to see more of the log entries that follow as well as a
>> packet capture.  There is not enough detail here to say what is going on.
>
> Do you mean a tcpdump capture or something else like wireshark? From the
> client or from the KDC?

A packet capture from the client would be fine.

>
> No, the KDC does not log more than that in this case - looks it the client
> retries 7 times:
>
> Jul 10 10:40:07 fred-vm1 kdc[12044]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
> Jul 10 10:40:07 fred-vm1 kdc[12044]: Server not found in database: krbtgt/[hidden email]: Success
> Jul 10 10:40:08 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
> Jul 10 10:40:08 fred-vm1 kdc[13458]: Server not found in database: krbtgt/[hidden email]: Success
> Jul 10 10:40:09 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
> Jul 10 10:40:09 fred-vm1 kdc[13458]: Server not found in database: krbtgt/[hidden email]: Success
> Jul 10 10:40:12 fred-vm1 kdc[12044]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
> Jul 10 10:40:12 fred-vm1 kdc[12044]: Server not found in database: krbtgt/[hidden email]: Success
> Jul 10 10:40:13 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
> Jul 10 10:40:13 fred-vm1 kdc[12047]: Server not found in database: krbtgt/[hidden email]: Success
> Jul 10 10:40:18 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
> Jul 10 10:40:18 fred-vm1 kdc[12047]: Server not found in database: krbtgt/[hidden email]: Success
> Jul 10 10:40:19 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email] [renewable, forwardable]
> Jul 10 10:40:19 fred-vm1 kdc[12047]: Server not found in database: krbtgt/[hidden email]: Success
The client is asking for a cross-realm ticket.  It is most likely doing
so because the local configuration provides for [domain_realm] mappings
and the ssh client (which you refer to later) is trying to connect to a
host in .cern.ch.

Heimdal is not finding a cross-realm ticket. Therefore it is failing the
request.  The question is what is the response and why is the MIT
Kerberos client concluding it must try again?

>>> This answer seems to make the client think the KDC is somehow
>>> malfunctioning
>>> and repeats the request with any KDC combination (all KDCs it finds in
>>> /etc/krb5.conf on ports 88 and 750 here). Of course, it causes long
>>> timeouts
>>> before the ssh client gives up and asks for a password.
>>>
>>> Any idea to restore the old "Heimdal-1.2-style" behaviour? Is this
>>> considered a bug or misconfiguration?
>> I can't tell you since I don't have enough information.
>>
>> What is MYREALM?
>
> IFH.DE
>
>> What is the client?
>
> lx64.zeuthen.desy.de (an SL7 system). It's the system's ssh client, so
> linked to MIT kerberos version krb5-libs-1.14.1-26.el7.x86_64
>
>> What is the configuration of the client?
>
> Broke it down to a one-kdc setup on the client to illustrate the problem:
>
> ---
> [libdefaults]
> default_realm = IFH.DE
> ticket_lifetime = 25h
> renew_lifetime = 30d
> forwardable = true
> noaddresses = true
> krb4_convert = false
> allow_weak_crypto = true
>
> [realms]
> IFH.DE = {
> kdc = kdc1.zeuthen.desy.de
> admin_server = kdc1.zeuthen.desy.de
> }
>
> [domain_realm]
> .ifh.de = IFH.DE
> .zeuthen.desy.de = IFH.DE
> .desy.de = DESY.DE
> .cern.ch = CERN.CH
> ---
>
>> What is the configuration of the KDC?
>
> ---
> [libdefaults]
> default_realm = IFH.DE
> ticket_lifetime = 25h
> renew_lifetime = 30d
> forwardable = true
> allow_weak_crypto = true
>
> [realms]
> IFH.DE = {
> kdc = kdc1.zeuthen.desy.de
> admin_server = kdc1.zeuthen.desy.de
> }
>
> [domain_realm]
> .ifh.de  = IFH.DE
> .zeuthen.desy.de = IFH.DE
> .desy.de = DESY.DE
> .cern.ch = CERN.CH
>
> [kadmin]
> default_keys = v5
>
> [kdc]
> require-preauth = true
> ---
>
>> My guess is the difference in behavior is related to Kerberos Referrals
>> and/or implicit hierarchical  capaths both of which are not present in
>> 1.2.
>
> OK. But I just see that kind of logs during AS-REQ. We have an integration
> with AFS using a heimdal-7.3 client here.
What is the relationship of the afs/[hidden email] request to the
cross-realm request?

>
> Jul 10 10:32:42 fred-vm1 kdc[13458]: AS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email]
> Jul 10 10:32:42 fred-vm1 kdc[13458]: Client sent patypes: REQ-ENC-PA-REP
> Jul 10 10:32:42 fred-vm1 kdc[13458]: Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ
> Jul 10 10:32:42 fred-vm1 kdc[12047]: AS-REQ [hidden email] from IPv4:141.34.15.17 for krbtgt/[hidden email]
> Jul 10 10:32:42 fred-vm1 kdc[12047]: Client sent patypes: ENC-TS, REQ-ENC-PA-REP
> Jul 10 10:32:42 fred-vm1 kdc[12047]: ENC-TS pre-authentication succeeded -- [hidden email]
> Jul 10 10:32:42 fred-vm1 kdc[12047]: Client supported enctypes: aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha384-192, aes128-cts-hmac-sha256-128, des3-cbc-sha1, des3-cbc-md5, arcfour-hmac-md5, des-cbc-md5, des-cbc-md4, des-cbc-crc, using aes256-cts-hmac-sha1-96/aes256-cts-hmac-sha1-96
> Jul 10 10:32:42 fred-vm1 kdc[12047]: Requested flags: renewable, forwardable
> Jul 10 10:32:42 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for afs/[hidden email] [renewable, forwardable]
> Jul 10 10:32:42 fred-vm1 kdc[13458]: Server not found in database: afs/[hidden email]: no such entry found in hdb
> Jul 10 10:32:42 fred-vm1 kdc[13458]: Failed building TGS-REP to IPv4:141.34.15.17
> Jul 10 10:32:42 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for afs/[hidden email] [canonicalize, renewable, forwardable]
> Jul 10 10:32:42 fred-vm1 kdc[13458]: Searching referral for ifh.de
> Jul 10 10:32:42 fred-vm1 kdc[13458]: Server not found in database: krbtgt/[hidden email]: no such entry found in hdb
> Jul 10 10:32:42 fred-vm1 kdc[13458]: Failed building TGS-REP to IPv4:141.34.15.17
> Jul 10 10:32:42 fred-vm1 kdc[12047]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for [hidden email] [renewable, forwardable]
>
>
> In the end we only have a cross-realm trust with the realm DESY.DE here. So,
> if possible we could disable any cross-realm stuff for the rest of the
> world. We nether had to care about that before, though.
There is no cross-realm principal for CERN.CH therefore there is no
cross-realm configuration.

Jeffrey Altman



smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Andreas Haupt-2
Hi Jeffrey,

On Mon, 2017-07-10 at 07:23 -0400, Jeffrey Altman wrote:

> On 7/10/2017 4:49 AM, Andreas Haupt wrote:
> > On Fri, 2017-07-07 at 15:01 -0400, Jeffrey Altman wrote:
> > >
> > > On 7/4/2017 3:05 AM, Andreas Haupt wrote:
> > > I would like to see more of the log entries that follow as well as a
> > > packet capture.  There is not enough detail here to say what is going
> > > on.
> > Do you mean a tcpdump capture or something else like wireshark? From the
> > client or from the KDC?
> A packet capture from the client would be fine.
OK, I do that later. Any preferences like wireshark or tcpdump with specific
options?

> The client is asking for a cross-realm ticket.  It is most likely doing
> so because the local configuration provides for [domain_realm] mappings
> and the ssh client (which you refer to later) is trying to connect to a
> host in .cern.ch.

Indeed, the behaviour changes, if one removes the [domain_realm] mapping. It
looks like this then:

Jul 10 13:27:36 fred-vm1 kdc[12044]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 10 13:27:36 fred-vm1 kdc[12044]: Searching referral for lxplus040.cern.ch
Jul 10 13:27:36 fred-vm1 kdc[12044]: Server not found in database: krbtgt/[hidden email]: Success


Learned something new here: just thought, the domain_realm stuff only helps
the client find the correct realm for hosts located in a domain (e.g. when
this domain lacks the "_kerberos" TXT record in its zone definition).

> Heimdal is not finding a cross-realm ticket. Therefore it is failing the
> request.  The question is what is the response and why is the MIT
> Kerberos client concluding it must try again?

My general question is rather this one: The KDC tries some referral/cross-
realm stuff all the time. But whereas it gives back a real error in cases
like this ("no such entry found in hdb" which results in "Failed building
TGS-REP to IPv4:141.34.15.17"):

Jul 10 10:32:42 fred-vm1 kdc[13458]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for afs/[hidden email] [canonicalize, renewable, forwardable]
Jul 10 10:32:42 fred-vm1 kdc[13458]: Searching referral for ifh.de
Jul 10 10:32:42 fred-vm1 kdc[13458]: Server not found in database: krbtgt/[hidden email]: no such entry found in hdb
Jul 10 10:32:42 fred-vm1 kdc[13458]: Failed building TGS-REP to IPv4:141.34.15.17


... it "succeeds" in the CERN.CH case:

Jul 10 13:27:36 fred-vm1 kdc[12044]: TGS-REQ [hidden email] from IPv4:141.34.15.17 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 10 13:27:36 fred-vm1 kdc[12044]: Searching referral for lxplus040.cern.ch
Jul 10 13:27:36 fred-vm1 kdc[12044]: Server not found in database: krbtgt/[hidden email]: Success

"Server not found" == "Success"? Is this really the expected answer? I would
guess, no - but not really sure ...

> > > > OK. But I just see that kind of logs during AS-REQ. We have an
> > integration
> > with AFS using a heimdal-7.3 client here.
> What is the relationship of the afs/[hidden email] request to the
> cross-realm request?

This client is located in the AFS cell "ifh.de". We still only have the
"old-style" principal named [hidden email], whereas the client always asks for
the "new-style" principal afs/<cell>@REALM first. Was this what you asked
for?

> > In the end we only have a cross-realm trust with the realm DESY.DE here.
> > So,
> > if possible we could disable any cross-realm stuff for the rest of the
> > world. We nether had to care about that before, though.
> There is no cross-realm principal for CERN.CH therefore there is no
> cross-realm configuration.

Yes.

Cheers,
Andreas
--
| Andreas Haupt            | E-Mail: [hidden email]
|  DESY Zeuthen            | WWW:    http://www-zeuthen.desy.de/~ahaupt
|  Platanenallee 6         | Phone:  +49/33762/7-7359
|  D-15738 Zeuthen         | Fax:    +49/33762/7-7216



smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Jeffrey Hutzelman
On July 10, 2017 8:16:05 AM EDT, Andreas Haupt <[hidden email]> wrote:

>... it "succeeds" in the CERN.CH case:
>
>Jul 10 13:27:36 fred-vm1 kdc[12044]: TGS-REQ [hidden email] from
>IPv4:141.34.15.17 for host/[hidden email] [canonicalize,
>renewable, forwardable]
>Jul 10 13:27:36 fred-vm1 kdc[12044]: Searching referral for
>lxplus040.cern.ch
>Jul 10 13:27:36 fred-vm1 kdc[12044]: Server not found in database:
>krbtgt/[hidden email]: Success
>
>"Server not found" == "Success"? Is this really the expected answer? I
>would
>guess, no - but not really sure ...

This is a bug in the kdc, or possibly two bugs. First, the database lookup failed and no entry was returned, but the error code was not set and so remained zero, which com_err translates as "Success".

Second, the kdc is not sending any response at all. That causes the client to eventually time out and try another kdc. When it runs out of kdcs, it reports an error (unable to contact any kdc in realm).

you can confirm this by watching traffic between your client and kdc on port 88, using your favorite packet-capture tool.

-- Jeff

Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Andreas Haupt-2
Hi Jeffrey,

On Mon, 2017-07-10 at 08:32 -0400, Jeffrey Hutzelman wrote:

> This is a bug in the kdc, or possibly two bugs. First, the database lookup
> failed and no entry was returned, but the error code was not set and so
> remained zero, which com_err translates as "Success".
>
> Second, the kdc is not sending any response at all. That causes the client
> to eventually time out and try another kdc. When it runs out of kdcs, it
> reports an error (unable to contact any kdc in realm).
>
> you can confirm this by watching traffic between your client and kdc on
> port 88, using your favorite packet-capture tool.
Exact! That's indeed the problem here! Just moved on to real test systems
now.

141.34.32.72 -> SL7 client (GSSAPI-enabled system's ssh)
141.34.22.251 -> Heimdal-7.3 server

The test client only knows about the test Heimdal-7.3 server 141.34.22.251.

---
[chap-vm1] ~ % ssh -vvv lxplus.cern.ch
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
[...]
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Cannot contact any KDC for realm 'IFH.DE'
[...]
Password:
---

This is what is captured as traffic between client and KDC:

---
[chap-vm1] /root # tshark host test-kdc -t a
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eth0'
  1 16:06:43.529842068 141.34.32.72 -> 141.34.22.251 KRB5 990 TGS-REQ
  2 16:06:44.532691656 141.34.32.72 -> 141.34.22.251 UDP 990 Source port: 48551  Destination port: loadav
  3 16:06:45.533928362 141.34.32.72 -> 141.34.22.251 TCP 74 52164 > kerberos [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=1035010913 TSecr=0 WS=128
  4 16:06:45.534376809 141.34.22.251 -> 141.34.32.72 TCP 74 kerberos > 52164 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1816942862 TSecr=1035010913 WS=128
  5 16:06:45.534427289 141.34.32.72 -> 141.34.22.251 TCP 66 52164 > kerberos [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=1035010914 TSecr=1816942862
  6 16:06:45.534545716 141.34.32.72 -> 141.34.22.251 KRB5 1018 TGS-REQ
  7 16:06:45.534903298 141.34.22.251 -> 141.34.32.72 TCP 66 kerberos > 52164 [ACK] Seq=1 Ack=953 Win=30976 Len=0 TSval=1816942863 TSecr=1035010914
  8 16:06:45.536931301 141.34.22.251 -> 141.34.32.72 TCP 66 kerberos > 52164 [FIN, ACK] Seq=1 Ack=953 Win=30976 Len=0 TSval=1816942865 TSecr=1035010914
  9 16:06:45.537047128 141.34.32.72 -> 141.34.22.251 TCP 66 52164 > kerberos [FIN, ACK] Seq=953 Ack=2 Win=29312 Len=0 TSval=1035010916 TSecr=1816942865
 10 16:06:45.542447536 141.34.22.251 -> 141.34.32.72 TCP 66 kerberos > 52164 [ACK] Seq=2 Ack=954 Win=30976 Len=0 TSval=1816942870 TSecr=1035010916
 11 16:06:48.536256407 141.34.32.72 -> 141.34.22.251 KRB5 990 TGS-REQ
 12 16:06:49.537370582 141.34.32.72 -> 141.34.22.251 UDP 990 Source port: 48551  Destination port: loadav
 13 16:06:54.542592820 141.34.32.72 -> 141.34.22.251 KRB5 990 TGS-REQ
 14 16:06:55.543675219 141.34.32.72 -> 141.34.22.251 UDP 990 Source port: 48551  Destination port: loadav
---

And the matching KDC logs:

---
Jul 11 16:06:43 chip-vm8 kdc[17992]: TGS-REQ [hidden email] from IPv4:141.34.32.72 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 11 16:06:43 chip-vm8 kdc[17992]: Searching referral for lxplus010.cern.ch
Jul 11 16:06:43 chip-vm8 kdc[17992]: Server not found in database: krbtgt/[hidden email]: Success
Jul 11 16:06:44 chip-vm8 kdc[17992]: TGS-REQ [hidden email] from IPv4:141.34.32.72 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 11 16:06:44 chip-vm8 kdc[17992]: Searching referral for lxplus010.cern.ch
Jul 11 16:06:44 chip-vm8 kdc[17992]: Server not found in database: krbtgt/[hidden email]: Success
Jul 11 16:06:45 chip-vm8 kdc[17992]: TGS-REQ [hidden email] from IPv4:141.34.32.72 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 11 16:06:45 chip-vm8 kdc[17992]: Searching referral for lxplus010.cern.ch
Jul 11 16:06:45 chip-vm8 kdc[17992]: Server not found in database: krbtgt/[hidden email]: Success
Jul 11 16:06:48 chip-vm8 kdc[17992]: TGS-REQ [hidden email] from IPv4:141.34.32.72 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 11 16:06:48 chip-vm8 kdc[17992]: Searching referral for lxplus010.cern.ch
Jul 11 16:06:48 chip-vm8 kdc[17992]: Server not found in database: krbtgt/[hidden email]: Success
Jul 11 16:06:49 chip-vm8 kdc[17992]: TGS-REQ [hidden email] from IPv4:141.34.32.72 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 11 16:06:49 chip-vm8 kdc[17992]: Searching referral for lxplus010.cern.ch
Jul 11 16:06:49 chip-vm8 kdc[17992]: Server not found in database: krbtgt/[hidden email]: Success
Jul 11 16:06:54 chip-vm8 kdc[17992]: TGS-REQ [hidden email] from IPv4:141.34.32.72 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 11 16:06:54 chip-vm8 kdc[17992]: Searching referral for lxplus010.cern.ch
Jul 11 16:06:54 chip-vm8 kdc[17992]: Server not found in database: krbtgt/[hidden email]: Success
Jul 11 16:06:55 chip-vm8 kdc[17992]: TGS-REQ [hidden email] from IPv4:141.34.32.72 for host/[hidden email] [canonicalize, renewable, forwardable]
Jul 11 16:06:55 chip-vm8 kdc[17992]: Searching referral for lxplus010.cern.ch
Jul 11 16:06:55 chip-vm8 kdc[17992]: Server not found in database: krbtgt/[hidden email]: Success
---

Opened bug report: https://github.com/heimdal/heimdal/issues/299

Cheers,
Andreas
--
| Andreas Haupt            | E-Mail: [hidden email]
|  DESY Zeuthen            | WWW:    http://www-zeuthen.desy.de/~ahaupt
|  Platanenallee 6         | Phone:  +49/33762/7-7359
|  D-15738 Zeuthen         | Fax:    +49/33762/7-7216



smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Greg Hudson
On 07/11/2017 10:37 AM, Andreas Haupt wrote:
> On Mon, 2017-07-10 at 08:32 -0400, Jeffrey Hutzelman wrote:
>> This is a bug in the kdc, or possibly two bugs. First, the database lookup
>> failed and no entry was returned, but the error code was not set and so
>> remained zero, which com_err translates as "Success".

>> Second, the kdc is not sending any response at all.

I think the bug was introduced by commit
4b4036c9a6697f0101c60845e19664f64fdd0810 and is that the value of ret is
squashed by the call to _krb5_find_capath() in tgs_build_reply().  In
this scenario, I believe the call succeeds, but doesn't find any
capaths, so we don't goto server_lookup, instead dropping down and going
to out with ret still 0.  _kdc_tgs_rep() doesn't create an error reply
if ret is 0, so the KDC sends no reply.
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Viktor Dukhovni-2
On Tue, Jul 11, 2017 at 10:19:48PM -0400, Greg Hudson wrote:

> I think the bug was introduced by commit
> 4b4036c9a6697f0101c60845e19664f64fdd0810 and is that the value of ret is
> squashed by the call to _krb5_find_capath() in tgs_build_reply().  In
> this scenario, I believe the call succeeds, but doesn't find any
> capaths, so we don't goto server_lookup, instead dropping down and going
> to out with ret still 0.  _kdc_tgs_rep() doesn't create an error reply
> if ret is 0, so the KDC sends no reply.

That looks plausible, does the below look like the right fix to you?

diff --git a/kdc/krb5tgs.c b/kdc/krb5tgs.c
index 98503812f..b6ccb68ed 100644
--- a/kdc/krb5tgs.c
+++ b/kdc/krb5tgs.c
@@ -1511,7 +1511,7 @@ tgs_build_reply(krb5_context context,
  AuthorizationData **auth_data,
  const struct sockaddr *from_addr)
 {
-    krb5_error_code ret;
+    krb5_error_code ret, ret2;
     krb5_principal cp = NULL, sp = NULL, rsp = NULL, tp = NULL, dp = NULL;
     krb5_principal krbtgt_out_principal = NULL;
     char *spn = NULL, *cpn = NULL, *tpn = NULL, *dpn = NULL, *krbtgt_out_n = NULL;
@@ -1677,10 +1677,12 @@ server_lookup:
  if ((req_rlm = get_krbtgt_realm(&sp->name)) != NULL) {
             if (capath == NULL) {
                 /* With referalls, hierarchical capaths are always enabled */
-                ret = _krb5_find_capath(context, tgt->crealm, our_realm,
-                                        req_rlm, TRUE, &capath, &num_capath);
-                if (ret)
+                ret2 = _krb5_find_capath(context, tgt->crealm, our_realm,
+                                         req_rlm, TRUE, &capath, &num_capath);
+                if (ret2) {
+                    ret = ret2;
                     goto out;
+                }
             }
             new_rlm = num_capath > 0 ? capath[--num_capath] : NULL;
             if (new_rlm) {

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Weird cross-realm behaviour after upgrade to Heimdal 7.3

Andreas Haupt-2
Hi Viktor,

On Wed, 2017-07-12 at 05:14 +0000, Viktor Dukhovni wrote:
> On Tue, Jul 11, 2017 at 10:19:48PM -0400, Greg Hudson wrote:
> I think the bug was introduced by commit
> > 4b4036c9a6697f0101c60845e19664f64fdd0810 and is that the value of ret is
> > squashed by the call to _krb5_find_capath() in tgs_build_reply().  In
> > this scenario, I believe the call succeeds, but doesn't find any
> > capaths, so we don't goto server_lookup, instead dropping down and going
> > to out with ret still 0.  _kdc_tgs_rep() doesn't create an error reply
> > if ret is 0, so the KDC sends no reply.
> That looks plausible, does the below look like the right fix to you?

Yes! Already had a similar patch ready and this indeed cures the KDC's
response behaviour to the client!

Cheers,
Andreas
-- 
| Andreas Haupt            | E-Mail: [hidden email]
|  DESY Zeuthen            | WWW:    http://www-zeuthen.desy.de/~ahaupt
|  Platanenallee 6         | Phone:  +49/33762/7-7359
|  D-15738 Zeuthen         | Fax:    +49/33762/7-7216



smime.p7s (6K) Download Attachment