MIT Kerberos for Windows failing with Windows 10 update 1803?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

MIT Kerberos for Windows failing with Windows 10 update 1803?

Ruurd Beerstra
Hi,

I'm developer of a Windows SSH/Telnet  client (called IVT) that supports
both GSSAPI authentication and Kerberized telnet.
I've noticed that the setup I use for regression testing now finds
errors for both protocols: Login fails.

After a lot of digging, I'm suspecting Windows 10 privacy update (1803)
that was pushed to my development workstation a short while ago.

The symptoms are that I can obtain a TGT from my KDC (which ends up in
de LSA of Windows), but every attempt to use that TGT to obtain a
service ticket yields an error:
Matching credential not found.

When I install a copy of the software on a Windows 7 Virtual Box machine
(same network, same KDC, same user/principal, same IVT version, same
Kerberos for Windows version 4.1, etc) it works flawlessly.
I was about to go single stepping through my code to find the problem,
but when I woke the PC to start work on that, I noticed that the MIT
software itself has the same problem!
This popup appeared:



So that is Kerberos for Windows trying to refresh my credentials and
running into the very same error.
Apparently it cannot access the TGT either.

I found this article
https://www.csoonline.com/article/3253899/windows/the-best-new-windows-10-security-features.html
about all sorts of new security features in Windows 10 and that sounds
like Microsoft may have changed something that breaks Kerberos?

When I use a sniffer on my network I can see that there is no
communication between my Telnet client and the KDC when it is supposed
to request a ticket for the host I'm logging in to.
So there is no error logged on the KDC either (I jusyt see an entry when
I login to get my TGT).

Some details about the environment:
- KDC is MIT version krb5-1.16.1
- kfw-4.1-amd64.msi, freshly (re)installed
- Target is a Linux box with a ktelnetd on it, but all that does is
saying "DO AUTH" and then when I try to get a ticket it fails.
- PC is Windows 10 Home edition, version 1803 build 17134.112

Everything worked until about two weeks ago (1803 was installed on 5th
of June).

I can get my TGT:

but that is all I ever see, no tickets for the host I'm trying to login to.

Insights very much appreciated, please reply to [hidden email].

     Regards,
     Ruurd Beerstra




________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Greg Hudson
On 06/17/2018 02:02 PM, Ruurd Beerstra wrote:
> The symptoms are that I can obtain a TGT from my KDC (which ends up in
> de LSA of Windows), but every attempt to use that TGT to obtain a
> service ticket yields an error:
> Matching credential not found.

Unfortunately, our mailing list server doesn't pass through attachments,
so while I briefly saw your screenshots before moderating through your
message, they didn't make it to the list (and I didn't keep a copy.)

I believe the correct short answer is to use the "API:" ccache instead
of the "MSLSA:" ccache for this setup.

For some time Windows has restricted access to TGT session keys in the
LSA, which means our libkrb5 code can't use a TGT from the LSA to get
service tickets.  Instead, our MSLSA ccache type requests service
tickets via Windows, but that only works if the realm is set up in the
LSA configuration.  Since you are using an MIT krb5 KDC, I am guessing
that it is not set up in the LSA configuration, so we fall back to
trying to get service tickets using the TGT.

The TGT session key restriction can be overridden by the registry value
HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos AllowTGTSessionKey,
which I believe our installer sets.  I would not be surprised if update
1803 disables this registry value so that the restriction is always in
effect.

I have also heard that Microsoft plans to disable access to service
ticket session keys from userspace, effectively preventing KfW from
using the LSA ccache.  I don't know if that restriction is present in
update 1803, and I believe it only applies if Credential Guard is
enabled.  (I don't know what determines whether Credential Guard is
enabled.)  We could conceivably work around this restriction in the
GSSAPI library by getting context establishment tokens via SSPI instead
of via our krb5 code, but I can't make any promises as to when that
might be implemented.  I don't believe this is the restriction at issue
in your test setup anyway.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Benjamin Kaduk-2
On Sun, Jun 17, 2018 at 04:35:50PM -0400, Greg Hudson wrote:

> On 06/17/2018 02:02 PM, Ruurd Beerstra wrote:
> > The symptoms are that I can obtain a TGT from my KDC (which ends up in
> > de LSA of Windows), but every attempt to use that TGT to obtain a
> > service ticket yields an error:
> > Matching credential not found.
>
> Unfortunately, our mailing list server doesn't pass through attachments,
> so while I briefly saw your screenshots before moderating through your
> message, they didn't make it to the list (and I didn't keep a copy.)
>
> I believe the correct short answer is to use the "API:" ccache instead
> of the "MSLSA:" ccache for this setup.
>
> For some time Windows has restricted access to TGT session keys in the
> LSA, which means our libkrb5 code can't use a TGT from the LSA to get
> service tickets.  Instead, our MSLSA ccache type requests service
> tickets via Windows, but that only works if the realm is set up in the
> LSA configuration.  Since you are using an MIT krb5 KDC, I am guessing
> that it is not set up in the LSA configuration, so we fall back to
> trying to get service tickets using the TGT.

Does this mean that you think setting the appropriate entries under
SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Domains would resolve
the issue?

-Ben
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Ruurd Beerstra
In reply to this post by Greg Hudson
Hi,

Thanks for the quick reply.
The first screenshot showed a popup from  MIT Kerberos (Title "Kerberos
Five") saying:
Matching credential not found. Kerberos error -1765328243.
krb5_get_renewed_creds() failed.

The 2nd was the ticket manager showing just my TGT and nothing else.

I probably should have mentioned I tried setting the ccache type to
"FILE", and that didn't work either.
I tried "API" just now (didn't know that one), the ticket manager shows
"API: Initial default ccache" in its "Credential cache" column.
But no joy: IVT and Putty both fail to use GSSAPI, and IVT also fails on
Kerberized telnet.
Tried reboot of my workstation: No change.

Both IVT and Putty cause the ticket-manager to pop up to obtain new
credentials, because they do not see my TGT (and so allow me to obtain
one).
The AllowTGTSessionKey is set in the registry, value 1.

A quick search on Credential Guard says: The Windows Defender Credential
Guard prevents these attacks by protecting NTLM password hashes,
Kerberos Ticket Granting Tickets, and credentials stored by applications
as domain credentials.

And that sounds like exactly my problem. I can't find any way to turn
that protection  off.
I don't understand why a FILE: or API: based ccache would be affected by
this Credential Guard, but I can't get any GSSAPI or Kerberized telnet
session to work anymore.
And up until recently (I have automated regression tests fror IVT) that
was not a problem.

Bedtime here now, maybe more tomorrow,

     Thanks again,
     Ruurd


Op 17-6-2018 om 22:35 schreef Greg Hudson:

> On 06/17/2018 02:02 PM, Ruurd Beerstra wrote:
>> The symptoms are that I can obtain a TGT from my KDC (which ends up in
>> de LSA of Windows), but every attempt to use that TGT to obtain a
>> service ticket yields an error:
>> Matching credential not found.
>
> Unfortunately, our mailing list server doesn't pass through
> attachments, so while I briefly saw your screenshots before moderating
> through your message, they didn't make it to the list (and I didn't
> keep a copy.)
>
> I believe the correct short answer is to use the "API:" ccache instead
> of the "MSLSA:" ccache for this setup.
>
> For some time Windows has restricted access to TGT session keys in the
> LSA, which means our libkrb5 code can't use a TGT from the LSA to get
> service tickets.  Instead, our MSLSA ccache type requests service
> tickets via Windows, but that only works if the realm is set up in the
> LSA configuration.  Since you are using an MIT krb5 KDC, I am guessing
> that it is not set up in the LSA configuration, so we fall back to
> trying to get service tickets using the TGT.
>
> The TGT session key restriction can be overridden by the registry
> value HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos
> AllowTGTSessionKey, which I believe our installer sets.  I would not
> be surprised if update 1803 disables this registry value so that the
> restriction is always in effect.
>
> I have also heard that Microsoft plans to disable access to service
> ticket session keys from userspace, effectively preventing KfW from
> using the LSA ccache.  I don't know if that restriction is present in
> update 1803, and I believe it only applies if Credential Guard is
> enabled.  (I don't know what determines whether Credential Guard is
> enabled.)  We could conceivably work around this restriction in the
> GSSAPI library by getting context establishment tokens via SSPI
> instead of via our krb5 code, but I can't make any promises as to when
> that might be implemented.  I don't believe this is the restriction at
> issue in your test setup anyway.
>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Greg Hudson
On 06/18/2018 12:25 PM, Ruurd Beerstra wrote:
> I probably should have mentioned I tried setting the ccache type to
> "FILE", and that didn't work either.

Just "FILE"?  You need to set it to "FILE:pathname" for some pathname.

I don't have a hypothesis to explain why "API:" wouldn't work.  I
updated a Windows 10 VM to 1803 and installed the kfw 4.1 MSI.  With the
API: ccache type I was able to acquire tickets, renew tickets, acquire
service tickets using kvno, and see the acquired service ticket with klist.

With the MSLSA: ccache type it does seem like I can't access the TGT
session key.  I can acquire a TGT and can view it in the graphical
ticket manager (but not with klist; not sure why).  Renewing the TGT
doesn't appear to work, although I only see an error message if I run
"kinit -R" from the command line (same error as you saw, "Matching
credential not found"); with the graphical ticket manager it seems to
silently fail.  I can obtain a service ticket and view that with klist,
but from tracing output it is clear that that is happening through the
LSA (which is probably able to find the realm I'm testing against using
SRV records in DNS).

> A quick search on Credential Guard says: The Windows Defender Credential
> Guard prevents these attacks by protecting NTLM password hashes,
> Kerberos Ticket Granting Tickets, and credentials stored by applications
> as domain credentials.

To be clear, my uncomfirmed hypothesis is that update 1803 made the
HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos AllowTGTSessionKey
registry variable inoperable, with or without Credential Guard.  An
additional restriction on access to service ticket session keys does not
seem to match the errors you're seeing.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Benjamin Kaduk-2
On Mon, Jun 18, 2018 at 05:31:33PM -0400, Greg Hudson wrote:

> On 06/18/2018 12:25 PM, Ruurd Beerstra wrote:
> > I probably should have mentioned I tried setting the ccache type to
> > "FILE", and that didn't work either.
>
> Just "FILE"?  You need to set it to "FILE:pathname" for some pathname.
>
> I don't have a hypothesis to explain why "API:" wouldn't work.  I
> updated a Windows 10 VM to 1803 and installed the kfw 4.1 MSI.  With the
> API: ccache type I was able to acquire tickets, renew tickets, acquire
> service tickets using kvno, and see the acquired service ticket with klist.
>
> With the MSLSA: ccache type it does seem like I can't access the TGT
> session key.  I can acquire a TGT and can view it in the graphical
> ticket manager (but not with klist; not sure why).  Renewing the TGT

I think the magic there is at
https://github.com/krb5/krb5/blob/master/src/windows/leash/KrbListTickets.cpp#L201

> doesn't appear to work, although I only see an error message if I run
> "kinit -R" from the command line (same error as you saw, "Matching
> credential not found"); with the graphical ticket manager it seems to
> silently fail.  I can obtain a service ticket and view that with klist,
> but from tracing output it is clear that that is happening through the
> LSA (which is probably able to find the realm I'm testing against using
> SRV records in DNS).

My recollection was that you needed something in the registry (at
the path I mentioned in my previous mail) even to get the LSA to
look for SRV records.  (Do I know what realm you're testing with?)
IIRC the KfW installer does prepopulate KDC entries for a couple of
realms, as an example if nothing else.)

-Ben

> > A quick search on Credential Guard says: The Windows Defender Credential
> > Guard prevents these attacks by protecting NTLM password hashes,
> > Kerberos Ticket Granting Tickets, and credentials stored by applications
> > as domain credentials.
>
> To be clear, my uncomfirmed hypothesis is that update 1803 made the
> HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos AllowTGTSessionKey
> registry variable inoperable, with or without Credential Guard.  An
> additional restriction on access to service ticket session keys does not
> seem to match the errors you're seeing.
> ________________________________________________
> Kerberos mailing list           [hidden email]
> https://mailman.mit.edu/mailman/listinfo/kerberos
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Ruurd Beerstra
In reply to this post by Greg Hudson
OK, I'm confused now....
2 days ago I tried FILE: by editing krb5.ini and setting default_cc_name
to FILE:c:/tmp/krb5cc_${uid}
I saw it uses the SID for my user as part of the filename in c:/tmp.
I SAW the file being made, and it STILL refused to work the way it
should. The ticket manager showed "FILE:.." in the "Credential cache"
column.
I tried your API suggestion last night, saw "API: Initial default cache:
in the ticket manager and it still did not work.

Then I tried again tonight, by using "kvno" to acquire a ticket. And all
of  sudden, the host service ticket appears in the ticket manager!
Fire up IVT: It uses the ticket and logs in with Kerberized telnet and
GSSAPI-over-SSH like it should.
Destroy the TGT and other tickets, restart IVT: It pops up the ticket
manager, I type the proper password, tickets are acquired: Everything works.
No clue as to why it started working all of a sudden, but I'm happy!

So I wanted to go through the proper steps again so I could document it
in IVT in case a user runs into this issue.
I change the krb5.ini file back to the default empty:
  Directory of C:\ProgramData\MIT\Kerberos5

19-06-2018  21:03    <DIR>          .
19-06-2018  21:03    <DIR>          ..
19-06-2018  21:04                 0 krb5.ini
                1 File(s)              0 bytes

Start the "MIT kerberos.exe", get a TGT: The Credential cache" still
shows "API: Initial default ccache". Huh? Why does it not go back to MSLSA?
Reboot. No difference. API.
Search Registry for things called MIT and/od ccache: Found nothing relevant.
Search file system for things called krb5.ini: Nothing relevant.

So now it keeps working despite everything I try to break it.
Admittedly, progress is made, but I want to understand what happens here!

Can someone here point me in the right direction?
Where am I supposed to configure MSLSA if I want to force the problem again?
Where does it store the current API setting?
What did I do wrong the time I tried the FILE: setting?

     Help very much appreciated,
     Ruurd



Op 18-6-2018 om 23:31 schreef Greg Hudson:

> On 06/18/2018 12:25 PM, Ruurd Beerstra wrote:
>> I probably should have mentioned I tried setting the ccache type to
>> "FILE", and that didn't work either.
>
> Just "FILE"?  You need to set it to "FILE:pathname" for some pathname.
>
> I don't have a hypothesis to explain why "API:" wouldn't work.  I
> updated a Windows 10 VM to 1803 and installed the kfw 4.1 MSI. With
> the API: ccache type I was able to acquire tickets, renew tickets,
> acquire service tickets using kvno, and see the acquired service
> ticket with klist.
>
> With the MSLSA: ccache type it does seem like I can't access the TGT
> session key.  I can acquire a TGT and can view it in the graphical
> ticket manager (but not with klist; not sure why). Renewing the TGT
> doesn't appear to work, although I only see an error message if I run
> "kinit -R" from the command line (same error as you saw, "Matching
> credential not found"); with the graphical ticket manager it seems to
> silently fail.  I can obtain a service ticket and view that with
> klist, but from tracing output it is clear that that is happening
> through the LSA (which is probably able to find the realm I'm testing
> against using SRV records in DNS).
>
>> A quick search on Credential Guard says: The Windows Defender Credential
>> Guard prevents these attacks by protecting NTLM password hashes,
>> Kerberos Ticket Granting Tickets, and credentials stored by applications
>> as domain credentials.
>
> To be clear, my uncomfirmed hypothesis is that update 1803 made the
> HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos AllowTGTSessionKey
> registry variable inoperable, with or without Credential Guard.  An
> additional restriction on access to service ticket session keys does
> not seem to match the errors you're seeing.
>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Greg Hudson
On 06/19/2018 03:51 PM, Ruurd Beerstra wrote:
> OK, I'm confused now....
> 2 days ago I tried FILE: by editing krb5.ini and setting default_cc_name
> to FILE:c:/tmp/krb5cc_${uid}
> I saw it uses the SID for my user as part of the filename in c:/tmp.
> I SAW the file being made, and it STILL refused to work the way it
> should. The ticket manager showed "FILE:.." in the "Credential cache"
> column.
I'm not sure if I can completely explain this, but it's [libdefaults]
default_ccache_name, not default_cc_name.

> Start the "MIT kerberos.exe", get a TGT: The Credential cache" still
> shows "API: Initial default ccache". Huh? Why does it not go back to MSLSA?

Three's a registry value HKCU\Software\MIT\Kerberos5 ccname, which the
ticket manager sets if you click "make default".
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos for Windows failing with Windows 10 update 1803?

Ruurd Beerstra
Thanks very much, that clarifies my problems.
I must have clicked on "Make default", and it has the API settings in
the registry.
I can't find any button to undo that - If I've changed something in
krb.ini, click "Make default", then afterwards I'm stuck with those
settings until I manually edit the registry?
Odd.

But If I use the API settings, all is well and logging in with GSSAPI or
Kerberized telnet works the way it should.
I'll update my docs to reflect that in case my users run into this problem.

Thanks again for the help,

     Ruurd

Op 19-6-2018 om 22:25 schreef Greg Hudson:

> On 06/19/2018 03:51 PM, Ruurd Beerstra wrote:
>> OK, I'm confused now....
>> 2 days ago I tried FILE: by editing krb5.ini and setting
>> default_cc_name to FILE:c:/tmp/krb5cc_${uid}
>> I saw it uses the SID for my user as part of the filename in c:/tmp.
>> I SAW the file being made, and it STILL refused to work the way it
>> should. The ticket manager showed "FILE:.." in the "Credential cache"
>> column.
> I'm not sure if I can completely explain this, but it's [libdefaults]
> default_ccache_name, not default_cc_name.
>
>> Start the "MIT kerberos.exe", get a TGT: The Credential cache" still
>> shows "API: Initial default ccache". Huh? Why does it not go back to
>> MSLSA?
>
> Three's a registry value HKCU\Software\MIT\Kerberos5 ccname, which the
> ticket manager sets if you click "make default".
>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos