MIT Kerberos Master principal deletion

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

MIT Kerberos Master principal deletion

Harshawardhan Kulkarni
Hi Team,

I basically need an advice on an ongoing issue I am currently stuck on.

We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
the nodes. We don't have a failover node for KDC server yet. On the KDC
admin server while doing a clean up activity for unwanted kdc principals, I
deleted the master key principal (K/[hidden email]) We never took a kdc dump
of the master key. So we don't have a backup to restore from.

Is there any way I can restore the master key principal?

I have tried creating with kdb5_util add_mkey but the error says that KDC
DB is not able to find a master key credential. I assume this would only
work when you want to create another master key without deleting the
primary key.

Another option for me would be to de-kerberise the cluster and create the
same REALM and kerberise the cluster again. But there could be serious
issues if this doesn't fix as this is a live cluster where people are using
this on a daily basis.

Can anyone help me here? Looking forward for your reply.

Thanks,
Harsh Kulkarni
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Greg Hudson
On 6/10/20 10:32 PM, Harshawardhan Kulkarni wrote:
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/[hidden email]) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?

Unfortunately, it doesn't look like our tools provide any good recovery
options for this case, so I think you're stuck recreating the Kerberos
database.

I will file a ticket that it shouldn't be possible to delete the K/M
principal entry.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Nico Williams
In reply to this post by Harshawardhan Kulkarni
On Thu, Jun 11, 2020 at 03:32:35AM +0100, Harshawardhan Kulkarni wrote:
> I basically need an advice on an ongoing issue I am currently stuck on.
>
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/[hidden email]) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?

If you have a running KDC you could use a debugger to recover that key.
It won't be easy.  It's not something anyone does on a regular basis, so
I don't have instructions to give you.

> I have tried creating with kdb5_util add_mkey but the error says that KDC
> DB is not able to find a master key credential. I assume this would only
> work when you want to create another master key without deleting the
> primary key.

Adding a new key won't help you: the existing records are encrypted in
the old key.

> Another option for me would be to de-kerberise the cluster and create the
> same REALM and kerberise the cluster again. But there could be serious
> issues if this doesn't fix as this is a live cluster where people are using
> this on a daily basis.

You could rebuild your realm, yes.  That's a flag day.  Users in that
realm will need to be re-enrolled, keytabs will need to be re-created
and distributed...

Nico
--
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re[2]: MIT Kerberos Master principal deletion

Chris Hecker

Maybe dump the core of the running process so you don't accidentally
crash it while trying to debug it live?  But that would make finding it
in memory even harder...

Chris


------ Original Message ------
From: "Nico Williams" <[hidden email]>
To: "Harshawardhan Kulkarni" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Sent: 2020-06-11 15:05:19
Subject: Re: MIT Kerberos Master principal deletion

>On Thu, Jun 11, 2020 at 03:32:35AM +0100, Harshawardhan Kulkarni wrote:
>>  I basically need an advice on an ongoing issue I am currently stuck on.
>>
>>  We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
>>  the nodes. We don't have a failover node for KDC server yet. On the KDC
>>  admin server while doing a clean up activity for unwanted kdc principals, I
>>  deleted the master key principal (K/[hidden email]) We never took a kdc dump
>>  of the master key. So we don't have a backup to restore from.
>>
>>  Is there any way I can restore the master key principal?
>
>If you have a running KDC you could use a debugger to recover that key.
>It won't be easy.  It's not something anyone does on a regular basis, so
>I don't have instructions to give you.
>
>>  I have tried creating with kdb5_util add_mkey but the error says that KDC
>>  DB is not able to find a master key credential. I assume this would only
>>  work when you want to create another master key without deleting the
>>  primary key.
>
>Adding a new key won't help you: the existing records are encrypted in
>the old key.
>
>>  Another option for me would be to de-kerberise the cluster and create the
>>  same REALM and kerberise the cluster again. But there could be serious
>>  issues if this doesn't fix as this is a live cluster where people are using
>>  this on a daily basis.
>
>You could rebuild your realm, yes.  That's a flag day.  Users in that
>realm will need to be re-enrolled, keytabs will need to be re-created
>and distributed...
>
>Nico
>--
>________________________________________________
>Kerberos mailing list           [hidden email]
>https://mailman.mit.edu/mailman/listinfo/kerberos


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Nico Williams
On Thu, Jun 11, 2020 at 10:19:39PM +0000, Chris Hecker wrote:
> Maybe dump the core of the running process so you don't accidentally crash
> it while trying to debug it live?  But that would make finding it in memory
> even harder...

I don't think it would make it harder.

BTW, we should make it much harder to delete important principals...
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re[2]: MIT Kerberos Master principal deletion

Chris Hecker

 > I don't think it would make it harder.

I just mean because you won't be able to set a breakpoint at a function
that uses the key, you'll have to actually chase it around in memory
(assuming you use something like gcore to dump it as fast as possible
without regard to where it is executing when it's dumped).

If I was doing this live, I'd set a breakpoint on some function that
used the key to decrypt and then inspect there, but with a core file
you'll need to make sure you can find all the structures first.

Is realm_mkey in the kdc_realm_data struct the one he wants?

Chris

------ Original Message ------
From: "Nico Williams" <[hidden email]>
To: "Chris Hecker" <[hidden email]>
Cc: "Harshawardhan Kulkarni" <[hidden email]>;
"[hidden email]" <[hidden email]>
Sent: 2020-06-11 15:31:28
Subject: Re: MIT Kerberos Master principal deletion

>On Thu, Jun 11, 2020 at 10:19:39PM +0000, Chris Hecker wrote:
>>  Maybe dump the core of the running process so you don't accidentally crash
>>  it while trying to debug it live?  But that would make finding it in memory
>>  even harder...
>
>I don't think it would make it harder.
>
>BTW, we should make it much harder to delete important principals...


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re[3]: MIT Kerberos Master principal deletion

Chris Hecker

Looks like it might also be in the global master_keyblock in the
server_kdb.c file.

Chris


------ Original Message ------
From: "Chris Hecker" <[hidden email]>
To: "Nico Williams" <[hidden email]>
Cc: "Harshawardhan Kulkarni" <[hidden email]>;
"[hidden email]" <[hidden email]>
Sent: 2020-06-11 15:54:32
Subject: Re[2]: MIT Kerberos Master principal deletion

>
> > I don't think it would make it harder.
>
>I just mean because you won't be able to set a breakpoint at a function that uses the key, you'll have to actually chase it around in memory (assuming you use something like gcore to dump it as fast as possible without regard to where it is executing when it's dumped).
>
>If I was doing this live, I'd set a breakpoint on some function that used the key to decrypt and then inspect there, but with a core file you'll need to make sure you can find all the structures first.
>
>Is realm_mkey in the kdc_realm_data struct the one he wants?
>
>Chris
>
>------ Original Message ------
>From: "Nico Williams" <[hidden email]>
>To: "Chris Hecker" <[hidden email]>
>Cc: "Harshawardhan Kulkarni" <[hidden email]>; "[hidden email]" <[hidden email]>
>Sent: 2020-06-11 15:31:28
>Subject: Re: MIT Kerberos Master principal deletion
>
>>On Thu, Jun 11, 2020 at 10:19:39PM +0000, Chris Hecker wrote:
>>>  Maybe dump the core of the running process so you don't accidentally crash
>>>  it while trying to debug it live?  But that would make finding it in memory
>>>  even harder...
>>
>>I don't think it would make it harder.
>>
>>BTW, we should make it much harder to delete important principals...


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: MIT Kerberos Master principal deletion

Jeff D'Angelo
In reply to this post by Chris Hecker
Would the stash file help here (if it exists)?


--

Jeff

________________________________
From: [hidden email] <[hidden email]> on behalf of Chris Hecker <[hidden email]>
Sent: Thursday, June 11, 2020 6:54 PM
To: Nico Williams <[hidden email]>
Cc: Harshawardhan Kulkarni <[hidden email]>; [hidden email] <[hidden email]>
Subject: Re[2]: MIT Kerberos Master principal deletion


 > I don't think it would make it harder.

I just mean because you won't be able to set a breakpoint at a function
that uses the key, you'll have to actually chase it around in memory
(assuming you use something like gcore to dump it as fast as possible
without regard to where it is executing when it's dumped).

If I was doing this live, I'd set a breakpoint on some function that
used the key to decrypt and then inspect there, but with a core file
you'll need to make sure you can find all the structures first.

Is realm_mkey in the kdc_realm_data struct the one he wants?

Chris

------ Original Message ------
From: "Nico Williams" <[hidden email]>
To: "Chris Hecker" <[hidden email]>
Cc: "Harshawardhan Kulkarni" <[hidden email]>;
"[hidden email]" <[hidden email]>
Sent: 2020-06-11 15:31:28
Subject: Re: MIT Kerberos Master principal deletion

>On Thu, Jun 11, 2020 at 10:19:39PM +0000, Chris Hecker wrote:
>>  Maybe dump the core of the running process so you don't accidentally crash
>>  it while trying to debug it live?  But that would make finding it in memory
>>  even harder...
>
>I don't think it would make it harder.
>
>BTW, we should make it much harder to delete important principals...


________________________________________________
Kerberos mailing list           [hidden email]
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C5ecb0ae46a0f4206310108d80e5b131f%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637275131630535798&amp;sdata=slErWkRJAvfE0nd%2BMESCEFY5Ucx8c79mIpMN%2BwFBMz8%3D&amp;reserved=0
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Harshawardhan Kulkarni
Hi Jeff,

I have found the stash file, can I recover the master key from this file? Do you know any good links to follow?

Thanks
Harsh

Sent from my iPhone

> On 16 Jun 2020, at 04:07, D'Angelo, Jeff C <[hidden email]> wrote:
>
> 
> Would the stash file help here (if it exists)?
>
> --
> Jeff
>
> From: [hidden email] <[hidden email]> on behalf of Chris Hecker <[hidden email]>
> Sent: Thursday, June 11, 2020 6:54 PM
> To: Nico Williams <[hidden email]>
> Cc: Harshawardhan Kulkarni <[hidden email]>; [hidden email] <[hidden email]>
> Subject: Re[2]: MIT Kerberos Master principal deletion
>  
>
>  > I don't think it would make it harder.
>
> I just mean because you won't be able to set a breakpoint at a function
> that uses the key, you'll have to actually chase it around in memory
> (assuming you use something like gcore to dump it as fast as possible
> without regard to where it is executing when it's dumped).
>
> If I was doing this live, I'd set a breakpoint on some function that
> used the key to decrypt and then inspect there, but with a core file
> you'll need to make sure you can find all the structures first.
>
> Is realm_mkey in the kdc_realm_data struct the one he wants?
>
> Chris
>
> ------ Original Message ------
> From: "Nico Williams" <[hidden email]>
> To: "Chris Hecker" <[hidden email]>
> Cc: "Harshawardhan Kulkarni" <[hidden email]>;
> "[hidden email]" <[hidden email]>
> Sent: 2020-06-11 15:31:28
> Subject: Re: MIT Kerberos Master principal deletion
>
> >On Thu, Jun 11, 2020 at 10:19:39PM +0000, Chris Hecker wrote:
> >>  Maybe dump the core of the running process so you don't accidentally crash
> >>  it while trying to debug it live?  But that would make finding it in memory
> >>  even harder...
> >
> >I don't think it would make it harder.
> >
> >BTW, we should make it much harder to delete important principals...
>
>
> ________________________________________________
> Kerberos mailing list           [hidden email]
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C5ecb0ae46a0f4206310108d80e5b131f%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637275131630535798&amp;sdata=slErWkRJAvfE0nd%2BMESCEFY5Ucx8c79mIpMN%2BwFBMz8%3D&amp;reserved=0
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Harshawardhan Kulkarni
In reply to this post by Harshawardhan Kulkarni
Hi Team,

I am reaching out back again with my existing issue regarding master key
deletion. I am trying ways to somehow restore it although I don't have a
dump of the KDC.
Re-creating is the last option for me as the cluster is live and a lot of
people are using it.

While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M@REALM principal
details in this file.

So is there any way to restore the master key using this principal.db file
or the .k5.... stash file?

Thanks,
Harsh


On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni <
[hidden email]> wrote:

> Hi Team,
>
> I basically need an advice on an ongoing issue I am currently stuck on.
>
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/[hidden email]) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?
>
> I have tried creating with kdb5_util add_mkey but the error says that KDC
> DB is not able to find a master key credential. I assume this would only
> work when you want to create another master key without deleting the
> primary key.
>
> Another option for me would be to de-kerberise the cluster and create the
> same REALM and kerberise the cluster again. But there could be serious
> issues if this doesn't fix as this is a live cluster where people are using
> this on a daily basis.
>
> Can anyone help me here? Looking forward for your reply.
>
> Thanks,
> Harsh Kulkarni
>
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Jeff D'Angelo
So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this.  I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try.

I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state.  This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump".  For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump).

Then I created a shorter dump file of just the header and K/M entry using grep [1]:
sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore'

  [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files.

Make sure it's just those two lines:
sudo cat kdb5.dump.km_restore


Then do a kdb5_util incremental (-update) load with that file:

sudo kdb5_util load -update kdb5.dump.km_restore


Surprisingly, it worked.  I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one.


Harsh, what version kerb are you running?


Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db.  I'm not sure yet what loading a different K/M would do.


--

Jeff

________________________________
From: [hidden email] <[hidden email]> on behalf of Harshawardhan Kulkarni <[hidden email]>
Sent: Thursday, June 18, 2020 6:27 PM
To: [hidden email] <[hidden email]>
Subject: Re: MIT Kerberos Master principal deletion

Hi Team,

I am reaching out back again with my existing issue regarding master key
deletion. I am trying ways to somehow restore it although I don't have a
dump of the KDC.
Re-creating is the last option for me as the cluster is live and a lot of
people are using it.

While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M@REALM principal
details in this file.

So is there any way to restore the master key using this principal.db file
or the .k5.... stash file?

Thanks,
Harsh


On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni <
[hidden email]> wrote:

> Hi Team,
>
> I basically need an advice on an ongoing issue I am currently stuck on.
>
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/[hidden email]) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?
>
> I have tried creating with kdb5_util add_mkey but the error says that KDC
> DB is not able to find a master key credential. I assume this would only
> work when you want to create another master key without deleting the
> primary key.
>
> Another option for me would be to de-kerberise the cluster and create the
> same REALM and kerberise the cluster again. But there could be serious
> issues if this doesn't fix as this is a live cluster where people are using
> this on a daily basis.
>
> Can anyone help me here? Looking forward for your reply.
>
> Thanks,
> Harsh Kulkarni
>
________________________________________________
Kerberos mailing list           [hidden email]
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&amp;sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&amp;reserved=0
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Harshawardhan Kulkarni
Hi Jeff,

Many thanks for giving this a try. Really appreciated :)

Great that it has worked for you. I never thought of this way. But, I was
not able to take the dump from the database (principal) file. Maybe I
cannot take the dump as the master principal (K/M) is already deleted.
I was not able to proceed beyond this step.

My Kerberos version is (Kerberos 5 version 1.12.5),
Below is the error I get,

kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump
kdb5_util: No such entry in the database while retrieving master entry

Did you create the dump after deleting the K/M principal? And were you able
to create a dump from the .db file?
You have clearly depicted my current situation but, I am not sure if this
is due to the Kerberos version that i am not able to create the dump file
using the principal.db file.

Thanks again,
Harsh

On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C <[hidden email]> wrote:

> So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux
> box seemed to find a possible solution to this.  I'd like to hear from the
> veterans if this is a good idea or not as I can guess doing this wrong may
> make things worse before I offer it as a suggestion to try.
>
> I deleted the K/M principal on a test database (note there's a speed bump
> in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS
> attribute prevents casual deletion over kadmin/kadmind and one would need
> kadmin.local to bypass it, so I used kadmin.local to `modprinc
> -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind
> and krb5kdc running, which is what I expect matches Harsh's state.  This is
> after I already made backup dump of the database using kdb5_util; let's
> call that file "kdb5.dump".  For Harsh, I'd be he'd also need to make a
> dump of the original db file before continuing (kdb5_util dump -d
> /path/to/old/var/krb5kdc/principal krb5.dump).
>
> Then I created a shorter dump file of just the header and K/M entry using
> grep [1]:
> sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore'
>
>   [1] Adding the sudo step here for when you are running a non root shell
> in a normal environment that has root ownership restrictions over the db
> and dump files.
>
> Make sure it's just those two lines:
> sudo cat kdb5.dump.km_restore
>
>
> Then do a kdb5_util incremental (-update) load with that file:
>
> sudo kdb5_util load -update kdb5.dump.km_restore
>
>
> Surprisingly, it worked.  I guess kdb5_util load would use the K/M it
> finds in the dump file instead of the living "principal" database file
> because it needs to handle the case that it is creating a brand new
> database and/or blowing out an exiting one.
>
>
> Harsh, what version kerb are you running?
>
>
> Disclaimer: This presumes you haven't changed (rekeyed) K/M since you
> created your database (well really since you made that backup copy) and
> that you are really sure that backup copy was from an earlier date of this
> existing db.  I'm not sure yet what loading a different K/M would do.
>
>
> --
>
> Jeff
>
> ------------------------------
> *From:* [hidden email] <[hidden email]> on behalf of
> Harshawardhan Kulkarni <[hidden email]>
> *Sent:* Thursday, June 18, 2020 6:27 PM
> *To:* [hidden email] <[hidden email]>
> *Subject:* Re: MIT Kerberos Master principal deletion
>
> Hi Team,
>
> I am reaching out back again with my existing issue regarding master key
> deletion. I am trying ways to somehow restore it although I don't have a
> dump of the KDC.
> Re-creating is the last option for me as the cluster is live and a lot of
> people are using it.
>
> While going through all the KDC related files, I came across all the files
> which get created while the kdc database was created for the first time.
> I was wondering is there any way to restore the master key using either the
> stash file? or either using the database file which resides in the
> /var/log/kerberos/krb5kdc location?
> We have both the stash files and the principal.db file. When I open the
> file although it's not text readable, I can see the K/M@REALM principal
> details in this file.
>
> So is there any way to restore the master key using this principal.db file
> or the .k5.... stash file?
>
> Thanks,
> Harsh
>
>
> On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni <
> [hidden email]> wrote:
>
> > Hi Team,
> >
> > I basically need an advice on an ongoing issue I am currently stuck on.
> >
> > We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one
> of
> > the nodes. We don't have a failover node for KDC server yet. On the KDC
> > admin server while doing a clean up activity for unwanted kdc
> principals, I
> > deleted the master key principal (K/[hidden email]) We never took a kdc
> dump
> > of the master key. So we don't have a backup to restore from.
> >
> > Is there any way I can restore the master key principal?
> >
> > I have tried creating with kdb5_util add_mkey but the error says that KDC
> > DB is not able to find a master key credential. I assume this would only
> > work when you want to create another master key without deleting the
> > primary key.
> >
> > Another option for me would be to de-kerberise the cluster and create the
> > same REALM and kerberise the cluster again. But there could be serious
> > issues if this doesn't fix as this is a live cluster where people are
> using
> > this on a daily basis.
> >
> > Can anyone help me here? Looking forward for your reply.
> >
> > Thanks,
> > Harsh Kulkarni
> >
> ________________________________________________
> Kerberos mailing list           [hidden email]
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&amp;sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&amp;reserved=0
>
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Jeff D'Angelo
Sorry, I must have misunderstood you.  When you said:
While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M@REALM principal
details in this file.
I thought you said you found another old copy of the principal database file in /var/log/kerberos/krb5kdc (or was that a typo for where the prod db lives in /var/lib/kerberos/krb5kdc?) from before the deletion of K/M occurred.  If you did, that would make this easier as it seems existing tools could remedy the problem (dump the K/M entry from that other DB and then load -update into your prod one).

The trick is to make sure it's the correct K/M with the original key you had, but I have some suggestions on how to test that before you affect prod.  If you happened to find the original password you used to create the database, that would also work.  Heck, even if you only have some guesses as to what the original password was, that could also help.

But still, finding the stash file (which is presumably the correct one) is a big help and avoids needing to hunt it down out of a core file.


--

Jeff

________________________________
From: Harshawardhan Kulkarni <[hidden email]>
Sent: Friday, June 19, 2020 1:16 PM
To: D'Angelo, Jeff C <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: MIT Kerberos Master principal deletion

Hi Jeff,

Many thanks for giving this a try. Really appreciated :)

Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted.
I was not able to proceed beyond this step.

My Kerberos version is (Kerberos 5 version 1.12.5),
Below is the error I get,

kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump
kdb5_util: No such entry in the database while retrieving master entry

Did you create the dump after deleting the K/M principal? And were you able to create a dump from the .db file?
You have clearly depicted my current situation but, I am not sure if this is due to the Kerberos version that i am not able to create the dump file using the principal.db file.

Thanks again,
Harsh

On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C <[hidden email]<mailto:[hidden email]>> wrote:
So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this.  I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try.

I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state.  This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump".  For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump).

Then I created a shorter dump file of just the header and K/M entry using grep [1]:
sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore'

  [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files.

Make sure it's just those two lines:
sudo cat kdb5.dump.km_restore


Then do a kdb5_util incremental (-update) load with that file:

sudo kdb5_util load -update kdb5.dump.km_restore


Surprisingly, it worked.  I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one.


Harsh, what version kerb are you running?


Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db.  I'm not sure yet what loading a different K/M would do.


--

Jeff

________________________________
From: [hidden email]<mailto:[hidden email]> <[hidden email]<mailto:[hidden email]>> on behalf of Harshawardhan Kulkarni <[hidden email]<mailto:[hidden email]>>
Sent: Thursday, June 18, 2020 6:27 PM
To: [hidden email]<mailto:[hidden email]> <[hidden email]<mailto:[hidden email]>>
Subject: Re: MIT Kerberos Master principal deletion

Hi Team,

I am reaching out back again with my existing issue regarding master key
deletion. I am trying ways to somehow restore it although I don't have a
dump of the KDC.
Re-creating is the last option for me as the cluster is live and a lot of
people are using it.

While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M@REALM principal
details in this file.

So is there any way to restore the master key using this principal.db file
or the .k5.... stash file?

Thanks,
Harsh


On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni <
[hidden email]<mailto:[hidden email]>> wrote:

> Hi Team,
>
> I basically need an advice on an ongoing issue I am currently stuck on.
>
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/[hidden email]<mailto:[hidden email]>) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?
>
> I have tried creating with kdb5_util add_mkey but the error says that KDC
> DB is not able to find a master key credential. I assume this would only
> work when you want to create another master key without deleting the
> primary key.
>
> Another option for me would be to de-kerberise the cluster and create the
> same REALM and kerberise the cluster again. But there could be serious
> issues if this doesn't fix as this is a live cluster where people are using
> this on a daily basis.
>
> Can anyone help me here? Looking forward for your reply.
>
> Thanks,
> Harsh Kulkarni
>
________________________________________________
Kerberos mailing list           [hidden email]<mailto:[hidden email]>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&amp;sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&amp;reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C24638044c9484278c86008d81474972a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281838273451942&sdata=nq3dpBcY%2BsC2gMW9bWw%2FnPMQiVkmz7RlYQfzQz48N3s%3D&reserved=0>
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: MIT Kerberos Master principal deletion

Jeff D'Angelo
In reply to this post by Harshawardhan Kulkarni
A couple points:
Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted.
I was not able to proceed beyond this step.
Same for me in 1.17.  It seems that the following actions fail when K/M is missing from the db:
(1) A running kadmind cannot ank (create prinicipals) nor cpw (change passwords [1]), but at least other read and metadata modification functions still work (the old mkey is still in memory)
(2) A running krb5kdc will continue to issue tickets (the old mkey is still in memory)
(3) kadmind cannot be [re]started
(4) krb5kdc cannot be [re]started
(5) kadmin.local cannot open the database
(6) kdb5_util dump will fail the same way you saw

  [1] I got a misleading error (yes, I had permission in kadm5.acl): change_password: Operation requires ``change-password'' privilege while changing password for "[hidden email]".

As I mentioned earlier, I was able to re-add K/M from a previous dump with [2]:
kdb5_util load -update file_with_just_km.dump

 [2] And this step doesn't seem to require you to have the correct stash file in place yet, but good to correct as an immediately next or previous step.

If it was the correct K/M (and I presume you still have the correct stash file), then life was seemingly back to normal.

I also found that when you use an "imposter" K/M (created from a different password/key) but with matching principal name, realm, enctypes and kvno (not sure those all need to match, necessarily, to achieve these results), and replace the stash file to match, I was able to do the following:
(1) kdb5_util dump - notably, only the K/M entry was different.  The other entries matched what I had before in a previous dump, so presumably still use the same encryption they had before with the old key; in other words, the current K/M does not appear to be used to decrypt or encrypt passwords as this point [3], but the content of the db file are simply dumped in a mostly readable text format. [4]
(2) No effect I saw on running krb5kdc [5]
(3) kadmind would again permit kadmin clients to ank and cpw again, however, they use the new master key to encrypt not the old, so restoring a database with mixed master keys at the same kvno is likely a recipe for further pain; I might next test using a different kvno (simulating a normal master key rotation) to see how that fares
(4) krb5kdc will [re]start again, but fail to issue tickets; kinit attempts are met with an immediate and slightly misleading but arguably technically correct error message "kinit: Password incorrect while getting initial credentials" (before I even get a chance to enter the password)
(5) kadmind will start again; I presume some functions may fail at this point such as `kadmin: ktadd -norandkey` due to the master key mismatch

  [3] Unless you used an option like -mkey_convert

  [4] If I left the stash file mismatched, it would still dump, but give the warning:
kdb5_util: Unable to decrypt latest master key with the provided master key
 while getting master key list
kdb5_util: Warning: proceeding without master key list

  [5] Not that I tested more than kinits.  I'm sure it might be worth testing the effect on updates to "Last successful authentication" and "Last failed authentication", but I presume that will succeed like a `kadmin: modprinc -expire now` did for me.

So it would seem a recovery procedure for you would best:
(1) first reclaim the correct K/M entry (or your guess at one) into a dump file
(2) verify it is the correct entry with a test database, probably best on another system so you don't accidentally break your operational server
(3) perhaps [temporarily?] load the correct (or an imposter) K/M onto your operation DB so you can make a dump for emergency backup purpose as well as to use to test on your test db; maybe delete it again so to avoid ank or cpw calls if you can't stop them otherwise
(4) See if you can fire up krb5kdc on your test system with a database loaded from prod and kinit against it with a known principal

Or something like it.  Again, it may be best to test the full scenario on your version of software before assuming it will work for you.

As for (1) reclaim the correct K/M entry, I wonder how hard it would be to extend `krb5_util create` to permit using an existing key in a keytab (or stash) file as alternative to the usual password prompt.  Otherwise, we may be getting into other esoterica like how to convert a keytab (stash) file to a kdb5_util load_dump format.

But if you can find an old copy of the database file, or guesses at the original password, that would also help.


--

Jeff

________________________________
From: Harshawardhan Kulkarni <[hidden email]>
Sent: Friday, June 19, 2020 1:16 PM
To: D'Angelo, Jeff C <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: MIT Kerberos Master principal deletion

Hi Jeff,

Many thanks for giving this a try. Really appreciated :)

Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted.
I was not able to proceed beyond this step.

My Kerberos version is (Kerberos 5 version 1.12.5),
Below is the error I get,

kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump
kdb5_util: No such entry in the database while retrieving master entry

Did you create the dump after deleting the K/M principal? And were you able to create a dump from the .db file?
You have clearly depicted my current situation but, I am not sure if this is due to the Kerberos version that i am not able to create the dump file using the principal.db file.

Thanks again,
Harsh

On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C <[hidden email]<mailto:[hidden email]>> wrote:
So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this.  I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try.

I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state.  This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump".  For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump).

Then I created a shorter dump file of just the header and K/M entry using grep [1]:
sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore'

  [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files.

Make sure it's just those two lines:
sudo cat kdb5.dump.km_restore


Then do a kdb5_util incremental (-update) load with that file:

sudo kdb5_util load -update kdb5.dump.km_restore


Surprisingly, it worked.  I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one.


Harsh, what version kerb are you running?


Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db.  I'm not sure yet what loading a different K/M would do.


--

Jeff

________________________________
From: [hidden email]<mailto:[hidden email]> <[hidden email]<mailto:[hidden email]>> on behalf of Harshawardhan Kulkarni <[hidden email]<mailto:[hidden email]>>
Sent: Thursday, June 18, 2020 6:27 PM
To: [hidden email]<mailto:[hidden email]> <[hidden email]<mailto:[hidden email]>>
Subject: Re: MIT Kerberos Master principal deletion

Hi Team,

I am reaching out back again with my existing issue regarding master key
deletion. I am trying ways to somehow restore it although I don't have a
dump of the KDC.
Re-creating is the last option for me as the cluster is live and a lot of
people are using it.

While going through all the KDC related files, I came across all the files
which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the
stash file? or either using the database file which resides in the
/var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M@REALM principal
details in this file.

So is there any way to restore the master key using this principal.db file
or the .k5.... stash file?

Thanks,
Harsh


On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni <
[hidden email]<mailto:[hidden email]>> wrote:

> Hi Team,
>
> I basically need an advice on an ongoing issue I am currently stuck on.
>
> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of
> the nodes. We don't have a failover node for KDC server yet. On the KDC
> admin server while doing a clean up activity for unwanted kdc principals, I
> deleted the master key principal (K/[hidden email]<mailto:[hidden email]>) We never took a kdc dump
> of the master key. So we don't have a backup to restore from.
>
> Is there any way I can restore the master key principal?
>
> I have tried creating with kdb5_util add_mkey but the error says that KDC
> DB is not able to find a master key credential. I assume this would only
> work when you want to create another master key without deleting the
> primary key.
>
> Another option for me would be to de-kerberise the cluster and create the
> same REALM and kerberise the cluster again. But there could be serious
> issues if this doesn't fix as this is a live cluster where people are using
> this on a daily basis.
>
> Can anyone help me here? Looking forward for your reply.
>
> Thanks,
> Harsh Kulkarni
>
________________________________________________
Kerberos mailing list           [hidden email]<mailto:[hidden email]>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&amp;data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&amp;sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&amp;reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C24638044c9484278c86008d81474972a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281838273451942&sdata=nq3dpBcY%2BsC2gMW9bWw%2FnPMQiVkmz7RlYQfzQz48N3s%3D&reserved=0>
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos