kprop with multiple or NATted IP address

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

kprop with multiple or NATted IP address

Jerry Shipman
Hello,

I’m trying to set up an additional slave KDC in a new location (different network), and I’m having trouble kprop’ing the database.

There is some tricky networking / routing going on between the network where the master KDC is and the network where the slave will be, that I am in the situation of needing to work with.

I can go into that more if necessary, but I think the salient point is that each machine has multiple network interfaces, one with a public IP and one with a private IP (10.x.y.z). I am trying to use the private IPs when I kprop the database to the slave. (I am convinced that I eventually got this working with an iptables postrouting snat rule; I see the 10space address in logs, etc.)

I am seeing this error on the slave when I try to push the database from the master:
  kpropd: Incorrect net address while decoding database size from client
From the master side, it looks like:
  kprop: Connection reset by peer while sending database block starting at 0

I think that kpropd is trying to look up the hostname of the master in DNS, and seeing the public IP, instead of the private IP which the connection is coming from, and then aborting because of that mismatch (or something like that).
On a lark I tried adding the master’s hostname with its private address to /etc/hosts on the slave, but it didn’t immediately seem to help.

Is there a way to do what I’m trying to do?
Or, is there a reason that it is dangerous to avoid verifying that IP match, and I shouldn’t try to work around it?

Thank you for your help,
Jerry Shipman


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Benjamin Kaduk-2
On Wed, 23 Dec 2015, Jerry Shipman wrote:

> I think that kpropd is trying to look up the hostname of the master in DNS, and seeing the public IP, instead of the private IP which the connection is coming from, and then aborting because of that mismatch (or something like that).
> On a lark I tried adding the master’s hostname with its private address to /etc/hosts on the slave, but it didn’t immediately seem to help.

Did you try setting rdns = false in the [libdefaults] of the krb5.conf on
both machines?  (You did not specify which version(s) of krb5 were
involved; that features is somewhat new.)

-Ben
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Greg Hudson
In reply to this post by Jerry Shipman
On 12/23/2015 03:50 PM, Jerry Shipman wrote:
> Is there a way to do what I’m trying to do?
> Or, is there a reason that it is dangerous to avoid verifying that IP match, and I shouldn’t try to work around it?

The only really useful purpose of checking addresses is preventing
reflection attacks, where an attacker takes a KRB-PRIV or KRB-SAFE
message from one of the parties and send it back to them as if it came
from the other party.  Many protocols aren't susceptible to reflection
attacks because they don't use similar formats for requests and
responses.  After verifying that the kprop protocol isn't vulnerable, we
could probably make changes similar to the ones we made to kpasswd to
allow it to work over NATs.

(Protocols using GSS don't have this problem because GSS tokens only use
direction bits, not addresses.  Well, unless they use IP address channel
bindings, which isn't common.)
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

RE: kprop with multiple or NATted IP address

Luca Rea
In reply to this post by Jerry Shipman
Hi,
you can use dnsmasq to resolv the local hostname correctly and forward the other requests to DNS.

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Jerry Shipman
In reply to this post by Greg Hudson
Hello,

It’s me again, who was trying to kprop through a NAT a month ago.

Hypothetically speaking… how bad of an idea would it be to make a cron job that `scp`s the database file to the slave KDC, or something like that? Does the slave KDC daemon need to restart after the file is updated, maybe? Or is this significantly less safe than using kprop? I think I would be relying on ssh instead of kerberos for the confidentiality and integrity. But I do that whenever I log into the machine anyway. I think I may risk getting the file in the middle of a write (so some records could be corrupted in the copy). It seems like this would be a bad idea; just checking.

Thanks again,
Jerry




> On Dec 24, 2015, at 12:21 AM, Greg Hudson <[hidden email]> wrote:
>
> On 12/23/2015 03:50 PM, Jerry Shipman wrote:
>> Is there a way to do what I’m trying to do?
>> Or, is there a reason that it is dangerous to avoid verifying that IP match, and I shouldn’t try to work around it?
>
> The only really useful purpose of checking addresses is preventing
> reflection attacks, where an attacker takes a KRB-PRIV or KRB-SAFE
> message from one of the parties and send it back to them as if it came
> from the other party.  Many protocols aren't susceptible to reflection
> attacks because they don't use similar formats for requests and
> responses.  After verifying that the kprop protocol isn't vulnerable, we
> could probably make changes similar to the ones we made to kpasswd to
> allow it to work over NATs.
>
> (Protocols using GSS don't have this problem because GSS tokens only use
> direction bits, not addresses.  Well, unless they use IP address channel
> bindings, which isn't common.)

> On Dec 23, 2015, at 3:50 PM, jes59 <[hidden email]> wrote:
>
> Hello,
>
> I’m trying to set up an additional slave KDC in a new location (different network), and I’m having trouble kprop’ing the database.
>
> There is some tricky networking / routing going on between the network where the master KDC is and the network where the slave will be, that I am in the situation of needing to work with.
>
> I can go into that more if necessary, but I think the salient point is that each machine has multiple network interfaces, one with a public IP and one with a private IP (10.x.y.z). I am trying to use the private IPs when I kprop the database to the slave. (I am convinced that I eventually got this working with an iptables postrouting snat rule; I see the 10space address in logs, etc.)
>
> I am seeing this error on the slave when I try to push the database from the master:
>  kpropd: Incorrect net address while decoding database size from client
> From the master side, it looks like:
>  kprop: Connection reset by peer while sending database block starting at 0
>
> I think that kpropd is trying to look up the hostname of the master in DNS, and seeing the public IP, instead of the private IP which the connection is coming from, and then aborting because of that mismatch (or something like that).
> On a lark I tried adding the master’s hostname with its private address to /etc/hosts on the slave, but it didn’t immediately seem to help.
>
> Is there a way to do what I’m trying to do?
> Or, is there a reason that it is dangerous to avoid verifying that IP match, and I shouldn’t try to work around it?
>
> Thank you for your help,
> Jerry Shipman

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Russ Allbery-2
Jerry Shipman <[hidden email]> writes:

> It’s me again, who was trying to kprop through a NAT a month ago.

> Hypothetically speaking… how bad of an idea would it be to make a cron
> job that `scp`s the database file to the slave KDC, or something like
> that? Does the slave KDC daemon need to restart after the file is
> updated, maybe? Or is this significantly less safe than using kprop? I
> think I would be relying on ssh instead of kerberos for the
> confidentiality and integrity. But I do that whenever I log into the
> machine anyway. I think I may risk getting the file in the middle of a
> write (so some records could be corrupted in the copy). It seems like
> this would be a bad idea; just checking.

If you're going to use scp, I strongly recommend generating a dump with
kdb5_util dump, scping that, and then loading it with kdb5_util load.
That's effectively what kprop/kpropd do.

Just copying the database file runs the risk of copying a corrupt database
because you happened to catch it in the middle of a write, as you note.

--
Russ Allbery ([hidden email])              <http://www.eyrie.org/~eagle/>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Jerry Shipman
(I thought about that about 5 minutes after I sent the email — oops.)
I guess my question is: does kprop do anything other than: secrecy of the data in transmission, integrity of the transmission, kdb5_util dump/load ? Or can I really do the same thing in a cron job (or maybe 2, one on each end) without missing anything important? I guess I would lose out on the possibility of doing incremental propagation.

Thanks again,
Jerry

> On Jan 27, 2016, at 6:43 PM, Russ Allbery <[hidden email]> wrote:
>
> Jerry Shipman <[hidden email]> writes:
>
>> It’s me again, who was trying to kprop through a NAT a month ago.
>
>> Hypothetically speaking… how bad of an idea would it be to make a cron
>> job that `scp`s the database file to the slave KDC, or something like
>> that? Does the slave KDC daemon need to restart after the file is
>> updated, maybe? Or is this significantly less safe than using kprop? I
>> think I would be relying on ssh instead of kerberos for the
>> confidentiality and integrity. But I do that whenever I log into the
>> machine anyway. I think I may risk getting the file in the middle of a
>> write (so some records could be corrupted in the copy). It seems like
>> this would be a bad idea; just checking.
>
> If you're going to use scp, I strongly recommend generating a dump with
> kdb5_util dump, scping that, and then loading it with kdb5_util load.
> That's effectively what kprop/kpropd do.
>
> Just copying the database file runs the risk of copying a corrupt database
> because you happened to catch it in the middle of a write, as you note.
>
> --
> Russ Allbery ([hidden email])              <http://www.eyrie.org/~eagle/>


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Russ Allbery-2
Jerry Shipman <[hidden email]> writes:

> (I thought about that about 5 minutes after I sent the email — oops.)

> I guess my question is: does kprop do anything other than: secrecy of
> the data in transmission, integrity of the transmission, kdb5_util
> dump/load ? Or can I really do the same thing in a cron job (or maybe 2,
> one on each end) without missing anything important? I guess I would
> lose out on the possibility of doing incremental propagation.

You lose incremental propagation, but other than that, I'm pretty sure
kprop/kpropd is just an authenticated copy of a dump and loading it on the
other end.

--
Russ Allbery ([hidden email])              <http://www.eyrie.org/~eagle/>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Tom Yu
Russ Allbery <[hidden email]> writes:

> Jerry Shipman <[hidden email]> writes:
>
>> (I thought about that about 5 minutes after I sent the email — oops.)
>
>> I guess my question is: does kprop do anything other than: secrecy of
>> the data in transmission, integrity of the transmission, kdb5_util
>> dump/load ? Or can I really do the same thing in a cron job (or maybe 2,
>> one on each end) without missing anything important? I guess I would
>> lose out on the possibility of doing incremental propagation.
>
> You lose incremental propagation, but other than that, I'm pretty sure
> kprop/kpropd is just an authenticated copy of a dump and loading it on the
> other end.

The existence of kprop as an independent Kerberos-authenticated service
probably has its roots in a few historical factors that might no longer
be relevant for some deployments.  (I could be misremembering some of
these.)

The krb4 rcp program did not originally provide any encryption of the
file contents.  Neither did the krb4 rsh program that the rcp program
relies on.  These were less of a factor for krb5, but kprop remained an
independent program anyway.

Some particularly cautious operators wanted a minimum amount of attack
surface in a program that handles Kerberos database dumps.  The rcp
program required using rsh, a general-purpose remote shell program.
Also, there was not originally a capability to restrict which commands
the rsh daemon could execute for a given principal.

Having a special-purpose kprop program helps mitigate these risks.  This
program could also be written to avoid ever invoking a general-purpose
shell by hardcoding the names of the programs it runs.

The scp and ssh software consist of considerably more code than
Kerberos-enabled rcp and rsh, so they have a larger attack surface.  You
could reasonably decide that this is an acceptable risk in your
environment.

-Tom

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Jerry Shipman
In reply to this post by Greg Hudson
I am continuing (sorry) my old 2016 thread (part of it below) about trying to kprop through a NAT.

I have some secondary KDCs in a different network than the primary, with a NAT in the way, and was having trouble getting kprop to work (it doesn't like the mismatch between the IP from the hostname dns lookup and the IP in the getsockname(), or something like that).

I wound up working around it by periodically dumping the database, copying it over to the secondaries, and importing it.
But this workaround is causing some other trouble (the master database is locked for a noticeable amount of time during the periodic exports).

Can you tell me if there has already been, or if there might be in the near future, a plan to update kprop to let it work (maybe with a command line switch or something) through the NAT, so that I can do incremental propagation through the NAT?

Or, maybe there is some way that I can fake out the name resolution (using /etc/hosts or dnsmasq or something) to make it work -- is that a reasonable thing to try?

Or, one of my co-workers suggested that I make a cron job to scan for recent password changes and dump/import just those, periodically (instead of doing the full database dumps). I can do that and I guess it would work... but it would be nice to not do that.

Or is there some better idea that we didn't think of?

Thank you again for your help,
Jerry


-----Original Message-----
From: Greg Hudson <[hidden email]>
Date: Thursday, December 24, 2015 at 12:21 AM
To: "Jeremiah E. Shipman" <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: kprop with multiple or NATted IP address

On 12/23/2015 03:50 PM, Jerry Shipman wrote:
> Is there a way to do what I’m trying to do?
> Or, is there a reason that it is dangerous to avoid verifying that IP match, and I shouldn’t try to work around it?

The only really useful purpose of checking addresses is preventing
reflection attacks, where an attacker takes a KRB-PRIV or KRB-SAFE
message from one of the parties and send it back to them as if it came
from the other party.  Many protocols aren't susceptible to reflection
attacks because they don't use similar formats for requests and
responses.  After verifying that the kprop protocol isn't vulnerable, we
could probably make changes similar to the ones we made to kpasswd to
allow it to work over NATs.

(Protocols using GSS don't have this problem because GSS tokens only use
direction bits, not addresses.  Well, unless they use IP address channel
bindings, which isn't common.)


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Greg Hudson
On 1/3/20 11:00 AM, Jerry Shipman wrote:
> I am continuing (sorry) my old 2016 thread (part of it below) about trying to kprop through a NAT.

Apologies that I didn't follow up on that.  In that thread, I wrote:

> Many protocols aren't susceptible to reflection
> attacks because they don't use similar formats for requests and
> responses.  After verifying that the kprop protocol isn't vulnerable,
> we could probably make changes similar to the ones we made to kpasswd
> to allow it to work over NATs.

I took at look at the kprop protocol just now.  Unfortunately, kprop and
kpropd use  the same format for the client-to-server and
server-to-client KRB-SAFE messages, and on successful completion the
payloads are expected to be identical.  (KRB-PRIV messages only flow
from client to server.)  So without protection from reflection attacks,
an attacker could potentially make a failed transmission look like a
successful transmission by reflecting the client's KRB-SAFE message.

I think that sequence numbers would generally thwart this attack, since
kprop and kpropd use mutual authentication and enable sequence number
checking.  But I would have to do some additional analysis to be
confident about that; these sequence numbers are only 32 bits, and there
is some fudging around past implementations which further reduces the
margin of safety.

> I wound up working around it by periodically dumping the database, copying it over to the secondaries, and importing it.
> But this workaround is causing some other trouble (the master database is locked for a noticeable amount of time during the periodic exports).

Fortunately, we have a couple of workarounds for this secondary problem.

Release 1.13 added support for unlocked database dumps with the db2 KDB
module: "kdb5_util -x unlockiter dump".  This will make the dump command
take longer, but not keep the database locked.

Release 1.17 added the lmdb KDB module, which does not suffer from
locking conflicts between dump operations and write operations.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Jerry Shipman
Aha! This (-x unlockiter) looks like it will solve my immediate problem. Thanks a lot.

Happy new year!
Jerry

-----Original Message-----
From: Greg Hudson <[hidden email]>
Date: Friday, January 3, 2020 at 11:53 AM
To: "Jeremiah E. Shipman" <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: kprop with multiple or NATted IP address

On 1/3/20 11:00 AM, Jerry Shipman wrote:
> I am continuing (sorry) my old 2016 thread (part of it below) about trying to kprop through a NAT.

Apologies that I didn't follow up on that.  In that thread, I wrote:

> Many protocols aren't susceptible to reflection
> attacks because they don't use similar formats for requests and
> responses.  After verifying that the kprop protocol isn't vulnerable,
> we could probably make changes similar to the ones we made to kpasswd
> to allow it to work over NATs.

I took at look at the kprop protocol just now.  Unfortunately, kprop and
kpropd use  the same format for the client-to-server and
server-to-client KRB-SAFE messages, and on successful completion the
payloads are expected to be identical.  (KRB-PRIV messages only flow
from client to server.)  So without protection from reflection attacks,
an attacker could potentially make a failed transmission look like a
successful transmission by reflecting the client's KRB-SAFE message.

I think that sequence numbers would generally thwart this attack, since
kprop and kpropd use mutual authentication and enable sequence number
checking.  But I would have to do some additional analysis to be
confident about that; these sequence numbers are only 32 bits, and there
is some fudging around past implementations which further reduces the
margin of safety.

> I wound up working around it by periodically dumping the database, copying it over to the secondaries, and importing it.
> But this workaround is causing some other trouble (the master database is locked for a noticeable amount of time during the periodic exports).

Fortunately, we have a couple of workarounds for this secondary problem.

Release 1.13 added support for unlocked database dumps with the db2 KDB
module: "kdb5_util -x unlockiter dump".  This will make the dump command
take longer, but not keep the database locked.

Release 1.17 added the lmdb KDB module, which does not suffer from
locking conflicts between dump operations and write operations.


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Jeffrey Hutzelman
In reply to this post by Greg Hudson
Rather than making complex changes to the protocol, why not switch to directional addresses? Certainly the client and server would have to agree on this, but for kprop, a command-line switch would be sufficient.


-- Jeff


________________________________
From: [hidden email] <[hidden email]> on behalf of Greg Hudson <[hidden email]>
Sent: Friday, January 3, 2020 11:53 AM
To: Jerry Shipman; [hidden email]
Subject: Re: kprop with multiple or NATted IP address

On 1/3/20 11:00 AM, Jerry Shipman wrote:
> I am continuing (sorry) my old 2016 thread (part of it below) about trying to kprop through a NAT.

Apologies that I didn't follow up on that.  In that thread, I wrote:

> Many protocols aren't susceptible to reflection
> attacks because they don't use similar formats for requests and
> responses.  After verifying that the kprop protocol isn't vulnerable,
> we could probably make changes similar to the ones we made to kpasswd
> to allow it to work over NATs.

I took at look at the kprop protocol just now.  Unfortunately, kprop and
kpropd use  the same format for the client-to-server and
server-to-client KRB-SAFE messages, and on successful completion the
payloads are expected to be identical.  (KRB-PRIV messages only flow
from client to server.)  So without protection from reflection attacks,
an attacker could potentially make a failed transmission look like a
successful transmission by reflecting the client's KRB-SAFE message.

I think that sequence numbers would generally thwart this attack, since
kprop and kpropd use mutual authentication and enable sequence number
checking.  But I would have to do some additional analysis to be
confident about that; these sequence numbers are only 32 bits, and there
is some fudging around past implementations which further reduces the
margin of safety.

> I wound up working around it by periodically dumping the database, copying it over to the secondaries, and importing it.
> But this workaround is causing some other trouble (the master database is locked for a noticeable amount of time during the periodic exports).

Fortunately, we have a couple of workarounds for this secondary problem.

Release 1.13 added support for unlocked database dumps with the db2 KDB
module: "kdb5_util -x unlockiter dump".  This will make the dump command
take longer, but not keep the database locked.

Release 1.17 added the lmdb KDB module, which does not suffer from
locking conflicts between dump operations and write operations.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kprop with multiple or NATted IP address

Greg Hudson
On 1/3/20 1:06 PM, Jeffrey T. Hutzelman wrote:
> Rather than making complex changes to the protocol, why not switch to directional addresses? Certainly the client and server would have to agree on this, but for kprop, a command-line switch would be sufficient.

I was considering a change like
https://github.com/krb5/krb5/commit/b91da5a4c7efc189dcfe57c4de2a8e8673102295which
is only complicated in the analysis.  And on further consideration,
removing kpropd's check of the client address should clearly be
safe--kpropd only receives one KRB-SAFE message, before it sends
anything to the client.

We never implemented directional addresses.  It's possible that they
would be trivial to implement.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos