Fwd: [Bug 1179820] New: Kerberos KDC connection limit too low

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: [Bug 1179820] New: Kerberos KDC connection limit too low

Roland Mainz

Hi!

----

Does anyone know which limit the reporter in the bug report below may be referring to ?

AFAIK we have the following default limits in krb5kdc when running on Linux:
- 1 process (by default)
- max. 45 connections ([1]) as defined via |max_tcp_or_rpc_data_connections| in ./krb5/src/lib/apputils/net-server.c (per process, right ?)
- 1024 sockets/files max. as defined via Linux default resource limit for file descriptors (see $ ulimit -n #)
- [optionally] a fd limit imposed when |select()| is used, see |FD_SETSIZE|. Other event processing methods may have other limits
- <... did I miss any limits ? ...>

[1]=The 45 connection limits defined per |max_tcp_or_rpc_data_connections| in ./krb5/src/lib/apputils/net-server.c may be (theory!) a result of the old 64 fd resource limit for user processes in UNIX SystemV+BSD4.3 (which still exists in modern Solaris/Illumos and maybe *BSD, too) ... guessing... maybe 45 was picked so there are at least 19 fds available for other purposes like config files and plugins.

If the "45 connections limit" is the issue... would a patch be acceptable which adds code to query the resource limit for file descriptors ($ ulimit -n #) and then do a |max_tcp_or_rpc_data_connections=MAX(result/2, 45)| ?

----

Bye,
Roland

----- Forwarded Message -----
> From: [hidden email]
> To: [hidden email]
> Sent: Wednesday, January 7, 2015 4:39:27 PM
> Subject: [Bug 1179820] New: Kerberos KDC connection limit too low
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1179820
>
>             Bug ID: 1179820
>            Summary: Kerberos KDC connection limit too low
[snip]
>          Component: krb5
[snip]

>
> Description of problem:
>
> The Kerberos KDC has a low, hard-coded limit on concurrent tcp connections.
> This results in large numbers of INFO level log entries about dropped
> connections when many clients are trying to contact the KDC. It could lead to
> additional problems with more clients.
>
> My current scenario is not one that should be followed in real life, but
> similar situations could occur with larger numbers of clients that are not
> misconfigured. I anticipate regularly having many hundreds, potentially
> thousands, of connected clients booting nearly simultaneously within the
> year.
>
[snip]

>
> How reproducible:
> Always
>
> Steps to Reproduce:
> 1. Configure 62 clients to use a KDC
> 2. Prevent replies from the KDC from reaching the clients
> 3. observe KDC logs as clients attempt to contact KDC
>
> Actual results:
>
> Logs fill up with dropped connection messages.
>
> Expected results:
>
> affected clients fail to authenticate, but do not exhaust all available open
> connections before they time out.
>
[snip]

--
  __ .  . __
 (o.\ \/ /.o) [hidden email]
  \__\/\/__/  IPA/Kerberos5 team
  /O /==\ O\  
 (;O/ \/ \O;)
 
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Bug 1179820] New: Kerberos KDC connection limit too low

Greg Hudson
On 01/16/2015 04:57 PM, Roland Mainz wrote:
> Does anyone know which limit the reporter in the bug report below may be referring to ?

Most likely the 45-connection limit in net-server.c.

> If the "45 connections limit" is the issue... would a patch be acceptable which adds code to query the resource limit for file descriptors ($ ulimit -n #) and then do a |max_tcp_or_rpc_data_connections=MAX(result/2, 45)| ?

I don't think the limit is there primarily to prevent fd exhaustion; it
also serves to limit user-space and kernel memory usage.  A limit of 45
seems pretty low for any modern host, though.

I believe we would take a patch to make the limit configurable, and to
make the default somewhat higher than it is today.  Configurable tuning
parameters aren't great, but I don't see any way to automatically choose
a limit in a way which approximates the highest load capacity while
preventing DOS attacks which disable the KDC through memory exhaustion.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 1179820] New: Kerberos KDC connection limit too low

Roland Mainz


----- Original Message -----

> From: "Greg Hudson" <[hidden email]>
> To: "Roland Mainz" <[hidden email]>, [hidden email]
> Sent: Sunday, January 18, 2015 6:39:15 PM
> Subject: Re: Fwd: [Bug 1179820] New: Kerberos KDC connection limit too low
>
> On 01/16/2015 04:57 PM, Roland Mainz wrote:
> > Does anyone know which limit the reporter in the bug report below may be
> > referring to ?
>
> Most likely the 45-connection limit in net-server.c.

Agreed.

> > If the "45 connections limit" is the issue... would a patch be acceptable
> > which adds code to query the resource limit for file descriptors ($ ulimit
> > -n #) and then do a |max_tcp_or_rpc_data_connections=MAX(result/2, 45)| ?
>
> I don't think the limit is there primarily to prevent fd exhaustion; it
> also serves to limit user-space and kernel memory usage.

Right... but AFAIK even with the (current) upper limit of 65535 fds in modern Unix it's hard to imagine to drive such machines (assuming a server...) into a wall from the point of memory consumption... 65535 fds with 1MB memory consumption would be ~~64GB. Likely other things like database/LDAP/etc. connections etc. would be a bottleneck long before you run out of memory.

> A limit of 45
> seems pretty low for any modern host, though.
>
> I believe we would take a patch to make the limit configurable, and to
> make the default somewhat higher than it is today.

Bumping the limit while ignoring the fd ulimit will likely cause trouble on Unix which still have (( $(ulimit -n) <= 64 )).
AFAIK options are:
1. ignore the fd ulimit issue and just bump the limit to something like 128
2. make default dynamic, based on the $(ulimit -n) value, but use |MIN(default, 45)|
3. make the default static but issue a warning when we're below a certain fd limit
4. set the ulimit for max. open files ourselves to 1024 when we're below that limit and issue a warning when the syscall fails
5. <... insert more options here...>

[1] will cause trouble on Unix which still have the low default fd ulimit
[3] is the least invasive change
[4] is likely a portability nightmare... ;-/

> Configurable tuning
> parameters aren't great, but I don't see any way to automatically choose
> a limit in a way which approximates the highest load capacity while
> preventing DOS attacks which disable the KDC through memory exhaustion.

Grumpf... I don't see a good way either... ;-(

----

Bye,
Roland

--
  __ .  . __
 (o.\ \/ /.o) [hidden email]
  \__\/\/__/  IPA/Kerberos5 team
  /O /==\ O\  
 (;O/ \/ \O;)
 
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 1179820] New: Kerberos KDC connection limit too low

Greg Hudson
On 01/21/2015 09:54 AM, Roland Mainz wrote:
> 1. ignore the fd ulimit issue and just bump the limit to something like 128
> 2. make default dynamic, based on the $(ulimit -n) value, but use |MIN(default, 45)|
> 3. make the default static but issue a warning when we're below a certain fd limit
> 4. set the ulimit for max. open files ourselves to 1024 when we're below that limit and issue a warning when the syscall fails
> 5. <... insert more options here...>

I think the best choice is to leave the hardcoded limit at 45 and add a
profile variable to change it.

We could choose a default based on getrlimit(RLIMIT_NOFILE) and an
imprecise estimate of fd usage by the rest of the KDC, bounded by some
maximum, but I think that's too complicated.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 1179820] New: Kerberos KDC connection limit too low

Roland C. Dowdeswell
On Wed, Jan 21, 2015 at 11:34:52AM -0500, Greg Hudson wrote:
>

> I think the best choice is to leave the hardcoded limit at 45 and add a
> profile variable to change it.
>
> We could choose a default based on getrlimit(RLIMIT_NOFILE) and an
> imprecise estimate of fd usage by the rest of the KDC, bounded by some
> maximum, but I think that's too complicated.

A simple heuristic approach that would likely do what you want
would be to simply check the fd returned by accept(2) and if it is
``too close'' to the ulimit then switch off accepting connexions.
Then switch them back on whenever a connexion is closed.  This
would be self-tuning and deal with the fact that you don't know a
priori how many file descriptors the rest of the code is going to
use.  It would, however, fail to work if any other code dup2(2)s
any fds up to the top of the range but that is likely a manageable
risk.

This approach has the added advantage of using existing configuration
to determine the limit rather than creating new configuration
settings that admins would need to learn how to manage.

--
    Roland Dowdeswell                      http://Imrryr.ORG/~elric/
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 1179820] New: Kerberos KDC connection limit too low

Greg Hudson
On 01/22/2015 09:56 AM, Roland C. Dowdeswell wrote:
> A simple heuristic approach that would likely do what you want
> would be to simply check the fd returned by accept(2) and if it is
> ``too close'' to the ulimit then switch off accepting connexions.

1. We don't currently use getrlimit(), and beginning to do so would
require thinking about its portability.  That's a source of complexity.

2. This method does not take into account memory limitations, only fd
limitations.

3. This method requires an estimate of how many fds the KDC might use
during processing of currently pending requests.

4. If we switch off accepting new connections, an attacker can DOS the
TCP portion of the KDC by simply filling up the connection table and
doing nothing.  If we discard old connections as we currently do, the
attacker has to keep creating connections, and some legitimate requests
might still get through.

> This approach has the added advantage of using existing configuration
> to determine the limit rather than creating new configuration
> settings that admins would need to learn how to manage.

We can't really inquire from the environment how much memory it is
reasonable for the KDC to use (or even how much it is currently using,
which isn't a simple number in the age of copy-on-write), so I don't
think we can automatically determine a good connection limit.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 1179820] New: Kerberos KDC connection limit too low

Nico Williams
On Thu, Jan 22, 2015 at 11:19:49AM -0500, Greg Hudson wrote:
> On 01/22/2015 09:56 AM, Roland C. Dowdeswell wrote:
> > A simple heuristic approach that would likely do what you want
> > would be to simply check the fd returned by accept(2) and if it is
> > ``too close'' to the ulimit then switch off accepting connexions.
>
> 1. We don't currently use getrlimit(), and beginning to do so would
> require thinking about its portability.  That's a source of complexity.

Or just garbage collect old[er] connections when EMFILE/ENFILE.  (That's
Roland's off-line suggestion to me.)

Roland's idea is probably easier to implement if you have plugins that
might make it difficult to distinguish resource limits/exhaustion errors
from other errors.

> 2. This method does not take into account memory limitations, only fd
> limitations.

Same thing here: hit resource limit -> close older connections (which
should free up some resources).

Of course, here it's harder to use getrlimit() and friends...

> 3. This method requires an estimate of how many fds the KDC might use
> during processing of currently pending requests.

Nah.  For Roland's proposal you'd need to define "close to limit" as
within some reasonable constant, say, 5, of the limit.

> 4. If we switch off accepting new connections, an attacker can DOS the
> TCP portion of the KDC by simply filling up the connection table and
> doing nothing.  If we discard old connections as we currently do, the
> attacker has to keep creating connections, and some legitimate requests
> might still get through.

They could always do that anyways.  The KDC could respond resource
exhaustion by randomly choosing to close the current connection or an
older one.  But at the end of the day, an attacker who can consume all
your available resources... will do it.

In any case, 45 is ridiculously small.  A dynamic self-tuning mechanism
is better than a configurable knob, and a configurable knob with a
reasonable default is better than a configurable knob that generally
needs tuning.

Nico
--
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev