kdc crashing every other day

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

kdc crashing every other day

Marcin Cieslak-3
My 7.4.0 kdc (running on FreeBSD 10.3) having been running after few days
it stops accepting requests.

As it turns out the main process is waiting on wait4():

[Switching to LWP 101828 of process 8498]
0x000000080204f64a in _wait4 () from /lib/libc.so.7
(gdb) bt
#0  0x000000080204f64a in _wait4 () from /lib/libc.so.7
#1  0x0000000801d5a0dc in ?? () from /lib/libthr.so.3
#2  0x0000000000403c7b in reap_kid (context=0x803c18180, config=0x803c280c0, pids=0x803c22070, max_kids=2,
    options=0) at connect.c:1029
#3  0x00000000004033bb in start_kdc (context=0x803c18180, config=0x803c280c0,
    argv0=0x7fffffffef10 "/usr/local/libexec/kdc") at connect.c:1179
#4  0x00000000004075e1 in main (argc=2, argv=0x7fffffffed50) at main.c:176
(gdb) info thread
  Id   Target Id         Frame
* 1    LWP 101828 of process 8498 0x000000080204f64a in _wait4 () from /lib/libc.so.7

but there is no child process, because it has crashed.

I found a core file that says:

Core was generated by `/usr/local/libexec/kdc --detach'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000801286820 in der_length_general_string (data=0x18) at der_length.c:209
209    return strlen(*data);
(gdb) bt
#0  0x0000000801286820 in der_length_general_string (data=0x18) at der_length.c:209
#1  0x00000008011f9aed in length_Realm (data=0x18) at asn1_krb5_asn1.c:783
#2  0x000000080122b001 in length_KRB_ERROR (data=0x7fffffffdb10) at asn1_krb5_asn1.c:12827
#3  0x0000000800cfbc5f in krb5_mk_error_ext (context=0x803c18180, error_code=-1765328324,
    e_text=0x800852c5b "No client in request", e_data=0x0, server=0x803c1f5e0, client_name=0x0,
    client_realm=0x18, client_time=0x0, client_usec=0x0, reply=0x7fffffffe4f0) at mk_error.c:85
#4  0x0000000800839d35 in _kdc_fast_mk_error (context=0x803c18180, r=0x7fffffffe0b8,
    error_method=0x7fffffffdf40, armor_crypto=0x0, req_body=0x7fffffffe0e8, outer_error=-1765328324,
    e_text=0x800852c5b "No client in request", error_server=0x803c1f5e0, error_client_name=0x0,
    error_client_realm=0x18, csec=0x0, cusec=0x0, error_msg=0x7fffffffe4f0) at fast.c:331
#5  0x000000080083f0bf in _kdc_as_rep (r=0x7fffffffe0b8, reply=0x7fffffffe4f0,
    from=0x803cbb7bc "IPv4:35.177.172.195", from_addr=0x803cbb730, datagram_reply=0) at kerberos5.c:2230
#6  0x00000008008511ed in kdc_as_req (context=0x803c18180, config=0x803c280c0, req_buffer=0x7fffffffe458,
    reply=0x7fffffffe4f0, from=0x803cbb7bc "IPv4:35.177.172.195", addr=0x803cbb730, datagram_reply=0,
    claim=0x7fffffffe454) at process.c:77
#7  0x0000000800850c34 in krb5_kdc_process_request (context=0x803c18180, config=0x803c280c0,
    buf=0x803c06c00 "j\201n0\201k\241\003\002\001\005\242\003\002\001\n\244\201^0\\\240\a\003\005",
    len=113, reply=0x7fffffffe4f0, prependlength=0x7fffffffe514, from=0x803cbb7bc "IPv4:35.177.172.195",
    addr=0x803cbb730, datagram_reply=0) at process.c:208
#8  0x0000000000405f29 in do_request (context=0x803c18180, config=0x803c280c0, buf=0x803c06c00, len=113,
    prependlength=1, d=0x803cbb700) at connect.c:435
#9  0x00000000004050c0 in handle_tcp (context=0x803c18180, config=0x803c280c0, d=0x803cb8000, idx=44,
    min_free=45) at connect.c:834
#10 0x0000000000404472 in loop (context=0x803c18180, config=0x803c280c0, d=0x803cb8000, ndescr=48,
    islive=6) at connect.c:955
#11 0x000000000040344e in start_kdc (context=0x803c18180, config=0x803c280c0,
    argv0=0x7fffffffef10 "/usr/local/libexec/kdc") at connect.c:1190
#12 0x00000000004075e1 in main (argc=2, argv=0x7fffffffed50) at main.c:176

Seems like I keep getting interesting "requests" from outside.

113 bytes from the "buf" - frame 7 look like this:

0x803c06c00: 0x6a 0x81 0x6e 0x30 0x81 0x6b 0xa1 0x03
0x803c06c08: 0x02 0x01 0x05 0xa2 0x03 0x02 0x01 0x0a
0x803c06c10: 0xa4 0x81 0x5e 0x30 0x5c 0xa0 0x07 0x03
0x803c06c18: 0x05 0x00 0x50 0x80 0x00 0x10 0xa2 0x04
0x803c06c20: 0x1b 0x02 0x4e 0x4d 0xa3 0x17 0x30 0x15
0x803c06c28: 0xa0 0x03 0x02 0x01 0x00 0xa1 0x0e 0x30
0x803c06c30: 0x0c 0x1b 0x06 0x6b 0x72 0x62 0x74 0x67
0x803c06c38: 0x74 0x1b 0x02 0x4e 0x4d 0xa5 0x11 0x18
0x803c06c40: 0x0f 0x31 0x39 0x37 0x30 0x30 0x31 0x30
0x803c06c48: 0x31 0x30 0x30 0x30 0x30 0x30 0x30 0x5a
0x803c06c50: 0xa7 0x06 0x02 0x04 0x1f 0x1e 0xb9 0xd9
0x803c06c58: 0xa8 0x17 0x30 0x15 0x02 0x01 0x12 0x02
0x803c06c60: 0x01 0x11 0x02 0x01 0x10 0x02 0x01 0x17
0x803c06c68: 0x02 0x01 0x01 0x02 0x01 0x03 0x02 0x01
0x803c06c70: 0x02

same uudecoded:

begin 644 /tmp/request
M:H%N,(%KH0,"`06B`P(!"J2!7C!<H`<#!0!0@``0H@0;`DY-HQ<P%:`#`@$`
MH0XP#!L&:W)B=&=T&P).3:41&`\Q.3<P,#$P,3`P,#`P,%JG!@($'QZYV:@7
7,!4"`1("`1$"`1`"`1<"`0$"`0,"`0(`
`
end

Marcin

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kdc crashing every other day

Greg Hudson
On 12/25/2017 09:03 AM, Marcin Cieslak wrote:
> I found a core file that says:
>
> Core was generated by `/usr/local/libexec/kdc --detach'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x0000000801286820 in der_length_general_string (data=0x18) at der_length.c:209
> 209    return strlen(*data);

It looks like this bug was fixed in 7.5.0.  The relevant commit on the
release branch is here:

https://github.com/heimdal/heimdal/commit/749d377fa357351a7bbba51f8aae72cdf0629592

> As it turns out the main process is waiting on wait4():

This might be a secondary bug, also fixed in 7.5.0:

https://github.com/heimdal/heimdal/commit/108b28874788e5d0aa6f7c5af16d6cc405ae8eac

(I am not a Heimdal developer, but I was curious enough to look into the
backtrace.)
Reply | Threaded
Open this post in threaded view
|

Re: kdc crashing every other day

Marcin Cieslak-3
On Mon, 25 Dec 2017, Greg Hudson wrote:

> On 12/25/2017 09:03 AM, Marcin Cieslak wrote:
> > I found a core file that says:
> >
> > Core was generated by `/usr/local/libexec/kdc --detach'.
> > Program terminated with signal SIGSEGV, Segmentation fault.
> > #0  0x0000000801286820 in der_length_general_string (data=0x18) at der_length.c:209
> > 209    return strlen(*data);
>
> It looks like this bug was fixed in 7.5.0.  The relevant commit on the
> release branch is here:
>
> https://github.com/heimdal/heimdal/commit/749d377fa357351a7bbba51f8aae72cdf0629592
Thank you, this looks like the fix to me, too. (Just figured it out independently).

> > As it turns out the main process is waiting on wait4():
>
> This might be a secondary bug, also fixed in 7.5.0:
>
> https://github.com/heimdal/heimdal/commit/108b28874788e5d0aa6f7c5af16d6cc405ae8eac
>
> (I am not a Heimdal developer, but I was curious enough to look into the
> backtrace.)

Thank you very much!

What wonders me is that someone out of AWS sends such requests in the wild.

> openssl asn1parse -i -inform der -in /tmp/request -dump
    0:d=0  hl=3 l= 110 cons: appl [ 10 ]      
    3:d=1  hl=3 l= 107 cons:  SEQUENCE          
    6:d=2  hl=2 l=   3 cons:   cont [ 1 ]        
    8:d=3  hl=2 l=   1 prim:    INTEGER           :05
   11:d=2  hl=2 l=   3 cons:   cont [ 2 ]        
   13:d=3  hl=2 l=   1 prim:    INTEGER           :0A
   16:d=2  hl=3 l=  94 cons:   cont [ 4 ]        
   19:d=3  hl=2 l=  92 cons:    SEQUENCE          
   21:d=4  hl=2 l=   7 cons:     cont [ 0 ]        
   23:d=5  hl=2 l=   5 prim:      BIT STRING        
      0000 - 00 50 80 00 10                                    .P...
   30:d=4  hl=2 l=   4 cons:     cont [ 2 ]        
   32:d=5  hl=2 l=   2 prim:      GENERALSTRING    
      0000 - 4e 4d                                             NM
   36:d=4  hl=2 l=  23 cons:     cont [ 3 ]        
   38:d=5  hl=2 l=  21 cons:      SEQUENCE          
   40:d=6  hl=2 l=   3 cons:       cont [ 0 ]        
   42:d=7  hl=2 l=   1 prim:        INTEGER           :00
   45:d=6  hl=2 l=  14 cons:       cont [ 1 ]        
   47:d=7  hl=2 l=  12 cons:        SEQUENCE          
   49:d=8  hl=2 l=   6 prim:         GENERALSTRING    
      0000 - 6b 72 62 74 67 74                                 krbtgt
   57:d=8  hl=2 l=   2 prim:         GENERALSTRING    
      0000 - 4e 4d                                             NM
   61:d=4  hl=2 l=  17 cons:     cont [ 5 ]        
   63:d=5  hl=2 l=  15 prim:      GENERALIZEDTIME   :19700101000000Z
   80:d=4  hl=2 l=   6 cons:     cont [ 7 ]        
   82:d=5  hl=2 l=   4 prim:      INTEGER           :1F1EB9D9
   88:d=4  hl=2 l=  23 cons:     cont [ 8 ]        
   90:d=5  hl=2 l=  21 cons:      SEQUENCE          
   92:d=6  hl=2 l=   1 prim:       INTEGER           :12
   95:d=6  hl=2 l=   1 prim:       INTEGER           :11
   98:d=6  hl=2 l=   1 prim:       INTEGER           :10
  101:d=6  hl=2 l=   1 prim:       INTEGER           :17
  104:d=6  hl=2 l=   1 prim:       INTEGER           :01
  107:d=6  hl=2 l=   1 prim:       INTEGER           :03
  110:d=6  hl=2 l=   1 prim:       INTEGER           :02


Marcin CieĊ›lak

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kdc crashing every other day

Viktor Dukhovni-2
In reply to this post by Greg Hudson


> On Dec 25, 2017, at 9:50 AM, Greg Hudson <[hidden email]> wrote:
>
> It looks like this bug was fixed in 7.5.0.  The relevant commit on the
> release branch is here:
>
> https://github.com/heimdal/heimdal/commit/749d377fa357351a7bbba51f8aae72cdf0629592
>
>> As it turns out the main process is waiting on wait4():
>
> This might be a secondary bug, also fixed in 7.5.0:
>
> https://github.com/heimdal/heimdal/commit/108b28874788e5d0aa6f7c5af16d6cc405ae8eac
>
> (I am not a Heimdal developer, but I was curious enough to look into the
> backtrace.)

Your analysis is very likely correct.  The OP should try 7.5.0.
Similar symptoms were solved for other users via the same upgrade.

--
        Viktor.