Hadoop Datanode service throws exception with Kerberos security enabled

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Hadoop Datanode service throws exception with Kerberos security enabled

Sonia Garudi


Hello team,
We have a Ambari cluster setup using Rhel 7.5 beta machines. We are facing
issues with start up of Hadoop Datanode on enabling Kerberos security.

Error logged in /var/log/krb5kdc.log -
Mar 27 14:48:17 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
(1 etypes {16}) 10.77.67.132: PROCESS_TGS: authtime 0,
dn/[hidden email] for
nn/[hidden email], Ticket expired
Mar 27 14:48:55 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
(4 etypes {18 17 16 23}) 10.77.67.132: PROCESS_TGS: authtime 0,
nn/[hidden email] for
nn/[hidden email], Ticket expired

Below error in service log:
2018-03-27 14:46:44,739 WARN  ipc.Client (Client.java:run(711)) - Couldn't
setup connection for dn/[hidden email] to
pts00433-vm38.persistent.co.in/10.77.67.132:8020
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Ticket
expired (32) - PROCESS_TGS)]

We have following packages installed :
Version-Release number of selected component (if applicable):
# yum list installed | grep krb
krb5-devel.ppc64le                 1.15.1-18.el7       installed
krb5-libs.ppc64le                  1.15.1-18.el7       @anaconda/7.5
krb5-pkinit.ppc64le                1.15.1-18.el7       installed
krb5-server.ppc64le                1.15.1-18.el7       installed
krb5-workstation.ppc64le           1.15.1-18.el7       installed

# krb5-config --version
Kerberos 5 release 1.15.1

System and Ambari cluster details :
# uname -a
Linux pts00433-vm38.persistent.co.in 3.10.0-830.el7.ppc64le #1 SMP Mon Jan
15 12:26:57 EST 2018 ppc64le ppc64le ppc64le GNU/Linux
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)

Ambari version : 2.6.1
HDP version installed : 2.6.4

We have noticed, with Kerberos build version 1.15.1-8.el7, the datanode
starts up without any issue.

Any help or suggestions on why it fails with the higher update would be
appreciated .

Regards,
Sonia Garudi
[hidden email]
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop Datanode service throws exception with Kerberos security enabled

Robbie Harwood
"Sonia Garudi" <[hidden email]> writes:

> Hello team,
> We have a Ambari cluster setup using Rhel 7.5 beta machines. We are facing
> issues with start up of Hadoop Datanode on enabling Kerberos security.
>
> Error logged in /var/log/krb5kdc.log -
> Mar 27 14:48:17 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
> (1 etypes {16}) 10.77.67.132: PROCESS_TGS: authtime 0,
> dn/[hidden email] for
> nn/[hidden email], Ticket expired
> Mar 27 14:48:55 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
> (4 etypes {18 17 16 23}) 10.77.67.132: PROCESS_TGS: authtime 0,
> nn/[hidden email] for
> nn/[hidden email], Ticket expired
>
> Below error in service log:
> 2018-03-27 14:46:44,739 WARN  ipc.Client (Client.java:run(711)) - Couldn't
> setup connection for dn/[hidden email] to
> pts00433-vm38.persistent.co.in/10.77.67.132:8020
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Ticket
> expired (32) - PROCESS_TGS)]
>
> We have following packages installed :
> Version-Release number of selected component (if applicable):
> # yum list installed | grep krb
> krb5-devel.ppc64le                 1.15.1-18.el7       installed
> krb5-libs.ppc64le                  1.15.1-18.el7       @anaconda/7.5
> krb5-pkinit.ppc64le                1.15.1-18.el7       installed
> krb5-server.ppc64le                1.15.1-18.el7       installed
> krb5-workstation.ppc64le           1.15.1-18.el7       installed
>
> # krb5-config --version
> Kerberos 5 release 1.15.1
>
> System and Ambari cluster details :
> # uname -a
> Linux pts00433-vm38.persistent.co.in 3.10.0-830.el7.ppc64le #1 SMP Mon Jan
> 15 12:26:57 EST 2018 ppc64le ppc64le ppc64le GNU/Linux
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)
>
> Ambari version : 2.6.1
> HDP version installed : 2.6.4
>
> We have noticed, with Kerberos build version 1.15.1-8.el7, the datanode
> starts up without any issue.
>
> Any help or suggestions on why it fails with the higher update would be
> appreciated .
Hi Sonia,

I've replied on your bug with us and provided updated packages.

The corresponding upstream fix for this issue is
31d5c854198ed91fc2bd0b9fb87ed0dcd5a40eb6

Thanks,
--Robbie

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop Datanode service throws exception with Kerberos security enabled

Robbie Harwood
Robbie Harwood <[hidden email]> writes:

> "Sonia Garudi" <[hidden email]> writes:
>
>> Hello team,
>> We have a Ambari cluster setup using Rhel 7.5 beta machines. We are facing
>> issues with start up of Hadoop Datanode on enabling Kerberos security.
>>
>> Error logged in /var/log/krb5kdc.log -
>> Mar 27 14:48:17 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
>> (1 etypes {16}) 10.77.67.132: PROCESS_TGS: authtime 0,
>> dn/[hidden email] for
>> nn/[hidden email], Ticket expired
>> Mar 27 14:48:55 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ
>> (4 etypes {18 17 16 23}) 10.77.67.132: PROCESS_TGS: authtime 0,
>> nn/[hidden email] for
>> nn/[hidden email], Ticket expired
>>
>> Below error in service log:
>> 2018-03-27 14:46:44,739 WARN  ipc.Client (Client.java:run(711)) - Couldn't
>> setup connection for dn/[hidden email] to
>> pts00433-vm38.persistent.co.in/10.77.67.132:8020
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Ticket
>> expired (32) - PROCESS_TGS)]
>>
>> We have following packages installed :
>> Version-Release number of selected component (if applicable):
>> # yum list installed | grep krb
>> krb5-devel.ppc64le                 1.15.1-18.el7       installed
>> krb5-libs.ppc64le                  1.15.1-18.el7       @anaconda/7.5
>> krb5-pkinit.ppc64le                1.15.1-18.el7       installed
>> krb5-server.ppc64le                1.15.1-18.el7       installed
>> krb5-workstation.ppc64le           1.15.1-18.el7       installed
>>
>> # krb5-config --version
>> Kerberos 5 release 1.15.1
>>
>> System and Ambari cluster details :
>> # uname -a
>> Linux pts00433-vm38.persistent.co.in 3.10.0-830.el7.ppc64le #1 SMP Mon Jan
>> 15 12:26:57 EST 2018 ppc64le ppc64le ppc64le GNU/Linux
>> # cat /etc/redhat-release
>> Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)
>>
>> Ambari version : 2.6.1
>> HDP version installed : 2.6.4
>>
>> We have noticed, with Kerberos build version 1.15.1-8.el7, the datanode
>> starts up without any issue.
>>
>> Any help or suggestions on why it fails with the higher update would be
>> appreciated .
>
> Hi Sonia,
>
> I've replied on your bug with us and provided updated packages.
>
> The corresponding upstream fix for this issue is
> 31d5c854198ed91fc2bd0b9fb87ed0dcd5a40eb6
Wrong hash, should be 54e58755368b58ba5894a14c1d02626da42d8003

Thanks,
--Robbie

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos

signature.asc (847 bytes) Download Attachment