iprop_iprop_replica_poll=2m default...

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

iprop_iprop_replica_poll=2m default...

Tareq Alrashid-2
How can we make it as close to realtime as possible?
what is the smallest value possible we can assign?

Background:

Master receives a newly provisioned user, or new password change/reset, and since we live in the instant-gratification times, users attempt to login onto services that configured to authenticate against replica servers which of course have not been propagated to yet…. failed login => open a help desk ticket…etc. waste of time and frustration.

How do you all deal with the latency in your hi-ed environment?

HNY! Thanks for any insights


_____
 Tareq M. Alrashid - IT Engineer 3
 CWRU - University Technology, [U]Tech - Enterprise Systems -
 10900 Euclid Avenue, Crawford 489, Cleveland, OH 44106-7072  U.S.A.

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Kenneth MacDonald
On Wed, 2020-01-08 at 13:38 -0500, Tareq Alrashid wrote:

> How can we make it as close to realtime as possible?
> what is the smallest value possible we can assign?
>
> Background:
>
> Master receives a newly provisioned user, or new password
> change/reset, and since we live in the instant-gratification times,
> users attempt to login onto services that configured to authenticate
> against replica servers which of course have not been propagated to
> yet…. failed login => open a help desk ticket…etc. waste of time and
> frustration.
>
> How do you all deal with the latency in your hi-ed environment?
>
> HNY! Thanks for any insights

We haven't reduced the polling interval, but have configured our web
single sign on hosts to authenticate against our master KDC in
preference to the slaves by listing their IP addresses in order in
/etc/krb5.conf.

Cheers,

Kenny.





--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Tareq Alrashid-2
Thanks for the reply, Kenny.

I have left out an important detail, on campus of course all is configured to master KDC first, the kerb2/kerb3…etc., no problem.

This affects users of our clouds services, for example in AWS where we have duplicated all/most of our infrastructure services, if a user changes her password using our web tools against master KDC on campus, said user will not able to login immediately until changes are replicated out to the replica Kerberos servers in AWS. Granted 2m is not long, but this reason for asking in the first place to see if 2m is the shorted time delta allowed.

Thanks,
Tareq

> On Jan 9, 2020, at 4:11 AM, Kenneth MacDonald <[hidden email]> wrote:
>
> On Wed, 2020-01-08 at 13:38 -0500, Tareq Alrashid wrote:
>> How can we make it as close to realtime as possible?
>> what is the smallest value possible we can assign?
>>
>> Background:
>>
>> Master receives a newly provisioned user, or new password
>> change/reset, and since we live in the instant-gratification times,
>> users attempt to login onto services that configured to authenticate
>> against replica servers which of course have not been propagated to
>> yet…. failed login => open a help desk ticket…etc. waste of time and
>> frustration.
>>
>> How do you all deal with the latency in your hi-ed environment?
>>
>> HNY! Thanks for any insights
>
> We haven't reduced the polling interval, but have configured our web
> single sign on hosts to authenticate against our master KDC in
> preference to the slaves by listing their IP addresses in order in
> /etc/krb5.conf.
>
> Cheers,
>
> Kenny.
>
>
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Kenneth MacDonald
Ah, OK.  I cannot answer whether 2m is the minumum value.

Cheers,

Kenny.

On Thu, 2020-01-09 at 09:26 -0500, Tareq Alrashid wrote:

> Thanks for the reply, Kenny.
>
> I have left out an important detail, on campus of course all is
> configured to master KDC first, the kerb2/kerb3…etc., no problem.
>
> This affects users of our clouds services, for example in AWS where
> we have duplicated all/most of our infrastructure services, if a user
> changes her password using our web tools against master KDC on
> campus, said user will not able to login immediately until changes
> are replicated out to the replica Kerberos servers in AWS. Granted 2m
> is not long, but this reason for asking in the first place to see if
> 2m is the shorted time delta allowed.
>
> Thanks,
> Tareq
>
> > On Jan 9, 2020, at 4:11 AM, Kenneth MacDonald <
> > [hidden email]> wrote:
> >
> > On Wed, 2020-01-08 at 13:38 -0500, Tareq Alrashid wrote:
> > > How can we make it as close to realtime as possible?
> > > what is the smallest value possible we can assign?
> > >
> > > Background:
> > >
> > > Master receives a newly provisioned user, or new password
> > > change/reset, and since we live in the instant-gratification
> > > times,
> > > users attempt to login onto services that configured to
> > > authenticate
> > > against replica servers which of course have not been propagated
> > > to
> > > yet…. failed login => open a help desk ticket…etc. waste of time
> > > and
> > > frustration.
> > >
> > > How do you all deal with the latency in your hi-ed environment?
> > >
> > > HNY! Thanks for any insights
> >
> > We haven't reduced the polling interval, but have configured our
> > web
> > single sign on hosts to authenticate against our master KDC in
> > preference to the slaves by listing their IP addresses in order in
> > /etc/krb5.conf.
> >
> > Cheers,
> >
> > Kenny.
> >
> >
> >
> >
> >
> > --
> > The University of Edinburgh is a charitable body, registered in
> > Scotland, with registration number SC005336.
> >
>
>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Greg Hudson
In reply to this post by Tareq Alrashid-2
On 1/8/20 1:38 PM, Tareq Alrashid wrote:
> How can we make it as close to realtime as possible?
> what is the smallest value possible we can assign?

You can assign a value as low as one second.

> Master receives a newly provisioned user, or new password change/reset, and since we live in the instant-gratification times, users attempt to login onto services that configured to authenticate against replica servers which of course have not been propagated to yet…. failed login => open a help desk ticket…etc. waste of time and frustration.

You could try configuring a master_kdc value in krb5.conf on the clients
(or, if you use DNS, adding _kerberos-master._udp.realm and
_kerberos-master._tcp.realm records).  If these are present, kinit will
retry with the master KDC if it gets an error from the first KDC it
tries, if the error could have resulted from propagation not having
happened yet.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Tareq Alrashid-2
Thanks Greg.
Final question if there is any negative impact for having replicas poll at
often as one second or maybe it is best to be at higher numbers of seconds?

On Thu, Jan 9, 2020 at 11:24 Greg Hudson <[hidden email]> wrote:

> On 1/8/20 1:38 PM, Tareq Alrashid wrote:
> > How can we make it as close to realtime as possible?
> > what is the smallest value possible we can assign?
>
> You can assign a value as low as one second.
>
> > Master receives a newly provisioned user, or new password change/reset,
> and since we live in the instant-gratification times, users attempt to
> login onto services that configured to authenticate against replica servers
> which of course have not been propagated to yet…. failed login => open a
> help desk ticket…etc. waste of time and frustration.
>
> You could try configuring a master_kdc value in krb5.conf on the clients
> (or, if you use DNS, adding _kerberos-master._udp.realm and
> _kerberos-master._tcp.realm records).  If these are present, kinit will
> retry with the master KDC if it gets an error from the first KDC it
> tries, if the error could have resulted from propagation not having
> happened yet.
>
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Tareq Alrashid-2
> You can assign a value as low as one second.

Maybe I am missing something but changing the kdc.conf to any value...

iprop_replica_poll=1s
or even...
iprop_replica_poll   = 0.016666666666667m
 (for 1s= 1/60min!)

Based on tailing the kadmind.log, it is showing the replica polling every 2m!?


> On Jan 9, 2020, at 11:32 AM, Tareq Alrashid <[hidden email]> wrote:
>
> Thanks Greg.
> Final question if there is any negative impact for having replicas poll at often as one second or maybe it is best to be at higher numbers of seconds?
>
> On Thu, Jan 9, 2020 at 11:24 Greg Hudson <[hidden email] <mailto:[hidden email]>> wrote:
> On 1/8/20 1:38 PM, Tareq Alrashid wrote:
> > How can we make it as close to realtime as possible?
> > what is the smallest value possible we can assign?
>
> You can assign a value as low as one second.
>
> > Master receives a newly provisioned user, or new password change/reset, and since we live in the instant-gratification times, users attempt to login onto services that configured to authenticate against replica servers which of course have not been propagated to yet…. failed login => open a help desk ticket…etc. waste of time and frustration.
>
> You could try configuring a master_kdc value in krb5.conf on the clients
> (or, if you use DNS, adding _kerberos-master._udp.realm and
> _kerberos-master._tcp.realm records).  If these are present, kinit will
> retry with the master KDC if it gets an error from the first KDC it
> tries, if the error could have resulted from propagation not having
> happened yet.

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Greg Hudson
On 1/10/20 8:22 PM, Tareq Alrashid wrote:
> Maybe I am missing something but changing the kdc.conf to any value...
>
> iprop_replica_poll=1s 
> or even...
> iprop_replica_poll   = 0.016666666666667m
>  (for 1s= 1/60min!)
>
> Based on tailing the kadmind.log, it is showing the replica polling
> every 2m!?

If you are running a release prior to 1.17, you need to use the old name
iprop_slave_poll.  (The old name still works in 1.17 as well.)

Also make sure to set the value on the machine running kpropd (not the
master KDC where kadmind is run), and to restart kpropd.

I don't think the delta time format supports floating point values, but
"1s" or just "1" should work.

> Final question if there is any negative impact for having replicas poll at often as one second or maybe it is best to be at higher numbers of seconds?
Polling every second will cause a little bit of work on the replica and
the master KDC each second, and use a little bit of network traffic.
With today's computers and networks it's probably going to have much impact.
________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: iprop_iprop_replica_poll=2m default...

Tareq Alrashid-2
Since my last message, I came across the documentation part were it mentions the iprop_slave_poll.
We’re running Kerberos 5 release 1.15.1! - Will make the proper change and the test results should make more sense!

Thanks Greg.



> On Jan 12, 2020, at 5:54 PM, Greg Hudson <[hidden email]> wrote:
>
> On 1/10/20 8:22 PM, Tareq Alrashid wrote:
>> Maybe I am missing something but changing the kdc.conf to any value...
>>
>> iprop_replica_poll=1s
>> or even...
>> iprop_replica_poll   = 0.016666666666667m
>>  (for 1s= 1/60min!)
>>
>> Based on tailing the kadmind.log, it is showing the replica polling
>> every 2m!?
>
> If you are running a release prior to 1.17, you need to use the old name
> .  (The old name still works in 1.17 as well.)
>
> Also make sure to set the value on the machine running kpropd (not the
> master KDC where kadmind is run), and to restart kpropd.
>
> I don't think the delta time format supports floating point values, but
> "1s" or just "1" should work.
>
>> Final question if there is any negative impact for having replicas poll at often as one second or maybe it is best to be at higher numbers of seconds?
> Polling every second will cause a little bit of work on the replica and
> the master KDC each second, and use a little bit of network traffic.
> With today's computers and networks it's probably going to have much impact.


________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

kpropd on replica core dumping at start! (was Re: iprop_iprop_replica_poll=2m default...)

Tareq Alrashid-2
RHEL 7.7

2am all of a sudden all my replicas kpropd process crashed with a core dump. Nothing has changed!  

systemctl status kprop.service

kprop.service: main process exited, code=dumped, status=6/ABRT
Feb 20 09:35:13 kerb-replica systemd[1]: Unit kprop.service entered failed state.
Feb 20 09:35:13 kerb-replica systemd[1]: kprop.service failed.


kdc.conf on replica
      ….
        iprop_port           = 754
        iprop_slave_poll     = 3s
    }

Any ideas would be most appreciate, as we are in the weeds now looking at what has happened.  

Thank you in advance

Tareq

> On Jan 12, 2020, at 9:45 PM, TAREQ ALRASHID <[hidden email]> wrote:
>
> Since my last message, I came across the documentation part were it mentions the iprop_slave_poll.
> We’re running Kerberos 5 release 1.15.1! - Will make the proper change and the test results should make more sense!
>
> Thanks Greg.
>
>
>
>> On Jan 12, 2020, at 5:54 PM, Greg Hudson <[hidden email]> wrote:
>>
>> On 1/10/20 8:22 PM, Tareq Alrashid wrote:
>>> Maybe I am missing something but changing the kdc.conf to any value...
>>>
>>> iprop_replica_poll=1s
>>> or even...
>>> iprop_replica_poll   = 0.016666666666667m
>>> (for 1s= 1/60min!)
>>>
>>> Based on tailing the kadmind.log, it is showing the replica polling
>>> every 2m!?
>>
>> If you are running a release prior to 1.17, you need to use the old name
>> .  (The old name still works in 1.17 as well.)
>>
>> Also make sure to set the value on the machine running kpropd (not the
>> master KDC where kadmind is run), and to restart kpropd.
>>
>> I don't think the delta time format supports floating point values, but
>> "1s" or just "1" should work.
>>
>>> Final question if there is any negative impact for having replicas poll at often as one second or maybe it is best to be at higher numbers of seconds?
>> Polling every second will cause a little bit of work on the replica and
>> the master KDC each second, and use a little bit of network traffic.
>> With today's computers and networks it's probably going to have much impact.
>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kpropd on replica core dumping at start! (was Re: iprop_iprop_replica_poll=2m default...)

Robbie Harwood
Tareq Alrashid <[hidden email]> writes:

> RHEL 7.7
>
> 2am all of a sudden all my replicas kpropd process crashed with a core
> dump. Nothing has changed!

If you're running RHEL, I recommend you open a support case.  That's
what you pay us for :)

> systemctl status kprop.service
>
> kprop.service: main process exited, code=dumped, status=6/ABRT
> Feb 20 09:35:13 kerb-replica systemd[1]: Unit kprop.service entered failed state.
> Feb 20 09:35:13 kerb-replica systemd[1]: kprop.service failed.
>
>
> kdc.conf on replica
>       ….
> iprop_port           = 754
>         iprop_slave_poll     = 3s
>     }
>
> Any ideas would be most appreciate, as we are in the weeds now looking
> at what has happened.
If you have a coredump, a backtrace would be helpful to move forward.

Thanks,
--Robbie

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kpropd on replica core dumping at start! (was Re: iprop_iprop_replica_poll=2m default...)

Tareq Alrashid-2
In reply to this post by Tareq Alrashid-2
Well!  we had a theory based on facts, that the last 1 update from the master KDC blew all other replicas out of the water!
So I performed a "kproplog -R” got reset log and perform a Full Sync. Restarted kprop and what do you know, the incrementals started working and kprop never crashed again!

Now we to wait on RHEL to see if coredump provides some insights on what lightening mysterious event took place for that last password change action by one insomniac at 02:11am that can cause this crash to ever happen again?

Could it be the 3 second interval poll being too frequent?...etc. we have RCA to come up with.

Shared this in case others have to deal with, or in case someone did experience this and can shed some light.

As always any and all are welcome to chime in.  

Thank you,
Tareq

> On Feb 20, 2020, at 10:19 AM, Tareq Alrashid <[hidden email]> wrote:
>
> RHEL 7.7
>
> 2am all of a sudden all my replicas kpropd process crashed with a core dump. Nothing has changed!  
>
> systemctl status kprop.service
>
> kprop.service: main process exited, code=dumped, status=6/ABRT
> Feb 20 09:35:13 kerb-replica systemd[1]: Unit kprop.service entered failed state.
> Feb 20 09:35:13 kerb-replica systemd[1]: kprop.service failed.
>
>
> kdc.conf on replica
>       ….
> iprop_port           = 754
>         iprop_slave_poll     = 3s
>     }
>
> Any ideas would be most appreciate, as we are in the weeds now looking at what has happened.  
>
> Thank you in advance
>
> Tareq
>
>> On Jan 12, 2020, at 9:45 PM, TAREQ ALRASHID <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> Since my last message, I came across the documentation part were it mentions the iprop_slave_poll.
>> We’re running Kerberos 5 release 1.15.1! - Will make the proper change and the test results should make more sense!
>>
>> Thanks Greg.
>>
>>
>>
>>> On Jan 12, 2020, at 5:54 PM, Greg Hudson <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>> On 1/10/20 8:22 PM, Tareq Alrashid wrote:
>>>> Maybe I am missing something but changing the kdc.conf to any value...
>>>>
>>>> iprop_replica_poll=1s
>>>> or even...
>>>> iprop_replica_poll   = 0.016666666666667m
>>>> (for 1s= 1/60min!)
>>>>
>>>> Based on tailing the kadmind.log, it is showing the replica polling
>>>> every 2m!?
>>>
>>> If you are running a release prior to 1.17, you need to use the old name
>>> .  (The old name still works in 1.17 as well.)
>>>
>>> Also make sure to set the value on the machine running kpropd (not the
>>> master KDC where kadmind is run), and to restart kpropd.
>>>
>>> I don't think the delta time format supports floating point values, but
>>> "1s" or just "1" should work.
>>>
>>>> Final question if there is any negative impact for having replicas poll at often as one second or maybe it is best to be at higher numbers of seconds?
>>> Polling every second will cause a little bit of work on the replica and
>>> the master KDC each second, and use a little bit of network traffic.
>>> With today's computers and networks it's probably going to have much impact.
>>
>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos
Reply | Threaded
Open this post in threaded view
|

Re: kpropd on replica core dumping at start! (was Re: iprop_iprop_replica_poll=2m default...)

Tareq Alrashid-2
Is there a protocol you all follow to be safe with incremental propagation?  Like doing Full Sync, every now and then, maybe once a day or once a week?

Can the ulog or God forbid the database get clobbered corrupted and get replicated everywhere?? that would be  huge problem, but I cant imaging this to be such a high risk.

Have backups in place of database, daily I believe. Maybe should be making hourly backups just in case.

Pondering away…
Tareq

> On Feb 20, 2020, at 2:04 PM, Tareq Alrashid <[hidden email]> wrote:
>
> Well!  we had a theory based on facts, that the last 1 update from the master KDC blew all other replicas out of the water!
> So I performed a "kproplog -R” got reset log and perform a Full Sync. Restarted kprop and what do you know, the incrementals started working and kprop never crashed again!
>
> Now we to wait on RHEL to see if coredump provides some insights on what lightening mysterious event took place for that last password change action by one insomniac at 02:11am that can cause this crash to ever happen again?
>
> Could it be the 3 second interval poll being too frequent?...etc. we have RCA to come up with.
>
> Shared this in case others have to deal with, or in case someone did experience this and can shed some light.
>
> As always any and all are welcome to chime in.  
>
> Thank you,
> Tareq
>
>> On Feb 20, 2020, at 10:19 AM, Tareq Alrashid <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> RHEL 7.7
>>
>> 2am all of a sudden all my replicas kpropd process crashed with a core dump. Nothing has changed!  
>>
>> systemctl status kprop.service
>>
>> kprop.service: main process exited, code=dumped, status=6/ABRT
>> Feb 20 09:35:13 kerb-replica systemd[1]: Unit kprop.service entered failed state.
>> Feb 20 09:35:13 kerb-replica systemd[1]: kprop.service failed.
>>
>>
>> kdc.conf on replica
>>       ….
>> iprop_port           = 754
>>         iprop_slave_poll     = 3s
>>     }
>>
>> Any ideas would be most appreciate, as we are in the weeds now looking at what has happened.  
>>
>> Thank you in advance
>>
>> Tareq
>>
>>> On Jan 12, 2020, at 9:45 PM, TAREQ ALRASHID <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>> Since my last message, I came across the documentation part were it mentions the iprop_slave_poll.
>>> We’re running Kerberos 5 release 1.15.1! - Will make the proper change and the test results should make more sense!
>>>
>>> Thanks Greg.
>>>
>>>
>>>
>>>> On Jan 12, 2020, at 5:54 PM, Greg Hudson <[hidden email] <mailto:[hidden email]>> wrote:
>>>>
>>>> On 1/10/20 8:22 PM, Tareq Alrashid wrote:
>>>>> Maybe I am missing something but changing the kdc.conf to any value...
>>>>>
>>>>> iprop_replica_poll=1s
>>>>> or even...
>>>>> iprop_replica_poll   = 0.016666666666667m
>>>>> (for 1s= 1/60min!)
>>>>>
>>>>> Based on tailing the kadmind.log, it is showing the replica polling
>>>>> every 2m!?
>>>>
>>>> If you are running a release prior to 1.17, you need to use the old name
>>>> .  (The old name still works in 1.17 as well.)
>>>>
>>>> Also make sure to set the value on the machine running kpropd (not the
>>>> master KDC where kadmind is run), and to restart kpropd.
>>>>
>>>> I don't think the delta time format supports floating point values, but
>>>> "1s" or just "1" should work.
>>>>
>>>>> Final question if there is any negative impact for having replicas poll at often as one second or maybe it is best to be at higher numbers of seconds?
>>>> Polling every second will cause a little bit of work on the replica and
>>>> the master KDC each second, and use a little bit of network traffic.
>>>> With today's computers and networks it's probably going to have much impact.
>>>
>>
>

________________________________________________
Kerberos mailing list           [hidden email]
https://mailman.mit.edu/mailman/listinfo/kerberos