ipropd question

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ipropd question

Martin Flemming

Hi !

I want to use the iprop-mechanism but i'm little confused
if all running really well ...

My setup looks like follow

3 heimdal-server with Centos 7.3.1611

heimdal-rpms from epel :

heimdal-libs-1.6.0-0.9.20140621gita5adc06.el7.x86_64
heimdal-server-1.6.0-0.9.20140621gita5adc06.el7.x86_64

On the first view everything seems to be ok :

master :

cat slaves-stats
Status for slaves, last updated: 2017-03-24T09:16:31

Master version: 7158

Name                             Address            Version  Status  Last Seen
iprop/[hidden email]  IPv4:192.168.6.34     7158  Up
2017-03-24T09:15:01
iprop/[hidden email]  IPv4:192.168.6.37     7158  Up
2017-03-24T09:15:01

On both Slaves seems also everything ok

Mar 24 09:25:02 server2 ipropd-slave[3001]: slave status change: up-to-date
with version: 7158 at 2017-03-24T09:25:02
Mar 24 09:27:32 server2 ipropd-slave[3001]: slave status change: up-to-date
with version: 7158 at 2017-03-24T09:27:32
Mar 24 09:30:01 server2 ipropd-slave[3001]: replaying entry 7159
Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change: up-to-date
with version: 7159 at 2017-03-24T09:30:01
Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change: up-to-date
with version: 7159 at 2017-03-24T09


But if i dump and count all Principals on these 3 heimdal-server,
i've got 3 differents results :-(

kadmin -l list  \* > kadmin_list.txt
cat kadmin_list.txt |wc


Master : 31027
Slave1 : 30453
Slave2 : 29311

Can somebody explain this behaviour or is the ipropd-mechanism buggy ?

Or have i to upgrade to Heimdal 7.1.0 (own build from tar-package :-( ) ?

Major changes

     ....
     ....
     iprop has been revamped to fix a number of race conditions that could lead to inconsistent replication.
     ....
     ....

thanks & cheers

        Martin


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipropd question

Thomas M. Payerle-3
My experience is that ipropd-slave will try to apply the change log
sent from the master, but if it cannot for some reason, it will ignore
the change but bump the database version (and log a warning/error in
its logs).  There are pros and cons of that behavior; briefly it makes
the ipropd service more "robust" but also means that if one is not monitoring
the logs the DBs can get out of sync.

The upshot is, monitoring DB synchronization by looking at
slaves-stats file is NOT sufficient.

We generally do a listing of the names, kvnos, and last mod
times for all principals on all the KDCs every night around 3-4 AM
(to minimize false positives from actual updates) and compare to
ensure the DBs stay synchronized.



On Fri, 24 Mar 2017, Martin Flemming wrote:

>
> Hi !
>
> I want to use the iprop-mechanism but i'm little confused
> if all running really well ...
>
> My setup looks like follow
>
> 3 heimdal-server with Centos 7.3.1611
>
> heimdal-rpms from epel :
>
> heimdal-libs-1.6.0-0.9.20140621gita5adc06.el7.x86_64
> heimdal-server-1.6.0-0.9.20140621gita5adc06.el7.x86_64
>
> On the first view everything seems to be ok :
>
> master :
>
> cat slaves-stats
> Status for slaves, last updated: 2017-03-24T09:16:31
>
> Master version: 7158
>
> Name                             Address            Version  Status  Last
> Seen
> iprop/[hidden email]  IPv4:192.168.6.34     7158  Up
> 2017-03-24T09:15:01
> iprop/[hidden email]  IPv4:192.168.6.37     7158  Up
> 2017-03-24T09:15:01
>
> On both Slaves seems also everything ok
>
> Mar 24 09:25:02 server2 ipropd-slave[3001]: slave status change: up-to-date
> with version: 7158 at 2017-03-24T09:25:02
> Mar 24 09:27:32 server2 ipropd-slave[3001]: slave status change: up-to-date
> with version: 7158 at 2017-03-24T09:27:32
> Mar 24 09:30:01 server2 ipropd-slave[3001]: replaying entry 7159
> Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change: up-to-date
> with version: 7159 at 2017-03-24T09:30:01
> Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change: up-to-date
> with version: 7159 at 2017-03-24T09
>
>
> But if i dump and count all Principals on these 3 heimdal-server,
> i've got 3 differents results :-(
>
> kadmin -l list  \* > kadmin_list.txt
> cat kadmin_list.txt |wc
>
>
> Master : 31027
> Slave1 : 30453
> Slave2 : 29311
>
> Can somebody explain this behaviour or is the ipropd-mechanism buggy ?
>
> Or have i to upgrade to Heimdal 7.1.0 (own build from tar-package :-( ) ?
>
> Major changes
>
>    ....
>    ....
>    iprop has been revamped to fix a number of race conditions that could
> lead to inconsistent replication.
>    ....
>    ....
>
> thanks & cheers
>
>       Martin
>
>
>

Tom Payerle
IT-ETI-EUS [hidden email]
4254 Stadium Dr (301) 405-6135
University of Maryland
College Park, MD 20742-4111
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipropd question

Adam Lewenberg


On 3/24/2017 7:21 AM, Thomas M. Payerle wrote:

> My experience is that ipropd-slave will try to apply the change log
> sent from the master, but if it cannot for some reason, it will ignore
> the change but bump the database version (and log a warning/error in
> its logs).  There are pros and cons of that behavior; briefly it makes
> the ipropd service more "robust" but also means that if one is not
> monitoring
> the logs the DBs can get out of sync.
>
> The upshot is, monitoring DB synchronization by looking at slaves-stats
> file is NOT sufficient.
>
> We generally do a listing of the names, kvnos, and last mod
> times for all principals on all the KDCs every night around 3-4 AM
> (to minimize false positives from actual updates) and compare to ensure
> the DBs stay synchronized.

That makes me think that doing a periodic "hprop" from the master to
each replica might not be a bad thing.

However, what happens when a slave gets an hprop dump at the same time
the slave is running ipropd-slave? Might there not be some contention?
For example, if in the middle of the hprop dump the master sends a
change through ipropd.




>
>
>
> On Fri, 24 Mar 2017, Martin Flemming wrote:
>>
>> Hi !
>>
>> I want to use the iprop-mechanism but i'm little confused
>> if all running really well ...
>>
>> My setup looks like follow
>>
>> 3 heimdal-server with Centos 7.3.1611
>>
>> heimdal-rpms from epel :
>>
>> heimdal-libs-1.6.0-0.9.20140621gita5adc06.el7.x86_64
>> heimdal-server-1.6.0-0.9.20140621gita5adc06.el7.x86_64
>>
>> On the first view everything seems to be ok :
>>
>> master :
>>
>> cat slaves-stats
>> Status for slaves, last updated: 2017-03-24T09:16:31
>>
>> Master version: 7158
>>
>> Name                             Address            Version  Status
>> Last Seen
>> iprop/[hidden email]  IPv4:192.168.6.34     7158  Up
>> 2017-03-24T09:15:01
>> iprop/[hidden email]  IPv4:192.168.6.37     7158  Up
>> 2017-03-24T09:15:01
>>
>> On both Slaves seems also everything ok
>>
>> Mar 24 09:25:02 server2 ipropd-slave[3001]: slave status change:
>> up-to-date with version: 7158 at 2017-03-24T09:25:02
>> Mar 24 09:27:32 server2 ipropd-slave[3001]: slave status change:
>> up-to-date with version: 7158 at 2017-03-24T09:27:32
>> Mar 24 09:30:01 server2 ipropd-slave[3001]: replaying entry 7159
>> Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change:
>> up-to-date with version: 7159 at 2017-03-24T09:30:01
>> Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change:
>> up-to-date with version: 7159 at 2017-03-24T09
>>
>>
>> But if i dump and count all Principals on these 3 heimdal-server,
>> i've got 3 differents results :-(
>>
>> kadmin -l list  \* > kadmin_list.txt
>> cat kadmin_list.txt |wc
>>
>>
>> Master : 31027
>> Slave1 : 30453
>> Slave2 : 29311
>>
>> Can somebody explain this behaviour or is the ipropd-mechanism buggy ?
>>
>> Or have i to upgrade to Heimdal 7.1.0 (own build from tar-package :-( ) ?
>>
>> Major changes
>>
>>    ....
>>    ....
>>    iprop has been revamped to fix a number of race conditions that
>> could lead to inconsistent replication.
>>    ....
>>    ....
>>
>> thanks & cheers
>>
>>       Martin
>>
>>
>>
>
> Tom Payerle
> IT-ETI-EUS                 [hidden email]
> 4254 Stadium Dr                (301) 405-6135
> University of Maryland
> College Park, MD 20742-4111

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipropd question

Martin Flemming
On Fri, 24 Mar 2017, Adam Lewenberg wrote:

>
>
> On 3/24/2017 7:21 AM, Thomas M. Payerle wrote:
>>  My experience is that ipropd-slave will try to apply the change log
>>  sent from the master, but if it cannot for some reason, it will ignore
>>  the change but bump the database version (and log a warning/error in
>>  its logs).  There are pros and cons of that behavior; briefly it makes
>>  the ipropd service more "robust" but also means that if one is not
>>  monitoring
>>  the logs the DBs can get out of sync.
>>
>>  The upshot is, monitoring DB synchronization by looking at slaves-stats
>>  file is NOT sufficient.
>>
>>  We generally do a listing of the names, kvnos, and last mod
>>  times for all principals on all the KDCs every night around 3-4 AM
>>  (to minimize false positives from actual updates) and compare to ensure
>>  the DBs stay synchronized.
>
> That makes me think that doing a periodic "hprop" from the master to each
> replica might not be a bad thing.
>
> However, what happens when a slave gets an hprop dump at the same time the
> slave is running ipropd-slave? Might there not be some contention? For
> example, if in the middle of the hprop dump the master sends a change through
> ipropd.
>

I wouldn't merge hprop and iprop together, we use hprop (all 15 minutes)
for years without any problems, but now we consider to switch to be little
more modern ;-)

Maybe we've to test the new version 7.1, unfortunatley without any
official packages support, only on selfmade packages ... or we still stay
in hprop-mechanism ...

Any further suggestions or experiences ?

thanks & have a nice weeekend


  martin

>
>
>>
>>
>>
>>  On Fri, 24 Mar 2017, Martin Flemming wrote:
>> >
>> >  Hi !
>> >
>> >  I want to use the iprop-mechanism but i'm little confused
>> >  if all running really well ...
>> >
>> >  My setup looks like follow
>> >
>> >  3 heimdal-server with Centos 7.3.1611
>> >
>> >  heimdal-rpms from epel :
>> >
>> >  heimdal-libs-1.6.0-0.9.20140621gita5adc06.el7.x86_64
>> >  heimdal-server-1.6.0-0.9.20140621gita5adc06.el7.x86_64
>> >
>> >  On the first view everything seems to be ok :
>> >
>> >  master :
>> >
>> >  cat slaves-stats
>> >  Status for slaves, last updated: 2017-03-24T09:16:31
>> >
>> >  Master version: 7158
>> >
>> >  Name                             Address            Version  Status
>> >  Last Seen
>> >  iprop/[hidden email]  IPv4:192.168.6.34     7158  Up
>> >  2017-03-24T09:15:01
>> >  iprop/[hidden email]  IPv4:192.168.6.37     7158  Up
>> >  2017-03-24T09:15:01
>> >
>> >  On both Slaves seems also everything ok
>> >
>> >  Mar 24 09:25:02 server2 ipropd-slave[3001]: slave status change:
>> >  up-to-date with version: 7158 at 2017-03-24T09:25:02
>> >  Mar 24 09:27:32 server2 ipropd-slave[3001]: slave status change:
>> >  up-to-date with version: 7158 at 2017-03-24T09:27:32
>> >  Mar 24 09:30:01 server2 ipropd-slave[3001]: replaying entry 7159
>> >  Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change:
>> >  up-to-date with version: 7159 at 2017-03-24T09:30:01
>> >  Mar 24 09:30:01 server2 ipropd-slave[3001]: slave status change:
>> >  up-to-date with version: 7159 at 2017-03-24T09
>> >
>> >
>> >  But if i dump and count all Principals on these 3 heimdal-server,
>> >  i've got 3 differents results :-(
>> >
>> >  kadmin -l list  \* > kadmin_list.txt
>> >  cat kadmin_list.txt |wc
>> >
>> >
>> >  Master : 31027
>> >  Slave1 : 30453
>> >  Slave2 : 29311
>> >
>> >  Can somebody explain this behaviour or is the ipropd-mechanism buggy ?
>> >
>> >  Or have i to upgrade to Heimdal 7.1.0 (own build from tar-package :-( )
>> >  ?
>> >
>> >  Major changes
>> >
>> >     ....
>> >     ....
>> >     iprop has been revamped to fix a number of race conditions that
>> >  could lead to inconsistent replication.
>> >     ....
>> >     ....
>> >
>> >  thanks & cheers
>> >
>> >        Martin
>> >
>> >
>> >
>>
>>  Tom Payerle
>>  IT-ETI-EUS                 [hidden email]
>>  4254 Stadium Dr                (301) 405-6135
>>  University of Maryland
>>  College Park, MD 20742-4111
>
>

Gruss

        Martin Flemming


______________________________________________________
Martin Flemming
DESY / IT          office : Building 2b / 008a
Notkestr. 85       phone  : 040 - 8998 - 4667
22603 Hamburg      mail   : [hidden email]
______________________________________________________
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipropd question

Nico Williams
In reply to this post by Thomas M. Payerle-3
On Fri, Mar 24, 2017 at 10:21:13AM -0400, Thomas M. Payerle wrote:
> My experience is that ipropd-slave will try to apply the change log
> sent from the master, but if it cannot for some reason, it will ignore
> the change but bump the database version (and log a warning/error in
> its logs).

I believe that's true.  It should probably trigger a full prop instead.
Perhaps it should schedule one for later so as to prevent a tight loop
of full props if some bug could trigger repeated failures to apply
incrementals.

I've opened issue #269 for this on github.

>             There are pros and cons of that behavior; briefly it makes
> the ipropd service more "robust" but also means that if one is not monitoring
> the logs the DBs can get out of sync.

I think it's just not robust.

Nico
--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipropd question

Nico Williams
In reply to this post by Adam Lewenberg
On Fri, Mar 24, 2017 at 08:09:10AM -0700, Adam Lewenberg wrote:
> That makes me think that doing a periodic "hprop" from the master to each
> replica might not be a bad thing.

Don't mix iprop and hprop.

To force a full prop with Heimdal 7.1 and up use

$ iprop-log truncate --reset

Otherwise kill the ipropd-slave process, remove the iprop log, and
restart the ipropd-slave process.

> However, what happens when a slave gets an hprop dump at the same time the
> slave is running ipropd-slave? Might there not be some contention? For
> example, if in the middle of the hprop dump the master sends a change
> through ipropd.

Right, don't do that.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipropd question

Henry B (Hank) Hotz, CISSP-2
In reply to this post by Adam Lewenberg

> On Mar 24, 2017, at 8:09 AM, Adam Lewenberg <[hidden email]> wrote:
>
>
>
> On 3/24/2017 7:21 AM, Thomas M. Payerle wrote:
>> My experience is that ipropd-slave will try to apply the change log
>> sent from the master, but if it cannot for some reason, it will ignore
>> the change but bump the database version (and log a warning/error in
>> its logs).  There are pros and cons of that behavior; briefly it makes
>> the ipropd service more "robust" but also means that if one is not
>> monitoring
>> the logs the DBs can get out of sync.
>>
>> The upshot is, monitoring DB synchronization by looking at slaves-stats
>> file is NOT sufficient.
>>
>> We generally do a listing of the names, kvnos, and last mod
>> times for all principals on all the KDCs every night around 3-4 AM
>> (to minimize false positives from actual updates) and compare to ensure
>> the DBs stay synchronized.
>
> That makes me think that doing a periodic "hprop" from the master to each replica might not be a bad thing.
>
> However, what happens when a slave gets an hprop dump at the same time the slave is running ipropd-slave? Might there not be some contention? For example, if in the middle of the hprop dump the master sends a change through ipropd.

I’d recommend forcing iprop to do a full download instead. This used to be entirely too easy to trigger, but there has been some work since. “False” downloads seemed to be more common when you have a lot of slaves.

I don’t know what the “recommended” way to trigger a re-download would be, but certainly if you delete the kdb and the log file on a slave and start ipropd-slave it will re-download. Just make sure you don’t have the slave’s kdc running until the new kdb is available or it will be telling people (including its own ipropd-slave!!) that principals don’t exist when it shouldn’t!

Personal email.  [hidden email]



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipropd question

Henry B (Hank) Hotz, CISSP-2
In reply to this post by Nico Williams

> On Mar 24, 2017, at 1:57 PM, Nico Williams <[hidden email]> wrote:
>
> To force a full prop with Heimdal 7.1 and up use
>
> $ iprop-log truncate —reset

Ah!


Personal email.  [hidden email]



Loading...