krb5-1.12 beta2 - New Feature - iprop notify

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

krb5-1.12 beta2 - New Feature - iprop notify

Richard Basch
This implements a framework whereby when kadmind processes an update, it
notifies the slaves of pending updates (akin to DNS notify events). This
facilitates more real-time synchronization, and when combined with my
tree-based propagation patches, should be quite scalable for most
environments, especially global environments where WAN usage/latency may be
of concern.  This latest patch does layer atop my prior contributions.

 

https://github.com/rbasch/krb5/commit/69b9221daa77679aca117e880a0eac6e8f44ef
28

 

So far, I am quite pleased with my preliminary tests. There are still fringe
conditions which might result in up to a 1 second processing lag, but this
lag only exists with multi-tier configurations or when performing database
edits using kadmin.local instead of via the kadm5 protocol.

 

As author (and since these were developed on personal time with no employer
intellectual property encumbrances), I hereby authorize MIT to publish these
patches within the Kerberos source under the MIT copyright.

 

 

Full set of patches:

 

Allow slaves to locally track ulog updates (facilitates switching roles/role
reversals without forcing full propagation)

https://github.com/rbasch/krb5/commit/22bbe9461e354399c8e1e8924be7596ad0520e
aa

https://github.com/rbasch/krb5/commit/33bb7a9cb79b31f227bbd863ddc4aec639285a
03

 

Tree-based replication (implements new command-line options in kadmind &
kpropd)

https://github.com/rbasch/krb5/commit/e976d70e2a682f8ba081cc8f6aaf5d6ed9123b
b1

 

Notify slaves of pending updates (implements new command-line option in
kprop)

https://github.com/rbasch/krb5/commit/69b9221daa77679aca117e880a0eac6e8f44ef
28

 

 

Please feel free to contact me with any questions regarding the above
contribution (I already provided my contact information under separate
cover).

 

 

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: krb5-1.12 beta2 - New Feature - iprop notify

Nico Williams
On Mon, Dec 2, 2013 at 12:15 AM, Richard Basch <[hidden email]> wrote:
> This implements a framework whereby when kadmind processes an update, it
> notifies the slaves of pending updates (akin to DNS notify events). This
> facilitates more real-time synchronization, and when combined with my
> tree-based propagation patches, should be quite scalable for most
> environments, especially global environments where WAN usage/latency may be
> of concern.  This latest patch does layer atop my prior contributions.

Nice idea.  A few questions and comments:

 - since the live notified slaves will contact kadmind again, why not
reset the list of slaves to notify every time, thus weeding out dead
slaves?

 - where are the notification child processes reaped?

 - use sigaction(), not signal()

 - running the slave notification from a signal handler seems... like
asking for trouble, largely because you have to make sure you're only
calling async-signal-safe code from signal handlers (at least the way
adding slaves to the list works there's no concern about racing
between that and the handler, but you'll need to be careful about how
you weed out dead slaves)

Nico
--
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

RE: krb5-1.12 beta2 - New Feature - iprop notify

Richard Basch
In reply to this post by Richard Basch
One additional patch (command-line option documentation):

https://github.com/rbasch/krb5/commit/3cc5b70eed0c99d95f6939559acd9fd334286c
c9

 

. . .

 

This implements a framework whereby when kadmind processes an update, it
notifies the slaves of pending updates (akin to DNS notify events). This
facilitates more real-time synchronization, and when combined with my
tree-based propagation patches, should be quite scalable for most
environments, especially global environments where WAN usage/latency may be
of concern.  This latest patch does layer atop my prior contributions.

 

https://github.com/rbasch/krb5/commit/69b9221daa77679aca117e880a0eac6e8f44ef
28

 

So far, I am quite pleased with my preliminary tests. There are still fringe
conditions which might result in up to a 1 second processing lag, but this
lag only exists with multi-tier configurations or when performing database
edits using kadmin.local instead of via the kadm5 protocol.

 

As author (and since these were developed on personal time with no employer
intellectual property encumbrances), I hereby authorize MIT to publish these
patches within the Kerberos source under the MIT copyright.

 

 

Full set of patches:

 

Allow slaves to locally track ulog updates (facilitates switching roles/role
reversals without forcing full propagation)

https://github.com/rbasch/krb5/commit/22bbe9461e354399c8e1e8924be7596ad0520e
aa

https://github.com/rbasch/krb5/commit/33bb7a9cb79b31f227bbd863ddc4aec639285a
03

 

Tree-based replication (implements new command-line options in kadmind &
kpropd)

https://github.com/rbasch/krb5/commit/e976d70e2a682f8ba081cc8f6aaf5d6ed9123b
b1

 

Notify slaves of pending updates (implements new command-line option in
kprop)

https://github.com/rbasch/krb5/commit/69b9221daa77679aca117e880a0eac6e8f44ef
28

 

 

Please feel free to contact me with any questions regarding the above
contribution (I already provided my contact information under separate
cover).

 

 

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

RE: krb5-1.12 beta2 - New Feature - iprop notify

Richard Basch
In reply to this post by Richard Basch
I finally got everything working, I think.

 

https://github.com/rbasch/krb5/compare/krb5-1.12?expand=1

 

The relevant commits are:

 

Allow slaves to locally track ulog updates (facilitates switching roles
without forcing full propagation)

https://github.com/rbasch/krb5/commit/22bbe9461e354399c8e1e8924be7596ad0520e
aa

https://github.com/rbasch/krb5/commit/33bb7a9cb79b31f227bbd863ddc4aec639285a
03

 

Tree-based replication (implements new command-line options in kadmind &
kpropd)

https://github.com/rbasch/krb5/commit/e976d70e2a682f8ba081cc8f6aaf5d6ed9123b
b1

 

Notify slaves of pending updates (implements new command-line option in
kprop)

https://github.com/rbasch/krb5/commit/69b9221daa77679aca117e880a0eac6e8f44ef
28

https://github.com/rbasch/krb5/commit/c5c0a1db657b7e3eabfb8b3bc13a3f9192fb68
8c

(NEW; removed timer code from prior patch and implements an event-based
trigger for downstream slaves)

 

Man page updates for new command-line switches

https://github.com/rbasch/krb5/commit/3cc5b70eed0c99d95f6939559acd9fd334286c
c9

 

 

Unfortunately, the GIT commits are more complicated than needed be since
c5c0a1d reverses part of 69b9221 (I reverted the timer-based trigger and
implemented an event-based trigger for the downstream slaves once I realized
I could use the NULLPROC method as a trigger).

 

Let me know if you have questions.

 

 

From: Richard Basch [mailto:[hidden email]]
Sent: Monday, December 02, 2013 1:16 AM
To: [hidden email]
Cc: [hidden email]; 'Richard Basch'; [hidden email]; [hidden email]
Subject: krb5-1.12 beta2 - New Feature - iprop notify

 

This implements a framework whereby when kadmind processes an update, it
notifies the slaves of pending updates (akin to DNS notify events). This
facilitates more real-time synchronization, and when combined with my
tree-based propagation patches, should be quite scalable for most
environments, especially global environments where WAN usage/latency may be
of concern.  This latest patch does layer atop my prior contributions.

 

https://github.com/rbasch/krb5/commit/69b9221daa77679aca117e880a0eac6e8f44ef
28

 

So far, I am quite pleased with my preliminary tests. There are still fringe
conditions which might result in up to a 1 second processing lag, but this
lag only exists with multi-tier configurations or when performing database
edits using kadmin.local instead of via the kadm5 protocol.

 

As author (and since these were developed on personal time with no employer
intellectual property encumbrances), I hereby authorize MIT to publish these
patches within the Kerberos source under the MIT copyright.

 

 

Full set of patches:

 

Allow slaves to locally track ulog updates (facilitates switching roles/role
reversals without forcing full propagation)

https://github.com/rbasch/krb5/commit/22bbe9461e354399c8e1e8924be7596ad0520e
aa

https://github.com/rbasch/krb5/commit/33bb7a9cb79b31f227bbd863ddc4aec639285a
03

 

Tree-based replication (implements new command-line options in kadmind &
kpropd)

https://github.com/rbasch/krb5/commit/e976d70e2a682f8ba081cc8f6aaf5d6ed9123b
b1

 

Notify slaves of pending updates (implements new command-line option in
kprop)

https://github.com/rbasch/krb5/commit/69b9221daa77679aca117e880a0eac6e8f44ef
28

 

 

Please feel free to contact me with any questions regarding the above
contribution (I already provided my contact information under separate
cover).

 

 

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: krb5-1.12 beta2 - New Feature - iprop notify

Nico Williams
On Mon, Dec 9, 2013 at 9:23 PM, Richard Basch <[hidden email]> wrote:
> Unfortunately, the GIT commits are more complicated than needed be since
> c5c0a1d reverses part of 69b9221 (I reverted the timer-based trigger and
> implemented an event-based trigger for the downstream slaves once I realized
> I could use the NULLPROC method as a trigger).

This is why git rebase and git add -p/-e are such a wonderful
things...  I'm not saying that re-writing history is easy or fun, but
that it sure is helpful.

What I usually do is I create a new branch same as the old then rebase
-i one of them, then publish both so that upstream can see the
differences between the two branches (if any, besides the history),
and still we get clean history (plus I get my old history as a backup
in case I screw up a rebase, and it's useful for, e.g., showing how I
went from A to B).

I don't know how familiar you are with git rebase -i and git add -p/e,
so I'll write you a crash-course assuming you're not at all familiar
with them.

(There's plenty of videos and wiki and stackoverflow pages on all of
this, but hopefully I can a good enough job to get you started.)

Running

    git rebase -i origin/master

will enter interactive rebase mode.  In this mode git first runs
$EDITOR with a recipe consisting of all the commits between the merge
base with origin/master and HEAD, then when you exit $EDITOR git
applies the recipe.

You get to edit this recipe as follows:

 - any commits you don't want, just delete the corresponding "pick" entries

 - any commits you want to re-order, just move the corresponding
"pick" entries about so they come in the desired order

 - any two or more consecutive commits you want to merge into one,
just change the "pick" verb of all but the first to "squash" (merges
the commit messages too) or "fixup" (discards the commit message of
the fixup commits)

 - any commit whose message you want to reword, s/pick/reword/

 - any commit you want to split or edit, s/pick/edit/ -- this last is
very special, more on that below.

FYI, "pick" refers to git cherry-pick.  A recipe of nothing but picks
is akin to manually executing git cherry-pick for each of those
commits, in the order that they appear in the recipe.

Exit without saving to abort -- it's safe.

Save and exit to start the rebase.  Git will apply the recipe, one
action at a time, picking commits and doing whatever additional action
(like squashing commits).

If at any point there's a merge conflict git status will show them and
exit.  You then get your shell back and you must edit the affected
files to clear up the conflicts.  When you've cleared up the conflicts
you must git add those files before continuing.  When conflicts are
resolved you then run git rebase --continue.

If at any point you need to start over just git rebase --abort.  It
will undo everything and leave your branch as it was.  If you finish a
git rebase and are unhappy with the result you can file the old HEAD
for your branch in the git reflog (or in any backup branches you
made).

As for commits you want to split or change, you get your shell back
right after picking the commit in question.  To split a commit to a
file, just git reset HEAD^ that file, edit it, then git add -p, git
commit, then git add and git rebase --continue.  To add a new commit
you 'edit' the commit you want to add after, edit files, git add them,
then git commit, then git rebase --continue.

As for git add -p/-e, that lets git add only portions of extant
changes in the workspace, and you even get to edit the diffs to add to
the index.

It's hard to overstate just how powerful a feature this is.  And how
wonderful.  This is _teh_ killer feature of git.

Nico
--
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev