Refactoring standard_services

[Nick Anderson]

The standard_services bundle is quite long and I find it a bit hard to
track all thats going on.

I have been thinking that perhaps we should break it up into
implementation specific bundles which standard_services can call as it
does the classic services bundle.

Pull request illustrating the idea for systemd:
https://github.com/cfengine/masterfiles/pull/754

Thoughts?

[Mike Svoboda]

I spent quite a bit of time refactoring standard_services to work better with systemd.

https://groups.google.com/forum/#!topic/help-cfengine/2Vd7op97Tz4

We actually spend 95% + of our CPU time in policy in standard_services.  I haven't had time to trace down where we're spinning at, but I'm guessing its probably the regex parsing with all of the process promises.

With the implementation above, we probe systemd's state once, raise namespace classes, and then use those classes for the rest of execution.   Basically, we only enter the policy once and use the cached raised classes from the first pass, instead of entering the policy a million times to re-evaluate process state.

Its orders of magnitude more efficient.  

--
You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
To post to this group, send email to dev-cf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/576C6297.8070000%40cfenigne.com.
For more options, visit https://groups.google.com/d/optout.

[Nick Anderson]

Oh, great read. Thanks Mike!

[Ted Zlatanov]

On Thu, 23 Jun 2016 21:07:53 -0400 Mike Svoboda <michael....@gmail.com> wrote:

MS> I spent quite a bit of time refactoring standard_services to work better
MS> with systemd.
MS> https://groups.google.com/forum/#!topic/help-cfengine/2Vd7op97Tz4

MS> We actually spend 95% + of our CPU time in policy in standard_services. I
MS> haven't had time to trace down where we're spinning at, but I'm guessing
MS> its probably the regex parsing with all of the process promises.

MS> With the implementation above, we probe systemd's state once, raise
MS> namespace classes, and then use those classes for the rest of execution.
MS> Basically, we only enter the policy once and use the cached raised classes
MS> from the first pass, instead of entering the policy a million times to
MS> re-evaluate process state.

MS> Its orders of magnitude more efficient.

If you're willing to make a PR that switches to your improved version
with systemd, it would benefit everyone using the stock masterfiles,
especially if it's backwards compatible.

Ted

[Mike Svoboda]

> If you're willing to make a PR that switches to your improved version with systemd, it would benefit everyone using the stock masterfiles, especially if it's backwards compatible.

Unfortunately its not backwards compatible.  Its not a "library" policy.  We actively execute it.   It has to be one of the first policies we execute, since the classes it raises after the initial systemd state probe are used throughout policy execution.  

So, a user cant simply do

services:

  "blah"  service_policy => "start"

Without executing our "discovery policy" first.  The discovery policy executes some python, which probes the entire systemd state and populates variables in policy via usemodule.

http://pastebin.com/Zb1CYHUk

I did include the source code here:

https://groups.google.com/d/msg/help-cfengine/8ZroF8EvJLQ/m_KF2i5lCQAJ

I can upload it somewhere in git, but its an architectural change to service management.   Cfengine probes the state of systemd via a policy actively early in execution, and thats it.   All policies used that cached information.  

Right now, standard_services is used more of a passive "library" than an actively executed policy.  Using our infrastructure is a change in service management architecture.  

--
You received this message because you are subscribed to the Google Groups "dev-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-cfengine...@googlegroups.com.
To post to this group, send email to dev-cf...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dev-cfengine/87eg7my01u.fsf%40lifelogs.com.

[Ted Zlatanov]

On Fri, 24 Jun 2016 13:45:11 -0400 Mike Svoboda <michael....@gmail.com> wrote:

MS> Unfortunately its not backwards compatible. Its not a "library" policy.
MS> We actively execute it. It has to be one of the first policies we
MS> execute, since the classes it raises after the initial systemd state probe
MS> are used throughout policy execution.

It sounds like core support for the systemd D-Bus API would be
beneficial, so all of this discovery can happen at the C level on
startup: https://www.freedesktop.org/wiki/Software/systemd/dbus/

That would also allow the services promises to control systemd directly
instead of using the clumsy command-line interface, which will probably
be even faster for you and everyone else. Finally, it potentially allows
CFEngine to subscribe to systemd events.

Ted