Defining classes from external sources

[tentaculartentacle]

I'm trying to find a scalable way to define my inputs/bundlesequences based on classes. I have a cmdb, and I've created a bundle to query this and output stuff like +devserver +webrole +dmznet +3rdpartyccess +likesthecolormauve.

Ideally I'd like to use dynamic bundlesequences and/or inputs based on those classes, but it seems the bundlesequences/inputs are parsed before the query bundle can be run....

Is there a way around this, or some other approach I can use? Please keep in mind, hard-coding the classes like every example shows is not an option for lots of good and some not-so-good reasons. I need to define classes dynamically from an external source.

[Marco Marongiu]

On 12/13/2014 10:45 AM, tentaculartentacle wrote:
> Is there a way around this, or some other approach I can use?

Yes. Have the bundlesequence call just the bundle that queries the CMDB
and then a "main" bundle, and in main you'll call other bundles as
methods promises, conditioned by classes.

Sounds good?

[Brian Bennett]

You might be interested in reading my article about autorun where I detail how to do just that.

https://digitalelf.net/2014/07/a-primer-on-cfengine-3-dot-6-autorun/

The methodology I use runs bundles based off of defined classes. In short, if you have bundles named the same as the classes you define then they will run on any instances where those classes are defined. This will combine well with your bundle that defines classes based on queries to the cmdb.

-- 

Brian Bennett

Looking for CFEngine training?

http://www.verticalsysadmin.com/

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[tentaculartentacle]

On Saturday, December 13, 2014 3:55:54 PM UTC+1, brontolinux wrote:

Yes. Have the bundlesequence call just the bundle that queries the CMDB
and then a "main" bundle, and in main you'll call other bundles as
methods promises, conditioned by classes.

Sounds good?

Yeah that works fine for the bundlesequence, thanks. Is there any way to do something similar for "inputs" - get the one bundle executed and before inputs, or add to inputs "on the fly" from within a bundle?

It may sound silly at first but I don't really like the idea that for ex dev/prod read from the same code. I'd like to allow looser ACLs and controls on dev for example.

Splitting into entirely separate cfengine instances seems overkill and not very scalable.

[Brian Bennett]

You should have different servers for dev and prod. Use the same code base and promote to prod rather than hand editing policies.

You should really have at least 3 environments. Dev, Staging and Prod. Keep your policies in version control (e.g. git) and promote commits and/or branches. Otherwise if you have only one set of policies and some of them apply to dev while others apply to prod then you need to hand edit prod policies for each deployment so you're moving untested code directly to prod. Would you let a developer do that?

-- 

Brian Bennett

Looking for CFEngine training?

http://www.verticalsysadmin.com/

[Neil Watson]

Try this:
https://groups.google.com/forum/?fromgroups#!searchin/help-cfengine/common$20bundles$20watson/help-cfengine/SIZuyk8iNws/QlCQVsxT5T0J

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[tentaculartentacle]

On Sunday, December 14, 2014 4:12:51 PM UTC+1, Brian Bennett wrote:

You should have different servers for dev and prod. Use the same code base and promote to prod rather than hand editing policies.

You should really have at least 3 environments. Dev, Staging and Prod. Keep your policies in version control (e.g. git) and promote commits and/or branches. Otherwise if you have only one set of policies and some of them apply to dev while others apply to prod then you need to hand edit prod policies for each deployment so you're moving untested code directly to prod. Would you let a developer do that?

I agree, of course, the code has to be separated, and code should not require any edits as it moves from dev upwards. Separate servers was my initial solution. But, it's not just 3 servers instead of one, it's 3 times the servers...which in some places I've worked means like 27 servers instead of 9 (heavy segregation of environments for technical, regulatory, security, redundancy, etc reasons....). In such scenarios I really want to keep everything as minimally complex as possible because the environment policy makes everything ridiculously complex by default. Every speck of added complexity in the initial design snowballs into a hellish avalanche.

 

That's why I thought of having a separate subdir for each environment and use "inputs" to control which servers read which source based on their class. It keeps all the code in one place (easy for me to control, audit, etc) while still having it impossible for a prod server to accidentally (or maliciously) use dev code. I'll check out code from git and validate it, deploy it in dev, and when I'm ready i just rsync the bundles from dev/ directory to staging/, and from there on to prod/. No maintenance or edits on staging+prod is ever needed, they're just copies of dev. Only a handful have the power to promote code like this and I have no extra servers to maintain (or pay for). I can also (if I must...) scale this out to an arbitrary amount of possible future classes/codebases, e.g. branch offices, 3rd party shared environments, etc...which I can't do easily with full server segregation.

My second best (hacky) idea was to not use cfengine for pushing policy. I could have these separate subdirs and then just use some other agent on the clients to fetch the policy files from the appropriate location. But I don't like that for relatively obvious reasons....

Third idea, was to actually use an input source like /var/cfengine/inputs/me and based on the class, symlink me to dev/staging/prod/etc.... but this just smells bad from the start.

[Danny Sauer]

I do this by creating a directory for each host, and putting a file in each directory named "classes."  The classes file - one class per line - is updated using a trigger on a database which identifies all known hosts which match the pattern where the class is to be applied, and regenerating the files for each modified host.  While I'm still hoping to replace the physical files with a FUSE implementation which dynamically queries the DB and generates the files on-demand (there are a few other files, including passwd/group/shadow), the current approach has scaled acceptably to tens of thousands of hosts.  It feels kind of brute-force, but is also simple to integrate with our external config system.

Anyway, each host pulls down "/config/$(sys.uqhost)" into "$(sys.workdir)/hostcfg".  So, the CFEngine policy in promises.cf simply reads the "/var/cfengine/hostcfg/classes" file into an slist and then defines a class for each element in the list.  Some of those classes (those which begin with "servertype_") are then used to define elements in arrays named b[] and i[].  The b and i arrays get passed into getindices() at the end fo promises.cf, which populates "bundle" and "include" slists, which are themselves then the only value within the bundlesequence and the include parameters in the common control body.

This worked really well up through 3.5.x, so long as everything in promises.cf was in the right execution order from the top down.  However, with 3.6.x, the behavior of promises.cf has seemingly changed a little.  I just noticed today, but it seems that one part of the "read a file, build an slist, define the classes" sequence isn't done until the second pass through promises.cf, which allows the bundle sequence to be modified later but does not seemingly alter the include list, The end result is that I'm seeing "bundle not defined" issues for bundles defined in included files which are controlled by those file-based classes.  I have a couple of ideas on how to work around the new behavior, but haven't actually implemented them yet.  Surely something will work. :)  Until then, if you were to include "everything" (perhaps with output from lsdir) and only make the bundle sequence dynamic, that'd work fine with the above mechanism.

--Danny

[Alex Georgopoulos]

I've also noticed the evaluation issues in 3.6 you describe.  I've had to resort to having to use two bundles to make things work like they did in 3.5.  You might be able to get the clases from your files by doing something like this.

body common control

{

 bundlesequence => {

                 "prep_classes",

                 "set_classes"

                };

}

bundle common prep_classes

{

vars:

    "host_classes_file" string => "/tmp/host_classes_file.txt";

    "host_classes" slist => readstringlist("$(host_classes_file)","\s*#[^\n]*","\n",99999,9999999);

classes:

    "set_test_class" expression => returnszero("/bin/echo my_test_class > $(host_classes_file)","useshell");

reports:

    linux::

    "$(host_classes_file)";

}

bundle common set_classes

{

classes:

    "$(prep_classes.host_classes)" expression => "any";

reports:

    my_test_class::

    "PASS";

}

 

[Nick Anderson]

Also I believe you can't use body file control to include files where those classification bundles live.

Several things going on here it's under active investigation.

Sent from my mobile device.

--