Autorun with hardclasses

[berntjernberg]

Hi all,

Do you see any pitfalls using hard classes as inputs to bundlesmatching?

body file control {
    agent::
        inputs => { @(my_autorun.inputs) };
}

bundle agent my_autorun {

    meta:
        "tags" slist => { "autorun" };

    vars:
        "bundle" slist => bundlesmatching(".*", "^(any|$(sys.class)|$(sys.uqhost))$");
        "inputs" slist => findfiles("$(this.promise_dirname)/.*\*.cf");

    methods:
        "$(bundle)" usebundle => "$(bundle)";

    reports:
        inform_mode::
            "autorun is executing";
            "$(this.bundle): found bundle $(bundle)";
}

bundle agent my_any
{
    meta:
        "tags" slist => { "any" };

    reports:
        any::
            "$(this.bundle): any bundle";
}

bundle agent my_solaris
{
    meta:
        "tags" slist => { "solaris" };

    reports:
        any::
            "$(this.bundle): solaris bundle";
}

bundle agent my_aix
{
    meta:
        "tags" slist => { "aix" };

    reports:
        any::
            "$(this.bundle): aix bundle";
}

bundle agent my_linux
{
    meta:
        "tags" slist => { "linux" };

    reports:
        any::
            "$(this.bundle): linux bundle";
}

bundle agent my_host
{
    meta:
        "tags" slist => { "ahostname" };

    reports:
        any::
            "$(this.bundle): host bundle";
}

Bernt Jernberg

[Ted Zlatanov]

On Thu, 14 May 2015 10:10:32 -0700 (PDT) berntjernberg <bernt.j...@gmail.com> wrote:

b> Do you see any pitfalls using hard classes as inputs to bundlesmatching?
...
b> vars:
b> "bundle" slist => bundlesmatching(".*",
b> "^(any|$(sys.class)|$(sys.uqhost))$");

I think that's really nice and readable :)

Martin Simons was doing something like that for the roadshow policies by
hand, so this way would simplify them significantly! And I'm sure there
are many other users who would benefit similarly.

Ted

[Bernt Jernberg]

I think that's really nice and readable :)

Thanks! :-)
Bernt

[Brian Bennett]

Not at all. This is exactly what I do. And I've always structured my policies around discovered hard classes.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Bernt Jernberg]

If we add sorting:

bundle agent my_autorun {

    meta:
        "tags" slist => { "autorun" };

    vars:

            "bundle_input" slist => bundlesmatching(".*", "^(any|$(sys.class)|$(sys.uqhost))$");

            "inputs" slist => findfiles("$(this.promise_dirname)/.*\*.cf");

            "bundle" slist => sort("bundle_input","lex");

    methods:
        "$(bundle)" usebundle => "$(bundle)";
}

put common stuff in a file called any.cf:

bundle agent my_solaris
{
    meta:
        "tags" slist => { "shelley" };

    reports:
        any::
            "$(this.bundle): solaris bundle";
}
bundle agent my_aix
{
    meta:
        "tags" slist => { "aix" };

    reports:
        any::
            "$(this.bundle): aix bundle";
}
bundle agent my_linux
{
    meta:
        "tags" slist => { "linux" };

    reports:
        any::

            "$(this.bundle): Linux bundle";

}
bundle agent my_any
{
    meta:
        "tags" slist => { "any" };

    reports:
        any::
            "$(this.bundle): any bundle";
}

globals in global.cf

bundle common 1_global

{
    meta:
        "tags" slist => { "any" };

    reports:
        any::

            "$(this.bundle): global bundle";
}

bundle common 2_global

{
    meta:
        "tags" slist => { "any" };

    reports:
        any::

            "$(this.bundle): global bundle";
}

things to run last in last.cf:

bundle common zz1_last

{
    meta:
        "tags" slist => { "any" };

    reports:
        any::

            "$(this.bundle): last bundle";
}
bundle common zz2_last

{
    meta:
        "tags" slist => { "any" };

    reports:
        any::

            "$(this.bundle): last bundle";
}

then on a Solaris node I get:

# /var/cfengine/bin/cf-agent -IKf ./promises.cf
R: 1_global: global bundle
R: 2_global: global bundle
R: my_any: any bundle
R: my_solaris: solaris bundle
R: zz1_last: last bundle
R: zz2_last: last bundle

I know, the bundle names aren't that exiting but ordering works.

Bernt

[Bernt Jernberg]

Arghhh.....

the last.cf bundles should read bundle agent, not common.

Bernt

[Brian Bennett]

This is almost precisely the same structure I use.

https://digitalelf.net/2014/07/a-primer-on-cfengine-3-dot-6-autorun/

https://digitalelf.net/2013/04/a-case-study-in-cfengine-layout/

On May 14, 2015, at 10:10 AM, berntjernberg <bernt.j...@gmail.com> wrote:

[Bernt Jernberg]

Thanks for your answers,

I have consulted your documents, they are great! :-)
I didn't want to "hardcode" to bundle names as you do with:

methods:
      "$(bundle)"
          usebundle => "$(bundle)",
          ifvarclass => "$(bundle)";

so I match on meta tags instead.

Bernt

[Ted Zlatanov]

On Fri, 15 May 2015 03:37:24 -0700 (PDT) Bernt Jernberg <bernt.j...@gmail.com> wrote:

BJ> I didn't want to "hardcode" to bundle names as you do with:

BJ> methods:
BJ> "$(bundle)"
BJ> usebundle => "$(bundle)",
BJ> ifvarclass => "$(bundle)";

BJ> so I match on meta tags instead.

Yeah, the reasons I like Bernt's approach are:

- the hostname can be a tag, but not a class (because of the dashes)

- you don't have to name your bundles in a special way, and can have multiples per tag

- it's much easier to retrofit into an existing setup

It's exciting to me to watch the autorun capabilities and the meta tags
that were only just added in 3.6 evolve like this!

Ted

[Brian Bennett]

> On May 15, 2015, at 4:54 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
>
> - the hostname can be a tag, but not a class (because of the dashes)

You can use host names with dashes, just write them as underscores. The same goes for the FQDN.

And hard classes aren't the *only* bundle names I use. I trigger additional bundles by defining whichever class names I want.

--
Brian

[Martin Simons]

Dear Bernt;

Very interesting indeed.

TZ> I think that's really nice and readable :)

TZ> Martin Simons was doing something like that for the roadshow policies by
TZ> hand, so this way would simplify them significantly!

I am always striving to abstact the policies and to simplify them.
https://github.com/Webhuis/CFEngine-Roadshow/blob/cfgmgmtcamp2015/policies/masterfiles/services/autorun/roadshow.cf

This is part of the entry point for The CFEngine Roadshow through the autorun facility:

bundle agent roadshow

{
  meta:
      "tags" slist => { "autorun" };

  methods:
      "" usebundle => classify;
      "" usebundle => do_roadshow;
      "" usebundle => cfe_roadshow_update;
      "" usebundle => ${do_roadshow.class_domain};
      "" usebundle => $(do_roadshow.os);
      "" usebundle => $(classify.role);

The idea behind the roadhow is to have a role based aproach, the role defines the machine. Classify takes care of deriving the roles from the host name standardized by the hostnaming convention. I hardly use classes in navigation, they have to be omnipresent and disclose the architecture, I prefer to use sys.variables and "soft" variables like the role.

The CFEngine now offers:
1. The plaform layer, CentOS x, Debian x and I am working on Suse.
2. The middleware layer
3. The applications

This will be demonstrated live at: https://www.nluug.nl/events/vj15/programma.html

You are more than welcome to introduce any improvements along the lines you mentioned in your post.

Best regards,
Martin.

Op donderdag 14 mei 2015 19:54:11 UTC+2 schreef Ted Zlatanov:

[Bernt Jernberg]

The idea behind the roadhow is to have a role based aproach, the role defines the machine. Classify takes care of deriving the roles from the host name standardized by the hostnaming convention. I hardly use classes in navigation, they have to be omnipresent and disclose the architecture, I prefer to use sys.variables and "soft" variables like the role.

The CFEngine now offers:
1. The plaform layer, CentOS x, Debian x and I am working on Suse.
2. The middleware layer
3. The applications

Actually, the subject of this thread is misleading, I use variables as well. I should have named it
Autorun with meta tags. Even though any is a hard class, it is just a string here in meta context.
The tags list of every bundle is usually a variable.

This will be demonstrated live at: https://www.nluug.nl/events/vj15/programma.html

You are more than welcome to introduce any improvements along the lines you mentioned in your post.

Thanks.

Bernt

 

[Ted Zlatanov]

On Sat, 16 May 2015 08:21:53 -0700 Brian Bennett <brian....@verticalsysadmin.com> wrote:

>> On May 15, 2015, at 4:54 AM, Ted Zlatanov <t...@lifelogs.com> wrote:
>>
>> - the hostname can be a tag, but not a class (because of the dashes)

BB> You can use host names with dashes, just write them as underscores. The same goes for the FQDN.

Sorry for the confusion. I was saying there's no transformation needed
when you use hostnames as tags. When you use them as class names or
bundle names, you need to canonify the hostname. It's thus a bit more
convenient to use a hostname as a tag (in addition to allowing multiple
bundles with the same tag).

BB> And hard classes aren't the *only* bundle names I use. I trigger
BB> additional bundles by defining whichever class names I want.

Right, yeah :)

Ted

[Ted Zlatanov]

On Tue, 19 May 2015 12:35:04 -0700 (PDT) Martin Simons <mjcm....@gmail.com> wrote:

MS> "" usebundle => ${do_roadshow.class_domain};
MS> "" usebundle => $(do_roadshow.os);
MS> "" usebundle => $(classify.role);

Bernt's approach would replace all three of these with a slist
iteration:

vars:
"mybundles" slist => bundlesmatching(".*", "^(${do_roadshow.class_domain}|$(do_roadshow.os)|$(classify.role))$");

methods:
"run" usebundle => $(mybundles);

...and then you can have multiple bundles with the same
`$(classify.role)` tag, for instance, instead of just one. It's much
more dynamic.

Bernt's approach also means that if there are no bundles named
`$(classify.role)`, you won't have a fatal error (you can just warn that
the role is unknown because the bundle list is empty). So it's much
safer to support a new role.

If you agree this is a useful change, I can show you in more detail how
it would work as a PR :)

Ted

[Martin Simons]

Dear Ted,

TZ>...and then you can have multiple bundles with the same
TZ> `$(classify.role)` tag, for instance, instead of just one.
TZ>  It's much more dynamic.

That is true.

In The CFEngine Roadshow, however, machines have a single role which is derived from the standardized host name. Roles can be complex and in that case a role has more than one component

Regards,
Martin.

[Martin Simons]

Dear Bernt,

BB> Actually, the subject of this thread is misleading, I use variables as well. I should have named it
BB> Autorun with meta tags. Even though any is a hard class, it is just a string here in meta context
BB> The tags list of every bundle is usually a variable.

All very interesting. Do you have a github repo, like I have for The CFEngine Roadshow. https://github.com/Webhuis/CFEngine-Roadshow

It would give a better overview than snippets of code.

By now there are quite a number interesting frameworks available for CFEngine. I know Nick Anderson is trying to get this organized. Why don we not have a corner on http://cfengine.com which provides an overview of all these frameworks. It would help people to get going with CFEngine and ease the always difficult first steps.

Regards,
Martin.

[Bernt Jernberg]

All very interesting. Do you have a github repo, like I have for The CFEngine Roadshow. https://github.com/Webhuis/CFEngine-Roadshow

Not at the moment.
 

It would give a better overview than snippets of code.

By now there are quite a number interesting frameworks available for CFEngine. I know Nick Anderson is trying to get this organized. Why don we not have a corner on http://cfengine.com which provides an overview of all these frameworks. It would help people to get going with CFEngine and ease the always difficult first steps.

I plan to write a blog post about it soon. Hopefully that will be more informative. :-)

Bernt

[Bernt Jernberg]

It would give a better overview than snippets of code.

Agree :-)
 

By now there are quite a number interesting frameworks available for CFEngine. I know Nick Anderson is trying to get this organized. Why don we not have a corner on http://cfengine.com which provides an overview of all these frameworks. It would help people to get going with CFEngine and ease the always difficult first steps.

A corner sounds good. :-)

Bernt

[Ted Zlatanov]

On Fri, 22 May 2015 06:15:45 -0700 (PDT) Martin Simons <mjcm....@gmail.com> wrote:

MS> Dear Ted,

TZ> ...and then you can have multiple bundles with the same
TZ> `$(classify.role)` tag, for instance, instead of just one.
TZ> It's much more dynamic.

MS> That is true.

MS> In The CFEngine Roadshow, however, machines have a single role which is
MS> derived from the standardized host name. Roles can be complex and in that
MS> case a role has more than one component

I don't mean machines should have multiple roles.

I mean the role could be implemented with multiple bundles, and you can
even share bundles between roles, by just setting tags appropriately.
Thus you can map one bundle to each component in a natural way.

Ted

[Bernt Jernberg]

Hi all,

My post is out:

https://cfengine.com/company/blog-detail/dynamic-bundlesequence-with-autorun-meta-tags-and-hard-classes/

Bernt

[Bernt Jernberg]

Hi,

I worked around the "problem" with "spam" parsing linux code on Solaris and vice versa.
I changed my_autorun.cf to look like this:

body file control {
  agent::
    inputs => { @(my_autorun.inputs) };
}

bundle agent my_autorun
{

  meta:
    "tags" slist => { "autorun" };

  vars:
    # Match all (meta) tags in bundles.
    #
    # any            = any
    # $(sys.class)   = linux|solaris|aix
    # $(sys.flavour) = redhat_6|oracle_6|sunos_5.10|aix_6.1
    # $(sys.uqhost)  = unqualified hostname
    # $(ostype)      = redhat,oracle
    # $(role)        = role of server. Ex web, app, db
    # $(function)    = primary_kdc, primary_ldap, policy_server.

    any::
      "role"        string => "none";
      "function"    string => "none";
      "zone_type"   string => "none";
      "domain_type" string => "none";
      "ostype"      string => "none";

    !pre_commit_check::
      "dirs" slist => {
                       "global",
                       "any",
                       "$(sys.class)",
                       "$(sys.flavour)",
                       "$(ostype)"
                      };

    pre_commit_check::
      "dirs" slist => lsdir("$(this.promise_dirname)","[0-9A-z_]+","false");

    # Function
    ps01|tps01::
      "function" string => "policy_server";
    ldap01|tldap01::
      "function" string => "primary_ldap";
    kdc01::
      "function" string => "primary_kdc";

    # Role
    web::
      "role" string => "web";
    app::
      "role" string => "app";
    db::
      "role" string => "db";

    # Global/local zone, control domain.
    control_domain::
      "domain_type" string => "control_domain";
    zone_global::
      "zone_type" string => "global_zone";
    solaris.!zone_global::
      "zone_type" string => "local_zone";

    # ostype
    ostype_oracle::
      "ostype" string => "oracle";
    ostype_redhat::
      "ostype" string => "redhat";

    # Search for matching meta tags.
    any::
      "bundle_input" slist => bundlesmatching(".*","^(any|$(sys.class)|$(sys.flavour)|$(sys.uqhost)|$(ostype)|$(role)|$(function)|$(zone_type)|$(domain_type))$");
      "inputs_$(dirs)" slist => findfiles("$(this.promise_dirname)/$(dirs)/*\.cf");
      "inputs_array[$(inputs_$(dirs))]" string => "true";
      "inputs" slist => getindices("inputs_array");

    list_populated::

      "bundle" slist => sort("bundle_input","lex");

  classes:
    # Stop spam from sort function.
    "list_populated"  expression => isvariable("bundle_input");

    # Create role class.
    "web"   or => { classmatch("web.*") };
    "app"   or => { classmatch("app.*") };
    "db"    or => { classmatch("db.*") };

    # Solaris control domain.
    "control_domain"  expression => fileexists("/usr/sbin/ldm");

    # Create ostype class
    "$(sys.flavour)_class" expression => "any";
    "ostype_oracle" expression => classmatch("oracle_.*_class");
    "ostype_redhat" expression => classmatch("redhat_.*_class");

  methods:
    "$(bundle)" usebundle => "$(bundle)",
    comment => "Run all discovered bundles";

  reports:
    inform_mode::
    "autorun is executing";
    "$(this.bundle): found bundle $(bundle)";
}

I run this as part of my pre commit hook to check syntax in all sub directories:

cf-agent -nKD pre_commit_check

Bernt Jernberg

[Nick Anderson]

On 11/03/2015 08:07 AM, Bernt Jernberg wrote:
> # Search for matching meta tags.
> any::
> "bundle_input"slist
> =>bundlesmatching(".*","^(any|$(sys.class)|$(sys.flavour)|$(sys.uqhost)|$(ostype)|$(role)|$(function)|$(zone_type)|$(domain_type))$");
> "inputs_$(dirs)"slist

Nice

[Neil Watson]

Please consider a more reader friendly regex. (?mxs)is your friend.
http://evolvethinking.com/build-better-regular-expressions-in-cfengine/

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com