Deploying CHANGING Configurations

[Todd Erwin]

Im trying to solve a problem with deploying a NEW version of a cf file that manages Vas in our environment.  The issues is that when we install this new version of software it has to re-authenticate against a central server, as you can imagine having your ENTIRE compute ENV hit that one server over 30 minutes could cause some problems.  So i want to GRADUALLY roll out this new version.

In my promises.cf I have something like this..

body common control

{

        inputs => {

                "update.cf",

                "server.cf",

                "cfengine_stdlib.cf",

                "/var/cfengine/packages/vas/1.0/vas.cf";

}

        bundlesequence => {

                modules(),

                update(),

                cfengine(),

                vas("vas","1.0),

}

I want to go to version 2.0 of vas.. so my promises.cf would need to look like this...

body common control

{

        inputs => {

                "update.cf",

                "server.cf",

                "cfengine_stdlib.cf",

                "/var/cfengine/packages/vas/2.0/vas.cf";

}

        bundlesequence => {

                modules(),

                update(),

                cfengine(),

                vas("vas","2.0),

}

Is there a way in CFEngine to Dynamically assign a NEW promises.cf across your compute farm say over 8 Hours?  IF I open up a window for a host to get the NEW file, once that window has closed they will go back and get the OLD file and thus revert back the version of VAS.  Hopefully what im asking for is clear if not please feel free to ask further questions.

In the end I want to deploy a new configuration to 100 Random Hosts per hour untill all hosts have the new configuration.

Thanks

[Nick Anderson]

Todd Erwin <toddw...@gmail.com> writes:

> Is there a way in CFEngine to Dynamically assign a NEW promises.cf across
> your compute farm say over 8 Hours? IF I open up a window for a host to
> get the NEW file, once that window has closed they will go back and get the
> OLD file and thus revert back the version of VAS. Hopefully what im asking
> for is clear if not please feel free to ask further questions.

Hi Todd,

It sounds to me like you are describing dynamic inputs. I would probably look to use body file control to achieve it. Note you can use body file control inside any policy file, so you can include one policy file from another without touching promises.cf itself.

bundle common vas_classification
{
  vars:
    any::
      "version" string => "1.0"; # the default version of policy to use

    host001.Afternoon::
      "version" string => "2.0"; # host 1 uses version 2 during the afternoon.

    host002.Morning::
      "version" string => "3.0"; # Host 2 uses version 3 during the Morning

    any::

      "inputs" slist => { "/var/cfengine/packages/$(version)/vas.cf" };
}

body file control
{
  inputs => { @(vas_classification.inputs) };
}

bundle agent main
{
  methods:

    "Vas Classification"
      usebundle => vas_classification;

    "VAS"
     usebundle => vas( "vas", $(vas_classification.version) );

}

You explicitly mentioned a NEW promises.cf for different subsets of
infrastructure. For that you can configure `exec<sub>command</sub>` in body executor
control. The default MPF uses different exec_commands for different platforms
but you could use it with your own classes. Note your custom classes would have
be be derived very early in order to be able to use them to effect the behaviour
of execd.

body executor control
{
host001.Afternoon::
exec_command => "$(sys.workdir/bin/cf-agent -f update.cf; $(sys.workdir)/bin/cf-agent -f /var/cfengine/differnt-inputs/promises.cf";
}

Does that make sense?

–
Nick Anderson
Doer of things, CFEngine

[Todd Erwin]

So My problem with this is what happens to host002 that Evening when it's no longer assigned to host002.afternoon?  It reverts back to the 1.0 version File control copies in the 1.0 version and im back to having that machine at version 1.0?

For full disclosure we have automated the building of our promises.cf.  we have a header file and a footer file and then fill in all the inputs and bundlesequence.  The perl script (cf-imports) reads a MANIFEST of who get's what when and then builds out the promises.cf.  The down side to this is that cf-imports is called DURING the cfengine run so the promises.cf is technically built for the NEXT run.  Not a big deal.

However the AUTO assignment based on classes comes and goes as I mentioned above, if I open up a WINDOW for host002 to get the new policy files, as soon as that window closes it reverts back to the NORMAL 1.0 version.

I want to deploy said new version based on IP.. and every 1Hour allow the next set of IP's to have the file.  So 1 hour ip 10-20 get the file AND KEEP USING IT.  then hour 2 ip 30-40 get the file and KEEP USING IT. So on and so forth.. Then when ALL Clients are updated ~1-2 weeks.  Move the GLOBAL marker to 2.0 and remove the Phased Marker.

Example from "MY" manifest that ends up getting built..

[any]

vas_client=1.0

[any.WEEKLY_DEPLOY_TIME]

vas_client=2.0

Some example code here of how I set the classes....

my $full_ip_address = `hostname -i`;

chomp $full_ip_address;

my($a,$b,$c,$d) = split (/\./, $full_ip_address);

my $date = `date +%w::%k::%U`;

my($day_of_week,$hour,$WW) = split (/::/, $date);

   if ($day_of_week eq 1) { #It's Monday 62 Ip's per Class C

      if ( ($hour >=8) && ($hour <=10 ) ) { #Monday Phase 1

         print "+WEEKLY_DEPLOY_TIME\n" if (($d>=1) && ($d<=10));     #4% Deploy

         print "+WEEKLY_DEPLOY_TIME\n" if (($d>=128) && ($d<=138));  #4% Deploy

      } elsif ( ($hour >=11) && ($hour <=13 ) ) { #Monday Phase 2

This continues for each day of the week but you get the idea.

These CLASSES all get set very early in update.cf  inside of "bundle agent update" inside of a commands promise..  the script cf-imports runs shortly after that so it's privy to these classes.

[Aleksey Tsalolikhin]

GitLab has Canary Deployments.  https://docs.gitlab.com/ee/user/project/canary_deployments.html   It would be nice if CFEngine had something like this.

In the meantime, we are handling this with policy channels (implemented with a version of masterfiles-stage.sh, see  https://github.com/cfengine/core/tree/master/contrib/masterfiles-stage) and manually assigning (subscribing) hosts to a new policy channel that has the new code.

Aleksey

-- 

Achieve real learning.  Email trai...@verticalsysadmin.com.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengine+unsubscribe@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Bas van der Vlies]

Todd,

In out environment we use def.json setup for every host. We use the ip
address or cf-key as unique indentifier. We have the same cf-key for
mulitple hosts (compute nodes in a cluster, see:
*
http://cfgmgmtcamp.eu/schedule/cfengine/augments-def-json-bas-van-der-vlies.html

But you can also used the setup in the MPF for def.json. You can
generate a def.json file that that sets classes based on hardclasses
like ip, hostname, dns, ..., eg:
{{{
"classes": {
"vas_client_1.0": [ "1.2.3.4", "host01", .... ],
"vas_client_2.0": [ "1.2.3.5", "host03", .... ],
}
}}}

Then you move the hardclass from vas_client_1.0 to vas_client_2.0 and
use these classes to set dynamic inputs like Neil suggested.

Maybe this helps you

> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to help-cfengin...@googlegroups.com
> <mailto:help-cfengin...@googlegroups.com>.

> To post to this group, send email to help-c...@googlegroups.com

> <mailto:help-c...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/help-cfengine.
> For more options, visit https://groups.google.com/d/optout.

--
---
Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098
XG Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surfsara.nl | www.surfsara.nl |

[Todd Erwin]

My ENV is well North of 100k spread out over many Geos.. Hostname is simply not an option, managing a list of hosts to tools and version is not doable.  DNS/IP are much better options.  A staged masterfiles is very similar to what we are doing today, we have a central policy server and then we also have GEO dispersed Policy servers.  The GEO ones pull from the Central SVN repo in order to replicate the package settings configiurations and other things.  In short my cf-imports works very similar to the masterfiles-stage.sh  script it reads a manifest with a list of classes and then assigns what version, which tools and which configurations to apply.  IF I tell it to deploy to something inside of a CLASS that is only set for 1-2 hours, once the hours pass, it then fails back to the DEFAULT that ALL systems get.  IF i set a PERMENAT class and deploy to that then eventually everyone will have that permeant class and will have full deployment.

Deploy to IP-RANGE A when it's time and continue to keep it at condition A untill the upgrade cycle is completed.  I need a way to tell all clients the upgrade started and the window closed via automation..  So when the deployment is happening don't revert back, then when finished Fall back to DEFAULT (which should be moved forward because its production) Rinse and repeat.

Canary deployments are exactly what Im looking for

[Nick Anderson]

On Friday, September 22, 2017 at 4:33:03 PM UTC-5, Todd Erwin wrote:

So My problem with this is what happens to host002 that Evening when it's no longer assigned to host002.afternoon?  It reverts back to the 1.0 version File control copies in the 1.0 version and im back to having that machine at version 1.0?

Yes, sorry I mis-understood and I thought that was what you were wanting to do.

Now I understand that you want to go for time period x and later into perpetuity. To implement that I would probably use `touch -t` to set the timestamp of a file to the start time and then use `isnewerthan()` to determine if the current time is after the timestamp of the file.