Download files if they exist on a remote host without promises not kept

1. Policy Explanation
2. Demo

This example shows downloading files from a server if they exist on the remote server. The policy promises that a local directory should contain copies of the files that match a list of regular expressions on a remote server. As noted in local and remote searches when depth_search is combined with copy_from qa remote search is made after the search over local base-path objects. Since we are promising to copy files opportunistically if they match a pattern this allows the absence of a remote file to not result in a promise not kept.

1 Policy Explanation

I like to use bodies from the standard library when possible, so I make sure that's included.

body file control
{
      inputs => { "/home/nickanderson/CFEngine/masterfiles/lib/stdlib.cf" };

}

Next I define the list of servers to download the assets from. If the promise fails it should be attempted on each subsequent server until it is successful. I also define the list of files I want to download. In this case I am downloading a policy file named for the unqualified hostname.

Note: Since I am writing a stand-alone policy I use bundle agent main since it is conveniently the default bundle if no bundlesequence is defined. If I were to integrate this into a larger policy set I would use a different bundle name.

bundle agent main
{
  vars:
      "servers" slist => { "localhost" };
      "assets"
        slist => { "apacryphon_v3.*" },
        comment => "Only files on the remote server matching this list of
                    regular expressions will be downloaded.";

/var/cfengine/deploy/ should contain a copy of any files matching one of the regular expressions defined for assets that exist on the first accessible remote servers. I define classes based on the results of the files promise to help illustrate the behavior.

Note: If you are going to be downloading policy or data files to be used with your main policy I suggest placing them outside /var/cfengine/inputs. If you have the cfengine_inturnal_purge_policies class defined so that local files that do not exist on the remote host are purged. This synchronization behavior helps to avoid potential duplicate definition of bundle errors when files are renamed. It also keeps things tidy which I find helpful if I need to debug something.

  files:
      "/var/cfengine/deploy/."
        copy_from => overlay( "/var/cfengine/srv/app", @(servers) ),
        depth_search => recurse_with_base(1), # [1]
        file_select => by_name( @(assets) ), # [2], [3]
        classes => results("namespace", "app_assets"); # [3]

     # @[1] This should probably be set to 1 If you recurse deeper than a single
     # directory, please understand that file_select is meatching on the leaf
     # file name and not the directory ath it is in.

     # @[2] Passing in as a list may have efficiency gains as oposed to
     # iterating over the files at this level.

     # @[3] The result here is based on the scope of the file iteration. For
     # example, when passing a list to file select as opposed to iterating with
     # a scalar the resulting classes will be defined based on any repair, not
     # based on any specific file. Be careful to not introduce iteration in the
     # definition of this class unless you are already iterating for the file
     # select. You would only do this if you wanted classes that identified the
     # specific files

  vars:
      "c" slist => classesmatching( "app_assets.*" );
  reports:
      "Result Classes:";
      "$(const.t)$(c)";
}

body copy_from overlay( path, servers )
# @brief Copy files from `path` on first accessible `servers`.
{
        servers => { @(servers) };
        source => "$(path)";
        compare => "digest";
        copy_backup => "false";
        collapse_destination_dir => "false";
        verify => "true";
        type_check => "false"; # We trust the upstream file tree
                               # if a file is changed to a directory
                               # we expect it was intentional.

   # Defaults specified for completeness.
        preserve => "false";
        trustkey => "false";
        purge => "false";
        encrypt => "false";
        check_root => "false";
        stealth => "false";
}

I define an access rule on hosts with the asset_server or policy_server class granting open access to /var/cfengine/srv.

bundle server assets_share
{
  access:
    asset_server|policy_server::
      "/var/cfengine/srv/"
        admit => { "0.0.0.0/32" };
}

Note: cf-serverd must be started with this policy on a host defining one of the noted classes before the copy will work. Here is an example of access summary logs from cf-serverd verbose output showing the controls that apply to the shared path.

verbose:  === BEGIN summary of access promises ===
verbose: 	Path: /var/cfengine/srv/
verbose: 		admit_ips: 0.0.0.0/32
verbose: 		admit_ips: 192.168.42.0/24
verbose: 		admit_hostnames: $(sys.policy_hub)/16

Note: I believe the agent has some optimization's with regard to making network connections to local resources. This is due to the lack of logging I have observe red in cf-serverd vebose and debug level outputs when connecting on localhost vs the logging I see from connecting to an IP bound to an interface on the executing host.

2 Demo

2.1 Setup Environment

First we need to have some content to serve.

mkdir -p /var/cfengine/srv/app
touch /var/cfengine/srv/app/apacryphon_v1.7z
touch /var/cfengine/srv/app/apacryphon_v3.5.7z
touch /var/cfengine/srv/app/apacryphon_v3.9.7z

tree /var/cfengine/srv

And let's see what assets have already been deployed to /var/cfenigne/deploy.

tree /var/cfengine/deploy

In fact we can see /var/cfengine/deploy doesn't even exist yet.

/var/cfengine/deploy [error opening dir]

0 directories, 0 files

2.2 See files get copied

Now let's run the policy and then check the contents of /var/cfengine/deploy.

/var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf

tree /var/cfengine/deploy

Note: If cf-serverd has not been started with the proper access_rules you may see errors. For example:

    info: Can't stat file '/var/cfengine/srv/app' on 'localhost' in files.copy_from promise
R: Result Classes:
R: 	app_assets_failed
R: 	app_assets_error
R: 	app_assets_reached
R: 	app_assets_not_kept

2.3 See that we don't copy files unnecessarily

If we run the policy again we find that no unnecessary repairs were made.

/var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf

2.4 See that we don't delete local files that have gone missing upstream

If we move the served file out of the way and run the policy we can see that no repairs are made and no promises are broken.

rm /var/cfengine/srv/app/apacryphon_v3.5.7z

/var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf

Note: The previously download file was not purged, for that behavior you would need a copy_from body with purge set to true.