Remote file copy conditional on file's existence?

[mike.w...@verticalsysadmin.com]

Hello,

I'm running into something that seems to be a language restriction, and I'm in search of workarounds or data that I may be missing:

I have a promise to copy a host-specific file from the policy server, overwriting the local copy of that file, if the file exists on the policy server.

The file is not expected to exist on the policy server for all hosts, but there is no deterministic way to know (from the host's point of view) whether it should expect the file to be there on the policy server or not.  This is because the file is an "override" and thus could be added on the policy server at any time, to override the setting (the local file on the host) currently used for that host.

The files "copy from" promise works fine as written, but it generates noise in the Mission Portal if the override file is not present on the policy server.  I need to prevent this noise, as it is not actually a "promise not kept" if a host doesn't get an override file from the policy server.

(This is copied as part of the "update.cf" run, by the way, not as part of the "promises.cf" run, though I don't think that makes any difference.  It shows up in the Mission Portal either way.)

The only options I currently see by which I can accomplish this all seem much more complex than they should be:

1. I could have the policy server dynamically generate an index of which hosts have override files present on the policy server, and have every host pick up the index and only attempt to copy the override file if "my hostname" is listed in the index.

Pros: Would definitely function as described.

Cons: Added complexity -- minimally two additional promises for the hosts to evaluate (copy index, and search the index), and the generation of the index on the policy server would mean another promise.  Also it adds another layer of indirection that I would need to investigate any time a host doesn't get its override file.  Performance hit, with thousands of hosts potentially being listed in the index.  (The performance hit could possibly be worked around by adding even more complexity.)  Doesn't operate with the "basic building blocks" specified by the CFEngine philosophy, but instead uses at least 4 promises where only 1 should theoretically be necessary.

2. I could skip the "files" promise entirely and instead use a "commands" promise to copy the file from the policy server, and set the "kept_return_codes" of the command such that it would be counted as a kept promise regardless of whether the file were there on the remote box (the policy server) or not.

Pros: Would only require a single promise in CFEngine.

Cons: May not work at all due to networking and authentication issues.  (I can't embed ssh credentials in the CFEngine policy, obviously, even if the firewall rules would allow scp from all of the bootstrapped hosts, which they probably don't.)  Doesn't make use of the fact that these override files are being served by cf-serverd on the policy server.  Is a bit of a twist since the promise is about a file, not about a command.  Is dependent on firing up another process than CFEngine (e.g. scp or rsync) which would impact performance even if I could get it to work.

What I would really want for this use case is a way to specify, for a single promise, "don't ever report this as not kept regardless of what happens."

Even better would be a way to specify, for a remote file copy specifically, that if the source file doesn't exist on the source server, the promise should be skipped (or counted as kept).

For a local copy this can be done:

  files:

    any::

      "/tmp/target_file"

        create => "true",

        copy_from => local_cp("/tmp/source_file"),

        ifvarclass => fileexists("/tmp/source_file");

But, I know of no way whatsoever to do this for a remote copy.

Perhaps there could be something like the following?

  files:

    any::

      "/tmp/target_file"

        create => "true",

        copy_from => remote_dcp("/var/cfengine/override_files/$(sys.fqhost)", "$(sys.policy_hub)"),

        action => "attempt_only";

(I'm using 3.6.6 but will soon be using 3.7.3.)

Thanks,

--Mike Weilgart

[Nick Anderson]

If you have a promise to copy a file,  and it can't find that specifically enumerated file on the remote host its definitely a promise not kept.

Your right that there is currently no direct way in the language to ask if a remote file exists.

In addition to building an index as you describe there are some functions that could be used. I have inspected remote classes (https://docs.cfengine.com/lts/reference-functions-remoteclassesmatching.html)  and I have used remote scalars (https://docs.cfengine.com/lts/reference-functions-remotescalar.html)  to perform these types of checks in the past.  While it worked it didn't feel very natural to use them for checking a remote files existence.

I just had a thought maybe it's clever maybe not.

Instead of asking for a specific file,  can you ask for the directory the file is in recursively,  and use file select to restrict the files your looking for to the host specific file.



Sent from my mobile device.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Nick Anderson]

The trick is to use depth_search and file_select.

https://gist.github.com/d235ddc3f49c8bc8aabd15ea4320b07e


# Created 2016-05-01 Sun 16:06
#+TITLE: Download files if they exist on a remote host without promises
not kept
#+AUTHOR: Nick Anderson
#+SETUPFILE: ~/src/org-html-themes/setup/theme-readtheorg.setup

This example shows downloading files from a server if they exist on the
remote
server. The policy promises that a local directory should contain copies
of the
files that match a list of regular expressions on a remote server. As
noted in
[[https://docs.cfengine.com/docs/master/reference-promise-types-files.html#local-and-remote-searches][local
and remote searches]] when =depth_search= is combined with =copy_from= qa
remote search is made after the search over local base-path objects.
Since we
are promising to copy files opportunistically if they match a pattern this
allows the absence of a remote file to not result in a promise not kept.

** Policy Explanation

I like to use bodies from the standard library when possible, so I make sure
that's included.

#+BEGIN_SRC cfengine3
body file control
{
inputs => {
"/home/nickanderson/CFEngine/masterfiles/lib/stdlib.cf" };

}
#+END_SRC

Next I define the list of servers to download the assets from. If the
promise
fails it should be attempted on each subsequent server until it is
successful. I
also define the list of files I want to download. In this case I am
downloading
a policy file named for the unqualified hostname.

*Note:* Since I am writing a stand-alone policy I use =bundle agent
main= since
it is conveniently the default bundle if no =bundlesequence= is defined.
If I
were to integrate this into a larger policy set I would use a different
bundle
name.

#+BEGIN_SRC cfengine3
bundle agent main
{
vars:
"servers" slist => { "localhost" };
"assets"
slist => { "apacryphon_v3.*" },
comment => "Only files on the remote server matching this list of
regular expressions will be downloaded.";
#+END_SRC

=/var/cfengine/deploy/= should contain a copy of any files matching one
of the
regular expressions defined for =assets= that exist on the first accessible
remote =servers=. I define classes based on the results of the =files=
promise
to help illustrate the behavior.

*Note:* If you are going to be downloading policy or data files to be
used with
your main policy I suggest placing them *outside*
=/var/cfengine/inputs=. If you
have the =cfengine_inturnal_purge_policies= class defined so that local
files
that do not exist on the remote host are purged. This synchronization
behavior
helps to avoid potential duplicate definition of bundle errors when
files are
renamed. It also keeps things tidy which I find helpful if I need to debug
something.

#+BEGIN_SRC cfengine3
files:
"/var/cfengine/deploy/."
copy_from => overlay( "/var/cfengine/srv/app", @(servers) ),
depth_search => recurse_with_base(1), # [1]
file_select => by_name( @(assets) ), # [2], [3]
classes => results("namespace", "app_assets"); # [3]

# @[1] This should probably be set to 1 If you recurse deeper
than a single
# directory, please understand that file_select is meatching on
the leaf
# file name and not the directory ath it is in.

# @[2] Passing in as a list may have efficiency gains as oposed to
# iterating over the files at this level.

# @[3] The result here is based on the scope of the file
iteration. For
# example, when passing a list to file select as opposed to
iterating with
# a scalar the resulting classes will be defined based on any
repair, not
# based on any specific file. Be careful to not introduce
iteration in the
# definition of this class unless you are already iterating for
the file
# select. You would only do this if you wanted classes that
identified the
# specific files

vars:
"c" slist => classesmatching( "app_assets.*" );
reports:
"Result Classes:";
"$(const.t)$(c)";
}

body copy_from overlay( path, servers )
# @brief Copy files from `path` on first accessible `servers`.
{
servers => { @(servers) };
source => "$(path)";
compare => "digest";
copy_backup => "false";
collapse_destination_dir => "false";
verify => "true";
type_check => "false"; # We trust the upstream file tree
# if a file is changed to a directory
# we expect it was intentional.

# Defaults specified for completeness.
preserve => "false";
trustkey => "false";
purge => "false";
encrypt => "false";
check_root => "false";
stealth => "false";
}
#+END_SRC

I define an access rule on hosts with the =asset_server= or =policy_server=
class granting open access to =/var/cfengine/srv=.

#+BEGIN_SRC cfengine3
bundle server assets_share
{
access:
asset_server|policy_server::
"/var/cfengine/srv/"
admit => { "0.0.0.0/32" };
}
#+END_SRC

*Note:* =cf-serverd= must be started with this policy on a host defining
one of
the noted classes before the copy will work. Here is an example of access
summary logs from =cf-serverd= verbose output showing the controls that
apply to
the shared path.

#+CAPTION: Snippet from ~cf-serverd -Fv~
#+BEGIN_EXAMPLE
verbose: === BEGIN summary of access promises ===
verbose: Path: /var/cfengine/srv/
verbose: admit_ips: 0.0.0.0/32
verbose: admit_ips: 192.168.42.0/24
verbose: admit_hostnames: $(sys.policy_hub)/16
#+END_EXAMPLE

*Note:* I believe the agent has some optimization's with regard to making
network connections to local resources. This is due to the lack of logging I
have observe red in =cf-serverd= =vebose= and =debug= level outputs when
connecting on =localhost= vs the logging I see from connecting to an IP
bound to
an interface on the executing host.

** Demo

*** Setup Environment

First we need to have some content to serve.

#+BEGIN_SRC sh
mkdir -p /var/cfengine/srv/app
touch /var/cfengine/srv/app/apacryphon_v1.7z
touch /var/cfengine/srv/app/apacryphon_v3.5.7z
touch /var/cfengine/srv/app/apacryphon_v3.9.7z
#+END_SRC

#+BEGIN_SRC sh
tree /var/cfengine/srv
#+END_SRC

#+RESULTS:
#+BEGIN_EXAMPLE
/var/cfengine/srv
└── app
├── apacryphon_v1.7z
├── apacryphon_v3.5.7z
└── apacryphon_v3.9.7z

1 directory, 3 files
#+END_EXAMPLE

And let's see what assets have already been deployed to
=/var/cfenigne/deploy=.

#+BEGIN_SRC sh
tree /var/cfengine/deploy
#+END_SRC

In fact we can see =/var/cfengine/deploy= doesn't even exist yet.

#+RESULTS:
#+BEGIN_EXAMPLE
/var/cfengine/deploy [error opening dir]

0 directories, 0 files
#+END_EXAMPLE

*** See files get copied

Now let's run the policy and then check the contents of
=/var/cfengine/deploy=.

#+BEGIN_SRC sh
/var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf
#+END_SRC

#+RESULTS:
#+BEGIN_EXAMPLE
info: Copying from 'localhost:/var/cfengine/srv/app/apacryphon_v3.5.7z'
info: Copying from 'localhost:/var/cfengine/srv/app/apacryphon_v3.9.7z'
R: Result Classes:
R: app_assets_reached
R: app_assets_kept
R: app_assets_repaired
#+END_EXAMPLE

#+BEGIN_SRC sh
tree /var/cfengine/deploy
#+END_SRC

#+RESULTS:
#+BEGIN_EXAMPLE
/var/cfengine/deploy
├── apacryphon_v3.5.7z
└── apacryphon_v3.9.7z

0 directories, 2 files
#+END_EXAMPLE

*Note:* If =cf-serverd= has not been started with the proper
=access_rules= you
may see errors. For example:

#+BEGIN_EXAMPLE
info: Can't stat file '/var/cfengine/srv/app' on 'localhost' in
files.copy_from promise
R: Result Classes:
R: app_assets_failed
R: app_assets_error
R: app_assets_reached
R: app_assets_not_kept
#+END_EXAMPLE

*** See that we don't copy files unnecessarily

If we run the policy again we find that no unnecessary repairs were made.

#+BEGIN_SRC sh
/var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf
#+END_SRC

#+RESULTS:
#+BEGIN_EXAMPLE
R: Result Classes:
R: app_assets_reached
R: app_assets_kept
#+END_EXAMPLE

*** See that we don't delete local files that have gone missing upstream

If we move the served file out of the way and run the policy we can see
that no
repairs are made and no promises are broken.

#+BEGIN_SRC sh
rm /var/cfengine/srv/app/apacryphon_v3.5.7z
#+END_SRC

#+BEGIN_SRC sh
/var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf
#+END_SRC

#+RESULTS:
#+BEGIN_EXAMPLE
R: Class defined as the result of promising the /tmp directory contained
any matching files
R: personality_file_reached
R: personality_file_kept
#+END_EXAMPLE

*Note:* The previously download file was not purged, for that behavior
you would
need a =copy_from= body with =purge= set to =true=.

[mike.w...@verticalsysadmin.com]

Thanks, Nick!  Excellent idea!  :)

Since I need the local copy of the file to have a fixed name (whereas the source file on the policy server is named according to the host on which it will be deployed), I believe I will need to have a second, local files promise to make the local copy.  Does that sound right to you?

Also, for anyone else looking at this in the future, the following "simplest case" proof of concept may be of use.  (It does not include the server access bundle; otherwise it is standalone.)

body common control {

  bundlesequence => { "main" };

}

body file control {

  inputs => { "/var/cfengine/inputs/lib/3.6/stdlib.cf" };

}

bundle agent main {

  files:

    "/tmp/"

      depth_search => recurse_with_base(1),

      file_select => by_name( "$(sys.fqhost).txt" ),

      copy_from => remote_dcp( "/var/cfengine/host_specific_files/", "$(sys.policy_hub)" );

    "/tmp/fixed_name.txt"

      copy_from => local_dcp( "/tmp/$(sys.fqhost).txt" ),

      ifvarclass => fileexists( "/tmp/$(sys.fqhost).txt" );

}

Nick, do you see anything missing from the above code?  It is much simplified from yours, I admit, but are there any caveats with this simpler version I should be aware of before using it?

Thanks so much for the quick and thorough answer you gave on this, very appreciated.  :)

Best,

--Mike Weilgart

[Nick Anderson]

I think you want single copy nirvana.

--

[mike.w...@verticalsysadmin.com]

:)

...I guess I'll take that as a "no" to my question, "Do you see anything missing from the above code?"  ;)

I'm happy with the two promises, one to get the file remotely (without throwing errors if it's not there) and the other to make a local copy with the correct name (without throwing errors if the source is absent).  Single-promise Nirvana would be wonderful, of course:

bundle agent main {

  boxes:

    "$(this.host)"

      configured => "correctly";

}

:D

Best,

--Mike Weilgart

[Ted Zlatanov]

On Tue, 3 May 2016 19:41:30 -0700 (PDT) mike.w...@verticalsysadmin.com wrote:

mw> ...I guess I'll take that as a "no" to my question, "Do you see anything
mw> missing from the above code?" ;)

mw> I'm happy with the two promises, one to get the file remotely (without
mw> throwing errors if it's not there) and the other to make a local copy with
mw> the correct name (without throwing errors if the source is absent).
mw> Single-promise Nirvana would be wonderful, of course:

mw> bundle agent main {
mw> boxes:
mw> "$(this.host)"
mw> configured => "correctly";
mw> }

Have you looked at the cf-serverd `shortcuts` feature?

https://docs.cfengine.com/docs/master/reference-promise-types-access.html#shortcut

You can use `connection.key` and `connection.hostname` to define
server-side mappings so the same file on the client side points to
different things on the server side.

There's a good discussion at https://stackoverflow.com/questions/36131088/how-to-use-parametrized-bundles-in-cf-serverd

Ted

[mike.w...@verticalsysadmin.com]

It looks like this isn't such a good solution after all.  With 1800 host-specific files in the directory on the Hub, the update run for a host takes over 4 minutes!  Evidently the filtering happens on the client end, not on the hub end, so ALL 1800 files get sent over the network (apparently), and then CFEngine discards all but one of them.  (The files are only two lines each, by the way.)

I'll have to restrategize.  :(

--Mike Weilgart

[Nick Anderson]

It transfers the files to the client and then deletes them?  If so file a bug.

Sent from my mobile device.

-----Original Message-----
From: mike.w...@verticalsysadmin.com
To: help-cfengine <help-c...@googlegroups.com>

[Mike Weilgart]

I didn't investigate with strace, so perhaps only the file metadata is being sent.  I also discovered I was looking in the wrong directory when I gave the 1800 count earlier on this thread; the actual count is about 15000 files.  So the mere fact of the filtering occurring on the client end is enough to severely impact performance, whether the actual contents of the file is sent or not.

What I do know is that the verbose output of:

cf-agent -f update.cf -IC -DDEBUG -v

...includes a listing of every single file from the directory on the hub, with a mention "skipping <filename>."  That's what I meant by "ALL the files getting sent over the network."

(These files are so small that the file metadata, if it includes timestamp and full "stat" information, may well be larger than the file contents anyway.)

--Mike Weilgart

[Ted Zlatanov]

On Mon, 9 May 2016 21:06:42 -0700 Mike Weilgart <mike.w...@verticalsysadmin.com> wrote:

MW> What I do know is that the verbose output of:

MW> cf-agent -f update.cf -IC -DDEBUG -v

MW> ...includes a listing of every single file from the directory on the hub, with a mention "skipping <filename>." That's what I meant by "ALL the files getting sent over the network."

MW> (These files are so small that the file metadata, if it includes timestamp and full "stat" information, may well be larger than the file contents anyway.)

Was the `shortcut` feature I pointed out earlier not working for you? It
seems ideal for this case and should definitely not transfer any
extraneous metadata.

Ted

[Enrico Scholz]

Ted Zlatanov <t...@lifelogs.com> writes:

> On Mon, 9 May 2016 21:06:42 -0700 Mike Weilgart <mike.w...@verticalsysadmin.com> wrote:
>
> MW> ...includes a listing of every single file from the directory on

> MW> the hub, with a mention "skipping <filename>." That's what I
> MW> meant by "ALL the files getting sent over the network."

>
> MW> (These files are so small that the file metadata, if it includes

> MW> timestamp and full "stat" information, may well be larger than the
> MW> file contents anyway.)

>
> Was the `shortcut` feature I pointed out earlier not working for you?
> It seems ideal for this case and should definitely not transfer any
> extraneous metadata.

I do not think that shortcuts are a good solution for this issue. I am
using opportunistic copy operations a lot (e.g. copying bin/, cron.d/,
systemd/, ... directories from global, site, group and host specific
folders, where some of them might be missing).

I would prefer to keep server configuration as simple and static as
possible and not to register every single file there.

IMO, cfengine3's file copy operations are broken/misdesgined (realpath()
on server side makes sharing of keys impossible, recursive copy does not
allow to set file permissions, missing source files cause a lot of noise)
and a big regression from cfengine2.


Enrico

[Ted Zlatanov]

On Tue, 10 May 2016 17:01:19 +0200 Enrico Scholz <enrico...@ensc.de> wrote:

ES> Ted Zlatanov <t...@lifelogs.com> writes:
>> On Mon, 9 May 2016 21:06:42 -0700 Mike Weilgart <mike.w...@verticalsysadmin.com> wrote:
>>
MW> ...includes a listing of every single file from the directory on
MW> the hub, with a mention "skipping <filename>." That's what I
MW> meant by "ALL the files getting sent over the network."
>>
MW> (These files are so small that the file metadata, if it includes
MW> timestamp and full "stat" information, may well be larger than the
MW> file contents anyway.)
>>
>> Was the `shortcut` feature I pointed out earlier not working for you?
>> It seems ideal for this case and should definitely not transfer any
>> extraneous metadata.

ES> I do not think that shortcuts are a good solution for this issue. I am
ES> using opportunistic copy operations a lot (e.g. copying bin/, cron.d/,
ES> systemd/, ... directories from global, site, group and host specific
ES> folders, where some of them might be missing).

ES> I would prefer to keep server configuration as simple and static as
ES> possible and not to register every single file there.

OK, but I thought Mike's use case was a single small file for every
host, not multiples. So maybe for him it works, even if it's not right
for your use case.

My personal preference is to run out of Git without a central server, so
certainly there's room for many approaches and CFEngine doesn't force
any of them.

Ted

[mike.w...@verticalsysadmin.com]

Hi Ted,

I took a look at the shortcuts approach and it looks like it could work, except for two things:

1. It appears that it would need to do a reverse DNS lookup for every single host connecting (even if there were no overrides file for that host) to map the shortcut name to the hostname-based filename for the connecting host; and

2. I don't see how this would solve the "promise not kept" issue if the overrides file is not present for a particular host.

(2) is more important than (1), since the system could be re-architectured to use the connection key in place of the hostname for the name of the host-specific overrides file (though it would be inconvenient), but there's no point from my perspective in making such a change if it doesn't also solve the "promise not kept" issue with the same stone.

Is there an open feature request somewhere for a "remote_fileexists()" function which I could vote for?  (Or perhaps an "is_served_filepath()" function, since the real question is whether a given file is available for download from the policy server, not just whether it exists or not.)

Or, what may be simpler, is there a feature request anywhere by which a "notkept" outcome can be ignored at the agent level for a specified promise?  (As far as I know this only exists for commands promises, but of course there is no commands promise that can access a file served by cf-serverd and download it, ignoring any errors.)

--Mike Weilgart

[Ted Zlatanov]

On Wed, 11 May 2016 00:59:13 -0700 (PDT) mike.w...@verticalsysadmin.com wrote:

mw> Is there an open feature request somewhere for a "remote_fileexists()"
mw> function which I could vote for? (Or perhaps an "is_served_filepath()"
mw> function, since the real question is whether a given file is available for
mw> download from the policy server, not just whether it exists or not.)

That would be useful in some cases but moves the burden to the client,
complicating the logic there and introducing new fun ways to fail.

Hmmmmm. Maybe the shortcuts feature can take an slist of possible paths,
from most specific to least specific, instead of a single path, and
serve the first one it finds. I think that would help your use case most
directly and keep the client side simple. What do you think?

Ted

[Nick Anderson]

I believe you could use files_single_copy to avoid a promise not kept,
so long as you have at least one file that you could copy.


https://docs.cfengine.com/docs/3.9/reference-components-cf-agent.html#files_single_copy

files_single_copy is an agent control option. You can list regular
expression patterns that match on the *executing* host (locally). The
agent keeps track of files it copies into place, and once it has copied
a file into place that matches files_single_copy it avoids future copies
to that destination within the same agent run.

I started writing an example the other day when I mentioned I think you
might want "single copy nirvana". I got busy with other things but you
should be able to google for other examples. Diego shows an example in
his book.

Here is what I did:

https://gist.github.com/83054c7cd9c81c5f73cd5c0036e133bc

[Enrico Scholz]

mike.w...@verticalsysadmin.com writes:

> Or, what may be simpler, is there a feature request anywhere by which a
> "notkept" outcome can be ignored at the agent level for a specified
> promise?

https://tracker.mender.io/browse/CFE-2365 (specialised for this case;
your general "notkept" sounds like a good idea).


Enrico
>>