Behavior of compare method "exists" on remote copies?

[mike.w...@verticalsysadmin.com]

I might end up answering this one myself, but is the behavior of the "exists" setting for "compare" in a "copy_from" body (https://docs.cfengine.com/lts/reference-promise-types-files.html#compare) such as to prevent network traffic if the file already exists?

Specifically, are the following two files promises equivalent from the perspective of the network traffic generated?

bundle agent files_example {

files:

  "/file1"

    copy_from => remote_cp("/file1", "$(sys.policy_hub)"),

    unless => fileexists("/file1");

  "/file2"

    copy_from => lazy_cp("/file2", "$(sys.policy_hub)");

}

body copy_from remote_cp(from,server)

{

      servers     => { "$(server)" };

      source      => "$(from)";

      compare     => "mtime";

}

body copy_from lazy_cp(from,server)

{

      servers     => { "$(server)" };

      source      => "$(from)";

      compare     => "exists";

}

I have a bunch of static files to "seed" and I'm concerned about the Hub and network performance.

--Mike Weilgart

P.S.: I believe the CFEngine community could be greatly enhanced if we all make an effort to follow this post: http://blog.stackoverflow.com/2011/07/its-ok-to-ask-and-answer-your-own-questions/

[Neil Watson]

I don't understand why one would want to do that. The existing of the
file is trivial next to its contents. A file promise shown is really
about the content. Otherwise why care?

--
Neil H Watson
CFEngine reporting: https://github.com/neilhwatson/delta_reporting
CFEngine policy: https://github.com/neilhwatson/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3

[mike.w...@verticalsysadmin.com]

The purpose is to only trigger the copy (across the network) if the file is actually missing.

With the specific examples I gave there is no reason to use the custom "copy_from" body rather than just the "unless", but if I want to control other attributes of the file—e.g. mode, owner, group—then the copying is only incidental (if the file doesn't happen to exist yet).  In that case a "lazy_rcp" could be useful.

So yes, I care about the contents, but I don't want it to be checked constantly over the network.  It's a performance question.  I only care about populating it if not existing.

Just like, in the standard update policy, all the .cf files in masterfiles are only checked and copied if one particular flag file is changed.  This is for performance.  The other side of the coin is that if you go and edit one of the .cf files in inputs on a particular host, it won't get updated ("healed") until the next time a change is made to masterfiles on the Hub.

The language feature compare => "exists" and its behavior with regard to the network is the subject of my question, though.  I understand that not all language features are useful to everyone.  :)

--Mike Weilgart

[Neil Watson]

When I update files, including update.conf (I write my own) I always
confirm its contents. I've had 5000 hosts on one server checking every
15 minutes and load was low. I would not worry about performance.
CFEngine is proven.

You can use a manifest policy instead. I've written one in EFL:
https://github.com/neilhwatson/evolve_cfengine_freelib/blob/master/doc/efl_update.md

[Nick Anderson]

On 07/19/2016 07:14 PM, mike.w...@verticalsysadmin.com wrote:
> setting for "compare" in a "copy_from" body
> (https://docs.cfengine.com/lts/reference-promise-types-files.html#compare)

> such as to prevent /network traffic/ if the file already exists?

My testing shows that currently it does not prevent any network traffic.
This could be a useful feature addition but I don't know how simple it
would be to add. But this is also really easy to do from policy (with
unless => filexists( $(this.promiser) )) (the syntax may not work
exactly but I hope you understand what I mean) so it probably wouldn't
add a lot of value.

> Specifically, are the following two files promises equivalent from the
> perspective of the network traffic generated?

Yes they are equivalent as far as I know.

[mike.w...@verticalsysadmin.com]

Thanks Nick, that is very helpful to know!

I think it would add some value (and match the expected behavior better).  Consider the following workaround:

files:

  "/file2"

    copy_from => remote_cp("/file2", "$(sys.policy_hub)"),

    unless => fileexists("$(this.promiser)");

  "/file2"

    perms => mog("755", "root", "root");

These could be combined into one promise, but it would not have quite the same behavior.  If 'compare => "exists"' worked across the network like I had expected, the following would be equivalent:

files:

  "/file2"

    copy_from => lazy_rcp("/file2", "$(sys.policy_hub)"),

    perms => mog("755", "root", "root");

In other words the "copy from" part would only be evaluated (incur network traffic) if the file were missing, but the "perms" part of the promise would be evaluated during every run.

Probably not crucial for most use cases, but it would be nice—cleaner.

Thanks for doing the testing!  :)

--Mike Weilgart

[Dimitrios Apostolou]

On Wed, Jul 20, 2016 at 4:34 AM, Nick Anderson <ni...@cmdln.org> wrote:

On 07/19/2016 07:14 PM, mike.w...@verticalsysadmin.com wrote:
> setting for "compare" in a "copy_from" body
> (https://docs.cfengine.com/lts/reference-promise-types-files.html#compare)
> such as to prevent /network traffic/ if the file already exists?

My testing shows that currently it does not prevent any network traffic.
This could be a useful feature addition but I don't know how simple it
would be to add. But this is also really easy to do from policy (with
unless => filexists( $(this.promiser) )) (the syntax may not work
exactly but I hope you understand what I mean) so it probably wouldn't
add a lot of value.

Strange, a look in the code shows that if the file exists and compare=>"exists", then it doesn't copy. From CfCopyFile():

        Log(LOG_LEVEL_VERBOSE, "Destination file '%s' already exists",
            destfile);

        if (attr.copy.compare == FILE_COMPARATOR_EXISTS)
        {
            Log(LOG_LEVEL_VERBOSE,
                "Existence only is promised, no copying required");
            return result;
        }

I'm wondering why this wouldn't work. The debug logs should show full network traffic, can you post the snippet around the copy_from files promise?

Thanks,

Dimitris

[Nick Anderson]

On 07/25/2016 01:52 PM, Dimitrios Apostolou wrote:
> I'm wondering why this wouldn't work. The debug logs should show full
> network traffic, can you post the snippet around the copy_from files
> promise?

It seems to get the remote hash before deciding that no copying is
required since the destination file already exists.

```
verbose: P: .........................................................
verbose: P: BEGIN promise 'promise_copy_from_unless_file_exists_cf_7'
of type "files" (pass 1)
verbose: P: Promiser/affected object: '/var/cfengine/inputs/promises.cf'
verbose: P: Part of bundle: copy_from_unless_file_exists
verbose: P: Base context class: any
verbose: P: Container path :
'/default/copy_from_unless_file_exists/files/'/var/cfengine/inputs/promises.cf'[0]'
verbose: P:
verbose: P: Comment: We only need to copy the file down if it does
not already exist.
verbose: P: .........................................................
verbose:
debug: Direct file reference '/var/cfengine/inputs/promises.cf', no
search implied
verbose: File '/var/cfengine/inputs/promises.cf' copy_from
'/var/cfengine/masterfiles/promises.cf'
verbose: FindIdle: no existing connection to '192.168.33.2' is established.
verbose: Connecting to host 192.168.33.2, port 5308 as address 192.168.33.2
verbose: Waiting to connect...
verbose: Setting socket timeout to 30 seconds.
verbose: Connected to host 192.168.33.2 address 192.168.33.2 port 5308
debug: TLSVerifyCallback: no ssl->peer_cert
debug: TLSVerifyCallback: no conn_info->key
debug: This must be the initial TLS handshake, accepting
verbose: TLS version negotiated: TLSv1.2; Cipher:
AES256-GCM-SHA384,TLSv1/SSLv3
verbose: TLS session established, checking trust...
verbose: Received public key compares equal to the one we have stored
verbose: Server is TRUSTED, received key
'SHA=30017fd4f85914d0fa2d8c733958b5508d8383363d937e6385a41629e06ca1ca'
MATCHES stored one.
debug: TLSRecvLines(): CFE_v2 cf-serverd 3.7.3.
debug: TLSRecvLines(): OK WELCOME USERNAME=root.
debug: SendTransaction header: t 59
debug: SendTransaction data: SYNCH 1469473897 STAT
/var/cfengine/masterfiles/promises.cf
debug: ReceiveTransaction header: t 71....
debug: ReceiveTransaction data: OK: 0 384 0 0 0 11257 1469390462
1469390461 1469390462 0 661354 1 64768
debug: ReceiveTransaction header: t 3.....
debug: ReceiveTransaction data: OK:
debug: Trying to create a parent directory for:
/var/cfengine/inputs/promises.cf
debug: Directory for '/var/cfengine/inputs/promises.cf' exists. Okay
verbose: Destination file '/var/cfengine/inputs/promises.cf' already exists
verbose: Existence only is promised, no copying required
debug: Searching for specific busy connection to: 192.168.33.2
debug: Busy connection just became free
verbose: Handling file existence constraints on
'/var/cfengine/inputs/promises.cf'
verbose: A: Promise was KEPT
verbose: P: END files promise (/var/cfengine/inputs/promises....)
```

[Dimitrios Apostolou]

On Mon, Jul 25, 2016 at 9:18 PM, Nick Anderson <nick.a...@cfengine.com> wrote:

On 07/25/2016 01:52 PM, Dimitrios Apostolou wrote:
> I'm wondering why this wouldn't work. The debug logs should show full
> network traffic, can you post the snippet around the copy_from files
> promise?

It seems to get the remote hash before deciding that no copying is
required since the destination file already exists.

Thanks for the logs Nick! So no request for the actual data is ever sent, but the STAT request is sent. Indeed, I can see it in VerifyCopy() that cf_lstat() is sent in the beginning, before all the checks in CfCopyFile() that I mentioned before. From verify_files_utils.c:VerifyCopy():

        found = cf_lstat(source, &ssb, attr.copy, conn);
    [ ... ]
        return CfCopyFile(ctx, sourcefile, destfile, ssb,
                          attr, pp, inode_cache, conn);

I believe it makes sense to avoid it in the case of compare=>exist, as Mike mentioned.

Dimitris

[Aleksey Tsalolikhin]

On Tue, Jul 26, 2016 at 4:09 AM, Dimitrios Apostolou <ji...@cfengine.com> wrote:

I believe it makes sense to avoid it in the case of compare=>exist, as Mike mentioned.

Yes, please!  It'll make agent runs faster.  

At one of our clients, Mike and I got the average agent run time down from over a minute to under 3 seconds and we want to keep it low.  :-)  Every bit helps.

ᐧ

[Dimitrios Apostolou]

FWIW I'm considering using compare=>exists a bit risky, since a corrupt file will never be refreshed. Thus it's rarely used so I'm not expecting many people to benefit from such a change.

Dimitris

[Neil Watson]

On Tue, Jul 26, 2016 at 03:10:06PM +0200, Dimitrios Apostolou wrote:
> FWIW I'm considering using compare=>exists a bit risky, since a corrupt
> file will never be refreshed. Thus it's rarely used so I'm not expecting
> many people to benefit from such a change.

Absolutely this. This idea is trading certainty for performance.

[Nick Anderson]

On 07/26/2016 08:10 AM, Dimitrios Apostolou wrote:
> FWIW I'm considering using compare=>exists a bit risky, since a corrupt
> file will never be refreshed. Thus it's rarely used so I'm not expecting
> many people to benefit from such a change.

I created a ticket: https://tracker.mender.io/browse/CFE-2437

Please watch and vote for it if you are interested. With the easy
workaround to avoid hitting the network and the relatively low usage of
compare => exists I don't expect this ticket to get prioritized.

[Aleksey Tsalolikhin]

Very good, thank you, all. Understood on all points. :)

--

Aleksey Tsalolikhin

Founder and Chief Trainer

Vertical Sysadmin, Inc.

+1-323-393-0779

ᐧ