purging files on clients

[Tim]

Hi,

With cfengine-community-3.3.5-1 on RedHat Enterprise 6:

I've run into an issue where if I remove a file on server:/var/cfengine/masterfiles, that file does not get removed on client:/var/cfengine/inputs with a cf-agent run.
I'm using the distributed cfengine_stdlib.cf "For Cfengine Core: 3.1.0", and failsafe.cf "Community Failsafe.cf 1.0.0".

In failsafe.cf, there is the following in the "files" section:

  am_policy_hub|validated_updates_ready::  # policy hub should always put masterfiles in inputs in order to check new policy
   "$(inputs_dir)"
           comment => "Copy policy updates from master source on policy server if a new validation was acquired",
            handle => "update_files_inputs_dir",
         copy_from => u_rcp("$(master_location)","$(sys.policy_hub)"),
      depth_search => u_recurse("inf"),
#     file_select  => u_input_files,
      file_select  => all,
        depends_on => { "check_valid_update" },
            action => u_immediate,
           classes => u_if_repaired("update_report");

Perhaps this needs a "purge => true" or similar directive to accomplish removing obsolete files on clients?

Thanks for any assistance with this.

[Nick Anderson]

On 01/03/2013 02:56 PM, Tim wrote:
> I've run into an issue where if I remove a file on
> server:/var/cfengine/masterfiles, that file does not get removed on
> client:/var/cfengine/inputs with a cf-agent run.
> I'm using the distributed cfengine_stdlib.cf "For Cfengine Core: 3.1.0",
> and failsafe.cf "Community Failsafe.cf 1.0.0".

...

> handle => "update_files_inputs_dir",
> copy_from => u_rcp("$(master_location)","$(sys.policy_hub)"),

...

> Perhaps this needs a "purge => true" or similar directive to accomplish
> removing obsolete files on clients?
>
> Thanks for any assistance with this.

Hi Tim,

Yes the default policy only copies from the server as you can see in
body copy_from u_rcp

body copy_from u_rcp(from,server)
{
source => "$(from)";
compare => "digest";
trustkey => "false";

!am_policy_hub::

servers => { "$(server)" };
}

You can of course change this behaviour by adding purge => true.

I think its worth bringing up the question if this default should be
changed to u_rsync or similar (yes the name might be misleading ...
rsync the tool) and include purge => "true". See sync_cp in the standard
library. I think that sync is probably more expected behaviour
especially for new users.

[Tim]

Hi Nick,

Thanks for your reply.  Adding "purge => true" to
body copy_from u_rcp
works as expected.

My vote would be for sync as the default behavior.

[Nick Anderson]

On 01/04/2013 12:01 PM, Tim wrote:
> My vote would be for sync as the default behavior.

Does anyone else have thoughts on this default update policy behaviour?
This is the second or third time this has come up iirc.

[Neil Watson]

IIRC Rsync does not purge unless expressly told to do so. I'd rather
the user expressly ask for purging. Purging by default could cause
destruction for inexperienced users.

--
Neil Watson
Linux/UNIX Consultant
http://watson-wilson.ca

[Nick Anderson]

On 01/04/2013 02:47 PM, Neil Watson wrote:
>
> IIRC Rsync does not purge unless expressly told to do so. I'd rather
> the user expressly ask for purging. Purging by default could cause
> destruction for inexperienced users.

I think that�s correct with regard to rsync the tool. But I think that
dangling files could cause more confusion. In the beginning when you are
only adding files it is deceiving that a clients inputs and the servers
masterfiles look the same. When you start moving policy around and
renaming things old files linger on clients after being removed from
masterfiles. I recall initially finding this confusing.

[Stephen Spencer]

For a general-use promise, I would prefer a non-destructive default.   There are far too many scenarios where I would rather see the necessity of managed file deletions and make a policy change than find the painful result of an unintended destructive one.

It makes perfect sense for the policy master's repository to be precisely mirrored on the client.   If there is concern regarding a discrepancy between the unit promise and the stdlib promise, it could be renamed--u_rmirror or some such.

-Stephen 

[berntjernberg]

Hi all,

I used to use the purge => "true" but I had an incident where all filesystems (zfs) on a host was 100% full and kept filling up.
The purging went ok but no new files could be downloaded/copied so we manually had to fix the filling up situation and then get CFEngine back to normal.

Regards
Bernt Jernberg

[Nick Anderson]

Sure, full filesystems happen. I don't think defaulting masterfiles distribution to purge would have helped you much here.

I am curious if you were surprised that purge was not the default for masterfiles distribution. Other than ruining out of disk did not having purge on cause any problems or confusion for you. Did you opt to leave purge on? Why?

--
Sent from Kaiten Mail. Please excuse my brevity.

[berntjernberg]

> I am curious if you were surprised that purge was not the default for masterfiles distribution.

I wasn't suprised, I just hadn't thought about it at the time.

> Other than ruining out of disk did not having purge on cause any problems or confusion for you. Did you opt to leave purge on? Why?

My file-names have stayed fairly the same so it wasn't a big problem. Like others have said, in the beginning you tend to rename files until you come up with something you like so purge was convenient for me. At the time of the incident we were swamped with work so I just did a quick fix removing the purging. I like to use the purging because it is less work keeping the client in sync with the hub. Maybe one could use the purge if the usage isn't 100%?

Regards
Bernt Jernberg

[Juris Krumins]

When we started using CFEngine we were kind of confused for the first time to see that file rename/delete/... operations leave things behind and clients get filled with artefacts. And so we decided to set purge => "true" explicitly.
I believe this is the behaviour, new users expect. Of course this can cause problems, by loosing files, still since CFEngine implement client/server arch, by having central place for policies (ideally keep all policies in VCS) , then modifying policies directly on client is not the way to go.