Requesting feedback on feature proposal CFE-2788

[Nick Anderson]

There is a feature proposal intended to make it easier to run individual policy files that are also part of a larger policy set.

Currently, main is the default bundle used if no bundlesequence is provided. This is very useful for small examples and for beginning to write a new policy. However, the bundle name must be changed in order to integrate multiple of them into the same policy set.

The proposal looks to introduce a feature similar to __main__ from python.

You can see a prototype of this feature on the dev mailing list here and a pull request including example usage here.

We would love to have your feedback in CFE-2788.

–
Nick Anderson
Doer of things, CFEngine

[Cory Coager]

I started playing with this recently.  For me, this is useful to run a specific bundle without specifying all the dependent bundles on the command line.  However, one thing I see lacking is the ignore_missing_bundles and ignore_missing_inputs parameters from common control.  Some of the dynamic inputs I read in may not exist and I don't care if that's the case.  Without these parameters it will result in an error.  I think I have a plan to work around it though.

This was a great addition, thanks for adding it!

[Ole Herman Schumacher Elgesem]

Glad you like it, Cory!

I don’t completely understand your request. Maybe you can create a ticket in our bug tracker [1] and show a minimal policy example of what doesn’t work currently / how you would like it to work?

-Ole Herman

[1] https://tracker.mender.io/

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Nick Anderson]

Ole Herman Schumacher Elgesem writes:

> Glad you like it, Cory!
>
> I don’t completely understand your request. Maybe you can create
> a ticket in our bug tracker [1] and show a minimal policy
> example of what doesn’t work currently / how you would like it
> to work?
>
> -Ole Herman

My understanding is that he is looking for a way to configure the
ignore_missing_bundles and ignore_missing_inputs in body common
control
when executing from a library entry.

I think it can be done with some juggling, and possibly some
re-organization of policy.

It seems like it should be possible to use body file control to
load a
policy file (only if the entry is not promises.cf or update.cf)
that has
a body common control to set those desired configs.

> [1] https://tracker.mender.io/ <https://tracker.mender.io/>

>> <https://groups.google.com/forum/#!msg/dev-cfengine/P9Agtz9S3z8/wtK7MHgcBQAJ>

>> and a pull request including example usage here

>> <https://github.com/cfengine/core/pull/3118>.

>>
>> We would love to have your feedback in CFE-2788

>> <https://tracker.mender.io/browse/CFE-2788>.
>>
>> –

>> Nick Anderson
>> Doer of things, CFEngine
>>
>>
>> --
>> You received this message because you are subscribed to the
>> Google Groups "help-cfengine" group.
>> To unsubscribe from this group and stop receiving emails from
>> it, send an email to help-cfengin...@googlegroups.com

>> <mailto:help-cfengin...@googlegroups.com>.

>> To post to this group, send email to
>> help-c...@googlegroups.com

>> <mailto:help-c...@googlegroups.com>.

>> Visit this group at
>> https://groups.google.com/group/help-cfengine

>> <https://groups.google.com/group/help-cfengine>.

>> For more options, visit https://groups.google.com/d/optout

>> <https://groups.google.com/d/optout>.


--
Nick Anderson| Doer of Things | (+1) 785-550-1767 |
https://northern.tech

[Cory Coager]

Nick explained that pretty well.  It doesn't seem possible to make use of ignore_missing_bundles and ignore_missing_inputs from body file control.  You also cannot have multiple body common control files.  I am attempting to load files dynamically and not care if they exist or not but it will error out if the file doesn't exist, e.g. $(sys.uqhost).cf.  I believe I have come up with a work around to this issue though.

[Nick Anderson]

On Wednesday, February 27, 2019 at 6:44:27 AM UTC-6, Cory Coager wrote:

Nick explained that pretty well.  It doesn't seem possible to make use of ignore_missing_bundles and ignore_missing_inputs from body file control.  You also cannot have multiple body common control files.  I am attempting to load files dynamically and not care if they exist or not but it will error out if the file doesn't exist, e.g. $(sys.uqhost).cf.  I believe I have come up with a work around to this issue though.

If you do, please share :)

I was thinking something along the lines of:

services/policy1/def.json # define values for ignore_missing_bundles and ignore_missing_inputs

services/policy1/controls.cf # cat controls/cf_*.cf > controls.cf and tailor

services/policy1/policy1.cf # load ../../controls/def.cf and ./controls.cf with body file control based on sys.policy_entry_filename matching this.promise_filename (have to use an intermediary bundle in order to leverage this.promsie_filename)

Then if you execute cf-agent -f services/policy1/policy1.cf it will load def.json next to it and the custom controls

[Cory Coager]

I seem to be running into some issues with "error: Duplicate definition of bundle".  Can you not reuse a bundle that's also part of update.cf/promises.cf with bundle agent __main__?  It seems to be complaining about all of my bundle common that I use as a dependency for every bundle.  If that's the case I'll need to write a new main.cf file in each bundle directory that's not part of update.cf/promises.cf.  If I have to do it that way I should be able to use "body common control" in main.cf which would eliminate the issues with ignore_missing_bundles and ignore_missing_inputs.  Am I understanding this correctly?

--
You received this message because you are subscribed to a topic in the Google Groups "help-cfengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/help-cfengine/4JAdzFLlEfY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Nick Anderson]

Cory Coager writes:

I seem to be running into some issues with "error: Duplicate definition of
bundle". Can you not reuse a bundle that's also part of

update.cf/promises.cf with bundle agent main? It seems to be

complaining about all of my bundle common that I use as a dependency for
every bundle. If that's the case I'll need to write a new main.cf file in
each bundle directory that's not part of update.cf/promises.cf. If I have
to do it that way I should be able to use "body common control" in main.cf
which would eliminate the issues with ignore_missing_bundles and
ignore_missing_inputs. Am I understanding this correctly?

Right, each bundle must be uniquely named. Also, if you are using the library
main bundle, there can not be any bundle named main.

For example, this doesn't work:

bundle agent main
{
   reports:
     "CFEngine $(sys.cf_version)";
}
bundle agent __main__
{
    methods:
      "main";
}

/home/nickanderson/org/cfengine3-13734XRb:2:0: error: Duplicate definition of bundle main with type agent
/home/nickanderson/org/cfengine3-13734XRb:7:0: error: Duplicate definition of bundle main with type agent
   error: There are syntax errors in policy files
   error: Policy failed validation with command '"/home/nickanderson/.cfagent/bin/cf-promises" -c "/home/nickanderson/org/cfengine3-13734XRb"'
   error: CFEngine was not able to get confirmation of promises from cf-promises, so going to failsafe
   error: CFEngine failsafe.cf: /home/nickanderson/.cfagent/inputs /home/nickanderson/.cfagent/inputs/failsafe.cf
   error: No suitable server found
   error: No suitable server found
   error: No suitable server found
   error: Method 'failsafe_cfe_internal_update' failed in some repairs
  notice: Q: ".../cf-agent" -f /":    error: No suitable server found
Q: ".../cf-agent" -f /":    error: No suitable server found
Q: ".../cf-agent" -f /":    error: No suitable server found
Q: ".../cf-agent" -f /":    error: Method 'cfe_internal_update_policy_cpv' failed in some repairs
R: Built-in failsafe policy triggered

Inputs is supposed to be smart enough that having the same file included from
multiple places will not be considered a duplicate bundle.

What is the name of the bundle that it's complaining about?
In how many different files does a bundle with that name exist?

–

[Cory Coager]

> Inputs is supposed to be smart enough that having the same file included from
> multiple places will not be considered a duplicate bundle.  

This is not what I'm seeing.  One example would be "bundle common classify" which would be included in "body file control" for every custom bundle as well as the bundlesequence for update.cf/promises.cf.

[Nick Anderson]

Cory Coager writes:

>> Inputs is supposed to be smart enough that having the same file
>> included
> from
>> multiple places will not be considered a duplicate bundle.
> This is not what I'm seeing. One example would be "bundle
> common classify"
> which would be included in "body file control" for every custom
> bundle as
> well as the bundlesequence for update.cf/promises.cf.

I think I need to see the file structure to better understand.

services/classify/main.cf contains body common classify
services/policy1/main.cf has body file control inputs include
../classify/main.cf
services/policy2/main.cf has body file control inputs include
../classify/main.cf

^ I would expect that to work

--

[Cory Coager]

I found the issue.  I was running cf-promises on a local git copy.  The body file control inputs had absolute paths to $(sys.inputdir) so it was loading the same file from two different locations.  I changed the inputs to a relative path and the duplicate errors went away.

[Nick Anderson]

Cory Coager writes:
#+BEGIN_QUOTE

I found the issue. I was running cf-promises on a local git
copy. The
body file control inputs had absolute paths to $(sys.inputdir)
so it was
loading the same file from two different locations. I changed
the inputs
to a relative path and the duplicate errors went away.

#+END_QUOTE

Great!

I think the new sys.policy_entry* vars will be very nice with
respect to these
kinds of things.

[Cory Coager]

Still having problems with this.  When I switched to relative paths, a normal cf-agent run showed a lot of "No such file or directory" errors, e.g. /var/cfengine/inputs/../../path/to/bundle.cf.  I think for this to work I'll have to switch the body file control for __main__, update.cf and promises.cf to all use absolute paths.  Any issue with doing that?  Is there a better way?

[Nick Anderson]

Cory Coager writes:

Still having problems with this. When I switched to relative paths, a
normal cf-agent run showed a lot of "No such file or directory" errors,

e.g. var/cfengine/inputs../../path/to/bundle.cf. I think for this to
work I'll have to switch the body file control for main, update.cf and

promises.cf to all use absolute paths. Any issue with doing that? Is
there a better way?

I would imagine that the other places in policy that are using full paths, or
different variables could be updated instead of having to rely on absolute
paths.

Inspecting both sys.policy_entry* and sys.inputdir or sys.workdir vars and you
should be able to determine the paths a file needs so that it works in either
case.

–

[Cory Coager]

Example of what I'm seeing:

mypromises.cf:

body common control

{

   bundlesequence => { depend1, dosomething1 };

   inputs => { "mystuff/depend1/depend1.cf", "mystuff/dosomething1/dosomething1.cf" };

}

mystuff/depend1/depend1.cf:

bundle agent depend1

{

  reports:

    "$(this.bundle)";

}

body file control

{

    inputs =>

  {

    "../depend1/depend1.cf",

  };

}

bundle agent __main__

{

  methods:

    "depend1" usebundle => depend1;

    "dosomething1" usebundle => dosomething1;

}

bundle agent dosomething1

{

  reports:

    "$(this.bundle)";

}

cf-agent -KIC -f mystuff/dosomething1/dosomething1.cf

R: depend1

R: dosomething1

cf-agent -KIC -f ./mypromises.cf

   error: Can't stat file './../depend1/depend1.cf' for parsing. (stat: No such file or directory)

   error: Policy failed validation with command '"/var/cfengine/bin/cf-promises" -c "./mypromises.cf"'

   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.

   error: Error reading CFEngine policy. Exiting...

[Nick Anderson]

body file control

{

    inputs =>

  {

    "../depend1/depend1.cf",

  };

}

Instead of using plain relative paths ( I think these are relative from $(sys.inputdir) ) try using paths relative to $(sys.policy_entry_dirname) also, fyi sys.policy_entry vars can be expanded in body file control directly, where as this.promise vars can not.

[Cory Coager]

The path

"../depend1/depend1.cf"

IS the correct relative path to $(sys.policy_entry_dirname) when called with -f.

--

[Nick Anderson]

I think the policy_entry variable resolves to a full path, if you use that instead, does it work?

To unsubscribe from this group and all its topics, send an email to help-cfengine+unsubscribe@googlegroups.com.

[Cory Coager]

You mean $(sys.policy_entry_dirname)?  Yes it is a full path:

/var/cfengine/inputs/mystuff/dosomething1/

However, I still need to go UP a directory to get to the dependency bundle:

/var/cfengine/inputs/mystuff/depend1/depend1.cf

so the relative path would be:

../depend1/depend1.cf

To unsubscribe from this group and all its topics, send an email to help-cfengin...@googlegroups.com.

To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "help-cfengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/help-cfengine/4JAdzFLlEfY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to help-cfengin...@googlegroups.com.

[Nick Anderson]

Cory Coager writes:

You mean $(sys.policy_entry_dirname)? Yes it is a full path:

var/cfengine/inputs/mystuff/dosomething1

However, I still need to go UP a directory to get to the dependency bundle:
/var/cfengine/inputs/mystuff/depend1/depend1.cf

so the relative path would be:
../depend1/depend1.cf

I mean do the relateive path from policy_entry: $(sys.policy_entry_dirname)/../depend1/depend1.cf

[Cory Coager]

Still fails

cf-agent -KIC -f mystuff/dosomething1/dosomething1.cf

R: depend1

R: dosomething1

cf-agent -KIC

   error: Can't stat file '/var/cfengine/inputs/../depend1/depend1.cf' for parsing. (stat: No such file or directory)

   error: Policy failed validation with command '"/var/cfengine/bin/cf-promises" -c "/var/cfengine/inputs/promises.cf"'

   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.

   error: Error reading CFEngine policy. Exiting...

[Nick Anderson]

On Friday, March 1, 2019 at 7:36:30 AM UTC-6, Cory Coager wrote:

Still fails

cf-agent -KIC -f mystuff/dosomething1/dosomething1.cf

R: depend1

R: dosomething1

cf-agent -KIC

   error: Can't stat file '/var/cfengine/inputs/../depend1/depend1.cf' for parsing. (stat: No such file or directory)

   error: Policy failed validation with command '"/var/cfengine/bin/cf-promises" -c "/var/cfengine/inputs/promises.cf"'

   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.

   error: Error reading CFEngine policy. Exiting...

Mmm, yeah. probably all the inputs in the MPF need to be moved to use sys.policy_entry, or you have to do some kind of special juggling so that if entry is not one of the main entries (upate.cf/promises.cf) that it uses a slightly different path for inputs.

[Cory Coager]

The only way I can see that working is using absolute paths for update.cf, promises.cf and __main__.  Alternatively, add an additional mybundle_standalone.cf file that is not part of update.cf or promises.

--

[Cory Coager]

I'm also not fond of __main__ not reading in $(sys.inputdir)/def.json.  I realize you can put def.json in the same directory as the bundle. However, when you are sharing variables in def across multiple bundles it doesn't make sense to me to duplicate data in different places.  Perhaps we can modify def to attempt to fall back to read $(sys.inputdir)/def.json if $(sys.policy_entry_dirname)/def.json doesn't exist?  It appears that part of def.json is written in C?  My attempts to modify this behavior in controls/def.cf have fallen in vain.

[Nick Anderson]

Cory Coager writes:

I'm also not fond of main not reading in $(sys.inputdir)/def.json. I

realize you can put def.json in the same directory as the bundle. However,
when you are sharing variables in def across multiple bundles it doesn't
make sense to me to duplicate data in different places. Perhaps we can
modify def to attempt to fall back to read $(sys.inputdir)/def.json if
$(sys.policy_entry_dirname)/def.json doesn't exist? It appears that part
of def.json is written in C? My attempts to modify this behavior in
controls/def.cf have fallen in vain.

Just to clarify, cf-agent reads $(sys.policy_entry_dirname)/def.json (not
sys.inputdir) and uses it, if it is present. Yes, augments is a feature
implemented in C. Originally, in 3.7 it was introduced as a feature implemented
in the MPF, but it was quickly moved directly inside the agent as a core feature
in order to improve its capabilities and behavior. The remnants in def.cf for
supporting that bridge time will be redacted before the next LTS release.

I have wondered how useful it would be to have an option to cf-agent that
disabled def.json and/or that allowed the user to specify a specific def.json
instead of the one next to the policy entry.