per-promise 'inform => "false"'?

[t.d...@servicemusic.org.uk]

I had a recollection from several years ago that a "files:" promise could take an attribute:

• inform => "false"

Is my memory playing tricks?  I've just tried it and got:

• Unknown attribute 'inform' for promise type 'files'

Background.  We have been working our way up from 3.10 to 3.15.  Now I'm trying 3.18.  Suddenly one of our promises has changed from producing a single diagnostic line for the file:

• info: Edit file '...'

to two diagnostic lines for each and every line changed in the file.  What pushes this over the acceptability boundary for us is that this is a local "state" file, which we deliberately empty and re-populate with 400-500 lines each and every run of "cf-agent" (so 800-1,000 diagnostic lines every run of cf-agent).

What I ideally wish to do is that apply something like "inform => false" to that single promise (with all other promises continuing to show their more modest diagnostic lines).

So... is my (supposed) memory of a per-promise "inform => false" (or similar) playing mind-games and tricks on me?  Or was there once such an attribute?  (Maybe even 15 years ago when, at another place, I had been using CFE version 2!)  Or, even better, is there such a per-promise attribute, but I simply haven't found it?

-- David Lee

[Nick Anderson]

I had a recollection from several years ago that a "files:" promise could take an attribute:

• inform => "false"

Is my memory playing tricks? I've just tried it and got:

• Unknown attribute 'inform' for promise type 'files'

Yes, probably a cosmic ray has flipped a bit in your memory.

There is a promise level inform attribute, but, it's not for files it's for commands.

Here is a fun snippet, show the promise types that have an attribute of inform.

exec 2>&1
cf-promises --syntax-description json | jq '.promiseTypes | to_entries[] | select(.value.attributes.inform != null) | .key'
:

"commands"

It's a relatively recent addition, introduced in 3.15.0 (2019).

https://docs.cfengine.com/docs/3.21/reference-promise-types-commands.html#inform

Background. We have been working our way up from 3.10 to 3.15. Now I'm trying 3.18. Suddenly one of our promises has changed from producing a single diagnostic line for the file:

• info: Edit file '…'

to two diagnostic lines for each and every line changed in the file. What pushes this over the acceptability boundary for us is that this is a local "state" file, which we deliberately empty and re-populate with 400-500 lines each and every run of "cf-agent" (so 800-1,000 diagnostic lines every run of cf-agent).

What I ideally wish to do is that apply something like "inform => false" to that single promise (with all other promises continuing to show their more modest diagnostic lines).

So… is my (supposed) memory of a per-promise "inform => false" (or similar) playing mind-games and tricks on me? Or was there once such an attribute? (Maybe even 15 years ago when, at another place, I had been using CFE version 2!) Or, even better, is there such a per-promise attribute, but I simply haven't found it?

Yes, the inform logging became more specific. It's not expected that each execution of cf-agent is done with inform logging, generally that's reserved for manual runs or specific cases. As a general rule there is an expectation that a regularly scheduled execution of cf-agent will produce no output unless there was some problem.

action bodies can specify log_level (https://docs.cfengine.com/docs/3.21/reference-promise-types.html#log_level) and report_level (https://docs.cfengine.com/docs/3.21/reference-promise-types.html#report_level). log_level influences what is sent to syslog and report_level influences standard output. These are useful for increasing, not decreasing the level of output from the global default. So, if you ran cf-agent with no log level specified you could for a single promise increase it's level to inform for example, but if you run cf-agent --log-level inform you could not reduce an individual promise log level to error.

Here is a small example to illustrate the behavior:

Listing 1: Example Policy

bundle agent __main__
{
    methods:
      "init";
      "example";
}
bundle agent init
{
  files:
    "/tmp/[1-3].txt"
      delete => default:tidy,
      action => default:report_level_x( "error" );
}
bundle agent example
{
  files:
    "/tmp/1.txt"
      content => "default report level";

    "/tmp/2.txt"
      content => "promise specific inform report level",
      action => default:report_level_x( "inform" );

    "/tmp/3.txt"
      content => "promise specific verbose report level",
      action => default:report_level_x( "verbose" );


  reports:
    "/tmp/1.txt" printfile => default:cat( "$(this.promiser)" );
    "/tmp/2.txt" printfile => default:cat( "$(this.promiser)" );
    "/tmp/3.txt" printfile => default:cat( "$(this.promiser)" );

}
body action report_level_x( x )
{
    report_level => "$(x)";
}

# cf-agent --no-lock --file /tmp/example.cf
    info: Created file '/tmp/2.txt', mode 0600
    info: Updated file '/tmp/2.txt' with content 'promise specific inform report level'
 verbose: P: .........................................................
 verbose: P: BEGIN promise 'promise_cfengine3_kw9dYH_25' of type "files" (pass 1)
 verbose: P:    Promiser/affected object: '/tmp/3.txt'
 verbose: P:    Part of bundle: example
 verbose: P:    Base context class: any
 verbose: P:    Stack path: /default/main/methods/'example'/default/example/files/'/tmp/3.txt'[1]
 verbose: Using literal pathtype for '/tmp/3.txt'
 verbose: No mode was set, choose plain file default 0600
 verbose: Additional promise info: source path '/home/nickanderson/org/roam/daily/work/cfengine3-kw9dYH' at line 25
    info: Created file '/tmp/3.txt', mode 0600
 verbose: Replacing '/tmp/3.txt' with content 'promise specific verbose report level'
 verbose: Additional promise info: source path '/home/nickanderson/org/roam/daily/work/cfengine3-kw9dYH' at line 25
    info: Updated file '/tmp/3.txt' with content 'promise specific verbose report level'
 verbose: Additional promise info: source path '/home/nickanderson/org/roam/daily/work/cfengine3-kw9dYH' at line 25
 verbose: files promise '/tmp/3.txt' repaired
 verbose: A: Promise REPAIRED
 verbose: P: END files promise (/tmp/3.txt)
R: /tmp/1.txt
R: default report level
R: /tmp/2.txt
R: promise specific inform report level
R: /tmp/3.txt
R: promise specific verbose report level
 verbose: P: .........................................................
 verbose: P: BEGIN promise 'promise_cfengine3_kw9dYH_25' of type "files" (pass 2)
 verbose: P:    Promiser/affected object: '/tmp/3.txt'
 verbose: P:    Part of bundle: example
 verbose: P:    Base context class: any
 verbose: P:    Stack path: /default/main/methods/'example'/default/example/files/'/tmp/3.txt'[1]
 verbose: Using literal pathtype for '/tmp/3.txt'
 verbose: P: .........................................................
 verbose: P: BEGIN promise 'promise_cfengine3_kw9dYH_25' of type "files" (pass 3)
 verbose: P:    Promiser/affected object: '/tmp/3.txt'
 verbose: P:    Part of bundle: example
 verbose: P:    Base context class: any
 verbose: P:    Stack path: /default/main/methods/'example'/default/example/files/'/tmp/3.txt'[1]
 verbose: Using literal pathtype for '/tmp/3.txt'

Then, when run again with inform as the default, we can see that the file deletion still comes out as info, it's not suppressed by setting it's report_level to error because this attribute can only increase the verbosity.

# cf-agent --no-lock --log-level inform --file /tmp/example.cf
    info: Deleted file '/tmp/1.txt'
    info: Deleted file '/tmp/2.txt'
    info: Deleted file '/tmp/3.txt'
    info: Created file '/tmp/1.txt', mode 0600
    info: Updated file '/tmp/1.txt' with content 'default report level'
    info: Created file '/tmp/2.txt', mode 0600
    info: Updated file '/tmp/2.txt' with content 'promise specific inform report level'
 verbose: P: .........................................................
 verbose: P: BEGIN promise 'promise_cfengine3_JFnoIJ_25' of type "files" (pass 1)
 verbose: P:    Promiser/affected object: '/tmp/3.txt'
 verbose: P:    Part of bundle: example
 verbose: P:    Base context class: any
 verbose: P:    Stack path: /default/main/methods/'example'/default/example/files/'/tmp/3.txt'[1]
 verbose: Using literal pathtype for '/tmp/3.txt'
 verbose: No mode was set, choose plain file default 0600
 verbose: Additional promise info: source path '/home/nickanderson/org/roam/daily/work/cfengine3-JFnoIJ' at line 25
    info: Created file '/tmp/3.txt', mode 0600
 verbose: Replacing '/tmp/3.txt' with content 'promise specific verbose report level'
 verbose: Additional promise info: source path '/home/nickanderson/org/roam/daily/work/cfengine3-JFnoIJ' at line 25
    info: Updated file '/tmp/3.txt' with content 'promise specific verbose report level'
 verbose: Additional promise info: source path '/home/nickanderson/org/roam/daily/work/cfengine3-JFnoIJ' at line 25
 verbose: files promise '/tmp/3.txt' repaired
 verbose: A: Promise REPAIRED
 verbose: P: END files promise (/tmp/3.txt)
R: /tmp/1.txt
R: default report level
R: /tmp/2.txt
R: promise specific inform report level
R: /tmp/3.txt
R: promise specific verbose report level
 verbose: P: .........................................................
 verbose: P: BEGIN promise 'promise_cfengine3_JFnoIJ_25' of type "files" (pass 2)
 verbose: P:    Promiser/affected object: '/tmp/3.txt'
 verbose: P:    Part of bundle: example
 verbose: P:    Base context class: any
 verbose: P:    Stack path: /default/main/methods/'example'/default/example/files/'/tmp/3.txt'[1]
 verbose: Using literal pathtype for '/tmp/3.txt'
 verbose: P: .........................................................
 verbose: P: BEGIN promise 'promise_cfengine3_JFnoIJ_25' of type "files" (pass 3)
 verbose: P:    Promiser/affected object: '/tmp/3.txt'
 verbose: P:    Part of bundle: example
 verbose: P:    Base context class: any
 verbose: P:    Stack path: /default/main/methods/'example'/default/example/files/'/tmp/3.txt'[1]
 verbose: Using literal pathtype for '/tmp/3.txt'

So, are you running with --inform by default, if so, why?

[t.d...@servicemusic.org.uk]

Many thanks, NIck.  Appreciated.

No we're not running with inform as default.   So that part of my message was not as considered as it should have been, and took us down a cul-de-sac.  Apologies.

Nevertheless, when we run cf-agent manually, we habitually run with "-I".  I suppose this qualifies as local convention.

For reference here is the problematic part (bundle slightly reduced here) where we dump all classes, several hundred of them, into a state file:

---------------------------------------------------------------

bundle agent dls_reports
{
  vars:
      "allclasses" slist => classesmatching(".*");

  files:
    cfengine_3::
      "$(sys.workdir)/state/allclasses.txt"
        create => "true",
        perms => mog("0644", "root", "root"),
        edit_defaults => empty,
        edit_line => insert_lines(@(allclasses));

}

---------------------------------------------------------------

Try that in a test framework.  Up to and including version 3.15.5, "cf-agent -I ..." (manually at the console) is decently quiet at this point (a single diagnostic line telling us the file has been edited).  But at 3.18 its output swamps us!

-- David Lee

[Vratislav Podzimek]

On Fri, 2023-07-14 at 02:54 -0700, t.d...@servicemusic.org.uk wrote:

Many thanks, NIck.  Appreciated.

No we're not running with inform as default.   So that part of my message was not as considered as it should have been, and took us down a cul-de-sac.  Apologies.

Nevertheless, when we run cf-agent manually, we habitually run with "-I".  I suppose this qualifies as local convention.

For reference here is the problematic part (bundle slightly reduced here) where we dump all classes, several hundred of them, into a state file:

---------------------------------------------------------------

bundle agent dls_reports
{
  vars:
      "allclasses" slist => classesmatching(".*");

  files:
    cfengine_3::
      "$(sys.workdir)/state/allclasses.txt"
        create => "true",
        perms => mog("0644", "root", "root"),
        edit_defaults => empty,
        edit_line => insert_lines(@(allclasses));

}

---------------------------------------------------------------

Try that in a test framework.  Up to and including version 3.15.5, "cf-agent -I ..." (manually at the console) is decently quiet at this point (a single diagnostic line telling us the file has been edited).  But at 3.18 its output swamps us!

A note here -- the insert_lines promise doesn't produce the info: lines if the promise is idempotent (i.e. if it makes no changes when run multiple times). Which version exactly are you running? Because since 3.18.1 there should be no info: messages if no changes are made to the file. OTOH, a single line of difference causes all of the individual lines to be logged, I think.

One tip -- it might be useful to do sort() on the slist to ensure better consistency of the results.

--

Vratislav

[Nick Anderson]

For reference here is the problematic part (bundle slightly reduced here) where we dump all classes, several hundred of them, into a state file:

bundle agent dls_reports { vars: "allclasses" slist => classesmatching(".*");

files: cfengine_3:: "$(sys.workdir)/state/allclasses.txt" create => "true", perms => mog("0644", "root", "root"), edit_defaults => empty, edit_line => insert_lines(@(allclasses)); }

---

Try that in a test framework. Up to and including version 3.15.5, "cf-agent -I …" (manually at the console) is decently quiet at this point (a single diagnostic line telling us the file has been edited). But at 3.18 its output swamps us!

Manual executions with -K and -I are super common, it's definitely embedded in my muscle memory.

First thing that I notice in that policy: since you are managing the full file content, I think you should use mustache or the content attribute instead of edit_line, any time you are using edit_default => empty it's worth re-evaluating the choice and considering something that was designed for full file management.

Instead of recording ALL classes you could be more selective and perhaps filter out classes that change with each run, like time_based classes.

Listing 1: Example Policy

bundle agent __main__
{
    

vars:
       "allclasses" slist => classesmatching(".*")

;
       "time_based_classes" slist => classesmatching(".*", "time_based");
       "allclasses_excluding_unstable"
         with => concat( "(", join( "|", time_based_classes), ")" ),
         slist => filter( "$(with)",         # Filter
                          "allclasses",      # source list
                          "true",            # is regex?
                          "true",            # invert result?
                          inf );             # max return

    reports:
      "allclasses has $(with) elements" with => length( "allclasses" );
      "allclasses_excluding_unstable has $(with) elements" with => length( "allclasses_excluding_unstable" );

}

Another option would be to use reports with report_to_file instead of files promises for managing the content of a file, you would have to be sure to cull the file first or it's basically a log.

Of course if you use a files promise and run with inform you will get noise about it's deletion.

bundle agent __main__

{
  vars:
      "allclasses" slist => classesmatching(".*")

;
      "time_based_classes" slist => classesmatching(".*", "time_based");

      "allclasses_excluding_unstable"
        with => concat( "(", join( "|", time_based_classes), ")" ),
        slist => sort( filter( "$(with)",         # Filter
                               "allclasses",      # source list
                               "true",            # is regex?
                               "true",            # invert result?
                               inf ),             # max return
                       "lex" );

  files:
      "/tmp/allclasses_excluding_unstable.txt"
        delete => default:tidy,
        handle => "cull_allclasses_statefile";

  reports:
      "allclasses has $(with) elements" with => length( "allclasses" );
      "allclasses_excluding_unstable has $(with) elements" with => length( "allclasses_excluding_unstable" );

      "$(allclasses_excluding_unstable)"
        report_to_file => "/tmp/allclasses_excluding_unstable.txt",
        depends_on => { "cull_allclasses_statefile" };
}

    info: Deleted file '/tmp/allclasses_excluding_unstable.txt'
R: allclasses has 281 elements
R: allclasses_excluding_unstable has 258 elements

So, you could use commands where you can suppress inform.

bundle agent __main__

{
  vars:
      "allclasses" slist => classesmatching(".*")

;
      "time_based_classes" slist => classesmatching(".*", "time_based");

      "allclasses_excluding_unstable"
        with => concat( "(", join( "|", time_based_classes), ")" ),
        slist => sort( filter( "$(with)",         # Filter
                               "allclasses",      # source list
                               "true",            # is regex?
                               "true",            # invert result?
                               inf ),             # max return
                       "lex" );

  commands:
      "/bin/rm /tmp/allclasses_excluding_unstable.txt"
        inform => "false",
        contain => silent,
        handle => "cull_allclasses_statefile",
        if => fileexists( "/tmp/allclasses_excluding_unstable.txt" );

  reports:
      "allclasses has $(with) elements" with => length( "allclasses" );
      "allclasses_excluding_unstable has $(with) elements" with => length( "allclasses_excluding_unstable" );

      "$(allclasses_excluding_unstable)"
        report_to_file => "/tmp/allclasses_excluding_unstable.txt",
        depends_on => { "cull_allclasses_statefile" };
}

R: allclasses has 281 elements
R: allclasses_excluding_unstable has 258 elements

It would be nice to be able to set the log level per promise and not just increase it, I think there is a ticket somewhere but I couldn't find in a couple minutes of searching.

And, finally, instead of managing that state file, you could get Enterprise reporting :D.

[t.d...@servicemusic.org.uk]

Thanks, Nick!

I was unaware of the "report_to_file" functionality on "reports::".  That's good to know.  I've just tried it and it seems to work very well.   (As you highlight, it's basically an append rather than a rewrite, so we need to re-create it.  But that's OK.)  So I now have a local MR awaiting review.

Yes, I certainly take the point about being more selective in the contents!  But that's a different topic, although I might try to raise it on the coat-tails of this MR.

Enterprise reporting?  Well, for ages I've been gently trying to push Enterprise here. A few months ago, we started pursuing it, but the cost, even after some negotiation was going to be prohibitive.  (In contrast to technical aspects of CFE, this aspect is out of my hands.)

Thanks again.

-- David Lee

On Friday, 14 July 2023 at 16:08:47 UTC+1 nick.a...@northern.tech wrote:

For reference here is the problematic part (bundle slightly reduced here) where we dump all classes, several hundred of them, into a state file:

bundle agent dls_reports { vars: "allclasses" slist => classesmatching(".*");

[Nick Anderson]

Thanks, Nick!

I was unaware of the "report_to_file" functionality on "reports::". That's good to know.

There are a few reports promise features that are little known. The printfile body comes to mind which emits a files content as a report.

Also the return_value_index attribute, which defines an associative array key for the variable named by the useresult attribute for the calling methods type promise in the calling bundle.

Enterprise reporting? Well, for ages I've been gently trying to push Enterprise here. A few months ago, we started pursuing it, but the cost, even after some negotiation was going to be prohibitive. (In contrast to technical aspects of CFE, this aspect is out of my hands.)

Yeah, that's unfortunate. But, I like to eat so I gotta suggest it :D.

[Nick Anderson]

Re-sent, there were supposed to be examples included, oops.

Thanks, Nick!

I was unaware of the "report_to_file" functionality on "reports::". That's good to know.

There are a few reports promise features that are little known. The printfile body comes to mind which emits a files content as a report.

Listing 1: Example illustrating body printfile cat

bundle agent __main__
{
  reports:
    "I am $(this.promise_filename).";
    "I contain" printfile => cat( $(this.promise_filename) );
}

R: I am $(this.policy_filename).
R: I contain
R: body file control{ inputs => { '$(sys.libdir)/stdlib.cf' };}
R: bundle agent __main__
R: {
R:   reports:
R:     "I am $(this.policy_filename).";
R:     "I contain" printfile => cat( $(this.promise_filename) );
R: }

Also the return_value_index attribute, which defines an associative array key for the variable named by the useresult attribute for the calling methods type promise in the calling bundle.

Listing 2: Example illustrating useresult and bundle_return_value_index

bundle agent __main__
{
  methods:

      "Call a bundle, define an associative array as a byproduct"
        usebundle => example_bundle_return_value_index_and_useresult("value 1"),
        useresult => "my_array";

      "Call a bundle, define an associative array as a byproduct"
        usebundle => example_bundle_return_value_index_and_useresult(""),
        useresult => "my_array";

  reports:
      "JSON representation of 'my_array' :$(const.n) $(with)"
        with => storejson( mergedata( my_array ) );
}
bundle agent example_bundle_return_value_index_and_useresult(param)
{
  reports:
      "Value 0"
        bundle_return_value_index => "0";

      "Value 2"
        bundle_return_value_index => "two";

      "$(with)"
        with => string_upcase( "$(param)" ), # Here we upcase the value of param and use it as the value for the key returned
        bundle_return_value_index => "$(param)"; # Here the parameter value is used to set the key that is returned

}

R: JSON representation of 'my_array' :
 {
  "0": "Value 0",
  "two": "Value 2",
  "value 1": "VALUE 1"
}

Enterprise reporting? Well, for ages I've been gently trying to push Enterprise here. A few months ago, we started pursuing it, but the cost, even after some negotiation was going to be prohibitive. (In contrast to technical aspects of CFE, this aspect is out of my hands.)

Yeah, that's unfortunate. But, I like to eat so I gotta suggest it :D.