EFL Delta Reporting setup questions

[Xander Cage]

so i'm in the process of testing this delta reporting thing. since exact setup instructions are literally not available i was forced to tinker (which i really hate), but now i reached a dead end.

how to i accomplish these two steps?  everything beside this i got already running more or less.


how do i do this with the efl_main method? my production policies are running within the standard cfengine autorun mech, and i want to keep this delta reporting voodoo away from them.
10:

1. Install delta_reporting.cf into your CFEngine policy. Run bundles in the following order. Use of EFL's main methods bundle is encouraged. Don't forget the null parameters.

   1. deltarep_prelogging as early as possible on all hosts.
   2. deltarep_postlogging as late as possible on all hosts.
   3. deltarep_client_get after deltarep_postlogging on policy servers only.

same as above, want to do this with efl.

12:

1. Install bin/dhlogmaker and configure CFEngine to install it on all hosts. /opt/delta_reporting/bin/dhlogmaker suggested.

[Neil Watson]

Hi Xander,

I'm glad you're trying EFL and DR, but your questions are vague to me.

The step 1 you mentioned:
DR has three CFEngine bundles. The first preps the agent host's policy
run to log promise outcomes. The second creates a final log of the
agent's run, thus at the end. The last one runs only on the policy
server; it fetches the created logs from each host and puts their data
in the DR database.

You can put these bundles in your input sequence, call them via a
method, or use the efl_main methods bundle.

Note: https://github.com/neilhwatson/delta_reporting#note-about-efl

How EFL and DR work:

https://github.com/neilhwatson/delta_reporting/blob/master/doc/architecture.odt

The DR program dhlogmaker processes logs created by cf-agent on each
host. Cf-agent's promise logging is poor, and dhlogmaker cleans up the
log prior to the server downloading them. This is described in the
above architecture doc.

Please offer up suggestions on how I can improve the documentation.

On Wed, Nov 23, 2016 at 02:12:46AM -0800, Xander Cage wrote:
>
>
> so i'm in the process of testing this delta reporting thing. since exact
> setup instructions are literally not available i was forced to tinker
> (which i really hate), but now i reached a dead end.
>
> how to i accomplish these two steps?  everything beside this i got
> already running more or less.
>
> how do i do this with the efl_main method? my production policies are
> running within the standard cfengine autorun mech, and i want to keep
> this delta reporting voodoo away from them.
> 10:
>

>  1. Install delta_reporting.cf into your CFEngine policy. Run bundles in

> the following order. Use of EFL's main methods bundle is encouraged.
> Don't forget the null parameters.
>

>  1. deltarep_prelogging as early as possible on all hosts.
>  2. deltarep_postlogging as late as possible on all hosts.
>  3. deltarep_client_get after deltarep_postlogging on policy

> servers only.
>
> same as above, want to do this with efl.
>
> 12:
>

>  1. Install bin/dhlogmaker and configure CFEngine to install it on all
> hosts. /opt/delta_reporting/bin/dhlogmaker suggested.
>
> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [1]help-cfengin...@googlegroups.com.
> To post to this group, send email to [2]help-c...@googlegroups.com.
> Visit this group at [3]https://groups.google.com/group/help-cfengine.
> For more options, visit [4]https://groups.google.com/d/optout.
>
>References
>
> Visible links
> 1. mailto:help-cfengin...@googlegroups.com
> 2. mailto:help-c...@googlegroups.com
> 3. https://groups.google.com/group/help-cfengine
> 4. https://groups.google.com/d/optout
>ELinks: No such file or directory


--
Neil H Watson @neil_h_watson
CFEngine reporting: https://github.com/neilhwatson/delta_reporting
CFEngine policy: https://github.com/neilhwatson/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3

[Xander Cage]

ok i made some progress,  i'm able to run the delta_reporting bundles, log files are collected from the clients but unfortunatly nothing shows up in the database. cfengine is trying to run the load script but something is not working as expected.

when i try to run the command manually it outputs the following.

root@nimmsag: /var/cfengine/outputs # perl /opt/delta_reporting/app/script/load /var/cfengine/delta_reporting/log/client_logs/172.17.14.5.log
[Wed Nov 23 15:40:09 2016] [debug] Reading configuration file "/opt/delta_reporting/app/DeltaR.conf"
hostname: Not a recognized flag: -
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536
ERROR: These inputs for Validator::client_log are invalid:  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line 536

sample of log file format..

root@nimmsag: /var/cfengine/delta_reporting/log/client_logs # cat 172.17.10.190.log
2016-11-23T15:30:52CET ;; have_fs_kind ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; November ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs___mounts_15__1__ ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; ipv4_172_16 ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; services_autorun ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs__ ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs_Licensed ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; _stdlib_path_exists_df ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; _stdlib_has_path_diff ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; www_alt_in_low ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs___mount ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; _stdlib_has_path_logger ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; ipv4_172 ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; 172_16_152_24 ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; diskfree_high ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; smtp_in_high ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs___mounts_125__1__ ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs___vol ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; _stdlib_has_path_df ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; _stdlib_path_exists_find ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs___mounts_17__2___ ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; __override_classes_ ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs_of ;; empty ;; empty ;; empty ;; empty
2016-11-23T15:30:52CET ;; have_fs_prevent ;; empty ;; empty ;; empty ;; empty
Enter code here...

delta reporting bundle sequence

root@nimmsag: /var/cfengine/masterfiles/itsv # cat deltarep_bundles.cf
bundle agent b0034_dist_dhlogmaker

{

 meta:
    "tags"  slist => { "itsv" };


  files:

        "/opt/delta_reporting/bin/."
         create => 'true',
         perms  => m( "644"  );

        "/opt/delta_reporting/bin/dhlogmaker"
         perms       => m( "755" ),
         copy_from   => secure_cp( "${sys.workdir}/bin/dhlogmaker", "${sys.policy_hub}");


 reports:


      "dist_dhlogmaker was running at ( $(sys.date) ).";


}

bundle agent A_deltarep_prelogging

{

 meta:
    "tags"  slist => { "itsv" };

 methods:
      "deltarep_prelogging" usebundle => deltarep_prelogging;

 reports:


      "deltarep_prelogging was running at ( $(sys.date) ).";

}

bundle agent b9998_deltarep_postlogging

{

 meta:
    "tags"  slist => { "itsv" };

 methods:
      "deltarep_postlogging" usebundle => deltarep_postlogging;

 reports:


      "deltarep_postlogging was running at ( $(sys.date) ).";

}


bundle agent b9999_deltarep_client_get

{

 meta:
    "tags"  slist => { "itsv" };


policy_server::

 methods:
      "deltarep_client_get " usebundle => deltarep_client_get;

 reports:


      "deltarep_client_get was running at ( $(sys.date) ).";

}

>   1. mailto:help-cfengine+unsub...@googlegroups.com

[Neil Watson]

The timestamps in those logs have a word timezone, CET, but the program
expects numbers, e.g. -100. I don't know how the letters got there. Can
you please open a bug on github. Include your agent platform, agent perl
version, and the DR tag you're running. Also, please include the head of
the raw logs from the client:

/var/cfengine/delta_reporting/log/classes
/var/cfengine/delta_reporting/log/promises
/var/cfengine/delta_reporting/log/final


The expected timestamp format is yyy-mm-ddThh:mm[+-]hhmmO

On Wed, Nov 23, 2016 at 06:51:02AM -0800, Xander Cage wrote:
> root@nimmsag: /var/cfengine/outputs # perl
> /opt/delta_reporting/app/script/load
> /var/cfengine/delta_reporting/log/client_logs/172.17.14.5.log
> [Wed Nov 23 15:40:09 2016] [debug] Reading configuration file
> "/opt/delta_reporting/app/DeltaR.conf"
> hostname: Not a recognized flag: -
> ERROR: These inputs for Validator::client_log are invalid:
>  policy_server timestamp, skipping record at lib/DeltaR/Query.pm line
> 536
>

> sample of log file format..
>
> root@nimmsag: /var/cfengine/delta_reporting/log/client_logs # cat
> 172.17.10.190.log
> 2016-11-23T15:30:52CET ;; have_fs_kind ;; empty ;; empty ;; empty ;;
> empty

[Xander Cage]

this has something to do with the ibm aix olson timezone setting, which is somewhat "recommended" from ibm. i will open a bug on github.

[Neil Watson]

FYI the date stamp format is set here:

https://github.com/neilhwatson/evolve_cfengine_freelib/blob/master/masterfiles/lib/3.7/EFL/efl_common.cf#L71

It's generated by cf-agent.

[Neil Watson]

For posterity, bug is
https://github.com/neilhwatson/delta_reporting/issues/31

and appears to be AIX related.