Policy testing using TAP

[Neil Watson]

Policy testing is something we don't see enough of. We need to know that
our policy works in the prototype stage and in development. Here is a
method using TAP (test anything protocol). First the policy. Recall that
you can test most things in CFEngine if you have a class. Classes can
represent a host or a promise outcome.

body common control
{
bundlesequence => { "main", };
}

bundle agent main
{
vars:
"tests"
comment => "Imagine your own classes and test here",
string => "
---
- class: any
class_to_test: debian
is_or_isnt: is
name: Test must be debian host
- class: any
class_to_test: redhat
is_or_isnt: is
name: Test must be redhat host
";

methods:
"any" usebundle => efl_test_simple( ${tests} );
}

bundle agent efl_test_simple(ref)
{
meta:
"Purpose" string => "Test class membership and outputs TAP protocol";

vars:
"test" data => parseyaml( ${ref} );
"next_test" slist => getindices( test );
"number_of_tests" int => length( "next_test" );
"test_number_${next_test}" string =>
format( "%1d", eval( "${next_test}+1", "math", "infix" ) );

classes:

"class_isnt_${next_test}"
ifvarclass => "${test[${next_test}][class]}",
expression => strcmp( "isnt", "${test[${next_test}][is_or_isnt]}" );
"class_is_${next_test}"
ifvarclass => "${test[${next_test}][class]}",
expression => strcmp( "is", "${test[${next_test}][is_or_isnt]}" );

"ok_${next_test}"
if => "class_isnt_${next_test}",
not => classmatch( "${test[${next_test}][class_to_test]}" );
"not_ok_${next_test}"
ifvarclass => "class_isnt_${next_test}",
expression => classmatch( "${test[${next_test}][class_to_test]}" );
"ok_${next_test}"
ifvarclass => "class_is_${next_test}",
expression => classmatch( "${test[${next_test}][class_to_test]}" );
"not_ok_${next_test}"
if => "class_is_${next_test}",
not => classmatch( "${test[${next_test}][class_to_test]}" );

reports:
# Prefix with new lines so TAP parser will ignore the CFEngine R: prefix.
"
1..${number_of_tests}";
"
ok ${test_number_${next_test}}, ${test[${next_test}][name]}"
if => "ok_${next_test}";
"
not ok ${test_number_${next_test}}, ${test[${next_test}][name]}"
if => "not_ok_${next_test}";
}

Let's run it:
R:
1..2
R:
ok 1, Test must be debian host
R:
not ok 2, Test must be redhat host

My test plan is to run two tests. Test one passes because I'm using a
debian host. Test two fails because it is not a redhat host. This output
is in the format of the TAP protocol. Anything that understand that
protocol can interpret it and offer better results. Now a test script,
this one is in Perl:

#!/usr/bin/perl

use strict;
use warnings;
use TAP::Harness;

my @tests = qw(
/home/neil/.cfagent/inputs/tap.cf
);

# Final test command is exec + element from test.
my $harness = TAP::Harness->new({
exec => [ qw{ /usr/bin/cf-agent -Kf } ],
verbosity => 1,
});

$harness->runtests( @tests );

This script runs /usr/bin/cf-agent -Kf /home/neil/.cfagent/inputs/tap.cf
and interprets the agent's output:

/home/neil/.cfagent/inputs/tap.cf ..
R:
1..2
R:
ok 1, Test must be debian host
R:
not ok 2, Test must be redhat host
Failed 1/2 subtests

Test Summary Report
-------------------
/home/neil/.cfagent/inputs/tap.cf (Wstat: 0 Tests: 2 Failed: 1)
Failed test: 2
Files=1, Tests=2, 1 wallclock secs ( 0.03 usr 0.00 sys + 0.03 cusr 0.01 csys = 0.07 CPU)
Result: FAIL

Now we know the results of our test.

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Marco Marongiu]

On 06/08/15 01:01, Neil Watson wrote:
> Policy testing is something we don't see enough of. We need to know that
> our policy works in the prototype stage and in development. Here is a
> method using TAP (test anything protocol).

Been there, done that:

> bronto@murray:~/Lab/hENC$ prove --exec "sudo cf-agent -KC -f" ./henc_test.cf
> ./henc_test.cf .. ok
> All tests successful.
> Files=1, Tests=8, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.07 cusr 0.00 csys = 0.10 CPU)
> Result: PASS

Thanks for the great idea, the next version of hENC will feature new
functionality, bug fixes and now also a TAP-compatible test suite :-)

Ciao!
-- bronto