Cleanup after running a test script?

[Craig Rodrigues]

Hi,

I have a very simple Kyuafile like this:

syntax(2)
test_suite('first')
plain_test_program{name='test1.sh'}


Is it possible to specify other programs or lua functions which run before and
after test1.sh which can do additional preparation and cleanup on the system,

but are not part of the test script itself?

I have a suite of test scripts, some of which do some destructive things.

When the test scripts are run in an automation run, cleanup needs to occur

after each script runs.

However, when the script is run individually, no cleanup should occur,
because the individual developer needs to be able to run the script
and see the system in the "non-cleaned up" state in order to debug the problem.

Thanks.

--
Craig

[Alan Somers]

Nope, not possible. When I had to do something like this, I created
ATF wrapper scripts. The ATF test case's body called the original
file (test1.sh) and the ATF testcase's cleanup function did the
cleanup. If you did the same, then developers could run test1.sh if
they want to see the noncleaned up state.

-Alan

[Craig Rodrigues]

That's too bad.  I wanted to avoid going down the path of writing wrapper scripts

around scripts which get called by Kyua, but looks like I can't avoid it.

kyua already provides some basic test-case isolation via doing things like creating a tmpdir
to run as the current working directory.  It would be nice if there were some pre and post-test hooks
that could be exposed for custom cleanup, not offered by the base kyua runtime.

--

Craig

[Craig Rodrigues]

On Fri, Sep 12, 2014 at 8:48 PM, Alan Somers <aso...@freebsd.org> wrote:

What was the structure of the wrapper scripts that you created?

A Kyuafile cannot have an atf_test_program or a plain_test_program

in a different directory from the Kyuafile.  So, did you copy the wrapper script into each directory with a Kyuafile?

Also, in a Kyuafile, is it possible to specify command-line arguments to an atf_test_program or plain_test_program?

I didn't seem to see a way to do it, so something like:

wrapper_cleanup.sh mytest1.sh

didn't seem possible.

Thanks.

--

Craig

[Julio Merino]

On Fri, Sep 12, 2014 at 8:47 PM, Craig Rodrigues <rod...@freebsd.org> wrote:
> Hi,
>
> I have a very simple Kyuafile like this:
>
> syntax(2)
> test_suite('first')
> plain_test_program{name='test1.sh'}
>
>
> Is it possible to specify other programs or lua functions which run before
> and
> after test1.sh which can do additional preparation and cleanup on the
> system,
> but are not part of the test script itself?

It's not possible and I'm not even sure it's a good idea to bundle
this in Kyua. As a general consideration, I don't want to have any
testing code in the Kyuafile; previous experiences with code in
Makefiles and elsewhere have proven this to be suboptimal.

In general, such logic belongs in the test program as the test program
is responsible for cleaning up after itself. (In my opinion, Kyua
already does too much trying to clean up mount points in the work
directory, for example, and I'd rather streamline this than add more
special cases to cope with sloppy test programs.)

> I have a suite of test scripts, some of which do some destructive things.
> When the test scripts are run in an automation run, cleanup needs to occur
> after each script runs.
>
> However, when the script is run individually, no cleanup should occur,
> because the individual developer needs to be able to run the script
> and see the system in the "non-cleaned up" state in order to debug the
> problem.

This is the "special case" and as such should be treated accordingly.
A test program should never leave the system in an inconsistent state
unless you explicitly told it to for debugging purposes. As an
alternative, have you considered adding a "DONT_CLEANUP_FOR_DEBUGGING"
environment variable to the test program and use that to determine
whether the cleanup parts should be run or not? Sounds like a simpler
alternative that doesn't require any plumbing.

[Craig Rodrigues]

Hi,

I mostly agree with your points.  I am working with a team that already has an existing suite of
unit tests that wants to put these tests under automation.  Kyua seems like a good fit for this, so I introduced Kyua
into this environment to get things going.  When I started modifying individual tests to provide more robust cleanup, I received pushback and was told NO.

The team does not want additional cleanup added to the tests other than what is already there.

I even proposed something similar to DONT_CLEANUP_FOR_DEBUGGING, but received pushback on this
also.  The team wants the cleanup to be done by the "test execution framework", and have it hidden from
the individual tests.  I'm basically being pushed in a direction where I cannot modify these unit tests.

It would be nice if Kyua could expose some hooks that could run maybe before and after each test,

to do custom preparation and cleanup.  If this can't be done, then I need to go down the

path that Alan did of coming up with wrapper scripts to do the custom cleanup, i.e.

cleanup_wrapper.sh    /path/to/test1.sh
cleanup_wrapper.sh    /path/to/test2.sh

This is really not ideal, but due to the constraints I have, I don't have many options.

--

Craig