Better reporting for spelling atf_init_test_cases() incorrectly?

[Craig Rodrigues]

Hi,

I created the following test case where I misspelled atf_init_test_cases().

If I did,
kyua list
kyua test

I could not tell what I did wrong.

However, if I did:

test1.sh test1

I got:

/usr/local/share/atf/libatf-sh.subr: line 761: atf_init_test_cases: command not found

Is there a way we can propagate this error up, so that we can
see it with "kyua list" or "kyua test"?

This will give folks a better idea of how to fix the problem in their
testcase.

--

Craig

[Julio Merino]

On Sun, Sep 28, 2014 at 12:19 AM, Craig Rodrigues <rod...@freebsd.org> wrote:
> /usr/local/share/atf/libatf-sh.subr: line 761: atf_init_test_cases: command
> not found
>
>
> Is there a way we can propagate this error up, so that we can
> see it with "kyua list" or "kyua test"?

Yes and no.

Propagating this error alone is insufficient because the test program
can fail to initialize for many more reasons. Kyua should be able to
display _any_ error that happens when obtaining the test case list.

Maybe we could extend the already-existing __test_cases_list__ hack to
include the output of the test program, so you'd inspect the output
just like with any other test case. It's not easy though, and I'm not
really excited about extending the hack further.

I think my current work on improving the test case execution code will
fix this in a better way, but I'm not completely sure yet. Need to
think a bit more.

[Craig Rodrigues]

On Sun, Sep 28, 2014 at 8:30 AM, Julio Merino <jm...@meroh.net> wrote:

I think my current work on improving the test case execution code will
fix this in a better way, but I'm not completely sure yet. Need to
think a bit more.

I think we need this.  I tried putting this inside ATF:

atf_init_test_cases()
{
    echo "${Prog_Name}: ERROR: ${@}" 1>&2
    echo "${Prog_Name}: atf_init_test_cases() not defined in test case." 1>&2
    echo "${Prog_Name}: See atf-sh-api(3) for more details." 1>&2
    exit 1
}

If I ran the testcase directly, that error message displayed.  But
if I ran "kyua list", it did not.

Here are some things you might want to test as you implement better error
reporting:

  -> atf_init_test_cases() is not there

  -> test_head() or test_body() are not there for a certain testcase

  -> atf_init_test_cases(), test_head(), test_body() are there, but

      they have an 'exit' or 'return' statement where they shouldn't

I tripped over some of these things when writing an initial ATF testcase,
and it was hard to figure out what was wrong.

--

Craig

[Garrett Cooper]

On Sep 28, 2014, at 11:19, Craig Rodrigues <rod...@freebsd.org> wrote:

> On Sun, Sep 28, 2014 at 8:30 AM, Julio Merino <jm...@meroh.net> wrote:
>
> I think my current work on improving the test case execution code will
> fix this in a better way, but I'm not completely sure yet. Need to
> think a bit more.
>
>
> I think we need this. I tried putting this inside ATF:
>
> atf_init_test_cases()
> {
> echo "${Prog_Name}: ERROR: ${@}" 1>&2
> echo "${Prog_Name}: atf_init_test_cases() not defined in test case." 1>&2
> echo "${Prog_Name}: See atf-sh-api(3) for more details." 1>&2
> exit 1
> }

I think that you’re right — it should have a meaningful message.

Maybe something like…?

atf_init_test_cases()
{
echo "${Prog_Name}: ERROR: ${@}" 1>&2

echo "${Prog_Name}: atf_init_test_cases() not defined in test script (no testcases)." 1>&2

echo "${Prog_Name}: See atf-sh-api(3) for more details." 1>&2
exit 1
}

> If I ran the testcase directly, that error message displayed. But
> if I ran "kyua list", it did not.
>
> Here are some things you might want to test as you implement better error
> reporting:
> -> atf_init_test_cases() is not there
> -> test_head() or test_body() are not there for a certain test case

test_body() is important; test_head should be omitted (but having a debug message in the logs reporting it “missing” would be a good idea), if need be to have functional parity with the C/C++ APIs.

> -> atf_init_test_cases(), test_head(), test_body() are there, but
> they have an 'exit' or 'return' statement where they shouldn’t

Overriding exit/return might be

> I tripped over some of these things when writing an initial ATF testcase,
> and it was hard to figure out what was wrong.

Thanks for the input!
-Garrett

[Garrett Cooper]

Craig has a valid point. This is one area that’s bit me and others on my team a few times. It might be good to add “FATAL” asserts to ATF, which would feedback a signal when run in isolation/list mode to dump the output for the command run.

It would be nice if the C/C++ APIs had similar feedback with compilation/linker errors with missing functions/expectations as well.

Thanks!
-Garrett

[Craig Rodrigues]

On Sun, Sep 28, 2014 at 1:23 PM, Garrett Cooper <yane...@gmail.com> wrote:

Craig has a valid point. This is one area that's bit me and others on my team a few times.

Garrett, I would like to see more feedback from your team, and
use that feedback to improve kyua.

There are some rough edges to kyua/ATF in terms of usability

and docs, which make it a bit hard to introduce to a QA team (which is
the experience I am having).

I think all these problems are solvable, and we can make things better.

Luckily, Julio is very good about fixing bugs and adding features,

such as junit XML reports for Jenkins.

Can you circulate this to your team and provide feedback:

https://github.com/rodrigc/kyua/wiki/Quickstart-Guide

Having feedback from QA people who are new to BSD and not familiar with kyua

would be valuable.

--

Craig

 

[Julio Merino]

On Sun, Sep 28, 2014 at 2:19 PM, Craig Rodrigues <rod...@freebsd.org> wrote:

On Sun, Sep 28, 2014 at 8:30 AM, Julio Merino <jm...@meroh.net> wrote:

I think my current work on improving the test case execution code will
fix this in a better way, but I'm not completely sure yet. Need to
think a bit more.

I think we need this.  I tried putting this inside ATF:

atf_init_test_cases()
{
    echo "${Prog_Name}: ERROR: ${@}" 1>&2
    echo "${Prog_Name}: atf_init_test_cases() not defined in test case." 1>&2
    echo "${Prog_Name}: See atf-sh-api(3) for more details." 1>&2
    exit 1
}

If I ran the testcase directly, that error message displayed.  But
if I ran "kyua list", it did not.

Right. That's because, due to the way Kyua internally handles the loading of test cases, it has no possibility of capturing and displaying this error early enough. This is a shortcoming indeed.

(Spent my 8 hour flight yesterday cleaning up parts of this, but there is a very significant amount of work left. All the restructuring I'm doing in this area is also a prerequisite for implementing parallel execution of tests, which is the "big thing" I wanna get done at the moment.)

Here are some things you might want to test as you implement better error
reporting:

  -> atf_init_test_cases() is not there

  -> test_head() or test_body() are not there for a certain testcase

  -> atf_init_test_cases(), test_head(), test_body() are there, but

      they have an 'exit' or 'return' statement where they shouldn't

I tripped over some of these things when writing an initial ATF testcase,
and it was hard to figure out what was wrong.

These are all good points. Please file a bug under the ATF tracker so the specifics don't get lost.

--

Julio Merino | BCD / PD SRE | jm...@google.com | +1 (347) 694-0576