Report of only failed Critical tests

[Magnus Smedberg]

Hi,
Does anyone out there know any good way to get a list, or report, of
only the failed, critical, tests of a test run?
I was looking through the options for rebot, but couldn't find
anything that fitted.

Background:
I've setup so that our RF tests are run automatically on a quite
frequent basis (for each new "snapshot" build to be precise), and for
each run, I'd like to send a brief, but informative, email to me and
some of the developers.
Since we're using non-critical tags to identify known issues etc, I'm
mostly just interested in the results of the critical tests.
Ideally, no critical tests would fail for most of the test runs, and
if/when any of them fails, I'd like to highlight these to make it
clearer to everyone what it was that stopped working.

Any help would be appreciated!

//Magnus

[Pekka Klärck]

2010/9/29 Magnus Smedberg <magnus....@gmail.com>:

> Hi,
> Does anyone out there know any good way to get a list, or report, of
> only the failed, critical, tests of a test run?
> I was looking through the options for rebot, but couldn't find
> anything that fitted.

You can use `rebot` to generate separate report and log with only the
critical tests. Depending did you originally use `--critical` or
`--noncritical` option, you should be able to use either

pybot --critical regression --log full_log.html --report
full_report --output full_output.xml tests.txt
rebot --include regression --log critical_log.html --report
critical_report.html full_output.xml

or

pybot --noncritical knownissue --log full_log.html --report
full_report --output full_output.xml tests.txt
rebot --exclude knownissue --log critical_log.html --report
critical_report.html full_output.xml

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Pekka Klärck]

2010/9/29 Pekka Klärck <pe...@iki.fi>:

> 2010/9/29 Magnus Smedberg <magnus....@gmail.com>:
>> Hi,
>> Does anyone out there know any good way to get a list, or report, of
>> only the failed, critical, tests of a test run?
>> I was looking through the options for rebot, but couldn't find
>> anything that fitted.
>
> You can use `rebot` to generate separate report and log with only the
> critical tests.

Jussi pointed out that the original question said `failed critical
tests`. Creating a report containing only them isn't possible with
`rebot` alone. We had an issue about this kind support [1], but we
decided not to implement it. The reason was that it is possible to do
this with a custom tool using the internal APIs of the framework [2].
If there's interest, the issue [1] could be reopened.

[1] http://code.google.com/p/robotframework/issues/detail?id=127
[2] http://robotframework.googlecode.com/svn/tags/robotframework-2.5.4/doc/userguide/RobotFrameworkUserGuide.html#executed-test-data

[Magnus Smedberg]

Thanks for the help.
I'm still not quite where I want to be, but getting there much faster thanks to this!
I looked through the issues for something related to this, but only looked at the open issues, I think, so I missed issue 127.
I do think re-opening this is a good idea. Or at least re-issue the same idea (since RF has changed a bit since the original issue). A "PASS/FAIL" argument to rebot would be very useful in my opinion.
The problem with using the internal APIs is that the criticality of a test case is not "correct" in the output.xml as noncritical tags are not taken into account.
Hence I have to first pre-process the output.xml to remove all passed cases, and then post-process the new output with rebot, excluding the non-critical tags (or similarly but reversed).
It's in no way impossible, but seems like a bit of an ugly workaround compared to doing it in one step in the "original" rebot-processing.

BRs
Magnus

[Pekka Klärck]

2010/9/30 Magnus Smedberg <magnus....@gmail.com>:

> Thanks for the help.
> I'm still not quite where I want to be, but getting there much faster thanks
> to this!

Great.

> I looked through the issues for something related to this, but only looked
> at the open issues, I think, so I missed issue 127.
> I do think re-opening this is a good idea. Or at least re-issue the same
> idea (since RF has changed a bit since the original issue). A "PASS/FAIL"
> argument to rebot would be very useful in my opinion.
> The problem with using the internal APIs is that the criticality of a test
> case is not "correct" in the output.xml as noncritical tags are not taken
> into account.
> Hence I have to first pre-process the output.xml to remove all passed cases,
> and then post-process the new output with rebot, excluding the non-critical
> tags (or similarly but reversed).
> It's in no way impossible, but seems like a bit of an ugly workaround
> compared to doing it in one step in the "original" rebot-processing.

I reopened issue 127 and added a reference to this discussion.
Implementing support for pseudo-tags `PASS` and `FAIL` ought to be
relatively simple, and other ideas mentioned in that issue aren't
probably that useful. Are you Magnus interested in looking at how to
implement this?

[1] http://code.google.com/p/robotframework/issues/detail?id=127

[Magnus Smedberg]

Thanks!
Would probably be a lot of fun, but not sure if my python skills (and
my calender) is up to the task.
Must admit, that I got a bit confused when trying to look at how the
post-processing worked... :-/
Might give it another go in a couple of days, but don't count on it...

/Magnus

On Oct 1, 12:04 pm, Pekka Klärck <p...@iki.fi> wrote:
> 2010/9/30 Magnus Smedberg <magnus.smedb...@gmail.com>: