Dynamic test case documentation/metadata/statistics/info

[Magnus]

Hi,
Does anybody have a good suggestion for how to use dynamic information
in the test case documentation (so that it shows up in the report)?

To be more precise, what I want to do is this:
Many of our test cases in RF are old manual tests that have been
automated.
These legacy test cases are described in a separate TCM system
(TestLink).
For these cases I would like to add a link in the RF-report to the
entry in the TCM. (like http://testlink/archiveData.php?targetTestCase=<TCM_TC_nr>)

Tagging each testcase with "testlink_<TCM_TC_nr>" and then using the --
tagstatlink option is almost perfect, except that the statistics table
gets huge, something I really cannot live with. (with some 200+ test
cases)
And if I use --tagstatexclude, the links disappears (obviously).

The perfect scenario would be if I could use something similar to --
teststatlink, but that would insert the link into the Documentation
field of the report instead.

Actually typing the link into the documentation entry of each test is
of course also a possibility, but it's not very appealing.

Any more ideas?

Thankful for any help,
BR,
Magnus

[Pekka Klärck]

2010/1/27 Magnus <magnus....@gmail.com>:

> Hi,
> Does anybody have a good suggestion for how to use dynamic information
> in the test case documentation (so that it shows up in the report)?
>
> To be more precise, what I want to do is this:
> Many of our test cases in RF are old manual tests that have been
> automated.
> These legacy test cases are described in a separate TCM system
> (TestLink).
> For these cases I would like to add a link in the RF-report to the
> entry in the TCM. (like http://testlink/archiveData.php?targetTestCase=<TCM_TC_nr>)
>
> Tagging each testcase with "testlink_<TCM_TC_nr>" and then using the --
> tagstatlink option is almost perfect, except that the statistics table
> gets huge, something I really cannot live with. (with some 200+ test
> cases)
> And if I use --tagstatexclude, the links disappears (obviously).

I wonder could we somehow enhance TagStatLink functionality to make it
work better. Could links be added somewhere else too so that even when
TagStatExclude is used to filter out excess statistics they would
still be available? Could, for example, tags themselves be made links
in test cases' info table in log and report?

> The perfect scenario would be if I could use something similar to --
> teststatlink, but that would insert the link into the Documentation
> field of the report instead.
>
> Actually typing the link into the documentation entry of each test is
> of course also a possibility, but it's not very appealing.
>
> Any more ideas?

Documentation (and other metadata) can contain variables that are
resolved when test case execution starts. You might be able to use
this somehow e.g. with the built-in variable ${TEST NAME} but even
then you needed to add that doc to all the test cases.

You should be able to build a fully dynamic solution by using the
internal APIs of the framework to post process the output. For example
something like below could work - after running a that you just need
to use `rebot` to generate logs and reports. See the User Guide and
various supporting available from the projects pages for more
information and usage examples of the internal APIs.

from robot.output import TestSuite
suite = TestSuite('output.xml')
for test in suite.tests:
test.doc += '\nTest link: http://test.link/%s/' % test.name
suite.write_to_file()

We've also been thinking about making it possible to change test
status and other information via the listener interface. That would
allow setting, for example, documentation or tags dynamically during
the test execution without needing extra post-processing step. We
already have en enhancement request about this in the tracker:
http://code.google.com/p/robotframework/issues/detail?id=177

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Magnus Smedberg]

Thanks for the suggestion with post processing Pekka, I'll give that a try.
Does the test objects in suite also have a tag variable (list?) - can I do test.tag[0]? I'm planning to use the tags instead (of test.name as in your example).
Is there any pydocs on the classes available anywhere?

I also support the idea of improving TagStatLink, or even making a new separate function. TagStatLink is an excellent function as it is now, we're using it to insert links to Bugzilla tickets that are linked to test cases (tagging testcases with ticket<nr>).

BRs
Magnus

[Janne Härkönen]

On Thu, Jan 28, 2010 at 11:26 AM, Magnus Smedberg
<magnus....@gmail.com> wrote:
> Does the test objects in suite also have a tag variable (list?) - can I do
> test.tag[0]? I'm planning to use the tags instead (of test.name as in your
> example).

Yes, the test cases have attribute `tags`, which can be modified:

from robot.output import TestSuite

s = TestSuite('output.xml')
for t in s.tests:
t.tags.append('My tag')
s.write_to_file()

> Is there any pydocs on the classes available anywhere?

Unfortunately not, this is something we need to take look at, latest
with RF 2.5.

regards,
__janne

[Magnus Smedberg]

Thanks!
I'll give it a shot.
And as a sign of my appreciation, I'll promise to create a page on the users-group, as discussed before. ;-)

BRs
Magnus