test case dependency processing in robot framework
Ulrich
lacking the knowledge how to do it, please bare with me):
Environment:
Using robot framework for products regression testing, various
products consists of several components, often a database is involved,
preparation of a test environment for a test case can take long, one
test case often depends on another, possibly long running test case
(e.g. think of executing a restore depends on having done a backup).
Motivation:
There are several levels in RF, the business level, the test suite
level, the test case level, the keyword level and the library/fixture
level. Here I’d like to concentrate on the first three levels (as I
don't want to implement the dependency processing in a library because
I think this is so important that it should be built into the robot
framework). On test suite and test case level it is possible to define
Setup and Teardown allowing to actually executing a keyword in order
to prepare an environment the test suite or test case is depending on.
It is not possible to specify test cases as being prerequisite of a
specific test case.
Using tags it is possible to express a business level dependency by
defining a tag standing for a business level processing, e.g. a use
case. All test cases required for testing a specific use case can be
tagged with a specific tag name. The knowledge about dependencies
would be represented by the different tags. Many tags are required and
the knowledge is spread across the whole regression test bed (imagine
a complex robot framework directory hierarchy). Some of those tagged
test cases would be valid test cases on their own (e.g. installation
of a product within a product regression test suite or create a new
product specific database) but at the same time a test case TC1 could
be a 'setup test case' for a subsequently executed test case TC2, when
test case TC1 depends on TC2.
Often regression test runs include a sub-set of the defined test cases
only, as there are time constraints and changes made to the software
under test might have affected just a few areas. If, for example, test
case TC2 is required, then the test case TC1 must be executed as well,
if TC2 depends on TC1. So, when using tags as a means for defining
dependencies, for each test case sub-set one has to define a tag sub-
set that needs to be set in order to successfully execute the test
case sub-set.
But there are just too many sub-sets in a complex regression test to
consider this an efficient option. Easier would be the approach to
define dependencies on a local basis, e.g. on test suite level or test
case level: 'Before test case TC2 can be executed, the test cases TC1,
TCx, TCy,… must have been executed'.
Idea:
In order to get pre- and post-conditions on business level (i.e.
defining test case level dependencies) instead of keyword level, could
specifying tag lists as Setup/Teardown be a solution?
A tag defined as pre-condition would mean that all test cases with
that tag must have been executed beforehand at some point in the past
and if that were not the case those test cases would be executed
before the test case in question would be started. Those test cases
could be called 'Setup Test Cases' regarding the test case in
question. These would not necessarily be executed exactly when the
test case in question starts, like the Setup keyword, but possibly
earlier.
A tag defined as post-condition would mean all test cases with that
tag must be executed eventually, if not in the normal test run phase
then as part of the test run tear down. Those test cases could be
called 'Teardown Test Cases' from viewpoint of the test case in
question. They would not necessarily be executed immediately at the
end of a test case like the Teardown keywords but normally at the end
of the test run.
Details:
Regression tests are normally defined on business level, i.e. testing
of a specific use case is covered by a set of test cases. Often there
are dependencies between use cases, e.g. data records must first be
added before they can be used.
In RF the Setup and Teardown configuration allows for one keyword with
arguments, that is why I'm saying the pre- and post-conditions are on
keyword level, not on test case level. Agreed, it is possible to
define a test suite Setup. But a regression test suite is often
related to one component being tested, not related to one set of test
data processed/manipulated by different components. Also, a test suite
would have to create test data for all test cases of that suite but
preparing test data not needed is a waste of time (in case of
different sub-sets of test cases being executed). Similarly, no-one
would do the installation of a product for each test case. Still,
different variants of a product installation are necessary test cases
of a product regression test.
In a test scenario where Setup creates data files and the creation of
those takes a long time, it is not possible to create data files for
each test case and destroy the data files after each test case.
Therefore the test data had to be created once per regression test
run, not once per test case. Still each test case needs to have the
Setup keyword to be able to run a regression test with one single test
case to ensure the specific test data needed by that test case
exists. Preventing duplicate data creation could be implemented by
having the Setup keyword set a global variable indicating that
specific test data has already been created (could be implemented by a
class with scope GLOBAL). If the Setup keyword is executed more than
once it would check the variable and avoid duplicate work. But this
makes the Setup keyword non-local (same data can be required by test
cases residing in different test suites) and complex. And there
doesn’t seem to be a key word 'Execute Keyword If Tag Exists', which
would help defining Setup and Teardown user keywords that do their
processing depending on tags.
Preparing the test data (or installing different components of a
product) could require many steps and some or all of these steps are
likely be part of other, already existing test cases, anyway. It would
be much easier if one could define a list of test cases as pre-
condition. Only if that list of prerequisite test cases executed
successfully, the test case in question would be executed. It is
clearly better to use a list of tags instead of a list of test cases
(how could these be specified when there is no unique test case id?).
Disadvantage:
There is an additional step required when using tags as pre- and post-
conditions, though. At the beginning of a test run the pre- and post-
conditions of all test cases had to be evaluated and all tags
mentioned as pre- and post-conditions would have to be treated as
global forced tags before the actual test case execution could begin.
The advantages:
- No duplication of setup/teardown steps in Setup/Teardown keywords as
well as in test cases
- Allowing definition of test case dependencies locally to a test
case, not hidden in a few huge Setup/Teardown keywords.
- Automated execution of all required 'setup test cases' for a
specific test case sub-set (e.g. defined by --include and --exclude
options).
Questions:
A. Is it completely off-track to think of tag lists for pre- and post-
conditions (besides the currently supported one keyword per test case/
test suite) in the future?
B. Is it possible to add key words 'Execute Keyword If Tags Exist/Not
Exist' (arguments: list of tags, keyword, and arguments for keyword)?
C. Would it be a huge effort to allow a list of Setup/Teardown
keywords instead of only one per test suite/test case?
After having this written up I could imagine that items B and C are
easier to implement and the above mentioned disadvantage of item A
would not exist (which is the two phase parsing to determine the tag
list, not only from --include and --exclude options but from the tags
defined in pre- and post-conditions as well). Still, without A, each
Setup step had to check if it had already been executed avoiding
multiple execution. Also, without A being implemented a new keyword
'Set Tag Globally' would be required as the currently existing ‘Set
Tag' only influences the current test case/current test suite. Without
the initial determination of depending test cases, any test case TC3
mentioned as a 'tear down test case' for some other test case T2 had
to follow after test case T2 in the regression test definition.
Using the new keyword 'Execute Keyword If Tags Exist' each setup step
could be filtered depending on tags and then the requirement/RFE
“using tag lists as pre- and post-conditions” could be removed.
If it is not feasible to implement C, the benefit of defining the
Setup/Teardown steps locally to a test case would no longer exist.
Still, if implementation of B existed it would be possible to have
Setup/Teardown steps executed in a global Setup/Teardown keyword
depending on tags allowing the definition of pre- and post conditions
of test cases on business level.
Quite possible that using tags as pre- and post-conditions for
expressing dependencies on business level is not a good idea and a
complete new feature (with new table entries) is required.
Any thoughts, comments, objections, better ideas?
Thanks,
Ulrich
Pekka Klärck
>
> Questions:
>
> A. Is it completely off-track to think of tag lists for pre- and post-
> conditions (besides the currently supported one keyword per test case/
> test suite) in the future?
This would be a huge change for how RF handles test execution and thus
not likely to happen. The current logic is pretty simple -- test cases
are executed in the order they are created inside files and suites in
alphabetical order -- and since it works in most cases very well we
won't change it. Having an alternative way could be possible, but this
would be a huge task requiring changes to most parts of the framework.
I'm not so sure would such a system be that useful even if
implementing it would be more simple, though. Having setup/teardown in
multiple levels is, IMFO, much more simple and nearly as flexible.
Personally I prefer simpler solutions because they are easier to
understand and maintain.
Notice also that you can already now run certain tests first just by
executing tests in multiple rounds. If you want a single report
afterwards you can combine results with rebot.
> B. Is it possible to add key words 'Execute Keyword If Tags Exist/Not
> Exist' (arguments: list of tags, keyword, and arguments for keyword)?
This is already possible using Run Keyword If/Unless and built-in
variable @{TEST_TAGS} (or ${TEST_TAGS in scalar context):
| Run Keyword If | 'spam' in @{TEST_TAGS} | Log | This test has tag spam |
> C. Would it be a huge effort to allow a list of Setup/Teardown
> keywords instead of only one per test suite/test case?
This is unlikely to happen because it is so easy to create a higher
level keyword that contains keywords you want to execute. The idea is
that setup/teardown is one action, but that action can have
sub-actions.
> After having this written up I could imagine that items B and C are
> easier to implement and the above mentioned disadvantage of item A
> would not exist (which is the two phase parsing to determine the tag
> list, not only from --include and --exclude options but from the tags
> defined in pre- and post-conditions as well). Still, without A, each
> Setup step had to check if it had already been executed avoiding
> multiple execution.
Checking has a keyword been already executed is really easy to do
inside a library and not too complicated with higher level keywords
either.
> Also, without A being implemented a new keyword
> 'Set Tag Globally' would be required as the currently existing 'Set
> Tag' only influences the current test case/current test suite.
Why not to use variables for this instead of tags? Variables already
have scopes and there are built-in keywords to handle them. Inside a
library there's no need for even that since libraries can have state,
and thus they can easily determine how many times a keyword is called.
Cheers,
.peke