Excluding test cases by name (via tag?)

[Bill]

We are hoping to reuse the same test suite in multiple environments where we know up front that certain tests will always fail with certain environments (but work in others), and we want to exclude those specific tests when running the respective environments.  For example, let's say we have tests 1-20 tests and environments A-E, and the following combinations we'd like to skip:

    A (skip 19,20)

    B (run them all)

    C (skip 4, 15, 16, 17, 20)

    D (skip 11)

    E (run them all)

We launch Robot from a command line using Pybot, and initially were including test cases by name, adding them to each environments list as they were proving to be stable.  However we're now to the point where most tests are stable in most environments, so the reverse (only listing the tests to skip) is preferable and more maintainable as new tests get added to the suite.  Plus we have a short list in our configuration file (which is used to launch Pybot) which kind of self-documents the tests that don't work in that environment.

There is the ability to tag tests, and exclude them via the command line using -Exclude, however the examples we see talk more in terms of using tags to categorize tests.  Nevertheless we looked into leveraging this to create a tag for each test that contained the test name, allowing us to exclude tests by name.  For example:

    *** Test Cases ***

    First test case - we use long names

        [Tags]    ${TEST NAME}

However it seems the tag for this test case doesn't get set until the running of the test case. The following should work, but of course we'd like to avoid duplicating each test case name and introducing error there:

    *** Test Cases ***

    First test case - we use long names

        [Tags]    First test case - we use long names

Certainly a few other ideas come to mind that are less desirable: 1. Create a separate test suite file per environment, and reference (via a superset resource file) the subset that we'd like to run. 2. Pass the "environment" as a variable to the test suite, and internally have it skip certain test cases conditionally based on the environment.

Does anyone have any other creative ideas for how to do this type of thing?  I'm hoping for the ability to exclude the tests by test case name via the command line, since as I say this allows our configuration file to "self-document" the tests cases we are skipping, but I'm open to other suggests as well.  And of course, yes, the Robot script could be this "documentation", but having a simple, clean list of test cases (outside a long Robot script) is going to be more easily consumable by others within the Dev/QA.

Thanks,
Bill

[Jari Nurminen]

Hi Bill,

 

I had a similar situation and was solving it with tags like this

TCs 19 & 20 get a "NoRunA" tag  --> pybot -e NoRunA

TCs 4, 15, 16, 17,  & 20 get a "NoRunC" tag --> pybot -e NoRunC

etc.

In this case the exclusion status becomes a part of the TC i.e. is not anymore in that config file.

 

The problem that I see with the "[tags]   test name" approach is that it'll mess up the reports: the report overview will have summaries (pass/fail) per tag --> with tag=name you'd end up with an overview that is not an overview anymore...

 

If you want to keep the excluded TCs in a config file then I'd probably try to do something in the test setup i.e. instead of skipping the tests I'd check, as the 1st step, in the test setup whether the currently running test case is in the exclusion list: If it's on the list --> abort the test execution either with "Pass Execution" or "Fail" KW, both of these KW have a message argument which can be used to log a meaningful message e.g. "Skipped - doesn't work in Environment A"

 

cheerio,

- Jari - 

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Pekka Klärck]

Hello,

Variables are resolved in tags in general only when tests are started
so trying to use their values with --include/--exclude doesn't work.

It seems that having an option like --notest to exclude certain named
tests would work splendidly for you. This has been proposed earlier,
but nobody has needed it badly enough to actually implement it:
https://github.com/robotframework/robotframework/issues/1966

Cheers,
.peke

> --
> You received this message because you are subscribed to the Google Groups
> "robotframework-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to robotframework-u...@googlegroups.com.
> To post to this group, send email to robotframe...@googlegroups.com.
> Visit this group at https://groups.google.com/group/robotframework-users.
> For more options, visit https://groups.google.com/d/optout.

--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Bill]

Thanks for your reply Jari,

I hadn't payed much attention to the report's summary of the "Statistics by tag", but I see your point about how having a tag per test throws that off.  Fortunately a lot of what we get from that report is programmatically parsed out, so we don't have to see that summary much.

I think your alternative idea sounds more robust, in particular being able to give more descriptive reasons as to why a test was skipped.  However the robot script (suite setup) would need to parse the config file to determine which tests are valid for which environment, and at this point I'd prefer not to add that connection to the robot script.  So for now I plan to use the "test case name as the tag" hack for simplicity.  I did add a little protection to make sure the hard coded tag name matched the test name, so that at least at runtime there is a check to catch any copy/paste errors.  It's a little ugly, but something I can live with. :)  Here's the example:

    *** Settings ***

    Test Setup        TestCaseSetup

    *** Test Cases ***

    II.1.1 Long And Descriptive Test Case Name

        [Tags]    II.1.1 Long And Descriptive Test Case Name

        # Test case detail omitted

    *** Keyword ***

    TestCaseSetup

        ConfirmTagContainsTestName

    ConfirmTagContainsTestName

        [Documentation]    Confirm that the test case is included in the tag list. Having the test case as a tag is by design

        ...    so that we can exclude tests individually by name.

        Should Contain    ${TEST TAGS}   ${TEST NAME}

 

Thanks,

Bill

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

[Bill]

Hi Peke, thanks for your response, that makes sense with the variable names not being resolved until the tests get started.  A --notest type feature would be the quick solution given my approach, however I may need to expand/change my way of thinking about tags.  Initially I was thinking in terms of the tags being broad categories of types of test. 

For example, "User Creation Tests", "User Modification Tests", "User Deletion Tests", where each of those has many tests. Let's say the "User Creation Tests" has two tests that fail in one of the environments.  That wouldn't work well as a tag since I wouldn't want to exclude all of those tests. But if I take a different approach and have a "doesn't work in environment" type tag PER environment, then that could work.

So using my original example:

     A (skip 19,20) 
     B (run them all) 
     C (skip 4, 15, 16, 17, 20) 
     D (skip 11) 
     E (run them all) 

With these 3 tags: "Skip In Env. A", "Skip In Env. C", "Skip In Env. D", I could tag the tests as follows:

    1-3, 5-10, 12-14, 18 (no tags)

    4, 15-17 (tagged with "Skip In Env. C")

    11 (tagged with "Skip In Env. D")

    19 (tagged with "Skip In Env. A")

    20 (tagged with "Skip In Env. A" and "Skip In Env. C")

It would move the "documentation" of which tests work in which environment from a configuration file into the Robot script...which has its tradeoffs...but would allow me to use a single --exclude tag (for example when running the test in environment C, --exclude "Skip In Env. C").  This avoids having to pass around the names of the test cases as command line arguments, which admittedly feels a little messy given such long multi-word names.  

Again, like using the test case name as the tag, this may be yet another unintended use of "tags" (although no doubt flexibility in their use was a goal), but definitely work taking a look at. :)

Thanks,
Bill

> email to robotframework-users+unsub...@googlegroups.com.