exclude test with dynamically set tag

[Vítězslav Ferko]

I am setting a tag in a Suite Setup and turns out, you can't --exclude those tags.

What I am trying to achieve in fact is to exclude whole family of test suites (whole folder).

Is that possible?

[Bryan Oakley]

Unfortunately there's nothing you can do. The core team has said more than once that they won't implement a feature that lets you skip tests at runtime. 

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at http://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Pekka Klärck]

2014-10-02 18:25 GMT+03:00 Vítězslav Ferko <nim...@gmail.com>:
> I am setting a tag in a Suite Setup and turns out, you can't --exclude those
> tags.

Yes, this is true. All such test case selection is done before test
execution is started. I can see that excluding tests based on tags
dynamically after suite setup would sometimes be useful. If someone is
interested to take a look at that I can help.

> What I am trying to achieve in fact is to exclude whole family of test
> suites (whole folder).

The easiest solution is just failing the suite setup and letting Robot
to automatically mark all tests in the suite failed without actually
running them. You probably would also want to change test tags so that
tests are considered non-critical. Alternatively you could add some
special tag and post-process your output.xml with Rebot so that you
exclude tests with that tag. The end result ought to be same as if
those tests were excluded dynamically already during execution.

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Pekka Klärck]

2014-10-02 21:10 GMT+03:00 Bryan Oakley <oak...@bardo.clearlight.com>:
> Unfortunately there's nothing you can do. The core team has said more than
> once that they won't implement a feature that lets you skip tests at
> runtime.

Both of these claims are somewhat wrong. In my previous reply I
already explained two ways to accomplish this, and we wouldn't be
against dynamically excluding those tests from execution either.

We have said no to a feature request to add separate SKIP state in
addition to PASS and FAIL, though. The main reason originally was that
there generally is no consensus what other states should possibly be
added (PENDING, POSTPONED, INVALID, ...). It seems PASS/FAIL/SKIP are
pretty commonly used these days by other tools, so this argument isn't
so strong anymore. Another reason why I don't think SKIP is needed is
that with PASS/FAIL and criticality you can already now emulate that
pretty well. I agree explicit SKIP would sometimes be more clear, but
having two related concepts would also be somewhat confusing. The
final reason is that adding a new state would be huge task.

If someone is anyway interested to see new SKIP state (or any other
new state), it's easy to fork the code on GitHub and start hacking. If
the functionality works well, doesn't cause backwards-incompatibility
problems, and is generally found useful, I wouldn't have anything
against merging it in later.

[Bryan Oakley]

My apologies if I misrepresented the situation.  Thanks for the clarification.

[Vítězslav Ferko]

Well, that is interesting, thank you!

I didn't want to exclude them at runtime, though the tag is only set at runtime, but to give all tests within a folder a certain tag, which I am then using with the -e parameter.
Is there then any other way to exclude whole folders from testing? Could possibly the tests within a folder automatically obtain a tag with it's folder name? They already are part of some hierarchy logic, members of one family of tests, why not work with this.

from this example, the test_to_run would have automatic tags `ex  ample`, the other two would both have tags `ex  clude`.
I could then very easily set the folder to run or to not run by a folder name and -i or -e parameters.


Also, I noticed, that folders named with dot as first character are ignored. Our Jenkins is downloading a fresh copy from repo each time a job is run, so if it is not possible using the robot framework logic or pybot arguments, renaming the folders should somewhat work, right?

Dne čtvrtek, 2. října 2014 21:37:42 UTC+2 Pekka Klärck napsal(a):

2014-10-02 21:10 GMT+03:00 Bryan Oakley <oak...@bardo.clearlight.com>:
> Unfortunately there's nothing you can do. The core team has said more than
> once that they won't implement a feature that lets you skip tests at
> runtime.

Both of these claims are somewhat wrong. In my previous reply I
already explained two ways to accomplish this, and we wouldn't be
against dynamically excluding those tests from execution either.

We have said no to a feature request to add separate SKIP state in

addition to PASS andFAIL, though. The main reason originally was that

[Pekka Klärck]

2014-10-03 15:49 GMT+03:00 Vítězslav Ferko <nim...@gmail.com>:
>
> I didn't want to exclude them at runtime, though the tag is only set at runtime, but to give all tests within a folder a certain tag, which I am then using with the -e parameter.

This you can do using Force Tags setting in suite initialization files:
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#initialization-files

Alternatively you could use Set Tags in suite setups, but those tags
cannot be used with --exclude.

> Is there then any other way to exclude whole folders from testing? Could possibly the tests within a folder automatically obtain a tag with it's folder name? They already are part of some hierarchy logic, members of one family of tests, why not work with this.

I don't think adding tags automatically is a good idea. I'd rather see
option like --nosuite that excludes a suite by its name. For
consistency reason there should then also be --notest option.