Petition for a "Pass" keyword
Kai Hackemesser
I have a handful of operations where keywords get complex and I would like to do some tests only under certain conditions that are tested in the begin of the keyword. If the condition doesn't match then the keyword should just pass. Therefore I need a builtin keyword named "Pass" (or "Return") that would then ignore the rest of the keyword, just as "Fail" does for the negative case. the current solution is to create an extra keyword for all that stuff that is executed if the condition matches, but as the keyword is useful at just this one special place only, it is just cluttering the test suite tree.
Cheers,
Kai
Dietrich Schulten
Is it possible that you need an ignore keyword, not pass?
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.
> To post to this group, send email to robotframe...@googlegroups.com.
> To unsubscribe from this group, send email to robotframework-u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/robotframework-users?hl=en.
>
Markus Gaertner
in general I avoid the usage of any conditional logic in my tests and
keywords. This involves if/then/else statements together with loops. I
do this since I had very bad experiences with such logic in a test
automation effort I was involved in in 2006/2007. Your tests become
unrepeatable, and you don't know what was executed and what is going
to be executed the next time. You start wasting more and more time on
reading your test results, and interpreting them, and making decisions
based on that. If you get a binary answers - fail vs. pass - and can
rely on that, then you can make way more meaning out of your tests and
their results. Final point: I figured that I didn't know which code
branches actually were executed, and with each run of the test suite,
this could and would change.
That warning given, you may want to use "No Operation", which does
nothing and should make a test pass when used.
Best
Markus
> --
> You received this message because you are subscribed to the Google Groups
> "robotframework-users" group.
> To post to this group, send email to robotframe...@googlegroups.com.
> To unsubscribe from this group, send email to
> robotframework-u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/robotframework-users?hl=en.
>
--
Dipl.-Inform. Markus Gärtner
http://www.shino.de/blog
http://www.it-agile.de
Twitter: @mgaertne
Kai Hackemesser
Cheers,
Kai
Kai Hackemesser
We do integration tests here with this robot. It's not completely avoidable to use If and For when we parse the results of a web query for example.
"The No Operation" keyword does nothing and the keyword processes with the next line of the script. I want something that doesn't continue but returns instantanious, like "Fail", just without failing.
Cheers,
Kai
Markus Gaertner
I found different teams refer to different things with the term
integration tests. What do you mean by that?
For example, I can write a test which checks the results of the query,
and I have to just forward whatever I got from the system under test
to robot, and robot will take care to fail the test if it does not
match the expected results. Even in "integration tests" you can fail
the test if a certain condition is not met, and otherwise continue.
Usually I prefer such Guard Clauses in Unit Tests.
Maybe it would help to get a bit more context for your tests.
Best
Markus
On Mon, May 30, 2011 at 11:34, Kai Hackemesser
Sajjad Malang
Regards,
\Sajjad.
http://www.babycareindia.com
Taylor, Martin
I asked for this back in 2008. There’s an RF issue for it: http://code.google.com/p/robotframework/issues/detail?id=174
If you think it would be useful, please add your comments and vote there.
Thanks,
Martin
Kai Hackemesser
Cheers,
Kai
Pekka Klärck
>
> I have a handful of operations where keywords get complex and I would like
> to do some tests only under certain conditions that are tested in the begin
> of the keyword. If the condition doesn't match then the keyword should just
> pass.
Could you clarify this situation a bit. First of all, is this a
keyword implemented as a user keyword in the test data and not a
keyword in a library? I guess the former as in the latter case you
could just use the return statement. Second, does "the keyword should
just pass" mean that the next user keyword should start or should the
whole test end with a pass status at that point?
> Therefore I need a builtin keyword named "Pass" (or "Return") that
> would then ignore the rest of the keyword, just as "Fail" does for the
> negative case.
As Martin commented, this has been proposed already some time ago and
there's also an issue in the tracker [1]. The main reason this hasn't
been implemented is that there hasn't been any use case for this
functionality that couldn't be solved using some other (and better)
approach. Perhaps you could explain your use case a bit more
thoroughly so that we can see does it make an exception.
By guess is that most people that want this functionality would like
to use it when the test cannot be executed for some external reason
such as no suitable execution environment being available. Marking a
test passed in this case doesn't sound right for me, though, as it
gives an impression that everything is alright when in fact we don't
know. An approach I recommend for these cases, and that I've used
several times myself, is first manipulating test's tags with Set Tags
or Remove Tags keywords so that it is not considered critical and then
failing it with a clear error message. This way your test fails but
the overall execution can still pass.
[1] http://code.google.com/p/robotframework/issues/detail?id=174
Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org
Pekka Klärck
>
> in general I avoid the usage of any conditional logic in my tests and
> keywords. This involves if/then/else statements together with loops. I
> do this since I had very bad experiences with such logic in a test
> automation effort I was involved in in 2006/2007. Your tests become
> unrepeatable, and you don't know what was executed and what is going
> to be executed the next time. You start wasting more and more time on
> reading your test results, and interpreting them, and making decisions
> based on that. If you get a binary answers - fail vs. pass - and can
> rely on that, then you can make way more meaning out of your tests and
> their results. Final point: I figured that I didn't know which code
> branches actually were executed, and with each run of the test suite,
> this could and would change.
I agree with this 100%. Or more if possible. A lot of the power of
automation is lost if you need to start looking at the results
manually to see are the failures really failures and did the passed
tests really pass.
Taylor, Martin
Cheers,
Martin
-----Original Message-----
From: robotframe...@googlegroups.com [mailto:robotframe...@googlegroups.com] On Behalf Of Pekka Klärck
Sent: Monday, May 30, 2011 8:36 PM
To: mgae...@googlemail.com
Cc: robotframework-users
Subject: Re: Petition for a "Pass" keyword
--
Pekka Klärck
> I agree that any complex logic in a Keyword should be implemented in a Python or Java library, not in an RF user-defined keyword. However, in my opinion, there is a strong case for "Early positive escape" logic as could be implemented by a "PASS" keyword. Heck, RF has a FAIL KW for immediate failure (http://robotframework.googlecode.com/hg/doc/libraries/BuiltIn.html#Fail), why not a PASS KW for immediate pass?
Could you clarify on what kind of situation "early positive escape"
would be useful? As I wrote in an earlier mail, it doesn't sound right
for me that a test is marked passed if it, for example, cannot be run
due to environment issues. I prefer marking the test non-critical and
doing "early negative escape".
Bryan Oakley
> 2011/5/31 Taylor, Martin <cmta...@ti.com>:
>> I agree that any complex logic in a Keyword should be implemented in a Python or Java library, not in an RF user-defined keyword. However, in my opinion, there is a strong case for "Early positive escape" logic as could be implemented by a "PASS" keyword. Heck, RF has a FAIL KW for immediate failure (http://robotframework.googlecode.com/hg/doc/libraries/BuiltIn.html#Fail), why not a PASS KW for immediate pass?
>
> Could you clarify on what kind of situation "early positive escape"
> would be useful? As I wrote in an earlier mail, it doesn't sound right
> for me that a test is marked passed if it, for example, cannot be run
> due to environment issues. I prefer marking the test non-critical and
> doing "early negative escape".
For me, the best idea is to introduce a third state: pass, fail and
SKIP. We would probably do this a lot -- skip certain tests under
certain conditions. For example, we might have selenium tests unique
to safari that should be skipped when using firefox (that's a poor
example, but an understandable one).
The "SKIP" state solves the debate of whether a "quick exit" is a pass
or a fail. Pekka seems to think it should be a non-critical fail,
others think it should be a pass. I say, mark it as "skipped". That
way you get an indication that something happened, but it's neither a
pass nor fail.
Now, you can say "just tell robot to exclude tests with the "safari"
tag" but then you have to tell someone else how to run your tests. It
should be a matter of "pybot mytestsuite" without requiring the user
to read documentation to find the specific set of tags to include or
exclude. Especially because it may not be as clear cut as safari
versus firefox. It may be based on timezones, presence/absence of
certain types of data, hosts running the test, environment (staging,
qa, etc), and so on.
Mark Levison
Mikko Korpela
of a test can not be satisfied is that the mechanism is unintuitive:
you need to set a not critical tag (that is set from command line to
not critical) dynamically to the test and fail it immediately after
that to skip the test. I think that there are other mechanisms to
achieve similar functionality but this one is the best I know of.
Basically a test could have (at least) 4 end results:
- PASS
- FAIL
- UNKNOWN (inconclusive or skipped or not yet run)
- ERROR (failure in test library or test code -- not in the SUT)
These are the ones that I've seen in literature so they are not made up :)
As you all know Robot supports only PASS and FAIL.
I think that at least the ERROR could be valuable (and based on this
discussion also the UNKNOWN) in RF too.
2011/5/31 Mark Levison <ma...@mlevison.com>:
> --
> You received this message because you are subscribed to the Google Groups
> "robotframework-users" group.
> To post to this group, send email to robotframe...@googlegroups.com.
> To unsubscribe from this group, send email to
> robotframework-u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/robotframework-users?hl=en.
>
--
Mikko Korpela
Pekka Klärck
>
> For me, the best idea is to introduce a third state: pass, fail and
> SKIP. We would probably do this a lot -- skip certain tests under
> certain conditions. For example, we might have selenium tests unique
> to safari that should be skipped when using firefox (that's a poor
> example, but an understandable one).
I understand the idea and know that many other tools have skip state.
> The "SKIP" state solves the debate of whether a "quick exit" is a pass
> or a fail. Pekka seems to think it should be a non-critical fail,
> others think it should be a pass. I say, mark it as "skipped". That
> way you get an indication that something happened, but it's neither a
> pass nor fail.
There are, in my opinion, two problems with introducing skip state to
Robot. First of all, it wouldn't be too clear what is the difference
between a skipped test and a failed non-critical test. This isn't a
problem with tools that don't differentiate tests to critical and
non-critical (I actually don't know any other tool that does that) but
I'm worried this would cause confusion with Robot. Notice also that
you can already now get the same effect by tagging tests you skip
dynamically with a certain tag (such as `skip`) that is considered
non-critical. This is illustrated with the example below.
*** Test Cases ***
Example
Skip test if using Safari
Actual test here
Example 2
Skip test if using IE
Actual test here
*** Keywords ***
Skip tests if using Safari
Run Keyword If "${BROWSER}" == "safari" Skip Test Test is
not compatible with Safari
Skip tests if using Safari
Run Keyword If "${BROWSER}" == "ie" Skip Test Test is not
compatible with IE
Skip Test
[Arguments] ${reason}
Set Tags skip
Fail ${reason}
The second reason why adding skip state is problematic is more
practical: it would simply be a really big task. The change would
affect test execution, produces logs and reports, all tools processing
outputs, documentation, etc. It is very unlikely that this ever gets
high enough priority at NSN, for example compared with making RIDE
better, that the core development team could spent time on it. We are,
however, more that happy to accept patches if someone from the
community is willing to implement this.
Pekka Klärck
> I think the problem with the current mechanism when the preconditions
> of a test can not be satisfied is that the mechanism is unintuitive:
> you need to set a not critical tag (that is set from command line to
> not critical) dynamically to the test and fail it immediately after
> that to skip the test.
Luckily this all can be encapsulated into a generic user keyword.
> I think that there are other mechanisms to
> achieve similar functionality but this one is the best I know of.
>
> Basically a test could have (at least) 4 end results:
> - PASS
> - FAIL
> - UNKNOWN (inconclusive or skipped or not yet run)
> - ERROR (failure in test library or test code -- not in the SUT)
>
> These are the ones that I've seen in literature so they are not made up :)
I've seen all these too. I've also seen states such as not-run,
not-ready, obsolete, etc. and all have worked well in certain
contexts. The problem is that a combination of all these states
wouldn't work anywhere.
> As you all know Robot supports only PASS and FAIL.
The reason Robot has only pass and fail is that these are the only
universally agreed states. Notice, though, that because Robot supports
tagging you can very easily emulate any states that happen to work in
your exact context.
> I think that at least the ERROR could be valuable (and based on this
> discussion also the UNKNOWN) in RF too.
Early versions of Robot actually had also error state but it was
causing more confusion than anything else. In practice it is very hard
to tell is a certain problem a failure or an error, and for the end
user the only interesting thing generally is did the test pass or not.
Bryan Oakley
> 2011/5/31 Mikko Korpela <mikko....@gmail.com>:
> The reason Robot has only pass and fail is that these are the only
> universally agreed states. Notice, though, that because Robot supports
> tagging you can very easily emulate any states that happen to work in
> your exact context.
I don't think that's a true statement. There is no working around the
fact that a test must be set to either pass or fail. There is simply
no getting around it, except to exclude it from running in the first
place.
Pekka Klärck
It's obviously true that a test must be set to either pass of fail.
That's the reason I wrote that you can _emulate_ other states with
tags. A problem with adding a true support for more states than pass
and fail is that we needed to first agree on what states to use.
Mikko Korpela
This would allow a BuiltIn keyword for skipping tests (Skip Test If ...) so that the skipped tests are always marked as non-critical failures.
--
Mikko Korpela
Bryan Oakley
>
> It's obviously true that a test must be set to either pass of fail.
> That's the reason I wrote that you can _emulate_ other states with
> tags. A problem with adding a true support for more states than pass
> and fail is that we needed to first agree on what states to use.
Maybe this is something that can really set robot ahead of the pack.
Let the third state be "indeterminate", and add a field (attribute,
whatever) for the reason. Then, each team is free to use that field
how they see fit, and aren't forced into any particular type of third
state.They could do a return("indeterminate", reason="skipped") or
"...reason="unknown error" or "... reason="feature not supported on
SUT", etc. That reason could then be printed out in the report and
counted as a separate result. (eg: 120 pass 0 fail 3 indeterminate)
Though, frankly, I like "skipped" better than "indeterminate" since
that's exactly what happens -- the test is skipped before a result is
determined :-)
I'm just thinking out loud while I enjoy my morning orange juice...