[Continue on failure] Making a regular ":FOR" loop behave as a Test Template

[bocadillodeatun]

Hi.

Let's say I want to test the behavior of 4 webservers.
Let's first try to login with an incorrect password.

One option would be to use a Test Template:


*** Test Cases ***
(Each webserver) Try invalid login password
[Template] Try invalid login password template
: FOR ${webserver} IN @{WEBSERVERS}
\ ${webserver}

*** Keywords ***
Try invalid login password template
[Arguments] ${modem}
Selenium.Open Browser http://${webserver["ip"]}
Selenium.Input Text AUTH_FORM ${BAD_PASSWORD}
Selenium.Submit Form
Selenium.Wait Until Page Contains Element AUTH_FORM



The idea is that when you enter the BAD_PASSWORD,
the original login password page (containing the
"AUTH_FORM" form) will be re-loaded.
I.e. when you enter a bad password, your end up in the
same web page.

This works, and thanks to using the Test Template, even
if one server fails the test, robot will keep testing the rest
of them so that, at the end, we have test results of all the
servers under test.

Well... now I want to do THIS SAME THING without
using Test Templates (the reason of this will be explained
later).

I first thought about using the "Run Keyword and Continue
on Failure" keyword, like this:


*** Test Cases ***
(Each webserver) Try invalid login password
: FOR ${webserver} IN @{WEBSERVERS}
\ Run Keyword and Continue on Failure Try invalid login
password ${webserver}

*** Keywords ***
Try invalid login password
[Arguments] ${modem}
Selenium.Open Browser http://${webserver["ip"]}
Selenium.Input Text AUTH_FORM ${BAD_PASSWORD}
Selenium.Submit Form
Selenium.Wait Until Page Contains Element AUTH_FORM



However, if one of the server fails, the others are not
tested.

In order to fix this, I had to change the "Run Keyword
and Continue on Failure" location, and place it before
the call to the last Selenium keyword, like this:


*** Test Cases ***
(Each webserver) Try invalid login password
: FOR ${webserver} IN @{WEBSERVERS}
\ Try invalid login password ${webserver}

*** Keywords ***
Try invalid login password
[Arguments] ${modem}
Selenium.Open Browser http://${webserver["ip"]}
Selenium.Input Text AUTH_FORM ${BAD_PASSWORD}
Selenium.Submit Form
Run Keyword and Continue on Failure Selenium.Wait Until Page
Contains Element AUTH_FORM


So... it looks like the "Run Keyword and Continue on
Failure" keyword can only "capture" exceptions from
direct python keywords, and not from user defined
keywords that make use of other python keywords
(ie. one or more levels of indirection).

This turns out to be a problem because:
1. I would have to place the "Run Keyword and
Continue on Failure" keyword in front of all
Selenium keywords (just in case some of the
others fail).
2. Because of 1), the test might enter inconsistent
state.

That's because, what I really want is to stop the
keywords execution when ANY of them fails, but
resume the test execution in the next :FOR loop
iteration (like in the Test Template case!)

So... in short, I want a way to "emulate" Test Templates
with a ":FOR" loop :)

Now... Why don't I want to use Test Templates?

Well, continuing with the webservers example, now
imagine that for each of them I want to test SEVERAL
incorrect password (and not just one).

I would need a "Test Template" of "Test Templates",
but this is not allowed in robot.

Instead, If the "Run Keyword and Continue on Failure"
keyword was able to "capture" all sub-exceptions,
I could just use that.

That way, all webservers would be tested agains all
invalid password, and tests would not be interrupted
when one of these tests fails on any of the webservers.

Does that make any sense? :)


PS: By the way, If I use "Run Keyword and Ignore
Error", it works! (but the final result shows "PASSED"
instead of "FAIL", which is not what I want).
Any idea of why this happens?


Thanks !

[bocadillodeatun]

Ok... I did some more testing and now I think it is something
related to the ":FOR" keyword itself.

If I don't make use of ":FOR", the execution is properly
continued:



*** Test Cases ***
Test case 1 that works
Run Keyword And Continue On Failure Try invalid login passwords $
{[WEBSERVERS[0]}
Run Keyword And Continue On Failure Try invalid login passwords $
{WEBSERVERS[1]}

Test case 2 that does NOT work

: FOR ${webserver} IN @{WEBSERVERS}

\ Run Keyword And Continue On Failure Try invalid login
passwords ${webserver}



Test case 1 runs tests in both webservers, even if the test in the
first webserver fails.

Test case 2 won't run tests on the second webserver if tests in the
first one go wrong.

Is this the intended behaviour (doesn't seem "logical" to me, but who
knows :)
Can this be fixed?

Thanks!

[bocadillodeatun]

I just found that this same problem was reported in issue 290 (http://
code.google.com/p/robotframework/issues/detail?id=290)

However, back then in 2009, this was marked as a duplicate of issue
137 (that's the issue where the "Run Keyword And Continue On Failure"
keyword was introduced).

Do you really think that's a duplicate? Maybe the one reporting the
issue was referring to the same problem I'm facing right now.

[Pekka Klärck]

2011/7/7 bocadillodeatun <bocadil...@gmail.com>:

> Ok... I did some more testing and now I think it is something
> related to the ":FOR" keyword itself.

This is actually a very stubble bug that occurs if you use Run Keyword
And Continue On Failure with a user keyword inside a FOR loop. Thanks
to your example it was easy to reproduce but figuring out the root
cause took some time. Now I understand why the bug occurs and also
know how to fix it. I submitted an issue about this [1] and will fix
it before next 2.6 pre-release.

By the way, the new keyword teardown functionality in 2.6 [2] might be
pretty useful for you too.

[1] http://code.google.com/p/robotframework/issues/detail?id=900
[2] http://code.google.com/p/robotframework/wiki/ReleaseNotes26#Keyword_teardown_functionality

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[bocadillodeatun]

Thanks!

I just tested your fix and now the ":FOR" loop works
as expected.

:)