Template Test Teardown only executes after all tests are run

[Tim Alexander]

Thought I would pose this here, as I'm not sure it's intended behavior.

If I have a test with a template set (so it's data driven), it seems that the test teardown only executes after all tests are complete and executed. I thought I remembered this being different in previous releases (running 2.7.1 right now), since I have tests utilizing this functionality that are now failing. Is this intended behavior? I'm just revisiting these tests after some time not running them, so I could be mistaken about them previously passing.

I would expect, if a test teardown is set, that it would run after each test has executed (thus, after each row of a test with a template set in it). I don't know of another way to conditionally execute something if the test just passed/failed, and this is the functionality I need.

[Tim Alexander]

FYI - here is a test that demonstrates this behavior (TXT format):

*** Test Cases ***

DD

    [Template]    DD

    1

    2

    3

    [Teardown]    log    THIS IS TEARDOWN    WARN

*** Keywords ***

DD

    [Arguments]    ${arg}

    Run Keyword If    ${arg} == 2    FAIL    THIS TEST FAILS

    LOG    This test didn't fail    INFO

[Pekka Klärck]

2012/4/17 Tim Alexander <dragon...@gmail.com>:

Robot Framework considers a test case with test template a single test
case even if it would contain multiple steps. Thus the teardown is
executed only after the whole test case is executed. This is intended
and in my opinion very useful behavior, and has worked the same way
since test template functionality was introduced in RF 2.5.

There is, however, separate keyword teardown functionality that was
introduced in RF 2.6 and works great in this case. I have modified the
example you sent in a separate mail to use both test and keyword
teardowns:

*** Test Cases ***
DD
[Template] DD
1
2
3

[Teardown] log THIS IS TEST TEARDOWN WARN

*** Keywords ***
DD
[Arguments] ${arg}
Run Keyword If ${arg} == 2 FAIL THIS TEST FAILS
LOG This test didn't fail INFO

[Teardown] log THIS IS KEYWORD TEARDOWN WARN

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[sybblow]

It seems that template keywords have "teardown" setting only but no "setup" setting.

Let's see how the issue is.

I have a test file(tsv format) with a keyword table as bellow:

Template Keyword	[Arguments]	${path}
	[Setup]	Log	enter case
	Directory Should Exist	${path}
	[TearDown]	Log	leave case

When I execute the test file, there is an error:

[ ERROR ] Error in file '/xxx/robot.tsv': Invalid syntax in keyword 'Template Keyword': Non-existing setting 'Setup'.

It's nonesense that only partial functionality is supported. I would expect "setup" setting to be supported either.

[Pekka Klärck]

2014-10-23 14:17 GMT+03:00 sybblow <silia...@gmail.com>:
>
> It seems that template keywords have "teardown" setting only but no "setup" setting.
>

[snip]

>
> It's nonesense that only partial functionality is supported. I would expect "setup" setting to be supported either.

User keywords in general can only have a teardown but no setup. The
reason they cannot have a setup is that it would provide absolutely no
benefit compared to just using the same keyword as the first normal
keyword. This is different with tests where you can use `Test Setup`
in the setting table to define a common setup to all tests, and
`[Setup]` with individual tests to override that common setup.

It can be argued that user keywords should support setups just for
consistency reasons, and I'm not totally against that. If someone
feels strongly about this issue, I'm glad to review pull request
implementing keyword setups.

[Kevin O.]

We have a resource file with lots of keywords that require the same setup and teardown. To avoid boilerplate, I created a listener library that invokes keywords before and after all keywords in the resource file.

I thought it was pretty cool that this was even possible. Essentially the listener decorates all keywords in a resource file.

Not sure if I would recommend it due to complexity, but it demonstrates the power and flexibility of the framework.

A library that can act as a listener is a great new feature.

Kevin

[Pekka Klärck]

2014-10-23 18:19 GMT+03:00 Kevin O. <korm...@gmail.com>:
> We have a resource file with lots of keywords that require the same setup
> and teardown.

Is this a common need? If it is, having a way to specify common
setup/teardown for all keywords in a resource file would be useful.
Then having separate [Setup] in keywords themselves would also make
sense.

[Kevin O.]

I do not think its worth it, especially since syntax changes would affect tools like RIDE.

The enter and exit keywords called by the listener is Switch Soap Client in SudsLibrary.

In tests that use more than one web service, its nice to have the user keywords that interact with the web service automatically switch the context and restore it. The listener also creates the SOAP client on the first keyword that needs it.

The same principle would also apply to SSHLibrary and Selenium2Library which also have switch keywords.

We prefer user keywords over Python due to the transparency in RIDE and the test logs as well as the ease of creation.

Writing custom libraries that hide the implementation would definitely avoid issues like switching, but they have their own drawbacks.

[sybblow]

listener interface is realy powful, but I think I will not try it since its complexity in my case.

I also found another way to do data-driven style test, that is declaring the template in the setting table, as show in http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#data-driven-style.

In consideration of RIDE, change the syntax may be a long-term work. But I think data-driven web service tests are quite common, where we need automatically switch the context and restore. It will be nice to have simple and elegant interface for this scenario.