How to execute the same set of Test Cases multiple times within same suite/run?

[Kyle Nolan]

I have a set of testcases that I want to execute multiple times and see a report of how many times they passed at the end, but there doesn't appear to be a way to do this in Robot.

For example it I have:

  - Action A

  - Action B

  - Action C

And I want to execute them say 25 times in that order

It seems like my only options are:

1 -  create 75 different test cases with different names 

        don't know if you can have 3 different templates and create 25 of each, and this is not very scalable if want to simply change number of times we run

2 -  turn the 3 actions into keywords and call them in order under one test case 

          would lose the ability to see the results of individual actions as test cases at the top level, and still not very scalable, and would still need to create 25 test cases or template calls

3 -  create a suite that just runs the 3 test cases and execute this multiple times via the command line and combine the results

          would run into issues with suite setup and teardown -- don't want that run before/after every test case, just once for the entire run

In general we would want to repeat the same test cases multiple times within the same suite run. We would basically like to do this repeatedly until we stop the test..or at least do a large number of iterations.

I can't believe this isn't something others haven't needed to do as well.

Is there way to create a simple for loop around the test cases in the robot file, or a setting that can be specified either within the robot file or on the command line for how many times to repeat the test cases? Or is there some other way to do what I am trying to do?

[Geist Eng]

 

Check out the BuildIn lib 

 

BuiltIn - Repeat Keyword 

[Kyle Nolan]

Thanks, but it looks like this only applies to Keywords and not Test Cases.

The issue I have is that I want to repeat the same series of Test Cases over and over again at the top level, instead of doing this all inside one testcase.

[Geist Eng]

Since it is important to repeat an action as a test case to get the failure rate then this has to be done outside of robot.  Something like a bash script to call robot in a loop:

call robot suite to run setup
for ...
    call robot suit to run tests giving output.xml a unique name e.g. output_run1.xml
call robot suite to run teardown
call rebot with list of output*.xml files to produce one report.

[Tatu Aalto]

Ugh

Running same test multiple is possible with pabot[1], please see the --argumentfile[INTEGER] [FILEPATH] option for more details.

-Tatu
Send from my mobile

[1] https://github.com/mkorpela/pabot/blob/master/README.md#command-line-options

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Ajith George]

Hi,

With pabot, it is possible to run the same test in parallel.

Sample Bash Code:

for /l %%A in (1,1,10) do copy "Test.robot"  "TestTemp%%A.robot" 

CALL timeout 5

CALL  pabot --processes 10 TestTemp*

del TestTemp*

pause

copy the above snippet to a bat file (so that it could be made executable by double clicking)

Steps:

Put Your Test Case into Test.robot. [Alter the value "10" in green color  to what is required, say 25 (for "25" keep the pabot process to 8-10 in red color)

[Kyle Nolan]

We are not trying to execute everything in parallel, just the same sequence of testcases over and over again.

One issue with the separating setup, testcase and teardown into 3 separate robot files is that we lose the environment from the the setup in the testcase.

If setup creates sessions on external test tools, then we need to try to figure out those sessions, and reestablish all this information and variable structures for every test loop.

Another possible approach: Is there a way to define multiple test templates in the same file? (This is not clear from the documentation)

For example, if I defined keywords ActionA, ActionB and ActionC as templates, is there a way to then specify which template to use when defining testcases:

*** Settings ***

Test Template    ActionA   ActionB    ActionC

*** Test Cases ***

[ActionA] ActionA1 args

[ActionB] ActionB1 args

[ActionC] ActionC1 args

[ActionA] ActionA2 args

[ActionB] ActionB2 args

[ActionC] ActionC2 args

*** Keywords ***

ActionA

   Do something

ActionB

   Do something else

ActionC

   Do yet another thing

[Pekka Klärck]

to 20. syysk. 2018 klo 19.41 Kyle Nolan (kyle...@gmail.com) kirjoitti:
>
> We are not trying to execute everything in parallel, just the same sequence of testcases over and over again.

Robot doesn't support that out-of-the-box and there are no plans to
add such orchestration features to the core either. What you can do is
creating a script that runs tests again and again, or creating a
script that generates a big batch of tests based on some external
model and then just running that in one go.

> One issue with the separating setup, testcase and teardown into 3 separate robot files is that we lose the environment from the the setup in the testcase.
> If setup creates sessions on external test tools, then we need to try to figure out those sessions, and reestablish all this information and variable structures for every test loop.

You could possibly avoid these problems by doing setup/teardown in
test suite initialization files (e.g. __init__.robot).

> Another possible approach: Is there a way to define multiple test templates in the same file? (This is not clear from the documentation)

No, there can be only one default template per test case file. Each
test can override that template if needed, though, but if a group of
tests needs a different template it's generally better to move them to
their own file.

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Tatu Aalto]

Ugh

Even pabot is meant to run test in parallel, it also allows one to repeat the same test in serial way. Define --argumetfile[INTEGER] as many times you want to test to run and --processes as one. In this way pabot will run test in serial manner.

-Tatu
Send from my mobile

[Michał Anglart]

Have you looked into Listener APIv3? With such listener you're able to e.g. dynamically create a test inside a suite when it (the suite) starts. See example in user guide: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#modifying-execution-and-results

I haven't dig into it but theoretically you could use it to define a suite with your 3 tests and then the listener could extend the suite in runtime by adding more of them - assuming that cloning of those tests is possible there.

Similarly pre run modifier RobotFrameworkUserGuide.html#pre-run-modifier may be used to modify structure of your suites before whole execution starts.