Organizing/initilalizing custom libraries, large number of tests

[Hitesh Patel]

Hello all:

I am new to robot-framework and have been struggling with how to organize work flow so that we are set-up to handle a large number of test cases, that could potentially be contributed by several team members.

First however, here is how I am currently setup:

- We are testing an embedded system set-up that has its own custom UI and connections to possible external h/w that control the behavior of the DUT 

- Using folder structure as proposed in http://blog.codecentric.de/en/2012/04/robot-framework-tutorial-a-complete-example/

- Test cases are all stored in one folder called "tests" with each test case is in its own .robot file

- These .robot files can have keyword based tests or, in cases where tests are complicated, calls to test case implemented in pure python (imported as library with a "run" keyword). 

- There is a setup.robot test that sets up our base python objects (as globals) that are passed to each Libray created in the .robot files

- All test cases are tagged with keywords such as "smoke", "regressions" ,etc

- Tests are executed by calling the "tests" folder with the appropriate tag

Here are some questions:

- Currently I am organizing each test case into its own .robot file to avoid a potentially very long .robot file with many test cases. What is the recommended practice to organize? Considering that each .robot file is a "test suite", is it OK to separate each test case into its own .robot file? 

- We have a library that needs to be initialized BEFORE any tests can run. This library (not a keyword library) sets up various objects for connectivity, h/w initialization, etc. The library returns various objects that ALL the various test cases will refer to while performing the tests. Thus I have a setup.py file that does this first setup and sets the various objects as global variables available to robot-framework. Then I have a setup.robot file that includes the Library call to setup.py. Each .robot file then initializes the various custom keyword libraries, etc with reference to the globals defined earlier in setup.robot (setup.py). The question is what is recommended practice here? My concern is that if we add more start-up objects to setup.py/setup.robot then EACH test case will have to be edited to pass this additional parameter. Also order of execution is important as setup.py MSUT be called first---I believe this is easily solved by adding "00_" to the file name to force robot to pick it up first---however, it will also need ALL keywords defined so it is picked up when using keywords.

- If we run a lot of tests there is potential that the report or log file can get very large. Is it possible to split output files to avoid potentially very large files that don't open easily in a browser?

Thanks,

Hitesh

[Tatu Aalto]

Ugh

Interesting questions, which doesn't have a single correct answer. I don't know what would the best solution for you, but I can tell what way has worked for me.

1) Organising test is complex question. The best solution which has worked for me is to organise test based on features. In a product, I have feature x, example user management. All the tests related to that feature are inside of user management folder, example user creation, modification and deletion are separate suites (at least files or can subfolders with files). In this way it's easy to find all the correct test for the feature and it eases the maintenance too. Also it helps me to use Robot Framework __init__ files efficiently.

2) There are many ways to overcome the problem with adding arguments to keywords. One is to use default values for keywords[1], it works nicely for many cases. But because many things are global in Robot Framework, I have been moving to towards extended variable syntax[2], where I pass single object to keywords. It has been useful for me with connect/setup type of issues, where the argument amount may change because environment or version changes.

I don't rely on the execution order of test suites. I do rely on test execution order in a file, but I would like to give up from that dependency too. When I did rely on suite execution order, it did soon become a maintenance nightmare. Example it might feel tempting to run user creation test suite as setup for user deletion. But it turned a maintenance problem, because I could not make changes to a test suite, because changes did randomly broken other suites.

3) The report and log file being too big can happen, although you may encounter other problems before you hit this problem. I think the Robot Framework has more than thousand tests and one can open the report and log files quite easily.

But if this comes a problem one can always post process the outputs[3].

-Tatu

[1] http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#default-values-with-user-keywords
[2] http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#extended-variable-syntax
[3] http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#post-processing-outputs

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Kevin O.]

If the logs and reports are getting very large, there is an argument --splitlog to split the logs. See 

http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#splitting-logs

[Hitesh Patel]

Tatu and Kevin

Many thanks for reading through this rather long post and making suggestions.

Hitesh