Test distribution and parallelisation

[Mikko Korpela]

Hello,

The problem of Robot Framework (RF) test execution distribution to multiple RF runs seems to come up in many places.

We have been discussing with the RF core team that it would be great to have a generic solution or at least a cookbook style guide for distributing test execution.

At this point I would be interested in hearing about existing solutions - How you have distributed your test execution and what kind of challenges did you encounter?

There are several things to consider and I'm more or less sure that there isn't an optimal solution that would suite all the situations.

Here are some notes about the problem (comments are welcome):

Situations that should be considered and most likely have there own specific problems:

1. multiple machines and all executions having there own system under test

   - how to communicate with other machines? .. many solutions exists

2. multiple machines and only one shared system under test

   - how to control that tests that can affect each other are not executed at the same time? -- or how to modify these tests so that they won't affect each other..

3. one machine but tests are executed in multiple pybot processes and all executions having there own system under test

   - where are the limits of the machine and how to control that parallelisation doesn't make things go even slower?

4. one machine but tests are executed in multiple pybot processes and only one shared system under test

   ..

The basic process:

1. Test execution is started from a CI-server

2. Tests/Suites are distributed between worker nodes (this needs some kind of a load balancer so that distribution is done evenly)

3. Worker nodes run the tests

4. Outputs from worker nodes are gathered to common place

5. rebot generates logs and reports from the outputs and they are published

6. CI execution passes / fails based on the result of this execution

[mummana subramanya]

Hi,Mikko
Hope you are in good spirit,
Well regarding to the above post ,I have a raw solution/appraoch which doesnt need any third party tools like CI or Jenkins for execution distribution and parallelization of multiple RF .The approach simply uses Selenium grid and python language which is much easier to execute multiple testsuites in multiple machines and multiple browsers at a time .
So, if you agree i'll be more happy to share the approach with you privately.
Hope you'll like it.Thank You

my skype_id:mummana vasu

[Tatu Aalto]

Ugh

And I have 5 solution.
5) I have runner/server machine and buch of server where SUT is installed. I use the runner machine to control which sut machines are used. In short, it is simple version of selenium grid. To communicate I use, ssh (library), selenium grid, or remote server (it depends on on the needs). But this controlling and maintaining the hole setup quite hard and maintenance cost is high.

-Tatu

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at http://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Sarah McK]

Mikko -- Thank you for all your hard work, we're having  a very good run with RF and are happy to be at the point where we have too many tests and too long run time

Curious -- Have you ever looked at Test Load Balancer?  We're looking at it for use with CI server parallel execution...to automatically create groups of tests to run in parallel based on historic runtimes -- to get to the fastest possible test run length overall by balancing the tests across jobs of similar length.

http://test-load-balancer.github.io/

-s

[OliverNZ]

Hi,

Currently building such a system myself. Using RF with Selenium and basically trying to scale Selenium and not RF. 

Setup:

1) pybot running on Ubuntu server

2) Server calling Selenium Grid on another machine, which then farms the test execution out.

3) Parallelization doen on a shell script level controlled by plain text config files, which control, dependencies between scripts and what can run in parallel and what not.

4) This is all based on the idea of parallelization of tests against one SUT. Against several SUT would eliminate the need of dependencies (i.e. higher parallelization) but add set-up and tear down tasks.

5) All results are filed away under /results/yyyy/mm/dd/hh-mm-ss_TestName/Test1 to ....TestNNN

6) Still need to tinker with how output.xml will look like.

7) CI integration will come.

8) Web interface to manage, control and monitor tests.

The thing that is really difficult (especially if you want to control from within RF on a granular level) is the locking of limited resources. Say email or DB table locks, waiting for data to be created or users to have become available. This is difficult and unstable at best and is the bit that needs to be thread overarching. The question is, if testing really needs that precise level of parallelization. In my case I am happy if I can run large parts of my regression independant over several instances but I don't care that there is wastage on a granular level. So the 1st question really is how far do you want/need to go down the rabbit hole.

There is no way btw to automagically determine what can run in parallel and what can't as it is 99% SUT dependant and needs to be done by the "automator". 

Glad to help with input. Done this a few times already with different frameworks and felt the pain ;-)

Cheers Oliver

[Ss Raghavan]

using 

Multithreaded Priority Queue:

lets us have all the testcases to the queue and in the thread run call frame python robot request with all necessary params. 

Now based on the number of threads in the pool, jobs will be taken automatically till the queue gets empty. say for eg, you have 10 threads in a pool and 100 tests to be executed,  first time 10 threads will be initiated by picking 10 from the queue and then from there on , whichever thread reports back next will be picked from the queue, so there will always be 10 threads running parallel. this number can be configurable based on the number of thread resources we keep.

Now, little drawback of it.if any of the thread gets hung, its not getting notified, so provide a timeout to thread so that after a specific time, the thread gets killed and the program gets back the control.

Also with this we can use along with GRID to achieve remote and faster execution. Currently i am doing this way.

On Wednesday, December 11, 2013 3:05:30 PM UTC+5:30, Mikko Korpela wrote:

[Mikko Korpela]

Hi everybody,

I implemented pabot to tackle the original post.

https://github.com/mkorpela/pabot

Currently it has something like 10 000 downloads at PyPi per month so the current version is already quite used and stable one.

Next version will include locking capabilities and resource distribution between worker processes, so you can also handle more advanced testing scenarios.

Best Regards,

Mikko Korpela

[Avihay Eyal]

I'll look into pabot, but I'm also thinking the mentioned below design (tl;dr - HTConcor for parallelization, and resource management, vms as the resources, and some minimal code from my side)

First of all, my problem looks like this:

I have a Robot tests suite, which has ~500 tests. For each test I start few application, on several machines via SSH, and then the tests start to run. At the teardown of
The test, I close the applications, so the setup will be clean for the next test (I also clean it in test setup, to be sure). Each setup and teardown will take ~15 seconds. The tests
Themselves are relevantly short.

I want to parallel the execution so I can run at least 10 robots simultaneously. I also want my solution to be scalable in the following manners:

1. It'll be easy to add resources (machines)
2. The resource pool will be used in the future by more tests suites, from different kind, and from various QA engineers in the same time.
3. There'll be a fair allocation of resources - i.e. no QA engineer will hog the resource pool


I'm thinking about implementing it like this:

1. Create a lot of vms (kvm in my case).
2. Use HTCondor as the resource manager.
3. Embed the Robot command within a script, that will first get resources (vms) from the HTCondor scheduler, and then will run a subset of the test suite.
4. I'll do it in parallel of course - in each setup I'll run dozens of tests.
5. At the end, I'll return the resources to the HTCondor.
6. Run rebot on the results, so the QA engineer will have a single report.

[Neeraj Vora]

Hello Mikko,

I work with Sarah on the technical side of things and we were wondering if your framework Pabot is built to acheive what you described above on top of RobotFramework (balanced distributions between worker nodes and consolidating results into one run). We have wrapped RobotFramework with a Java Shell (including many Java libraries) and we our one execution correspond to one test suite that has several test cases.

[Mikko Korpela]

Hi Neeraj,

Pabot distributes work in balance (note that smallest unit of work currently is one test suite file) as it is using a thread pool.

A combined report and log file are generated from the results.

Did this answer your question?

[Neeraj Vora]

Thanks Mikko, yes it does, so what you are saying is that Pabot will work with one test suite and execute it using thread pool. On our side we have wrapped Robot Framework within our own custom Java shell (we execute it using java -jar command). I could not find any documentation as to how to integrate Pabot within our customization. Can you please help us out here? Is Pabot compatible with Java invocation. How do we have it integrated within our Java shell so it does this distribution of work.

[Neeraj Vora]

Mikko, please let us know if you can point us to any documentation. In particular we are looking for pointers as to how to make this work with the Java shell we have built on top of Robot Framework. 

[sara bahiddi]

Hello!
I have a question about using RF when the test concern multiple Queues.
So We need to use Robot Framework to send requests to different apps in a parallel way, the issue is that tha apps use different queues, so we need to write a test that will make sure that each response goes to the right app. Is that possible? If Yes, is there any examples that explain that in a clear way?


Thank you

[Anne kootstra]

As this topic is more than three years old and your question is unrelated to the above topic can you please post a new question. 

Thanks in advance.

Anne

Op maandag 16 oktober 2017 11:50:24 UTC+2 schreef sara bahiddi:

[Jayanthan Kuppusamy]

Hi, 

I'm having two suites and running using Pabot

I like to get the value of one robot variable from the 1st suite and pass it to the 2nd suite. 

I need to communicate between two suites.

Is it possible in Pabot

Thanks

Jayanthan

[Tatu Aalto]

Ugh

It should be possible, please see the documentation: https://cdn.rawgit.com/mkorpela/pabot/master/PabotLib.html#Set%20Parallel%20Value%20For%20Key

-Tatu
Send from my mobile

--

You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.

Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Ruud Prijs]

I use distributed testing in a multi platform(win, linux , ce) enviorment, in combination with jenkins nodes and pipeline,  if you like to have some information about my setup
pls contact me directly

[Suresh Cod4Testing]

Hello Ruud, 

Do you have any blog or a website where I can read about distributed testing in CI/CD using robot framework

[Sapna Arohi]

Hello Ruud,

Can you help me with the distributed Testing you use through Jenkins?

On Monday, April 23, 2018 at 10:39:40 PM UTC+5:30, Ruud Prijs wrote: