how to use Robotframework to generate traffic in performance test

[Marek (hybris.com)]

Hello,
I've already seen few threads concerning use of Robotframework for
performance testing. I see that tool is recommended rather for
functional tests nevertheless maybe we can exchange selenium with some
other library or use some high performance browser and avoid cost of
introduction and integration of another tool in company.
I test with robot framework e-commerce platform for building complex
stores. All tested modules are web-applications, some of them are RIA-
apps built on ZK-framework. I'm going to use DynaTrace server to
monitor traffic and collect metrics. I would like to use Robot to
built scenarios, which will generate users' traffic.

Do you have any experience with use of robot framework for generating
multiuser access with at least 150 parallel sessions?
Have you tried to use different library than selenium for performance
tests?

I would appreciate your help.

[Daniel Aquino]

Have you seen HtmlUnit ?

http://htmlunit.sourceforge.net/

Browser mob is a good solution too...

https://browsermob.com/performance-testing

> --
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.
> To post to this group, send email to robotframe...@googlegroups.com.
> To unsubscribe from this group, send email to robotframework-u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/robotframework-users?hl=en.
>

[David]

There may be an issue of scale, on how much and how easily you can
scale RF tests, like you say to 150 concurrent sessions (and beyond).

Selenium can do that, whether with RF, or by itself, if you use
Selenium Grid, or equivalent parallel test deployment (e.g. Jenkins
slaves), and assuming you have enough resources to expand that far in
the parallel deployment. 150 sessions can take up quite a lot of
computing power, if you assume 4-5 browsers max per machine/node. You
can make up for computing power if you use Amazon EC2 or SauceLabs to
run the Selenium nodes.

For high scalability, you'd want to test with a tool/library that was
designed for load testing. But I don't know of any that has good API
along with already being bulit to integrate with RF. Some options in
that area:

* JMeter (good tool, not sure about integrate-ability with RF)
* Multi-Mechanize (Python based w/ API)
* Pylot (old version of Multi-Mechanize, less good for API/library
integration)
* there are probably others

On Jan 11, 11:06 pm, "Marek (hybris.com)" <marek.n...@gmail.com>
wrote:

[David]

> Have you seen HtmlUnit ?
>
> http://htmlunit.sourceforge.net/

This would require Selenium2Library (Web Driver) right, if using with
RF? Or will it also work with SeleniumLibrary?

[Daniel Aquino]

As far as performance testing check out the link I suggested:
https://browsermob.com/performance-testing

And for HtmlUnit yes it's a WebDriver implementation so you would have
to use it with Selenium2Library.

I would think that going forward shouldn't attention be moved towards
WebDriver as the main interface?

[David]

> I would think that going forward shouldn't attention be moved towards
> WebDriver as the main interface?

Yes, I would think so too, but I'm sure there's still a large user
base on SeleniumLibrary (and for non-RF, still on Selenium RC).

And Selenium2Library currently is not as mature as the RC version.
Some limitations at present:

* no Grid2 support
* no remote web driver support (local execution only)
* I think no Safari support (which requires WebDriver Java's
WebDriverBackedSelenium plus SeleneseExecutor classes)

So until limitations like those are resolved, one has to make an
assessment whether their testing needs are meet with the new library
or not.

In the context of performance testing in this thread, the lack of
Grid2 and remtoe web driver support may limit to some extent the
scalability of RF Selenium2Library tests as you then have to farm out
or parallelize the tests differently as a result of those limitations.

[Daniel Aquino]

Great info.. Thanks!

[Tset Noitamotua]

If asked the same question today in 2016. Which options are there now?

What libraries or tools would you suggest for load  and performance testing with Robot Framework?

Cheers

Tset

Great info.. Thanks!

>> > To unsubscribe from this group, send email to robotframework-users+unsub...@googlegroups.com.

>> > For more options, visit this group athttp://groups.google.com/group/robotframework-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.
> To post to this group, send email to robotframe...@googlegroups.com.

> To unsubscribe from this group, send email to robotframework-users+unsub...@googlegroups.com.

[David]

Not much has changed since then, but there are likely more options.

For parallel execution to help distribute the load, can use:

https://github.com/mkorpela/pabot

or RF's parallel library, this was the old link I found: http://robotframework.googlecode.com/hg/proto/parallel/doc/Parallel.html

and Selenium2Library has more features since then, I believe it support Grid, SafariDriver, and all that good stuff now.

for real load though, might want to try

https://pypi.python.org/pypi/robotframework-jmeterlibrary/1.0

https://sourceforge.net/projects/rf-jmeter-py/

but seriously though, if you go that far with Jmeter library, why not just run the performance tests standalone with Jmeter instead.

Though if the trouble for some people is in building the performance load test script, then the easy solution is to run the RF Selenium tests while having a JMeter/Gatling proxy/recorder running in the background, the RF test's web requests made via Selenium will be captured by the proxy and converted into that tool's native script format (jmeter, gatling). One can then use "as is" for load testing, or tweak it further for customization.

Also, another option though not ideal either is to convert the RF Selenium test (hitting the different pages) into making HTTP requests against those pages and other resources (CSS, JS, image files, etc.) via any one of the RF HTTP request libraries and then run that at scale with parallel execution. That's still bad design but HTTP requests scale better/higher than Selenium browsers. This approach is doing what JMeter/Gatling do, but at lower scale.

[Tatu Aalto]

Ugh

I also agree David, if I would need to performance testing, I would choose a different tool. Like jmeter or Gatling.

-Tatu

Cheers

Great info.. Thanks!

>> > To unsubscribe from this group, send email to robotframework-u...@googlegroups.com.

>> > For more options, visit this group athttp://groups.google.com/group/robotframework-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.
> To post to this group, send email to robotframe...@googlegroups.com.

> To unsubscribe from this group, send email to robotframework-u...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/robotframework-users?hl=en.
>

--

You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.

To post to this group, send email to robotframe...@googlegroups.com.

Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Wladislaw Wagner]

Thankx for your answers. I will check out suggested options. I understand that RF was not designed for load or performance testing but I like the idea of having the results of all kind of tests in one place, e.g. in the RF report/log.

Cheers
Tset

[David]

>> I like the idea of having the results of all kind of tests in one place, e.g. in the RF report/log.

It wouldn't make sense nor generally be practical to combine functional and performance/load tests under the same report unless you've orchestrated them to run together, otherwise, you would merge them together after their individual test runs with the RF reporting tools against their output files. But I do see value in having same reporting format across tools/tests, and having a master test suite results that cover functional & performance tests.

To that effect, one could always write an external test output parser/converter that converts the test format of other tools to RF format so that the RF reporting tools can then merge them together or post process them. This might be feasible for test result formats in XML, and perhaps HTML, simply XLST transform them to RF format.

But I don't think anyone wrote test result output converters for say JMeter, gatling, etc. Perhaps a worthy 3rd party project here ;-)

Cheers

Great info.. Thanks!

>> > To unsubscribe from this group, send email to robotframework-users+unsub...@googlegroups.com.

>> > For more options, visit this group athttp://groups.google.com/group/robotframework-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.
> To post to this group, send email to robotframe...@googlegroups.com.

> To unsubscribe from this group, send email to robotframework-users+unsub...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/robotframework-users?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

[Gu Ru]

A basic idea of Load or Performance test framework:

a. Load test framework has to create 'number of requests' (in jmeter language: Number of Samplers) and need to check the response of the SUT for each of those requests and produce a meaningful output e.g response time, based on the responses.

The requests (input) could be a most common protocol http, or ftp, or be whatever protocol, the api of the SUT supports, and that is applicable for performance test

b. Load test framework can be configured and creates multiple threads (or processes) of requests either parallely or serially 

Based on the idea above, one could import the external libraries of RF e.g SSHLibrary, and /or DBLibrary , and/or RequestsLibrary for http, and/or other python packages, applicable to test your SUT and develop a 'test-script'. This test-script could be managed by a Load/Performance Test Framework (and not RF)

Performance Test Framework now can create many instances of test-script (Threads / processes), monitor each of those thread / process, for completion, and produce an output e.g response-time 

Please check in python world the following Load Test Framework

- multimechanize

- locust tool using inbuilt http

- locust examples for non-http

Using Robot framework as Load Test Framework will not be an ideal solution to assimilate, plot, and infer performance of the SUT. RF is ideal for functional testing

p.s: In Jmeter, instead of writing a 'test-script', the logic is realized from the UI of Jmeter corresponding to selected sampler