What tools, libraries do you use for performance testing?

[Mohan R]

I understand that robot framework is not well suited for load/stress/scalability testing. I understand that there are dedicated tools like JMeter, Mechanize for that.

My requirements are different. It is performance regression testing. Following is my rough workflow.

Run a set of tests for each build.

These tests have to be run n times sequentially (where n is between 10 to 20) to minimize noise in collected data.

Outliers have to be analysed and eliminated and averages/means/SDs determined from the n runs. 

The collected data has to be compared with the past runs to determine pass/fail status (depending upon whether the deviations are within predetermined cutoffs).

Trend analysis has to be performed with past n builds to see how the performance varies over time.

I can see having RF run the tests in the above workflow. But once the tests have been run what libraries/tools do I use to process the data. I see that there are some python libraries to process this kind of performance data. But none seems mature and having a large following. What do you use to accomplish a similar workflow? Any suggestions, pointers are welcome. Thanks,

[David]

First of all, you need to clarify performance regression testing. What
kind of performance? Functional performance? System performance?
System reliability/stability? Or said another way, performance of
what? Such as functionality, CPU usage, memory usage, response times,
network bandwidth, system usability?

If it's functionality and usability, then RF might be of use. And the
methodology in my mind would be to run RF tests in loop X times, then
run custom report analyzer and check how often tests pass (individual
tests or the whole suite). And so the individual tests or whole suite,
etc. has to have a certain pass threshold (e.g. some % or amount of
tests must pass on average of X runs), to meet regression pass
criteria. You could have RF output tests in XML, then run an XML
parser to gather the x XML reports and analyze them all.

If it's system CPU/memory usage and stability, then RF might not be
the tool of choice. You monitor the system with monitoring tools
during test run and run test w/ tools like JMeter with simple checks
like verify HTTP 200 ok (for web requests) against requests to
commonly used web pages or web service APIs, etc. And access pass rate
from that as well as CPU/memory performance from the monitoring tools.

[David]

Also, if you're testing response time performance regression, then you
could run RF tests in N test runs, and collect logs assessing the
runtime of test suite, or individual tests or keyword commands and
compare whether they vary between runs or from past test results (e.g.
gotten slower). I think RF logs test runtime so you can assess that.
Otherwise, you could log it yourself with custom keywords in tests.

You most likely will need a custom report analyzer, don't think any
existing one may meet your needs right off the bat.

[Mohan Raj Rajamanickam]

Thanks for your answers David.

I need to measure a bunch of metrics: application memory, execution time of functions, total execution time of a test .. etc for the application under test. 

These metrics will be measured for n times for a build.

I am just wondering about python libraries that will help me process the collected test data and determine pass/fail status (e.g. something similar to or built upon numpy, pandas etc) that will help me determine outliers, calculate SDs, and do trend analysis etc.

This seems like a common workflow in performance testing and it is hard for me to believe that there aren't common tools / libraries out there that helps to implement this workflow.

Thanks,

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at http://groups.google.com/group/robotframework-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mohan Raj

[David]

I too would like to know of available tools for doing what you want. I
don't work with Python and reporting much, so can't help you there.

I'm thinking some of the Python math & scientific libraries might be
useful in this case. Are you also restricting to Python only? Because
wondering if there are such tools for other language platforms.

I'm thinking what you ask for might be available to some extent in
commercial test tools, but don't know about the open source space. It
might also be available to some extent but targeted or integrated w/
specific tools like JMeter, etc. so you'd have to retrofit the library
code to work w/ RF output for example. Same with math/science Python
libraries, probably not meant to handle software test results, so need
to be adapted for your use.

On Jan 30, 12:14 pm, Mohan Raj Rajamanickam <mohanraj....@gmail.com>
wrote: