rfswarm update v0.5.0-beta - Performance testing with Robot Framework

[Dave Amies]

Hi All,

Yesterday I released v0.5.0-beta. (https://github.com/damies13/rfswarm/releases/tag/v0.5.0-beta)

I have labeled this a beta release for several reasons:

- As through my own testing I grow more confident in the software, I have now successfully run several tests with 100 robots over 2 hours

- No new issues reported by others trying out rfswarm and all the issues previously reported have now been resolved

- All the features needed to make rfswarm usable for performance testing of an application are now implemented, so as far as I know there is nothing preventing it's use.

- There is a known cosmetic bug (Issue #24), This issue is what currently prevents me from progressing to a non beta release and is what I will be working on next.

New features in v0.5.0:

 - Returning results during test execution instead of at the end of each iteration

 - Command line switches to enable integration with CI/CD builds, other automation, or just temporary configuration

 - Now the summary also provides a Standard Deviation as well as the Percentile measurement. I realised this important statistic was missing, so added it with this release.

Again, thanks for the support I've received so far for this project, if you find issues please log them here https://github.com/damies13/rfswarm/issues or post to this group I am always happy to help you get going with this tool and fix any issues you find. 

Dave.

[Jirayu Kaewprateep]

How to limited the break pt?
I will not reach the stress point and hiw to create a close system?

[Dave Amies]

Hi Jirayu,

I'm not sure I understand your questions, but I'll try:

On Sunday, 15 December 2019 11:55:36 UTC+10, Jirayu Kaewprateep wrote:

How to limited the break pt?

I guess you want to know how to find the break point of your application? 

Usually (regardless of perf test tool) you would ramp up 20-50 users and let the system settle for 15-30 minutes then add another 20-50 users and run another 15-30 minutes and repeat, this type of test can take quite a long time (days) so if you are already fairly confident that your application was able to handle a certain number of users (e.g. 1000) then you would ramp-up to that first and then gradually add more after that.

 

I will not reach the stress point and hiw to create a close system?

Not sure what hiw is? or what you are trying to ask here? do you mean you don't have enough hardware to stress your application servers? 

If so this is a common problem for performance testers, there is really only 2 things you can do:

1) Request additional hardware, if your organisation really wants performance testing done then they should be willing to provide the resources to do the job.

2) Test after hours and seconder all the desktop machines you can from the organisation for your after hours testing use, you will probably need permission to do this but in some organisations this is more palatable than buying the hardware required for performance testing, every organisation works differently

Dave.

 

[Jirayu Kaewprateep]

What are we measure from this test? The system load capacity? If yes then we should define the breaking pt. somehow and reflect ton the testing program.

[Dave Amies]

Hi Jirayu,

On Sunday, 15 December 2019 13:58:02 UTC+10, Jirayu Kaewprateep wrote:

What are we measure from this test? The system load capacity? If yes then we should define the breaking pt. somehow and reflect ton the testing program.

Yes, the test scenario I gave you earlier is for finding the system load capacity and the breaking point(s?) but how the system load capacity and the breaking points are defined vary from project to project, for example one project I worked on the system load capacity was defined as all the servers not exceeding 80% cpu or memory, for others it was 70% and 90%. Another project I worked on it was that end user response time was not to exceed 8 seconds at 80%ile

As for breaking point I have worked on projects where it was defined as the web server returning any 5xx error and others where the application response time was greater than 10 seconds regardless of whether it eventually responded or not. 

Defining the what the system load capacity and breaking points are is something you will need to do with your project

FYI though it's web focused, this is a good resource reference for user response times https://developers.google.com/web/fundamentals/performance/rail#ux the table "User Perception Of Performance Delays" actually can be applied to any application web, desktop or mobile.

I hope that helps.