Performance testing with Robot Framework

[Dave Amies]

Hi All,

This post is to gauge if there is any interest here for people to be able to use Robot Framework as the scripting language for performance testing?

I have started a project that is designed to enable this to happen, so if this is something that you are interested in please comment in this thread.

I have called this project rfswarm, short for Robot Framework Swarm and is available here https://github.com/damies13/rfswarm

I believe what I have built so far to be stable, however I have labeled the initial release as Alpha as it is still missing features I consider necessary to be ready for every day use. 

I am still working on the documentation, so I will be working on that over the next week or so before going back to adding features to the application.

I haven't figured out the install instructions yet, if your keen to try it out running the python file and using pip to install any library that it complains is missing, should get you going.

Dave.

[Hélio Guilherme]

Congratulations.

Your project is very interesting.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/352c5e8b-4d24-449f-979d-b6b5655b7dd2%40googlegroups.com.

[Anthony fromtheuk]

This is interesting, will give this a try next week, thanks for sharing

[Sachin Dhiman]

Hey Dave ,

Amazing work,

Will try this library and revert back to you.

We can definitely extend this framework to add more testing capabilities 

For eg 

Security testing,

Generic pdf testing  and pdf to pdf comparison

( this is something I m currently working on) 

Can we brainstorm more on adding AI capabilities to this Framework 

For eg. Automatic Updation of locators whenever there is change in webpage.

For  AI capabilities in Automation Testing ,you can take reference from tools like Mabl, Testim .

There are plenty of good ideas with me,

I m looking for collaboration.

Drop me an email for discussion ( sachinn...@gmail.com)

On Sat, 2 Nov, 2019, 8:06 PM Anthony fromtheuk, <anthonyf...@gmail.com> wrote:

This is interesting, will give this a try next week, thanks for sharing

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/32e95f28-0430-4a0b-83d8-115687d5e43e%40googlegroups.com.

[Deepak Pareek]

 Nice work. Will try this.

Cheers

Deepak

Sent from my iPhone

On 2 Nov 2019, at 11:47, Hélio Guilherme <heliox...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/CABNCgGQeNj6WWc%3DMXJpYtu_8ZPz6MoKuwRnLHXWvRhDfjG8HgA%40mail.gmail.com.

[Dave Amies]

Hi Sachin,

Firstly I would be happy if you find this tool so useful, i'm not entirely sure how that will work.

> Security testing,

I guess this could be quite easy, if you set up a security focused robot test but needed to run it hundreds of times with different data variations to find out if a system was vulnerable, then rfswarm would easily enable that.

> Generic pdf testing  and pdf to pdf comparison

This I have no idea what you are thinking, as I would expect the comparison is something you would do in robot framework? but I guess if you had a lot of PDF's that needed comparison, you could use rfswarm to ramp up your test case with many parallel comparisons to speed up testing.

> Can we brainstorm more on adding AI capabilities

This I have no idea what you have in mind here? I am open minded but want to keep the GUI / Server component pretty focused on Performance testing, I try to work on the principle of building a tool to bet the best at it's specific task rather than mediocre at many tasks.

Having said that, I guess if you wanted to build an AI specific server that used the agents to trigger running of specific test cases based on reaction to certain conditions, that would be quite easy to do. The Communication between the Agents and the GUI application is a basic JSON (REST) API, which I guess I need to document. If you created an AI server that responded appropriately to the Agents I guess you could get them to run test cases on all sorts of triggers.

> Automatic Updation of locators whenever there is change in webpage

Again I have no idea what you are thinking here, and I expect that this is something that would be handled in your test case? However I am planning to build a tool similar to the VTS that is provided with LoadRunner, I plan to make something that is a little more flexible and less clunky than VTS, to make Data management for performance testing easier with rfswarm. I guess if you wrote a test case that detected a new element in a page it could be configured to write new test data to the data table for another robot script to pick up and test that element, is this along the lines of what you were thinking? 

I look forward to hearing how you make use of this tool, If you want specific features then add a feature request in the issues on the Github project or fork the project, add the feature and submit a pull request.

Dave.

On Sunday, 3 November 2019 00:50:30 UTC+10, Sachin Dhiman wrote:

Hey Dave ,

Amazing work,

Will try this library and revert back to you.

We can definitely extend this framework to add more testing capabilities 

For eg 

Security testing,

Generic pdf testing  and pdf to pdf comparison

( this is something I m currently working on) 

Can we brainstorm more on adding AI capabilities to this Framework 

For eg. Automatic Updation of locators whenever there is change in webpage.

For  AI capabilities in Automation Testing ,you can take reference from tools like Mabl, Testim .

There are plenty of good ideas with me,

I m looking for collaboration.

Drop me an email for discussion ( sachinn...@gmail.com)

On Sat, 2 Nov, 2019, 8:06 PM Anthony fromtheuk, <anthonyf...@gmail.com> wrote:

This is interesting, will give this a try next week, thanks for sharing

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

[Dave Amies]

Hi Deepak,

I look forward to hearing how you go.

Dave.

On Sunday, 3 November 2019 01:52:04 UTC+10, Deepak Pareek wrote:

 Nice work. Will try this.

Cheers

Deepak

Sent from my iPhone

On 2 Nov 2019, at 11:47, Hélio Guilherme <heliox...@gmail.com> wrote:



Congratulations.

Your project is very interesting.

On Sat, Nov 2, 2019 at 11:36 AM Dave Amies <dami...@gmail.com> wrote:

Hi All,

This post is to gauge if there is any interest here for people to be able to use Robot Framework as the scripting language for performance testing?

I have started a project that is designed to enable this to happen, so if this is something that you are interested in please comment in this thread.

I have called this project rfswarm, short for Robot Framework Swarm and is available here https://github.com/damies13/rfswarm

I believe what I have built so far to be stable, however I have labeled the initial release as Alpha as it is still missing features I consider necessary to be ready for every day use. 

I am still working on the documentation, so I will be working on that over the next week or so before going back to adding features to the application.

I haven't figured out the install instructions yet, if your keen to try it out running the python file and using pip to install any library that it complains is missing, should get you going.

Dave.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/352c5e8b-4d24-449f-979d-b6b5655b7dd2%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

[Dave Amies]

Hi Anthony,

I look forward to hearing your results.

Dave.

[Dave Amies]

Thanks Hélio

On Saturday, 2 November 2019 21:47:25 UTC+10, Hélio Guilherme wrote:

Congratulations.

Your project is very interesting.

On Sat, Nov 2, 2019 at 11:36 AM Dave Amies <dami...@gmail.com> wrote:

Hi All,

This post is to gauge if there is any interest here for people to be able to use Robot Framework as the scripting language for performance testing?

I have started a project that is designed to enable this to happen, so if this is something that you are interested in please comment in this thread.

I have called this project rfswarm, short for Robot Framework Swarm and is available here https://github.com/damies13/rfswarm

I believe what I have built so far to be stable, however I have labeled the initial release as Alpha as it is still missing features I consider necessary to be ready for every day use. 

I am still working on the documentation, so I will be working on that over the next week or so before going back to adding features to the application.

I haven't figured out the install instructions yet, if your keen to try it out running the python file and using pip to install any library that it complains is missing, should get you going.

Dave.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.

[Pekka Klärck]

Hi,

This sounds super interesting. I agree there's a lot of synergy
between functional test automation and performance testing. Being able
to reuse functional tests or just being able to use the same test data
syntax and same libraries would be great.

I know people have been using Robot Framework for performance testing
earlier, but I've been worried that RF itself has so high overhead
that it wouldn't scale if performance requirements are high. This kind
of server/agent architecture can mitigate that problem at least
partly, and it was interesting to read from the docs that the initial
tests show that performance is comparable with JMeter and even
LoadRunner in some scenarios. Impressive!

Let me know if you notice needs for new APIs or other features in
Robot that would help rfswarm!

Cheers,
.peke

> --
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/352c5e8b-4d24-449f-979d-b6b5655b7dd2%40googlegroups.com.

--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Anthony fromtheuk]

I am unable to use this on Windows, selection of a test suite works but then am unable to select a test, not sure what I am doing wrong here?

As soon as I posted that I realised what I was doing wrong, I only had a Test Case section not Test Cases, only a simple visit google test, got it running but agent errors repeatedly like so...

Exception in thread Thread-1426:

Traceback (most recent call last):

  File "C:\Users\me\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner

    self.run()

  File "C:\Users\me\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run

    self._target(*self._args, **self._kwargs)

  File ".\rfswarm_agent.py", line 472, in run_process_output

    tree = ET.parse(outputFile)

  File "C:\Users\me\AppData\Local\Programs\Python\Python37\lib\xml\etree\ElementTree.py", line 1197, in parse

    tree.parse(source, parser)

  File "C:\Users\mel\AppData\Local\Programs\Python\Python37\lib\xml\etree\ElementTree.py", line 587, in parse

    source = open(source, "rb")

FileNotFoundError: [Errno 2] No such file or directory: 'C:\the correct path\\Scenario_1572857539\\test_1_9_1572857577\\Visit_Google_output.xml' 

[Dave Amies]

Hi Anthony,

To be honest I haven't tried it on any os other than mac osx yet, but I have been careful to only use python libraries that are platform agnostic. 

Would It be possible to share your robot file with me so I can see if I can reproduce it? 

Also if you ran it from a command window can you see if there is any errors there? I just dump exceptions to the stdout for now, cause I haven't set up proper logging. 

On Mon., 4 Nov. 2019, 18:45 Anthony fromtheuk, <anthonyf...@gmail.com> wrote:

I am unable to use this on Windows, selection of a test suite works but then am unable to select a test, not sure what I am doing wrong here?

On Sunday, 3 November 2019 07:33:29 UTC, Dave Amies wrote:

Hi Anthony,

I look forward to hearing your results.

Dave.

On Sunday, 3 November 2019 00:36:41 UTC+10, Anthony fromtheuk wrote:

This is interesting, will give this a try next week, thanks for sharing

--
You received this message because you are subscribed to a topic in the Google Groups "robotframework-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/robotframework-users/dKrLwPoIDgQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to robotframework-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/1cdf1576-2ed1-4666-b856-0f24a22b0308%40googlegroups.com.

[Anthony fromtheuk]

Original issue was I had *** Test Case *** instead of *** Test Cases ***, at least switching to that meant I could select and run the test in the suite.

Test I ran was simple open browser to google sleep 4 seconds and close browser using seleniumlibrary.

Terminal showed this repeatedly for each test ran

Exception in thread Thread-1426:
Traceback (most recent call last):
File "C:\Users\me\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\me\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File ".\rfswarm_agent.py", line 472, in run_process_output
tree = ET.parse(outputFile)
File "C:\Users\me\AppData\Local\Programs\Python\Python37\lib\xml\etree\ElementTree.py", line 1197, in parse
tree.parse(source, parser)

File "C:\Users\me\AppData\Local\Programs\Python\Python37\lib\xml\etree\ElementTree.py", line 587, in parse

[Dave Amies]

Hi Anthony,

1) Is *** Test Case *** valid in robot? if it is then I will need to fix that.

2) I don't own a recent Windows license (most recent one is NT4), so I spent quite a bit of time today installing ReactOS on a VM, only to find out it is only at a Windows XP OS level and so only Supports Python 3.4.4, but doing this helped me find out that the minimum python version for the Agent is 3.6 and 3.7 for the GUI, so It was productive anyway.

> FileNotFoundError: [Errno 2] No such file or directory: 'C:\the correct path\\Scenario_1572857539\\test_1_9_1572857577\\Visit_Google_output.xml'

I also installed the Agent on linux Mint and got basically the same error as you below, this is a File Not Found Error, it means for some reason robot framework didn't create the xml file when it ran, for me the solution was obvious "/bin/sh: 1: robot: not found" Install robot.

RFSwarmAgent: mainloop: Running 2019-11-05 21:21:41 ( 1572952901 )isrunning: True isconnected: True

/bin/sh: 1: robot: not found

Exception in thread Thread-72:

Traceback (most recent call last):

  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner

    self.run()

  File "/usr/lib/python3.6/threading.py", line 864, in run

    self._target(*self._args, **self._kwargs)

  File "rfswarm_agent.py", line 472, in run_process_output

    tree = ET.parse(outputFile)

  File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1196, in parse

    tree.parse(source, parser)

  File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 586, in parse

    source = open(source, "rb")

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/rfswarmagent/logs/Scenario_1572952880/OC_Demo_2_1_1_1572952901/Load_Store_Landing_Page_output.xml'

But in addition to that, if you look at the path provided:

'/tmp/rfswarmagent/logs/Scenario_1572952880/OC_Demo_2_1_1_1572952901/Load_Store_Landing_Page_output.xml'

On the machine you are running the agent, try opening a command line to '/tmp/rfswarmagent/scripts' or in your case "C:\the correct path\..\scripts" (because "C:\the correct path" should include "\logs\"), from here try running "robot test.robot" and see if that works without any of my code.

when I did that after installing robotframework and robotframework-seleniumlibrary, I found that my agent machine (fresh install of Linux Mint) was missing geckodriver (for firefox)

Hopefully that will get you going.

Dave.

[Pekka Klärck]

ti 5. marrask. 2019 klo 14.04 Dave Amies <dami...@gmail.com> kirjoitti:

1) Is *** Test Case *** valid in robot? if it is then I will need to fix that.

Yes it is. The format is perhaps too flexible:

- Both "Test Cases" and "Test Case" are allowed.

- In RPA mode also "Tasks" and "Task" are OK.

- It's case-insensitive.

- Spaces between the text and asterisks are optional.

- Only one leading asterisk is mandatory. Others are optional and there can even be more that three.

- There can be other content on the same line after this header. Just need to separate it with two or more spaces.

Many Robot editors also have problems supporting all the above. Being more strict is fine, though, but should be documented.

The new parsing API provided by the forthcoming RF 3.2 will hopefully make this situation better. The idea is that external tools can just use that API and let Robot handle all the quirks in the syntax. If rwswarm needs to parse Robot's data more, it could switch to using this API as well.

Cheers,

    .peke

[Anthony fromtheuk]

The location show in error in last message - C:\the correct path\..\scripts - may not be a correct path after all, its looking here C:\\Users\\A6BDA~1.ODO\\AppData\\Local\\Temp\\rfswarmagent

When I think it should be looking here - C:\\Users\\a.odonnell\\AppData\\Local\\Temp\\rfswarmagent

I cannot run the test script from the scripts folder, it fails to load SeleniumLibrary (and perfresource.py file buts that my mistake using relative path in script), guess path to SeleniumLibrary is not in my environment vars? I did pip install to get it and works everywhere else fine.

When I try and run script in C:\Users\a.odonnell\AppData\Local\Temp\rfswarmagent\scripts -  I get the following... 

PS C:\Users\a.odonnell\AppData\Local\Temp\rfswarmagent\scripts> robot .\test.robot

[ ERROR ] Error in file 'C:\Users\a.odonnell\AppData\Local\Temp\rfswarmagent\scripts\test.robot': Importing test library 'SelenimuLibrary' failed: ModuleNotFoundError: No module named 'SelenimuLibrary'

Traceback (most recent call last):

  None

PYTHONPATH:

  C:\Users\a.odonnell\AppData\Local\Programs\Python\Python37\Scripts\robot.exe

  c:\users\a.odonnell\appdata\local\programs\python\python37\python37.zip

  c:\users\a.odonnell\appdata\local\programs\python\python37\DLLs

  c:\users\a.odonnell\appdata\local\programs\python\python37\lib

  c:\users\a.odonnell\appdata\local\programs\python\python37

  c:\users\a.odonnell\appdata\local\programs\python\python37\lib\site-packages

  c:\users\a.odonnell\appdata\local\programs\python\python37\lib\site-packages\win32

  c:\users\a.odonnell\appdata\local\programs\python\python37\lib\site-packages\win32\lib

  c:\users\a.odonnell\appdata\local\programs\python\python37\lib\site-packages\Pythonwin

[ ERROR ] Error in file 'C:\Users\a.odonnell\AppData\Local\Temp\rfswarmagent\scripts\test.robot': Resource file '..\Robot_Resources\perftest.resource' does not exist.

[Dave Amies]

Thanks, I will need to update rfswarm (logged Issue #6), this is great, finding new bugs already.

[Dave Amies]

Are you using the initial release (v0.1-alpha)? I pushed v0.2-alpha yesterday, relative paths was one of the issues I fixed, cause I had the exact same problem with:

 [ ERROR ] Error in file 'C:\Users\a.odonnell\AppData\Local\Temp\rfswarmagent\scripts\test.robot': Resource file '..\Robot_Resources\perftest.resource' does not exist.

The relative paths work now, but have some limitations, doing something like ..\..\..\..\something might take you out of the the temp directory and may not even be possible as it might try to go directories above c:\ or / (on unix systems) which obviously is not possible, going up 1 or 2 directories should be fine now though.

Dave.

[Anthony fromtheuk]

I fixed my script, I had spelt Selenium incorrectly... also added full path to resource file, can now run script from temp dir successfully (does the script in the temp directory not get updated if the original script has been?). 

Now have newest version, still missing xml output file and failing in same manner

[Dave Amies]

Hi All,

I have updated to version 0.3 

The most important change in this version is the ability to Save a Scenario an Open it again at a later time.

Also fixed bug, so now the following are also recognised:

*** Test Case ***

*** Tasks ***

*** Task ***

if you find issues please let em know here or post a new issue at https://github.com/damies13/rfswarm/issues

Dave.