Re: Best practices for automated web testing with Robot Framework of a Java-based project

[Florian Heine]

Hello,

We are using Robot Framework as the basis for our automated test framework. One of the main applications we are targeting is a Java app with a web and Swing front-end.

On of the important things to remember is that Jython has significantly less support from library developers. We have found that because of this issue, we need to keep Robot running in CPython and use the remote interface to talk to the "Java World". You can still use Jython to implement the Java end of things and expose it as a remote interface.

Having said this, there is no requirement for us to integrate with Maven or any significant benefit from doing so. If there is benefit for you, then the trade-off may well be different for you.

Florian

On Tuesday, 26 March 2013 17:00:40 UTC, Radu-Mihail Obadă wrote:

    Hello,

    I am currently evaluating Robot Framework for developing a test automation framework. The would-be system-under-test is a client/server system with a web front-end and Java EE on the server side.
   
    My first goal is to develop a collection of functional tests for the web front-end. From what I read so far I think that Robot Framework would be very appropriate for this project, especially given its hybrid keyword and data-driven design. I know for sure that, at one time or another, I will need to use Selenium for driving the browser. Given that a secondary but very important goal is to also integrate with/access the back-end Java code, what would be the best approach to pursue?

    The way I see it, there are several options:

• Use the Python version with Selenium2Library and integrate with the Java code, when required, using the Remote Library Interface.
• Use the Jython version with Selenium2Library-java and integrated directly with Maven and Java. I am not sure of the cons of this approach.
  
• Use the Python version and expose higher level keywords from Java (use Selenium directly from Java) through the Remote Library Interface.
• Use the Jython version and expose higher level keywords from Java (use Selenium directly from Java) using the native interface.

    What do you think? What approach would you take? Is there anything that I'm missing?

    Thank you for your time,
    Radu-Mihail

[Radu-Mihail Obadă]

On Wednesday, March 27, 2013 9:28:58 AM UTC+1, Florian Heine wrote:

Hello,

We are using Robot Framework as the basis for our automated test framework. One of the main applications we are targeting is a Java app with a web and Swing front-end.

On of the important things to remember is that Jython has significantly less support from library developers. We have found that because of this issue, we need to keep Robot running in CPython and use the remote interface to talk to the "Java World". You can still use Jython to implement the Java end of things and expose it as a remote interface.

Having said this, there is no requirement for us to integrate with Maven or any significant benefit from doing so. If there is benefit for you, then the trade-off may well be different for you.

Florian

    Hi Florian,

    Thank you very much for your reply. Hoping not to be too inquisitive, can I just ask you, please, if you integrated Robot with your continuous build system? Do you, if you don't mind, use Python to write keyword libraries? It would be very interesting also, if it's not too much of a better, to know a bit more of your automated testing setup.

    Thank you,
    Radu-Mihail

[Kevin O.]

This is definitely not an easy decision to make.

We went the route of staying in CPython and putting the Java code in a remote library. It has worked well, but I miss the ability to control the build environment with Maven.

Our remote library is built with Maven in its own Jenkins project and we use the Copy Artifact plugin for Jenkins to grab the latest build in other projects that depend on it.

I would also consider using Selenium2Library in Jython with the Maven plugin using the beta release of Jython 2.7. GA soon, hopefully.

Selenium2Library-java is very new so it is difficult to make a judgement on that.

I would definitely not suggest using Selenium directly from Java without some kind of framework on top of the WebDriver API. I think it is low in productivity to code with such a low-level API. Its also a much harder learning curve.

FYI I'm the one who rewrote jrobotremoteserver to be Mavenized and support JavaLib Core libraries (dynamic API libraries).

Kevin

[Radu-Mihail Obadă]

    Hello Kevin,

    Thank you very much for taking the time to reply. As you say, this is not an easy decision to make. For the moment, I think I will stick with CPython and the native Python libraries while waiting for Jython 2.7 to go GA, when I'll consider making the switch.
    Also, thank you for the great work on jrobotremoveserver and JavaLibCore, they might come in handy pretty soon. :)

    Best regards,
    Radu-Mihail

    P. S. I'd like to have some more opinions from other members of the group regarding the way you implemented Robot Framework for your automated tests. I think this the possibility of turning into a very interesting conversation.

[Tatu Aalto]

Ugh

When we did start using Robot Framework, we did live purely in Java world. At first we did start by using Robot Framework with Maven plugin, because it did provide easy way to try new stuff.
The test where (and currently are) in a separated project, which in looking back was a mistake. But there is cron job that runs all the automated test during the night and we use rebot to combine a one report and risto to draw pretty pictures. There is plan to combine the source code and test projects but because that part of the product is not currently in active development, it is quite bottom of our to do list.

Now we live in mix world, where we use Python and Java to produce our products. We use Robot Framework with Selenium2Library to automate our UI testing and do visualization how our product behaves over time period. In our case, the  UI is quite simple, user inputs few things and product/UI visualizes data feeds. Now days our main testing focus is on the data visualization and many times the time periods are quite long (usually over 12h, sometimes up to 48h).

So we have automated most of the test setup, (there are still some to do, related to configuration that lies on the server hard drive, some UI elements that are not used often and updating the SUT software), with Selenium2Library and some user defined keywords. Then Robot Framework basically just sits (there is some error detection/recovery logic implemented but is it quite simple) and takes screen captures how the data is visualized to the user. After the defined the time period has passed (12h - 48h), it tear downs the test. Because looking individual screen captures is quite boring and there is too many of them (usual there is around 1000 - 2000 screen captures), a video [1] [2] is created from the screen capture to shorten time time period from many hours to few minutes. It is quite useful the see how our product behaves, when there are changes in the data feed and how those changes are visualized. This has revealed quite many things from our product and have helped to improve the product.

Currently we are in a state where there is a dedicated server, that has all required software installed (Python, Robot Framework and so on), then there are few dedicated test server with predefined configurations. When user wants to start a test, he/she just runs a script and Robot Framework does the rest, within the limitations mentioned above.  There are different scripts for different data feeds and different user defined configurations.

I agree that in some cases, our situation is unique. I think it is quite rare that the time period that is taken to run one test takes is this long. And yes, we use CI, with Robot Framework to do basic checkups and yes there are test that do not take long time (not run by the CI), all that is usually quite normal in agile world. But it is they way how our product is build and what we are trying to do, forces us to run these long lasting test. We run them quite often, at least we try to start one test each day and see the result next morning. 

I do know how much this would help you but least it has tough for me to think outside of the box. I never though that looking videos could be testing :-)

-Tatu

[1] http://www.mplayerhq.hu
[2] http://www.mplayerhq.hu/DOCS/HTML/en/mencoder.html

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at http://groups.google.com/group/robotframework-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Radu-Mihail Obadă]

On Thursday, March 28, 2013 10:11:21 AM UTC+1, Tatu aalto wrote:

Ugh

When we did start using Robot Framework, we did live purely in Java world. At first we did start by using Robot Framework with Maven plugin, because it did provide easy way to try new stuff.
The test where (and currently are) in a separated project, which in looking back was a mistake. But there is cron job that runs all the automated test during the night and we use rebot to combine a one report and risto to draw pretty pictures. There is plan to combine the source code and test projects but because that part of the product is not currently in active development, it is quite bottom of our to do list.

Now we live in mix world, where we use Python and Java to produce our products. We use Robot Framework with Selenium2Library to automate our UI testing and do visualization how our product behaves over time period. In our case, the  UI is quite simple, user inputs few things and product/UI visualizes data feeds. Now days our main testing focus is on the data visualization and many times the time periods are quite long (usually over 12h, sometimes up to 48h).

So we have automated most of the test setup, (there are still some to do, related to configuration that lies on the server hard drive, some UI elements that are not used often and updating the SUT software), with Selenium2Library and some user defined keywords. Then Robot Framework basically just sits (there is some error detection/recovery logic implemented but is it quite simple) and takes screen captures how the data is visualized to the user. After the defined the time period has passed (12h - 48h), it tear downs the test. Because looking individual screen captures is quite boring and there is too many of them (usual there is around 1000 - 2000 screen captures), a video [1] [2] is created from the screen capture to shorten time time period from many hours to few minutes. It is quite useful the see how our product behaves, when there are changes in the data feed and how those changes are visualized. This has revealed quite many things from our product and have helped to improve the product.

Currently we are in a state where there is a dedicated server, that has all required software installed (Python, Robot Framework and so on), then there are few dedicated test server with predefined configurations. When user wants to start a test, he/she just runs a script and Robot Framework does the rest, within the limitations mentioned above.  There are different scripts for different data feeds and different user defined configurations.

I agree that in some cases, our situation is unique. I think it is quite rare that the time period that is taken to run one test takes is this long. And yes, we use CI, with Robot Framework to do basic checkups and yes there are test that do not take long time (not run by the CI), all that is usually quite normal in agile world. But it is they way how our product is build and what we are trying to do, forces us to run these long lasting test. We run them quite often, at least we try to start one test each day and see the result next morning. 

I do know how much this would help you but least it has tough for me to think outside of the box. I never though that looking videos could be testing :-)

-Tatu

    Hello Tatu,

    Thank you so much for your reply, it was most interesting a read!
    In your case, do you need to directly integrate/interface the Java code from your Robot Framework test? Have you developed any keyword libraries in Java? What do you think are the cons of your current set-up?

    Sincerely,
    Radu-Mihail

[Tatu Aalto]

Ugh

1) In your case, do you need to directly integrate/interface the Java code from your Robot Framework test?
With Maven plugin, there are Java keywords to interact with the product, made by the development. 

Now days, there are not Java or Python keywords to interface directly with the product. But there are interfaces that exposes some internals of the product. One cannot influence the product state from these interfaces but one can query status of the product or/and get some other data out of the product which otherwise could be difficult. But this not only intended for the testing, it is also intended for the customer support team for trouble shooting and error recovery or it is internal API used by the product it self. 

2) Have you developed any keyword libraries in Java
I do not code Java, but I do some coding with Python. Example coding some keywords that decode some output are quite easy do to in Python. It is a mix in our company, we use both Java and Python to build our products, so using Python is quite natural for me. But the bottom line is that I am tester and not coder, if it is complex someone from dev will do it or assist me doing it.

3) Pros and cons
When we directly interface with the product, test set up was fast, test where fast to run and feed back was fast. But it lacks the big picture, that was difficult to see. Of course other (manual) test did cover this part during that time.

Currently test setup takes long time, 20mins about to get screen captures taken, test takes long to run and therefore feed back is slow. I can not imagine to put this in any CI. But it always test the full application, it gives good overall view what is going on. You do not know all the small details, but sometimes that is not required.

Our testing department is so small, just me, that there is not resource to do deep level test by me and also test the hole product. So we did kind a choose to test to hole product and we are, well not constantly, now and then making improvements to get a faster feedback.

-Tatu

[Radu-Mihail Obadă]

On Thursday, March 28, 2013 12:41:53 PM UTC+1, Tatu aalto wrote:

Ugh

1) In your case, do you need to directly integrate/interface the Java code from your Robot Framework test?
With Maven plugin, there are Java keywords to interact with the product, made by the development. 

Now days, there are not Java or Python keywords to interface directly with the product. But there are interfaces that exposes some internals of the product. One cannot influence the product state from these interfaces but one can query status of the product or/and get some other data out of the product which otherwise could be difficult. But this not only intended for the testing, it is also intended for the customer support team for trouble shooting and error recovery or it is internal API used by the product it self. 

2) Have you developed any keyword libraries in Java
I do not code Java, but I do some coding with Python. Example coding some keywords that decode some output are quite easy do to in Python. It is a mix in our company, we use both Java and Python to build our products, so using Python is quite natural for me. But the bottom line is that I am tester and not coder, if it is complex someone from dev will do it or assist me doing it.

3) Pros and cons
When we directly interface with the product, test set up was fast, test where fast to run and feed back was fast. But it lacks the big picture, that was difficult to see. Of course other (manual) test did cover this part during that time.

Currently test setup takes long time, 20mins about to get screen captures taken, test takes long to run and therefore feed back is slow. I can not imagine to put this in any CI. But it always test the full application, it gives good overall view what is going on. You do not know all the small details, but sometimes that is not required.

Our testing department is so small, just me, that there is not resource to do deep level test by me and also test the hole product. So we did kind a choose to test to hole product and we are, well not constantly, now and then making improvements to get a faster feedback.

-Tatu

    Thank you for your patience and for your very informative response, Tatu!

    Radu-Mihail

[David]

Just wondering, if you are able to allocate the time for proof of concept & evaluation of Robot Framework, wouldn't it be good idea to try all the options you mention and see which seems to work out best for you in terms of usability with your team and which doesn't give you much problem and works mostly out of the box and needs least amount of integration work?

Getting a discussion going is good to get experiences of others and a feel of what to expect but you'd not know for sure until you try things out.

[Radu-Mihail Obadă]

On Saturday, March 30, 2013 6:46:06 AM UTC+1, David wrote:

Just wondering, if you are able to allocate the time for proof of concept & evaluation of Robot Framework, wouldn't it be good idea to try all the options you mention and see which seems to work out best for you in terms of usability with your team and which doesn't give you much problem and works mostly out of the box and needs least amount of integration work?

Getting a discussion going is good to get experiences of others and a feel of what to expect but you'd not know for sure until you try things out.

    Well, it turns out that that's what I started doing. :-)
    Thank you for your time, patience and shared knowledge, everyone!

    Radu-Mihail