RF - Inconsistent on execution

[Anand K L]

Hi,

 

I have done automated testing on Salesforce.com, using Robot framework (mostly Sel2Lib and Builtin Keywords). If I execute a suite which contains say around 20 test cases, the probablity of sucessfull end to end execution is less than 50%. The entire suite which passes first time, second time fails. Surprisingly the failure is due to "Locator did not match any element", thought he element in on the webpage rendered. This is happening randomly on any page for any element. The same test case or suite if I execute in another system, it may work perfectly fine.

 

I would like to know if any of you faced similar issue with using RF. If yes, what was your approach to overcome. It is making me think aloud "is RF a dependable framework to get consistent results on SaaS apps like SFDC" ?

 

Any view on this would help me.

 

Regards,

Anand KL

[Markus Bernhardt]

Hi Anand,

1. short answer:

- RF is consistent

- Your testcases are faulty

2. long answer:

In the wonderful world of dynamic HTML, JavaScript, AJAX, etc. web content in the browser is not static any more, but very often generated on the fly. Furthermore there is no control of the timing of events or durations of actions.

Example:

In our application all buttons are most of the time simple images. First when the mouse pointer enters a button the element gets removed and replaced by a clickable button with a slightly different image. If you use now simply the keyword Click Element on such a button, you have exactly such a unstable behavior.  Why? Because Click Element has to do internally two things. It moves the mouse pointer over the button and then immediately clicks the button. Now look at the timing:

When the testcase calls Click Element the following happens:

• Your testcase moves the mouse over the button.
• After some milliseconds in the browser the onEnter event-handler gets called.
• The event handler removes the dummy button, adds the real button, makes the new button visible, enables the new button
• Parallel your testcase clicks on the button.

Depending on the timing of the click the following could happen:

• The click gets executed, before the dummy element is removed:
  No failure is logged, but nothing happens, because the button is a dummy.
• The click gets executed, after the dummy element has been removed, but before the real button has been added:
  A "Locator did not match any element" error occurs
• The click gets executed, after the real button has been added, but before it is visible
  A "Element not visible" error occurs
• The click gets executed, after the real button is made visible, but before it is enabled
  A "Element not clickable" error occurs
• The click gets executed, after the real button is enabled.
  It works as expected

In real life depending on browser, browser version, operating system, cpu load, network load, network latency, file system load and another thousand things you will see one of the five things happen. Some more often than others, but in the end it is non-deterministic.

Selenium 1 (RC) tried to fix that by introducing a default wait between every action. The hope was, that the browser finished all actions, before Selenium executed the next action. There were only two problem. First, increasing the wait times made the test execution awfully slow and second, there were always test executions where the wait was to short.

In Selenium 2 (WebDriver) all that sleep stuff was replaced by explicit waits on properties. So it is possible to wait fort all required actions to finish and to write deterministic testcases.

In our application we are using the following keywords to click on a button:

• Mouse Over                         <locator of the button>
• Wait Until Element Is Clickable    <locator of the button with correct image>
• Click Element                      <locator of the button>

This works stable for our application.

I hope you understand now, why your testcases are faulty. From experience I can say there is no tool that saves you from understanding your system under test when you want to automate testexecution.

Hope that helps,

Markus

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at http://groups.google.com/group/robotframework-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kevin O.]

Seems like Markus is perhaps being a little harsh, but I agree with him.

I see lots of failures, but most are due to Selenium or bad test code or bugs in the AUT and not due to Selenium2Library or RF.

Many of the challenges Markus is talking about apply just as well to other bindings for Selenium (e.g. Java). I believe that Selenium2Library is more productive as well as having a much easier learning curve than using WebDriver directly or using lower-level libraries like Watir.

In addition to manually syncing with the AUT using keywords like Wait Until Element Is Clickable, which is unfortunately not in Selenium2Library yet, there is also another approach that might help:

Selenium has a feature called implicit wait that can help avoid the failures you are seeing. By default, implicit wait is turned off.

Here are the keywords to use this feature:

Set Selenium Implicit Wait

Set Browser Implicit Wait

Kevin

[Markus Bernhardt]

Hi Kevin,

yes, my mail yesterday was a little bit too harsh. Sorry for that. To my defense, I can only say it was 1 a.m in the morning and I have been driving 15 hours through Europe in my Smart yesterday.

Unfortunately I have to sincerely advise against using implicit wait on AJAX pages, because webdriver is IMHO broken here by design. When you switch implicit wait on, only exceptions of the type NoSuchElementException are caught, but not for example a ElementNotVisibleException. This renders implicit wait useless and its usage will complicate things even more.

Only my 2 cent.

Cheers,
Markus

[Jason Zhao]

I ran into the ElementNotVisibleException ,but fortunately your analysis just told me what might be the cause.  yep, the timing of the click was just not right, so I let it sleep 1s and then, Ta-Dah, no exception.

so thank you

To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-users+unsub...@googlegroups.com.