Why is WebDriver slower than selenium-client? (Ruby)

[Eric Pierce]

Greetings all. I'm one of the authors of a Ruby tool called Rsel[1]
that uses the older selenium-client gem. I've been hearing rumblings
for some time now that selenium-webdriver is the new hotness, and have
run into more than a few bugs in selenium-client / RC that were
classified by the Selenium team as "won't fix because we're using
WebDriver now".

So today I decided to investigate my options for switching over. I've
read through and tried some of the examples on the RubyBindings[2]
page, beginning with "WebDriver-backed Selenium". After switching to
this, I tried running Rsel's unit test suite of ~500 tests. I got
about 100 failures, which didn't really surprise me (I expected a few
wrinkles to need ironing), but what bothered me was how slow it ran--
it took over 9 times longer to run with WebDriver-backed Selenium as
it did using straight-up selenium-client.

OK, I thought, this is a kind of intermediary wrapper, and there are
bound to be some performance problems. What if I use native WebDriver
idioms? Will it work better to drive the browser (Firefox, in this
case) directly, instead of via the selenium-server I have running on
localhost? To help answer this, I wrote up a couple of super-simple
test programs[3] to measure the performance of Selenium::Client and
Selenium::WebDriver side-by-side. The results were depressing; after
several runs, testing various basic things such as clicking links,
navigating to a page, and filling in text fields (both by ID and by
xpath), I've concluded that WebDriver is, on average, 50% to 100%
slower than the older Client. There was no appreciable difference
between driving Firefox directly from WebDriver, and driving Firefox
through the "remote" with my selenium-server.

After much googling (as well as some searching through this group), I
haven't been able to find much explanation for this. I've seen a few
rumors that WebDriver can be faster (and I am keen on trying out the
Grid thing to do distributed testing), but so far my experience hasn't
been encouraging. It would be a substantial amount of work to convert
Rsel over to WebDriver, and if the only gain to be had is in the
number of coffees I can drink while running my automated tests, I am
not likely to attempt it.

To the point: How can I get WebDriver to be faster? I would really
like to switch to WebDriver--I don't want Rsel to have a dependency on
an unmaintained, deprecated library. I don't mind a slight performance
drop, but a doubling of test execution time is pretty hard to stomach.
I'd welcome any tips or suggestions!

Best regards,
Eric

[1] https://github.com/a-e/rsel
[2] http://code.google.com/p/selenium/wiki/RubyBindings
[3] https://gist.github.com/2047316

[Mark Collin]

If you are comparing a run where all the tests pass to a run where a load of
test fail you are comparing apples and oranges.

Secondly WebdriverBackedSelenium will be slower as you are not really using
WebDriver.

Best regards,
Eric

--
You received this message because you are subscribed to the Google Groups
"webdriver" group.
To post to this group, send email to webd...@googlegroups.com.
To unsubscribe from this group, send email to
webdriver+...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/webdriver?hl=en.

[Eric Pierce]

On Mar 16, 5:23 am, "Mark Collin" <mark.col...@lazeryattack.com>
wrote:

> If you are comparing a run where all the tests pass to a run where a load of
> test fail you are comparing apples and oranges.

Yes, that's true. That's why I opted for a simpler test, keeping all
variables the same except the choice of backend.

> Secondly WebdriverBackedSelenium will be slower as you are not really using
> WebDriver.

That's what I figured. Again, that's why I tried using native
WebDriver methods--but the performance issue remains. All other things
being equal, WebDriver was substantially slower. I am hoping there is
some configuration option I could use, or a Firefox profile tweak, or
something else (short of switching to pure Java) that would bring
WebDriver's performance up to par with Selenium-client's.

[David]

Related to this thread. Has anyone done performance benchmarks for the
other bindings as well? Java, .NET/C#, Python.

And on a side note, be interesting to benchmark unsupported 3rd party
bindings as well (PHP, Perl, etc.).

[David]

Also, what if you compare the RC version against RemoteWebDriver?
After all RemoteWebDriver better approximates RC in that you also have
to start up a server, rather than use the native drivers that have the
server built in.

And that way, you can also test RC version against WebDriver using
your own code to call the JSONWireProtocol rather than the WebDriver
API methods.

On Mar 16, 8:45 am, Eric Pierce <wapcaple...@gmail.com> wrote:

[Eric Pierce]

On Mar 16, 11:32 am, David <manga...@gmail.com> wrote:
> Also, what if you compare the RC version against RemoteWebDriver?
> After all RemoteWebDriver better approximates RC in that you also have
> to start up a server, rather than use the native drivers that have the
> server built in.

I've tried using both the builtin driver and the remote driver
(Selenium::WebDriver.for(:firefox) and
Selenium::WebDriver.for(:remote)), and have not observed a significant
difference in execution time. If anything, the direct-Firefox driver
is slightly faster. I've posted another gist showing the results I'm
getting with those two, compared with the deprecated
Selenium::Client::Driver API (https://gist.github.com/2187982). My
findings consistently show WebDriver taking nearly 2x as long to
execute the same commands. This is the closest I can get to an apples-
to-apples test, and so far WebDriver's apples don't taste so good.

> And that way, you can also test RC version against WebDriver using
> your own code to call the JSONWireProtocol rather than the WebDriver
> API methods.

I hadn't thought of that; I'm not familiar with the JSONWireProtocol,
but it looks pretty low-level. One of the reasons the Ruby
Selenium::Client interface has always appealed to me is its relatively
high-level, idiomatic-Ruby approach. The newer WebDriver API is
different, but still comparably high-level. Are you suggesting that
the intermediary calls between Selenium::WebDriver's top-level API and
the eventual remote calls that execute commands on the server are
responsible for the poor performance?

I'm not interested in doing a great deal more performance testing
right now. I still cannot see any advantage to switching to WebDriver;
aside from a couple of edge cases, the old Client works just fine (and
blows WebDriver out of the water in terms of speed). I am hoping
someone can convince me otherwise :-)

If there's another forum where I should be raising this issue, please
let me know.

[David]

I mentioned JSONWireProtocol as it may be a way to narrow down the
issue and see if issue is with the Ruby driver implementation or the
protocol itself. To test, make direct HTTP JSON formatted calls from
Ruby to the protocol to drive browser with the remote web driver
server.

Another idea is to port Ruby Selenium RC benchmark test to other
language driver bindings for WebDriver (Java, Python, etc.) and see if
Ruby for RC still outperforms WebDriver as well as whether the other
languages beat Ruby WebDriver binding or not.

And you might want to submit an issue to the Selenium project for them
to look into, besides this post? But I dunno... If you do submit
issue, you can attach benchmark reference RC test and the WebDriver
version for others to reproduce and evaluate. Hopefully your benchmark
test doesn't use non-public sites.

[David]

Is there sample Ruby code provided by Selenium project? Might be useful as reference tests to use those (for the WebDriver vs Selenium) benchmark. Something the Selenium developers can work with as well. But this is assuming the sample code can reproduce the issue.

Another question is which browsers and element locators are we comparing with? I've noticed that switching from RC in PHP to WebDriver in Java, that with FF the performance difference is not really noticeable. But with IE you may notice it seems to run faster under WebDriver. I also notice it runs faster for Safari in WebDriver as well (sometimes maybe too fast).

Can't really compare Chrome (limited or no support in RC?) but it runs fast there too.

And in a recent benchmark test using long complex XPath locators, it produced similar timings for FF and Safari for both WebDriver and RC. Couldn't get benchmarks for IE as the WebDriver port has some kind of issues where browser stalls and then crashes.

Maybe this is an issue with Ruby bindings then...

Perhaps best to provide a test for both RC and WebDriver that easily reproduces the issue and submit as issue to Selenium project. If test not using archived HTML files, then use against a public site.

And in the meantime if you can bear with it, switch to WebDriver with another binding like Python or Java over Ruby until performance issue is fixed.

Maybe could also opt to try and see if you could us JRuby (executing the Java bindings in Ruby or Ruby bindings in JRuby) to see if that makes any difference?

On Wednesday, July 18, 2012 5:31:46 PM UTC-7, dara wrote:

I'm seeing the same issue. I'm migrating ruby-based tests from Selenium::Client::Driver to Selenium::WebDriver and am seeing substantial slowdown in my scripts (I don't have hard benchmarks, but my scripts now take approximately twice as long to run). Everything I read led me to suggest WebDriver would be faster but this is not my experience, I'm seeing the opposite.

As with Eric, I am going straight to WebDriver, not using WebDriver-backed Selenium.

Writing my scripts in Ruby, using Firefox 13 as my browser.

[dara]

After spending a bunch more time looking at this, I realised a lot of the slowness I'm perceiving is due to Firefox starting up very slowly. Using WebDriver, on my machine at least, Firefox takes approximately 20 seconds to start up. IE and Chrome both start up in approximately 5 seconds, which is similar to the time it takes in Selenium. Not sure if this is a problem specific to my machine or a general issue.

[David]

I haven't logged startup and execution times, but I have observed that
Firefox with WebDriver doesn't appear all that fast compared to
Selenium RC, though the start up is often faster. IE appears slightly
faster on startup and execution. Safari by far is the browser that
runs the fastest in WebDriver, faster than in Selenium RC, so fast
that you have to do extra synchronization or waits to have test not
fail from execution commands too fast.

So in a way perhaps best way to compare WebDriver vs Selenium RC for
any binding may be to try it against Safari (use RemoteWebDriver to
access Safari if not using Java SafariDriver).

[Vishal]

I am sharing the results for the suite which has higher difference in execution

Webdriver Result

Name : Tests : Errors : Failures : Time(s) : Time Stamp  

AccountLockoutTest : 15 : 0 : 0 : 1935.291 : 2013-04-12T13:30:07

Selenium1 Result

Name : Tests : Errors : Failures : Time(s) : Time Stamp

AccountLockoutTest : 15 : 0 : 0 : 372.717 : 2013-04-04T09:42:05

On Monday, 15 April 2013 00:31:46 UTC+5:30, Vishal wrote:

Hi,

I am using the JAVA for my web-driver tests and as compared to the results with RC there is high performance issue.
I executed a single test suite with both Selenium RC on remote machine and the same suite with web-diver on same remote machine. Performance of execution is too low.

[Simon Stewart]

The main reason is that the webdriver-backed selenium aims for accuracy of emulation rather than speed. There's a lot of additional work it does to try and keep things ticking along as you'd expect which means that it performs poorly.

The best thing to do is to identify possible bottlenecks. I just wrap my CommandProcessor with one that collects timings and invocation statistics to make that easier, but there's a few obvious candidates for improvements:

1) Never, ever call "getBodyText". WebDriver attempts to be far more consistent than RC about the returned text and so performs a lot more calculations. Typically, this is used for checking to see if a certain string has appeared. Instead, find the element that should contain the text and do a "WebElement.getText" on that.

2) Avoid repeated lookups. A common pattern in RC is to wait for an element to be present, then to check an attribute, then to (perhaps) perform an additional operation on the element. This results in three element look ups. Since the same element is used each time, you can collapse this into a single element lookup and separate method calls on that. Your way is is the WebDriverWait and the ExpectedConditions.

3) Avoid XPath. Just as a general principle. FTR, webdriver uses a faster JS xpath engine than RC by default, but all the same....

4) If you're on Windows and testing with IE, avoid typing LOTS OF THINGS IN CAPITALS. The code path for handling this is particularly paranoid, so each character appears more slowly than is desirable.

Also, note that WebDriver is more conservative about when it believes a page is loaded. This leads to tests being more stable, but in the case of your simple sample programs suggests that webdriver has a speed issue.

HTH,

Simon

[Jim Evans]

Such coarse numbers are next to useless when talking about Selenium performance. There's a ton of context missing. You don't mention the browser you're using, or its version. You don't say what version of Selenium you're using.

You don't give any indication what code you're actually running either, or the structure of the HTML you're executing against. For example, RC and WebDriver have slightly different algorithms for finding the visible text of an element. WebDriver's is much more well-defined and rigorous, but that makes it computationally expensive, especially for a deeply-nested DOM. Determining the visibility of an element has similar performance characteristics.

While it's true that WebDriver can *in general* be faster than RC, as with nearly all performance benchmarks in computer science, the actual performance you'll see will depend on a very wide array of factors.