Re: Why Internet WebDriver is so slow!!!
Michael B
The subject should say “Why Internet Explorer WebDriver is so slow!!!”
I have a selenium script which gets table content from dynamically populated table with 10 rows and 4 columns. The table has pagination which allows going to the next page, each time you go to the next page new data is populated lazily from the server using ajax. The script is working fine but it takes 56 seconds to complete collection of 10 pages using Internet Explorer driver. Using Fire Fox and Chrome drivers it takes only 15 seconds.
I’m working with Internet Explorer automation for more than 10 years and I created a lot of automation tools and frameworks using IE automation and I never experienced such a bad performance, so I decided to build new WebDriver as a proof of concept. I t took me a day to create IEWebDriver using Jacob (java to com bridge) open source library I didn’t implemented selenium wire protocol so my driver extends WebDriver and WebElement directly (no RemoteWebDriver) no XPath and no CSS Locators (I’m not using this staff in my scripts anyway). After driver was completed I tried the mentioned script without any modifications and the performance was unbelievable, instead of 56 seconds it completed successfully in 13 seconds in 2 second less than Fire Fox and Chrome drivers.
Unfortunately I’m not a c++ developer and I can’t review the current Internet Explorer web driver code but it works very slow, maybe it using xpath to find the elements or any other not native to IE and COM automation technologies I don’t know but here is proof that it can ran much faster.
Michael B
Daniel Wagner-Hall
> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To post to this group, send email to selenium-...@googlegroups.com.
> To unsubscribe from this group, send email to
> selenium-develo...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/selenium-developers?hl=en.
>
Jason Leyba
- Share your test script that is running slowly?
- Share your IEWebDriver implementation that is faster?
Yuri Kl
exact though,
and I can't compare with a run on Firefox because our application
won't run on anything but IE.
However, the driver is visibly and rather embarrassingly slow. Until
now I have just assumed it was the fault of my WebDriver based
framework, and put
it off because speed has not become critical for me yet.
I will try to re-create the issue.
On Jun 25, 1:39 am, Jason Leyba <jmle...@gmail.com> wrote:
> Would you care to
>
> 1. Share your test script that is running slowly?
> 2. Share your IEWebDriver implementation that is faster?
> Honestly, right now this is just a rant without anything we can act upon.
>
> -- Jason
>
>
>
>
>
>
>
> > The subject should say *“Why Internet Explorer WebDriver is so slow!!!”*
Michael B
Attached is an Eclipse project with a test (Selenium jar is not attached so add it by your self). Run this test as java application using Test class and remove comments from the specific webdriver to run this test with different webdrivers. Execution time is measured and presented at the end of the test. This test composed from several classes which are part of my new PrimeFaces framework. Test maybe a little bit complicated but I’ll try to explain what is going on: Currently there are several Java enterprise technologies which are a set of standards for building java enterprise solutions; JSF (Java Server Faces) is one of those technologies for creating web User Interfaces. JSF allows third party companies to provide JSF UI component libraries, there are several open source projects that provide those libraries, and one of the most successful JSF component libraries is PrimeFaces. Almost every project that I worked for in recent years where using JSF and PrimeFaces.
JSF components are very complicated and they are using a lot of JavaScript libraries and ajax interactions but they are generated by the framework in a unified way. My idea was to create Selenium based framework for PrimeFaces components to encapsulate test patterns in single reusable library so every project that are using Primefaces can be easily tested and other test automation business frameworks can reside on the top of this or other similar UI libraries.
The Included test is connecting to PrimeFaces demo site and interacting with DataTable component, this test is executing very slow using current InternetExplorer implementation. Speed is very critical for DataTable component as you need to retrieve data from several table pages and then validate it against database very quickly. Majority of our clients are using IE browser and that why I decided to create my own version of Internet Explorer driver. My IEDriver is working faster than any other real browser Selenium webdriver (IE, Firefox and Chrome) and you can experience the difference by running this test. The driver is not complete so there is still a lot of work to be done, do not expect full featured webdriver.
Jim Evans
greatly help me to understand the kinds of operations you're
attempting that you are finding non-performant in the existing IE
driver. However, in order to accurately compare your implementation to
the current C++ implementation, we really need the source code of your
WebDriver implementation as well. Would it be to much to ask you to
provide it?
--Jim
On Jun 26, 4:54 am, Michael B <braim...@gmail.com> wrote:
> Attached is an Eclipse project with a test (Selenium jar is not attached so
> add it by your self). Run this test as java application using Test class and
> remove comments from the specific webdriver to run this test with different
> webdrivers. Execution time is measured and presented at the end of the test.
> This test composed from several classes which are part of my new PrimeFaces
> framework. Test maybe a little bit complicated but I’ll try to explain what
> is going on: Currently there are several Java enterprise technologies which
> are a set of standards for building java enterprise solutions; JSF (Java
> Server Faces) is one of those technologies for creating web User Interfaces.
> JSF allows third party companies to provide JSF UI component libraries,
> there are several open source projects that provide those libraries, and one
> of the most successful JSF component libraries is PrimeFaces. Almost every
> project that I worked for in recent years where using JSF and PrimeFaces.
>
> 326KViewDownload
>
> - Show quoted text -
Michael B
Dear Jim,
My code is very simple and it’s just using Jacob (JAVA-COM Bridge) open source project library. I’m just invoking MS Internet Explorer COM Automation API behind the JACOB’s ActiveXComponent class.
If you would like to correlate your c++ code to my IE webdriver implementation it’s not the right way to go (my source code is plain Java), you should correlate your code to Jacob’s JNI c++ implementation because this is a source with performance issues.
David Burns
David Burns
URL: http://www.theautomatedtester.co.uk/
Simon Stewart
check the state of an element before interactions. One of these checks
(for visibility) is actually quite expensive, but not making it leads
to a lot of hard to trace errors. I'll take a look at your code, but
that may well be part of the problem.
For the record, the very first version of anything that resembled the
IE driver was based on JacobIE.
Simon
Michael Braiman
When I submitted my proof of concept project I explicitly said what technologies I was using to try and solve IE slowness. There are two established open source projects that I’ve been using for years, JACOB and JNA. Those two projects are the building blocks for all my and many others COM and WIN32API solutions when you don’t want to use C++ code in your Java projects to interact with native OS libraries. Selenium is also using JNA open source in some extent, it is just invoking the start and stop method for remote webdriver server which is written in c++. This remote webdriver internally suppose to use COM bindings to automate IE browser through well known and published by Microsoft, COM Automation API. I’m using the same technologies but without writing any C++ code using JACOB open source project. The JACOB project started in 1999 and is being actively used by thousands of developers worldwide.
Simon Stewart
There were two main problems:
1) Stability --- any unhandled exceptions in the code that JNA called
could wipe out the JVM
2) Ease of development for multiple languages
For a while we were supporting the IE driver using JNA or similar
technologies across multiple different languages. The cost of
maintenance was unacceptably high, and the level of user satisfaction
when the JVM went out (for java users) was also unacceptably high. By
rewriting to use the normal remote wire protocol, we've reduced the
pain of both of these.
FWIW, the reason we abandoned using JacobIE was because it was
painfully slow in use beyond anything but the trivial case.
Simon
Michael Braiman
Simon,
My IEDriver dose have this mechanism to verify if element is visible, you can try to invoke findElements() method and see by yourself that only visible components are returned by this function. And it done using COM without any javascript libraries injections or invocations.
Now for JacobIE it has different level of abstraction. I’m using JACOB library directly in most simple and efficient way.
Michael Braiman
Simon,
I didn’t want to post those numbers but if you run my test you will see performance boost by 10 times (Current IE web driver implementation takes 99 second to execute this test script and my IEDriver takes only 10 seconds). I agree that native c++ dll implementation will be much stable and theoretically faster but not in factor of 10.
Simon Stewart
* You call "getText" in a loop. The algorithm we use to normalize the
text values across browsers makes a ton of JS calls to determine
whether elements are visible, and we don't use caching. This means
that getting the entire text for a table in a large DOM can be very
time consuming.
Suggestion: replace "getText" with a call to the JavascriptExcecutor
and execute the JS "return arguments[0].innerText", passing in the
element who's text you want as the argument.
* There's a lot of nested iterations of webelements going on. As a
rule of thumb, each and every call to the webdriver APIs is an RPC. We
used to be in-process, but we're now out-of-process. Most modern OSs
make optimizations across localhost, but even if that is the case,
there's extra traffic to process.
Suggestion: Create a wrapper around Tables (as you've done) and look
for bottlenecks in performance. Address these by carefully targeted
enhancements to efficiency.
As an example of this second point, have a look at the Select class.
This started off with a very naive implementation, but now works very
well even with large lists.
My understanding is that this really is just a problem with IE, and
that the other drivers perform acceptably rapidly? If so, it points to
a problem tied to the JS engine. Have you tried the same tests with IE
9 on a machine where Chakra works in the optimized mode? From memory,
this will be IE9 on 32bit Win7.
Simon
Jim Evans
The manipulation of IE is almost always done via COM calls, no matter
what the technology used. When you call JavaScript code in IE, you're
manipulating IE via COM, IE just hides that implementation from you.
The current IE driver uses COM to manipulate IE, but does so from C++
code. Your implementation manipulates IE via COM, but uses a Java-to-
COM bridge to do so. I want to see the code of your implementation to
see if it's semantically equivalent to the existing IE driver. Simply
invoking the methods on the IHTMLDocument2 and IHTMLElement interfaces
often seems like it's enough to get the driver to work, but the
contract of the WebDriver API implies more functionality than that.
Comparing your Java code to the existing driver's combination of C++
and JavaScript code is exactly the right way to go.
I hear your frustration at the performance of the existing IE driver.
It sounds like you're pretty proud of the work you've done to improve
the experience. If you've mananged to come up with a better solution,
I'd love for all Selenium users to be able to take advantage of it.
But in order to do that, it needs to be (1) implemented in a language-
neutral way and (2) semantically equivalent to the existing driver
implementation.
--Jim
On Jun 26, 11:44 pm, Michael B <braim...@gmail.com> wrote:
> Dear Jim,
>
> My code is very simple and it’s just using Jacob (JAVA-COM Bridge) open
> source project library. I’m just invoking MS Internet Explorer COM
> Automation API behind the JACOB’s ActiveXComponent class.
>
> If you would like to correlate your c++ code to my IE webdriver
> implementation it’s not the right way to go (my source code is plain Java),
> you should correlate your code to Jacob’s JNI c++ implementation because
> this is a source with performance issues.
>
Simon Stewart
directly, rather than posting code to the list for everyone to see.
It's also pretty easy to hook a new driver into the test suites we
have for webdriver, so you can find out how precisely you're meeting
the expected behaviours of a "WebDriver"
Simon
Michael Braiman
Hey Simon,
If you look carefully in my test I’m caching the table so I’m not trying to find all the elements in the loop I’m just calling getText method of cached web element reference by calling COM method getText. There is no need to use JS it’s very slow compared to the direct API call.
The performance bottleneck was findElement method and not getText, my first attempt was using JS and I didn’t cached the table and the test was executed 10 times slower than caching the table solution so I boosted performance by caching the table and not calling findElement method again and again. The slowest point in my test is when I fill my table cache. (I thing you’re not expecting that will execute findElement function using JS ;-))
Paul Hammant
Michael Braiman
I would like to stress one point. I like Selenium project very much and I appreciate the work that was done to bring this project to this point. In my current position I decided to use Selenium as primary tool for test automation by throwing away other solutions that where in plaice which cost thousands of dollars to the company I work for (the company is big Canadian financial corporation which doesn’t care to throw away tens of thousands of dollars for test automation tools). It took me a lot of efforts to convince management that Selenium is the way to go. Now if there are any issues with Selenium I’m responsible for those problems directly and not Selenium community, so I’m taking this project very seriously and I would like to see Selenium as the next standard in the test automation field.
When I first met Selenium webdriver project I got impression that Selenium is using native technologies to talk and manipulate with web browsers and that what caught my interest in this project as I didn’t liked the Selenium 1 way of injecting javascript in to application under the test. But now I understand that in the new WebDriver solution you are still using Javascript which is not a native way to automate IE. I’m against javascript execution on application under the test as you are modifying behavior of your business application. In my proof of concept web driver I’m not using single line of javascript code and currently I’m using JACOB library only (no JNA yet but I will have to use it for mouse and keyboard interaction which is not implemented yet). JACOB project is currently used in our production environment and it’s very stable and agile. It is currently used as temporary solution to wrap legacy COM services to be used with Java, but this temporary solution is last already for 6 years.
I’m not trying to compete with Selenium I’m trying to help! I can’t post my unfinished web driver right now because of several reasons; first of all I’m get paid for this solution but I don’t think that my company will be against contributing my work to open source community once it will be fully implemented, secondly my solutions are based on technologies that are not inline with current Selenium solution, as I understand from Simon there was attempt to use those technologies in the past but it created more problems than benefits.
Simon Stewart
Thanks for setting me straight, and thanks for the example code. Guess
I should read things a little more closely :)
It's going to be a while until I get a chance to dig into this
properly, but please feel free to give me a gentle reminder if I go
completely quiet. It's obviously important to me to make webdriver as
efficient as possible, so having these test cases that highlight
problems is very useful indeed.
The way that we implement "getText" in webdriver is via one of our
Automation Atoms, which are JS fragments. These are minified and used
by the IE driver behind the scenes, but despite this the algorithm is
pretty complex, and does lots of visibility checks (which are
expensive). This means that what looks like a single method call in
fact causes a whole heap of JS to execute. I guess you've done
something a little different with your jacob-based approach.
Thanks,
Simon
Simon Stewart
I totally understand the reasoning about not making the code public.
It's a fairly common conundrum, and I'd always suggest erring on the
side of caution (just as you're doing).
The other thing that I have a lot of sympathy for is that the overall
design of the current webdriver implementations looks nothing like the
what a person would imagine writing if starting from a clean slate.
Although it misses out a lot of the history, an explanation of why
things are the way they are is available in the recently released
"Architecture of Open Source Applications" book[1]. The whole thing is
online for you to read[2]. The short form: the current design is a
balance between ease and speed of development, consistency and
performance.
Though apparently that performance piece needs some work.... :)
Simon
[1] http://www.aosabook.org
[2] Though if you buy a copy then Amnesty International get the royalties.
Michael Braiman
Thanks Simon for the link. Unfortunately Selenium webdriver project documentations are lacking the whole design approach and human being intend to build the whole picture from the small bits and pieces of information they could find without going further in to the research. For now I have to be bound to my webdriver solution because of security constrains of our financial business applications, which are “suppose” to block arbitrary javascript execution.
Simon Stewart
back including the chapter in our main docs for a while longer for
hopefully obvious reasons. I know that the docs team are working
furiously to improve the quality of what's there, and I'm sure that
they'd love to hear from people about what needs to be improved. It
sounds like the piece you were missing was a detailed overview of the
architecture, right?
I'm interested in how your site is blocking arbitrary JS execution.
Because of the way that we inject code, we appear to come from the
main page rather than from a different site, and it sounds as if until
you started on your own IE driver things were working properly (albeit
slowly)
Regards,
Simon
Michael Braiman
During the process of incorporating Selenium for automation test I had spoken to security architect and he said that there are some mechanism to block javascript injections in our system but I’m not sure it was implemented at all that why I put quotes around “SUPPOSE” in my previous post. I tried Selenium for most of our applications as proof of concept and everything works fine.
Michael Braiman
Hi Simon,
Here is a list of things that are required in documentation:
1) Architecture - for sure is needed at least some overview about JS atoms.
2) Summary that lists full set of features. (This is very important as I can compare those features with other automation solutions and present them to stakeholders so they can make decisions)
3) And good Examples for API usage of course.
Selenium 2 is currently causing a lot of turbulences in the QA community but not a lot of QA people know what Selenium 2 is. One of the major misconceptions is that Selenium 2 is a tool and not an API. One of the QA managers asked me to send Selenium 2 test reports and graphs to see the test case coverage. It took me a while to explain that it is an API and not a tool and why Selenium (TestNG) report was very dull and not graphical enough. ;-)
Another point is that a lot of companies trying to block arbitrary JS execution and as I understand it’s done through some kind of HTML page integrity check which is done as JS script on the client side to track additions and modifications of script elements on the page. I don’t think it’s effective but how Selenium 2 will behave in this situation I don’t know because I don’t have examples of this technology.
What are your thoughts?