HtmlUnit and Javascript
Simon Stewart
Two things have happened recently which are making me consider
changing my stance on turning on support for javascript in the
HtmlUnitDriver: Firstly, it's been pointed out to me that it's
sufficiently advanced to allow it to run Selenium Core without muss or
fuss. Secondly, one of the HtmlUnit maintainers has been kind enough
to email me pointing out that the javascript support in HtmlUnit is
pretty capable.
My question to you is: should we enable it in the HtmlUnitDriver? I'm
keen to hear what you all have to say.
Regards,
Simon
Mirko Nasato
> My question to you is: should we enable it in the HtmlUnitDriver?
It's already fairly easy to enable JavaScript on the HtmlUnitDriver:
WebDriver driver = new HtmlUnitDriver() {
protected void newWebClient() {
super.newWebClient();
getWebClient().setJavaScriptEnabled(true);
}
};
It's true that JavaScript support in HtmlUnit is now good enough for
many cases, but it's also true that with JavaScript enabled tests run
noticeably slower. I haven't done any benchmarks, but my guess is that
on some pages the FirefoxDriver may be even faster than the
HtmlUnitDriver with JavaScript on.
That's why in practice I tend to leave JavaScript disabled by default
when using HtmlUnit, and only enable it for tests that absolutely
requires it. So, ideally, the best option would be to be able to
enable/disable JavaScript on a per test basis.
I also wonder if it would make sense to have a general option to
enable/disable JavaScript, implemented by all drivers?
Kind regards
Mirko
Simon Stewart
On Fri, Mar 7, 2008 at 2:53 PM, Mirko Nasato <mirko....@gmail.com> wrote:
> I haven't done any benchmarks, but my guess is that
> on some pages the FirefoxDriver may be even faster than the
> HtmlUnitDriver with JavaScript on.
That's interesting to hear. I'll have to put some benchmarks together
to find out what's happening. A quick comparison of the
HtmlUnitDriverTestSuite with javascript on and off shows that they run
at about the same speed. I'll enable the Javascript tests and see what
happens.
> I also wonder if it would make sense to have a general option to
> enable/disable JavaScript, implemented by all drivers?
That'd be neat, but it might not be possible for every driver --- the
safari driver does its magic by sending an apple event with a chunk of
javascript to execute. I'll do some experiments (again) to see what
happens. It's starting to sound like it would be nice to have
programmatic access to the capabilities of each driver, so you can ask
it whether it supports this or that feature.
Simon
Mirko Nasato
On 10/03/2008, Simon Stewart <simon.m...@gmail.com> wrote:
>
> On Fri, Mar 7, 2008 at 2:53 PM, Mirko Nasato <mirko....@gmail.com> wrote:
>
> > I haven't done any benchmarks, but my guess is that
> > on some pages the FirefoxDriver may be even faster than the
> > HtmlUnitDriver with JavaScript on.
>
>
> That's interesting to hear. I'll have to put some benchmarks together
> to find out what's happening. A quick comparison of the
> HtmlUnitDriverTestSuite with javascript on and off shows that they run
> at about the same speed. I'll enable the Javascript tests and see what
> happens.
>
That's probably because almost every page here imports a relatively
big JavaScript library (jQuery), and that exposes Rhino's slowness
compared to e.g. SpiderMonkey. Running a simple suite with 4 tests:
HtmlUnitDriver[js=off]: 2.0 secs
HtmlUnitDriver[js=on]: 6.8 secs
FirefoxDriver: 4.3 secs
Kind regards
Mirko
Simon Stewart
On Mon, Mar 10, 2008 at 3:01 PM, Mirko Nasato <mirko....@gmail.com> wrote:
> That's probably because almost every page here imports a relatively
> big JavaScript library (jQuery), and that exposes Rhino's slowness
> compared to e.g. SpiderMonkey. Running a simple suite with 4 tests:
>
> HtmlUnitDriver[js=off]: 2.0 secs
> HtmlUnitDriver[js=on]: 6.8 secs
> FirefoxDriver: 4.3 secs
That's an impressive difference. Thanks for taking the time to take
those measurements.
Regards,
Simon
Simon Stewart
Further to the discussion we've been having, I've been chatting with
Marc Guillemot, who's one of the HtmlUnit developers, and he says:
"The fact that Rhino isn't used in any significant browser doesn't mean
that it can't be used to simulate browser behavior and achieve a very
high compatibility even with Web 2.0 websites. We've integrated the unit
tests of different AJAX libraries in HtmlUnit's tests and a large number
of these tests already pass (currently only Sarissa is 100% completed).
Work continues and I hope that the next releases will bring 100%
compatibility with other complex AJAX libraries. Concerning WebDriver, I
guess that you could explain your users what you've written to me and
trust them to be able to make the difference between real bugs and
HtmlUnit limitations.
"HtmlUnit tries to mimic browser's behavior as good as it can. Due to the
size of the team we currently focus only on Firefox and Internet
Explorer (making small differences between 6 and 7). This means that we
expect it to behave like the browsers as much as it makes sense. Filing
a bug is really easy: provide the smallest possible code working with
the browser but not with HtmlUnit. An example of a fix from yesterday
could have been just "CollectGarbage()" which works with IE (but not
with FF). Of course patches with unit tests are even better ;-)"
In light of this response, I've updated the HtmlUnitDriver wiki
page[1] to provide a more rounded answer to the "why don't you use
javascript in the htmlunit driver?" question.
As it stands, it sounds like the major issues that we have are the
speed and a tendency (on my part) to be a little skittish about
enabling javascript in an environment which normal users will never
see. Does that sounds about right? And how do people feel about
Mirko's suggestion of having javascript capability be a configurable
option?
Regards,
Simon
Marc Guillemot
> ...
> big JavaScript library (jQuery), and that exposes Rhino's slowness
> compared to e.g. SpiderMonkey. Running a simple suite with 4 tests:
>
> HtmlUnitDriver[js=off]: 2.0 secs
> HtmlUnitDriver[js=on]: 6.8 secs
> FirefoxDriver: 4.3 secs
faster than Firefox to run the test suites of the AJAX libraries that
we have included in the build (for instance JQuery). I'm not 100% sure
and this would be an interesting comparison.
I've looked quickly at the WebDriver code that drives HtmlUnit and
I've seen that it calls Page.initialize() whereas it shouldn't as it
is done automatically. The consequence is probably that all js stuff
like onload is run a second time. Additionally this shows that our
JavaDoc is not good enough ;-).
Cheers,
Marc.
--
Blog: http://mguillem.wordpress.com
Simon Stewart
On Mon, Mar 17, 2008 at 2:53 PM, Marc Guillemot <mguil...@yahoo.fr> wrote:
> I've looked quickly at the WebDriver code that drives HtmlUnit and
> I've seen that it calls Page.initialize() whereas it shouldn't as it
> is done automatically. The consequence is probably that all js stuff
> like onload is run a second time. Additionally this shows that our
> JavaDoc is not good enough ;-).
Either that or it shows that I don't read them often enough :) I see
that I'm using some deprecated methods too, which I should fix.
Thanks for pointing out the problem; I've just checked in a fix for
the Page.initialize() problem, so hopefully it'll perform a little
better now.
Regards,
Simon
Mirko Nasato
On 17/03/2008, Marc Guillemot <mguil...@yahoo.fr> wrote:
>
> On Mar 10, 4:01 pm, "Mirko Nasato" <mirko.nas...@gmail.com> wrote:
>
> > Running a simple suite with 4 tests:
> >
> > HtmlUnitDriver[js=off]: 2.0 secs
> > HtmlUnitDriver[js=on]: 6.8 secs
> > FirefoxDriver: 4.3 secs
>
>
> I'm a bit surprised because as far as I can remember, HtmlUnit is
> faster than Firefox to run the test suites of the AJAX libraries that
> we have included in the build (for instance JQuery). I'm not 100% sure
> and this would be an interesting comparison.
>
Well don't take the figures above as a proper benchmark of course,
it's just a quick test I ran on the project I'm working on at the
moment.
On 17/03/2008, Simon Stewart <simon.m...@gmail.com> wrote:
>
> On Mon, Mar 17, 2008 at 2:53 PM, Marc Guillemot <mguil...@yahoo.fr> wrote:
>
> > I've looked quickly at the WebDriver code that drives HtmlUnit and
> > I've seen that it calls Page.initialize() whereas it shouldn't as it
> > is done automatically. The consequence is probably that all js stuff
> > like onload is run a second time. Additionally this shows that our
> > JavaDoc is not good enough ;-).
>
> Thanks for pointing out the problem; I've just checked in a fix for
> the Page.initialize() problem, so hopefully it'll perform a little
> better now.
>
It doesn't make much difference here honestly, running the same tests
as above. But it may well be that there's something weird with my
current setup. I'll investigate when I have some more time...
Kind regards
Mirko
John Lockhart
re turning on javascript support in HTMLUnit:
The current packaged version of webdriver includes htmlunit 1.11
right? The javascript support has improved massively since then in
terms of completeness and speed I believe. I'm surprised no one has
mentioned what version they are testing with above (especially Marc!)
- am I missing something?
My vote is "yes" by the way, simply because there are other ways I can
work round speed issues e.g. running from multiple machines. Would
like an option to easily turn it on and off though, and ideally for
all browsers.
cheers,
John
Michael Tamm
Because there was another "yes" vote, I vote with "no" for enabling JavaScript in the HtmlUnitDriver.
JavaScript should be disabled by default for the HtmlUnitDriver, because:
- Even if HtmlUnit very closly mimics Firefox/InternetExplorer,
I don't think it will show identical behaviour in every case -
so Simons argument remains: don't let us hunt bugs (if our tests fail) which don't exists.
- Tests will run faster.
But I absolutley agree, that it would be nice to easily enable JavaScript for the HtmlUnitDriver.
And furthermore I agree that it would ne nice to add the ability to turn JavaScript on/off the the WebDriver core API.
Just my 2 cents,
Michael Tamm
Marc Guillemot
I think that this is a lack of coherence to have it disabled by default
only in the HtmlUnitDriver: how can you expect that your tests to behave
the same using different drivers with such a fundamental difference.
HtmlUnit doesn't (yet ;-)) simulate 100% the behavior of the simulated
browsers, but it's surprising to use it only partially (and errors can
occur with js disabled too due to different parsing of malformed html).
In fact JS support is not the only missing feature in the HtmlUnitDriver
which should for instance surely allow to specify the browser to simulate.
FYI:
- we have some plan to *prove* that HtmlUnit JS execution path is
similar to FF's one (with the limitation of the JS that we currently
support)
- HtmlUnit 2.0 is out... but we have some performance problems in CSS
processing, therefore WebDriver is best advised to wait a bit to upgrade
(but could already fix the usage of deprecated API)
Cheers,
Marc.
--
Blog: http://mguillem.wordpress.com
Mirko Nasato
The current packaged version of webdriver includes htmlunit 1.11
right? The javascript support has improved massively since then in
terms of completeness and speed I believe. I'm surprised no one has
mentioned what version they are testing with above (especially Marc!)
- am I missing something?
I was always using HTMLUnit 1.14 in my tests, even when the WebDriver's had 1.11 in its svn repo (now it's 1.14 in there too).
(We use Maven 2 as our build tool, so I packaged WebDriver for our internal Maven repository and declared webdriver-htmlunit to depend on htmlunit-1.14. I can publish the Maven stuff somewhere if other people are interested.)
Kind regards
Mirko
Mirko Nasato
I think that this is a lack of coherence to have it disabled by default
only in the HtmlUnitDriver:
I agree that the API is inconsistent on this point, and I'd like an explicit option to turn javascript on/off.
OTOH in practice I would almost always turn javascript off on the HtmlUnitDriver anyway.
how can you expect that your tests to behave
the same using different drivers with such a fundamental difference.
Well, I don't. ;-)
Currently, we're using the FirefoxDriver to simulate real users, and the HtmlUnitDriver to simulate search spiders. We want to test that our pages are navigable even by bots that don't understand javascript, but we don't expect all the features to work in that case.
Of course, different projects may have different requirements. But it's a real world example.
Kind regards
Mirko
Michael Tamm
Currently, we're using the FirefoxDriver to simulate real users, and the HtmlUnitDriver to simulate search spiders. We want to test that our pages are navigable even by bots that don't understand javascript, but we don't expect all the features to work in that case.
Of course, different projects may have different requirements. But it's a real world example.
But I buy into the argumen that it is a lack of coherence to have JavaScript disabled by default only in the HtmlUnitDriver.
Therefore I think we really need an API to easily turn on/off JavaScript in any WebDriver implementation.
Greetings, Michael
John and Pip
John and Pip
Whoops sorry - that email was not meant for this list at all - replied to wrong post - sorry!
To try to make conversation so it isn't a complete waste of time, I was just trying out webdriver with Fitnesse, both of which I love conceptually but as a non-java-programmer had never really used.
To my surprise I was able to in an hour or two get Fitnesse driving WedDriver driving HTMLUnit driving a trivially simple google search test. That seems pretty damn cool to me considering I really can't write java code!
cheers,
John
Simon Stewart
To summarise:
* It'd be great to support to Javascript in the HtmlUnitDriver.
* It would desirable to switch Javascript support on and off at runtime
* It's considered inconsistent for some drivers to not support Javascript
Taking each of these in turn:
I've created a new issue for tracking progress on the JS-enabled
HtmlUnitDriver: http://code.google.com/p/webdriver/issues/detail?id=72
Some of the browsers (Safari, for example) explicitly require JS
support to be enabled in order to work. Having the ability to turn JS
support off and on for at runtime could therefore be problematic :) My
preference is currently to leave this out of the core WebDriver
interface and to expose it through another interface which drivers can
chose to implement (though please see below)
Finally, I don't see whether or not a driver supports Javascript as
being inconsistent. WebDriver attempts to provide a uniform interface
over the browser implementations, but each underlying browser has its
own quirks and capabilities. As an example, it's easy to imagine a
driver which sits on something that doesn't render content and doesn't
even attempt to pretend to (say something based on "curl" or "wget",
or a theoretical Python implementation written against WSGI) --- would
it be inconsistent if this driver doesn't report element positioning?
Or for it to be unable to execute javascript?
To me, it sounds like some sort of API to query the capabilities of a
driver implementation would be useful. That way, you can query a given
driver to find out whether your expectations and assertions are
reasonable to make. Does that sound reasonable? And is that something
you would use?
Regards,
Simon
Mirko Nasato
Some of the browsers (Safari, for example) explicitly require JS
support to be enabled in order to work. Having the ability to turn JS
support off and on for at runtime could therefore be problematic :) My
preference is currently to leave this out of the core WebDriver
interface and to expose it through another interface which drivers can
chose to implement (though please see below)
My preference would be to add that feature anyway, and mark the fact that Safari doesn't support it as a "known limitation" of that driver.
I'd rather hope somebody comes up with a creative way to implement that feature in Safari, than adapt to the lowest common denominator.
Mirko