Native events: I need your help

[Simon Stewart]

Hi everyone,

If you're using Linux and are feeling adventurous, please continue
reading. If you're not, this probably won't be very interesting for
you :)

We recently landed native typing for Firefox on Linux into trunk. This
means that rather than synthesizing typing by firing events at the DOM
level, we instead send events via X's event system. This means that we
more accurately emulate the behaviour of a Real User. In order to make
this work without requiring the firefox window to have focus, we had
to jump through some hoops.

Now, native events work fine on my test machine, but I'd really
appreciate a slightly wider audience trying it out and providing
feedback before we do an official release. So, if you're running any
variant of Linux, and you're using the Firefox driver, and you're
willing to try something out that may not work, could you please
download the (experimental, possibly buggy, may not work) version of
the firefox driver from:

http://selenium.googlecode.com/files/experimental-webdriver-firefox-7075.zip

And try out a quick typing test? I'd love to know whether or not it
works for you, and which distro of Linux you're using (a stack trace
would be nice too, if the firefox instance fails to start _at all_)

To re-iterate: this is _not_ an official release!

Many thanks,

Simon

[Marc Guillemot]

Hi Simon,

can you provide a test that you would like to be executed?

Is support for Linux 64bits included?

Cheers,
Marc.

Simon Stewart a écrit :

[Simon Stewart]

Hi Marc,

a simple test would be (using JUnit 4):

@Test public void typingShouldWork() {
WebDriver driver = new FirefoxDriver();
driver.get("http://www.google.com");
WebElement q = driver.findElement(
By.name("q"));
q.sendKeys("cheese");

Assert.assertEquals("cheese", q.getValue());
}

The driver should work on 64bit Linux versions (I've tried it with a
64 bit Ubuntu install)

Cheers,

Simon

[Marc Guillemot]

Hi Simon,

Firefox doesn't even start :-(
------------------
org.openqa.selenium.WebDriverException:
org.openqa.selenium.WebDriverException: Failed to connect to binary
FirefoxBinary(/home/marc/bin/firefox3) on port 7055; process output
follows:
Failed to dlopen /usr/lib/libX11.so
dlerror says: /usr/lib/libX11.so: Ne peut ouvrir le fichier d'objet
partagé: Aucun fichier ou dossier de ce type
�Failed to dlopen /usr/lib/libX11.so
dlerror says: /usr/lib/libX11.so: Ne peut ouvrir le fichier d'objet
partagé: Aucun fichier ou dossier de ce type
�
System info: os.name: 'Linux', os.arch: 'i386', os.version:
'2.6.28-14-generic', java.version: '1.6.0_13'
Driver info: driver.version: firefox
System info: os.name: 'Linux', os.arch: 'i386', os.version:
'2.6.28-14-generic', java.version: '1.6.0_13'
Driver info: driver.version: firefox
at
org.openqa.selenium.firefox.internal.ExtensionConnectionFactory.connectTo(ExtensionConnectionFactory.java:46)
at
org.openqa.selenium.firefox.FirefoxDriver.connectTo(FirefoxDriver.java:128)
at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:118)
at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:85)
at TmpTest.typingShouldWork(TmpTest.java:11)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
at
org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
at
org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
at
org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
at
org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
at
org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:66)
at
org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
at
org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
at
org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
at org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:45)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Caused by: org.openqa.selenium.WebDriverException: Failed to connect to
binary FirefoxBinary(/home/marc/bin/firefox3) on port 7055; process
output follows:
Failed to dlopen /usr/lib/libX11.so
dlerror says: /usr/lib/libX11.so: Ne peut ouvrir le fichier d'objet
partagé: Aucun fichier ou dossier de ce type
�Failed to dlopen /usr/lib/libX11.so
dlerror says: /usr/lib/libX11.so: Ne peut ouvrir le fichier d'objet
partagé: Aucun fichier ou dossier de ce type
�
System info: os.name: 'Linux', os.arch: 'i386', os.version:
'2.6.28-14-generic', java.version: '1.6.0_13'
Driver info: driver.version: firefox
at
org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.connectToBrowser(NewProfileExtensionConnection.java:60)
at
org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.<init>(NewProfileExtensionConnection.java:49)
at
org.openqa.selenium.firefox.internal.ExtensionConnectionFactory.connectTo(ExtensionConnectionFactory.java:44)
... 24 more
Caused by: org.openqa.selenium.firefox.NotConnectedException: Failed to
start up socket within 45000
at
org.openqa.selenium.firefox.internal.AbstractExtensionConnection.connectToBrowser(AbstractExtensionConnection.java:143)
at
org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.connectToBrowser(NewProfileExtensionConnection.java:58)
... 26 more
------------------

I'm running Ubuntu 9.04 64bits.

[Valdis Rigdon]

Same here. It fails to start Firefox. I'm running Ubuntu 9.04 64bit in a VM.

uname -a
Linux vrigdon-desktop 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25
01:19:55 UTC 2009 x86_64 GNU/Linux

Stack trace:

org.openqa.selenium.WebDriverException:
org.openqa.selenium.WebDriverException: Failed to connect to binary

FirefoxBinary(/usr/bin/firefox) on port 7055; process output follows:
Failed to dlopen /usr/lib/libX11.so
dlerror says: /usr/lib/libX11.so: cannot open shared object file: No
such file or directory
�Failed to dlopen /usr/lib/libX11.so
dlerror says: /usr/lib/libX11.so: cannot open shared object file: No
such file or directory
�
System info: os.name: 'Linux', os.arch: 'amd64', os.version:
'2.6.28-14-generic', java.version: '1.6.0_14'
Driver info: driver.version: firefox
System info: os.name: 'Linux', os.arch: 'amd64', os.version:
'2.6.28-14-generic', java.version: '1.6.0_14'

Driver info: driver.version: firefox
at org.openqa.selenium.firefox.internal.ExtensionConnectionFactory.connectTo(ExtensionConnectionFactory.java:46)
at org.openqa.selenium.firefox.FirefoxDriver.connectTo(FirefoxDriver.java:128)
at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:118)
at org.openqa.selenium.firefox.FirefoxDriver.<init>(FirefoxDriver.java:85)

at test.Test.typingShouldWork(Test.java:14)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Caused by: org.openqa.selenium.WebDriverException: Failed to connect

to binary FirefoxBinary(/usr/bin/firefox) on port 7055; process output

follows:
Failed to dlopen /usr/lib/libX11.so

dlerror says: /usr/lib/libX11.so: cannot open shared object file: No
such file or directory
�Failed to dlopen /usr/lib/libX11.so
dlerror says: /usr/lib/libX11.so: cannot open shared object file: No
such file or directory
�
System info: os.name: 'Linux', os.arch: 'amd64', os.version:
'2.6.28-14-generic', java.version: '1.6.0_14'

Driver info: driver.version: firefox
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.connectToBrowser(NewProfileExtensionConnection.java:60)
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.<init>(NewProfileExtensionConnection.java:49)
at org.openqa.selenium.firefox.internal.ExtensionConnectionFactory.connectTo(ExtensionConnectionFactory.java:44)

... 28 more

Caused by: org.openqa.selenium.firefox.NotConnectedException: Failed
to start up socket within 45000
at org.openqa.selenium.firefox.internal.AbstractExtensionConnection.connectToBrowser(AbstractExtensionConnection.java:143)
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.connectToBrowser(NewProfileExtensionConnection.java:58)

... 30 more

[Simon Stewart]

Hi Marc,

Which version of Linux are you using? And is it 32 or 64 bit? Looks 32
bit to me from the stack trace, but I could be wrong.

Simon

[Marc Guillemot]

Hi Simon,

next time, I won't put any info below the stacktrace ;-)

[Simon Stewart]

My apologies :)

Simon

[Marc Guillemot]

I've just seen that I was running the test with a 32bits JVM... but the
same exception occurs with a 64bits JVM :-(

[Simon Stewart]

It's certainly been considered. At some point, we'll break out a
proper API for this user interaction stuff. I see three layers of
event models:

1) Synthesized on the DOM: works everywhere, fiddly to get right and
is distinctly limited. Machine can be used for something else as tests
run.

2) Native events to window handles: easier to get events right, but
technically challenging and still not 100% perfect (I can go into
details, if people want, but it's deeply tedious) Machine can be used
for something else as tests run.

3) Native events to front-most window: less challenging than native
events to window handles, but demands focus and is far more fragile
because the machine must be dedicated to a single test run.

Type 3 events would/could be handled using the wonder of VNC.

It'd be nice to be able to flip between these various modes, as
needed. For now, it's possible to switch between 1 and 2 with the
FirefoxDriver like so:

FirefoxProfile profile = new FirefoxProfile();
profile.setEnableNativeEvents(false);
new FirefoxDriver(profile);

Once we've tidied up the event generation in Selenium Core and broken
it out into a module, I'm planning on using that as the basis for a
synthesized interaction library.

Cheers,

Simon

On Thu, Aug 13, 2009 at 4:37 PM, Patrick Lightbody<plig...@gmail.com> wrote:
>
> I'd be happy to help out! One question though: have we ever consudered
> using VNC as a way to send native cliks and mouse movement? Not sure
> what the downsides are but i'm sure there are many. The upside is it's
> cross platform :)
>
> Sent from my iPhone
>
> On Aug 13, 2009, at 3:38 AM, Simon Stewart <simon.m...@gmail.com>

[Eran]

I'm looking into the problem of running it on Ubuntu right now - it
looks like a distro-specific issue.
It should work with the official 32-bit Firefox binaries (but note
that you will need to install some extra packages for 32-bit
compatibility on a 64 bit system), if you're feeling extra
adventurous (of course, we'll appreciate any feedback if you do
decide to go ahead and try it). Remember to specify the path to the
non-default Firefox using -Dwebdriver.firefox.bin=...

Eran

> On Thu, Aug 13, 2009 at 4:37 PM, Patrick Lightbody<pligh...@gmail.com> wrote:
>
> > I'd be happy to help out! One question though: have we ever consudered
> > using VNC as a way to send native cliks and mouse movement? Not sure
> > what the downsides are but i'm sure there are many. The upside is it's
> > cross platform :)
>
> > Sent from my iPhone
>

> > On Aug 13, 2009, at 3:38 AM, Simon Stewart <simon.m.stew...@gmail.com>

> > wrote:
>
> >> Hi everyone,
>
> >> If you're using Linux and are feeling adventurous, please continue
> >> reading. If you're not, this probably won't be very interesting for
> >> you :)
>
> >> We recently landed native typing for Firefox on Linux into trunk. This
> >> means that rather than synthesizing typing by firing events at the DOM
> >> level, we instead send events via X's event system. This means that we
> >> more accurately emulate the behaviour of a Real User. In order to make
> >> this work without requiring the firefox window to have focus, we had
> >> to jump through some hoops.
>
> >> Now, native events work fine on my test machine, but I'd really
> >> appreciate a slightly wider audience trying it out and providing
> >> feedback before we do an official release. So, if you're running any
> >> variant of Linux, and you're using the Firefox driver, and you're
> >> willing to try something out that may not work, could you please
> >> download the (experimental, possibly buggy, may not work) version of
> >> the firefox driver from:
>

> >>http://selenium.googlecode.com/files/experimental-webdriver-firefox-7...

[Kamil Dworakowski]

The zip file contains three jar files. I am using the python bindings.

Do I understand correctly that native events are implemented on the
level of the java firefox driver? That would mean that to use them
from the python bindings I would have to go through the remote driver,
right?

On Aug 13, 11:38 am, Simon Stewart <simon.m.stew...@gmail.com> wrote:

> Hi everyone,
>
> If you're using Linux and are feeling adventurous, please continue
> reading. If you're not, this probably won't be very interesting for
> you :)
>
> We recently landed native typing for Firefox on Linux into trunk. This
> means that rather than synthesizing typing by firing events at the DOM
> level, we instead send events via X's event system. This means that we
> more accurately emulate the behaviour of a Real User. In order to make
> this work without requiring the firefox window to have focus, we had
> to jump through some hoops.
>
> Now, native events work fine on my test machine, but I'd really
> appreciate a slightly wider audience trying it out and providing
> feedback before we do an official release. So, if you're running any
> variant of Linux, and you're using the Firefox driver, and you're
> willing to try something out that may not work, could you please
> download the (experimental, possibly buggy, may not work) version of
> the firefox driver from:
>

> http://selenium.googlecode.com/files/experimental-webdriver-firefox-7...

[Simon Stewart]

Right now, the Java bindings are the most commonly used, which is why
I put out the JARs. When we're happy with them, we'll push the
necessary pieces into the official python bindings. That is, this
version of the firefox driver is just so that we can get some
feedback. Having said that...

If you're really keen on playing with it, then you can try the
following. Contained in the "webdriver-firefox.jar" there's a file
called "webdriver-extension.zip". This is the webdriver firefox
extension (as the name implies). There are also two files
"x_ignore_nofocus.so" in the firefox jar. Extract both of those to
separate directories, and add those directories to your
LD_LIBRARY_PATH. Finally, modify your LD_PRELAD variable to be
"x_ignore_nofocus.so"

Yeah, it's a little convoluted: that's why we'll be wrapping it up in
the main python driver :)

Cheers,

Simon

[Cameron Braid]

The code below woked fine for me on ubuntu

$ lsb_release  -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 9.04
Release:    9.04
Codename:    jaunty

$ uname -a
Linux beast 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64 GNU/Linux

$java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)

$ firefox --version
Mozilla Firefox 3.5.3pre, Copyright (c) 1998 - 2009 mozilla.org

I'm using firefox from the following PPA
deb http://ppa.launchpad.net/ubuntu-mozilla-daily/ppa/ubuntu jaunty main

If you need something more thorough tested, feel free to ask.

Cheers

Cameron

2009/8/13 Simon Stewart <simon.m...@gmail.com>

[Simon Stewart]

Lovely. Thank you!

Simon

[Eran M.]

Thanks everybody for testing it. Apparently the issue came up on systems without libx11-dev installed. A fix will be uploaded into trunk tomorrow, probably.

If you're interested in the technical details of the problem:

The shared library that prevents focus stealing from Firefox (x_ignore_nofocus) needs a function inside libX11.so (in fact, it over-rides the XNextEvent function from libX11.so and discards FocusOut events). This library exists under the name libX11.so.6 and when a user installs libx11-dev, a symlink is created: libX11.so -> libX11.so.6. The fix is simple - have the library open libX11.so.6.

libx11-dev is a dependency of the openjdk-jdk, so if you've installed openjdk-jdk you wouldn't encounter this problem.

Eran

2009/8/16 Cameron Braid <cam...@braid.com.au>

[Simon Stewart]

Nice work, Eran. Thanks for looking into this, and thank you to
everyone who's taken the time out to try it. I'm looking forward to
seeing the native events in everyone's hands :)

Cheers,

Simon