Vote to cut off native events support in latest Firefox versions

[Alexei Barantsev]

Hi, Devs,

Preamble:

Mozilla broke native lib compilation again with the release of Gecko 35, see https://bugzilla.mozilla.org/show_bug.cgi?id=1122727

This time the damage looks serious. We used to call the deleted method to get the native window HWND that is required to send native events.

And there is no simple alternative because of other compilation issues described by Jim Evans in the comments to the mentioned issue.

Suggestion:

Let's stop providing native libs for Firefox 35 and the forthcoming Firefox versions.

Discussion:

1) Marionette is coming soon. I hope :)

2) People who need native events can use Firefox 31 ESR

3) With this cutoff decision we'll not be forced to ship a new Selenium version after Firefox release.

4) We can spend more effort fixing remaining glitches in synthesized events instead of fighting C code of the native libs

5) Synthesized events are not too bad, at the moment we have more tests passed with synthesized events than with native ones

Please vote and comment!

Regards,

-- 

Alexei Barantsev

Software-Testing.Ru

Selenium2.Ru

[Santiago Suarez Ordoñez]

+1, no doubt about it.

The reasoning makes perfect sense to me and the need for a change is pretty clear based on the amount of effort that seems to have been spent in this problem.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/2aa25e2e-eb89-4381-808c-9848d58b9d86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Inman-Semerau]

Santi,

Is there any way that saucelabs can see a percent of jobs that are
against Firefox using / not using native events? (excluding the
selenium project if you could).

I know that salesforce does not use native events... so I'm in favor
of dropping it.

-Luke

> https://groups.google.com/d/msgid/selenium-developers/CADn8k68E3KxQKQMW%3DbmsKa3vctevqNJ%2B0E_Fx9wX5ukAPnftLg%40mail.gmail.com.

[Santiago Suarez Ordoñez]

We could definitely pull something out, from stored data if available or just calculate for a few days if not. 

What would be a way to detect the usage of native vs synthesised by users?

>> email to selenium-developers+unsub...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/selenium-developers/2aa25e2e-eb89-4381-808c-9848d58b9d86%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-developers+unsub...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/selenium-developers/CADn8k68E3KxQKQMW%3DbmsKa3vctevqNJ%2B0E_Fx9wX5ukAPnftLg%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAL97Zu7KXWve2T3c5W068nC4bDsmftey%3DVM-KbnVDpBXqUGOnw%40mail.gmail.com.

[Luke Inman-Semerau]

it should be in the initial desired capabilities.... although I'd
imagine that most users don't set anything explicitly?

On Wed, Jan 28, 2015 at 2:32 PM, Santiago Suarez Ordoñez

>> >> email to selenium-develo...@googlegroups.com.

>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/selenium-developers/2aa25e2e-eb89-4381-808c-9848d58b9d86%40googlegroups.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Selenium Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to selenium-develo...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/selenium-developers/CADn8k68E3KxQKQMW%3DbmsKa3vctevqNJ%2B0E_Fx9wX5ukAPnftLg%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Selenium Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to selenium-develo...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/selenium-developers/CAL97Zu7KXWve2T3c5W068nC4bDsmftey%3DVM-KbnVDpBXqUGOnw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-develo...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/selenium-developers/CADn8k6-xf9Vzcafqz0VVHq681dp5yZQnTdwj%3D6VWPfuGsnrr2A%40mail.gmail.com.

[Santiago Suarez Ordoñez]

That was my first guess. I'd be surprised if anyone is actually explicitly requesting this. I could start recording a metric about it, but I'm skeptical about it being very useful.

>> >> email to selenium-developers+unsub...@googlegroups.com.

>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/selenium-developers/2aa25e2e-eb89-4381-808c-9848d58b9d86%40googlegroups.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Selenium Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to selenium-developers+unsub...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/selenium-developers/CADn8k68E3KxQKQMW%3DbmsKa3vctevqNJ%2B0E_Fx9wX5ukAPnftLg%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Selenium Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to selenium-developers+unsub...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/selenium-developers/CAL97Zu7KXWve2T3c5W068nC4bDsmftey%3DVM-KbnVDpBXqUGOnw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to selenium-developers+unsub...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/selenium-developers/CADn8k6-xf9Vzcafqz0VVHq681dp5yZQnTdwj%3D6VWPfuGsnrr2A%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAL97Zu6%2B%2BK_%2BfA%3DFsT5dCcjFBnh-BBMG9PKYE-bVpYvSbr0%2Bmw%40mail.gmail.com.

[Seva Lo]

Hi,

At Google only a small minority of teams enable native events in their tests with FirefoxDriver. I have not yet heard back from them, but I don't expect it to be a disaster.

Given that no solution is easy to build (which is my understanding), I support dropping native events in FirefoxDriver.

Thanks,

Seva

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CADn8k6-N26hSD8RKcfO31EM0-pvoPjEVazW6Wwjzb7CLU2UpBA%40mail.gmail.com.

[Seva Lo]

Do advanced user interactions with ForefoxDriver require native events?

Seva

[Daniel Wagner-Hall]

From the tone of the bug, it sounds like continuing to support native events is going to continue to be a road of pain, and so sadly is probably worth ditching.

I believe that IME support currently only works with native events. I don't know how many people use IME.

--

You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/2aa25e2e-eb89-4381-808c-9848d58b9d86%40googlegroups.com.

[Louis-Dominique Dubeau]

The lack of native events severely impairs Selenium's ability to straightforwardly mimic how a user interacts with a web application running in a browser.

I've attached a Python script that shows an example of the issues that the lack of native events cause. Everything works as expected when native events are turned on:

1. The input receives a "blur" event which brings up a modal that prevents the click on the "Submit" button, and prevents entering more data into the input field.

2. Clicking "Submit" brings up a modal that prevents entering more data into the input field. (So the field ends with the value "foo" because neither "bar" nor "baz" could be added.)

When the capabilities are set so that "nativeEvents" is false, the events are sent directly to the elements. So the input field is never blurred and the appearance of modals does not prevent adding data to the input field, which ends with the value "foobarbaz".

On Tuesday, January 27, 2015 at 4:28:35 PM UTC-5, Alexei Barantsev wrote:

1) Marionette is coming soon. I hope :)

And in the meantime?
 

2) People who need native events can use Firefox 31 ESR

Seeing as users don't generally stick to ESR releases, this constitutes a major constraint.
 

3) With this cutoff decision we'll not be forced to ship a new Selenium version after Firefox release.

This is an advantage, but at what cost?
 

4) We can spend more effort fixing remaining glitches in synthesized events instead of fighting C code of the native libs

I don't consider the issues with synthesized events to be "glitches."
 

5) Synthesized events are not too bad, at the moment we have more tests passed with synthesized events than with native ones

I abhor synthesized events. I don't know how close this is to what actually is happening behind the scenes but what I imagine is going on with a click is something like this JavaScript:

   <element of interest>.dispatchEvent(new Event("click"));

<element of interest> is the actual DOM Element. This obviously does not result in the emission of any events that would be generated as a secondary effect of the user's action (e.g. "blur", as illustrated above), and it completely bypasses the event propagation machinery.

Regards,
--
Louis-Dominique Dubeau
Director of Software Development, BTW Project
Mangalam Research Center for Buddhist Languages

[LeHack]

+1 for Louis-Dominique Dubeau and not removing native events

A month ago when an update to Firefox 36 broke all of our tests (we *do* use native events for the reasons outlined by Louis-Dominique) I reviewed the issue and here's what I found:
1. Switching to non-native events requires some work (it doesn't just work out of the box) and even makes some things impossible (like relaying on proper event propagation)
2. Synthesized event are about 3 times slower when compared with native events (at least for our tests) which turns the ~2 hour regression build into a 6 hour one - this is simply unacceptable
3. We cannot just stick with Firefox 35 forever (like we do temporarily) as our customers expect us to fully support the latest 2 versions of Firefox (currently already 36 and 37)

So yes, to me native events are an important part of the FirefoxDriver.

[Alexei Barantsev]

Thank you and Louis-Dominique for the feedback!

1. First of all, I appeal to you -- if you have issues with synthesized evens please report them to the issue tracker

https://github.com/SeleniumHQ/selenium/issues

and we'll fix them eventually.

2. Synthesized events are faster than the native ones. One can easily observe this fact from the Selenium CI test run results:

http://ci.seleniumhq.org:8080/job/Firefox%2031%20Linux%20Native%20Java%20Tests/lastCompletedBuild/testReport/history/

http://ci.seleniumhq.org:8080/job/Firefox%2031%20Linux%20Synthesized%20Java%20Tests/lastCompletedBuild/testReport/history/

The performance degradation you mentioned is a totally different issue, see https://code.google.com/p/selenium/issues/detail?id=8551

3. Synthesized events are intended to emulate the full set of events. If something is not fired -- it is a bug, please report an issue!

4. The most interesting part is the question why "appearance of modals does not prevent adding data to the input field". Yes, synthesized events implementation does not include layout check at the moment. It is quite controversial topic if we should do this check or not. I believe it is an issue and should be fixed. I hope I'll be able to target this goal soon.

Regards,

-- 

Alexei Barantsev

Software-Testing.Ru

Selenium2.Ru

[Alexei Barantsev]

Hi, devs,

I'm back to this topic. Now I feel more confident about deleting native events, and I'm going to prone them away very soon.

1) The latest Firefox version with native events support is Firefox 34, and I can't see many issues in the tracker with complains about missing native events. Say more, I can't see one!

2) The latest "officially" supported version that can run with native events is Firefox 31 (previous ESR), its usage rate is close to zero.

3) FirefoxDriver with native events passes *less* tests than the one with synthesized events.

There are only 8 tests that are failed with synthesized events:

I18nTest.testShouldBeAbleToActivateIMEEngine

I18nTest.testShouldBeAbleToInputJapanese

TypingTest.testNonPrintableCharactersShouldWorkWithContentEditableOrDesignModeSet (fails on Linux only)

BasicMouseInterfaceTest.testMovingMouseBackAndForthPastViewPort

CombinedInputActionsTest.testPlainClickingOnMultiSelectionList

CombinedInputActionsTest.testShiftClickingOnMultiSelectionList

CombinedInputActionsTest.testControlClickingOnMultiSelectionList

DragAndDropTest.testDragTooFar

(and, by the way, all these tests are failed in IE and Chrome too)

The tests failed with native events are much more numerous and much more serious:

AlertsTest.testShouldHandleAlertOnWindowClose (crashing the browser in x_ignore_nofocus)

AlertsTest.testSwitchingToMissingAlertInAClosedWindowThrows

ClickTest.testShouldBeAbleToClickOnAnElementInFrameGreaterThanTwoViewports

ClickTest.testShouldBeAbleToClickOnASpanThatWrapsToTheNextLine

I18nTest.testEnteringSupplementaryCharacters

PageLoadingTest.testShouldTimeoutIfAPageTakesTooLongToLoadAfterClick

TypingTest.testChordControlCutAndPaste

BasicKeyboardInterfaceTest.canGenerateKeyboardShortcuts

BasicKeyboardInterfaceTest.testSelectionSelectAll

BasicKeyboardInterfaceTest.testSelectionSelectByWord

BasicMouseInterfaceTest.testHoverPersists

BasicMouseInterfaceTest.testMovingIntoAnImageEnclosedInALink

CombinedInputActionsTest.testChordControlCutAndPaste

CombinedInputActionsTest.testCombiningShiftAndClickResultsInANewWindow

CombinedInputActionsTest.testControlClickingOnCustomMultiSelectionList

CombinedInputActionsTest.testHoldingDownShiftKeyWhileClicking

DragAndDropTest.testDragAndDropElementWithOffsetInScrolledDiv

In view of these facts, I think it is safe enough to get rid of native events now.

Regards,

-- 

Alexei Barantsev

Software-Testing.Ru

Selenium2.Ru

[Louis-Dominique Dubeau]

On Wednesday, September 16, 2015 at 1:35:48 AM UTC-4, Alexei Barantsev wrote:

Hi, devs,

I'm back to this topic. Now I feel more confident about deleting native events, and I'm going to prone them away very soon.

1) The latest Firefox version with native events support is Firefox 34, and I can't see many issues in the tracker with complains about missing native events. Say more, I can't see one!

You missed some:

https://github.com/SeleniumHQ/selenium/issues/813

https://github.com/SeleniumHQ/selenium/issues/825

https://github.com/SeleniumHQ/selenium/issues/862

I think I still have a case that I have not submitted yet.

 

3) FirefoxDriver with native events passes *less* tests than the one with synthesized events.

My experience of synthetic events is that they do not accurately reproduce what happens when a real user is using a browser. My experience of native events is that they are able to accurately reproduce what happens when a real user is using a browser.

 

In view of these facts, I think it is safe enough to get rid of native events now.

I disagree.

With cross-platform native event support I am able to have a test suite that is largely going to operate the same way for Chrome, FF and IE. There are *some* differences that must be accounted for but they are limited and, for the most part, boil down to inherent differences among browsers. In other words, the differences are usually not due to Selenium itself. (Except in the case of bugs in Selenium that sometimes exist only for one platform) If you take native events out, then you force me to compensate for Selenium's deficiencies. You've pointed out how the native and synthetic events fail different tests. Why? Because they behave differently. Let's take my go-to example of overlapping elements, and let's suppose that I'm testing that a transparent overlay is able to intercept a user's attempt to click in an area of the window. (This is exactly how Bootstrap traps clicks outside of modals. This is also a common way with DataTables, for instance, to block user interaction with a table while it is refreshing. This is not an outlandish scenario.) With native events, the test is trivial and intuitive: I choose some element that a user could conceivably try to manipulate, and I instruct Selenium to click it. With native events, the browser will get a mouse click at coordinates X,Y that correspond to the element I chose and will compute which element should receive the click. Since the overlay is supposed to block clicking on the forbidden element, the overlay will get the click. Great! With synthetic events, the click will be delivered directly to the element I chose, completely bypassing the overlay. Okay, so I can instruct Selenium to send the event to the overlay directly but if I do this what on Earth am I testing now?? The overlay could have erroneous coordinates, that don't actually overlap with what I'm trying to protect from manipulation, or it could have a size of 0x0 and thus be ineffective at blocking anything, but if I send the event directly to it, it *will* get the event and the test *will* pass, even though the application is buggy. I actually ran into a bug which I was able to detect thanks to native events: on IE an overlay would not catch any clicks if its background color was not explicitly set. (It worked fine on Chrome and FF, just not IE.) If I had been forced to use synthetic events, the event would have been sent to the overlay anyway and I would not have caught the issue. At the end of the day, if I want to maximize the usefulness of my test, it should contain two paths: one which relies on the presence of native events and will catch the maximum number of problems, and one which is used when native events are not present.

-- Louis

[Alexei Barantsev]

Hi, Louis,

I see you point, these are legit issues.

But when I recollect the old issues that appeared after each Firefox release with many comments like "everything is broken", "fix this asap!" -- I don't want to go back :)

We have no *such* issue reports novadays.

To be honest, so called "native" events are simulated too.

But they are simulated on the OS level, whereas so called "synthesized" events are simulated on the browser level.

And I hope we'll be able to fix all the remaining issues in the "synthesized" simulation to make it as close to real user as possible.

Regards,

-- 

Alexei Barantsev

Software-Testing.Ru

Selenium.Ru

[Louis-Dominique Dubeau]

On Friday, September 18, 2015 at 1:28:05 AM UTC-4, Alexei Barantsev wrote:

Hi, Louis,

I see you point, these are legit issues.

Thank you. 

 

But when I recollect the old issues that appeared after each Firefox release with many comments like "everything is broken", "fix this asap!" -- I don't want to go back :)

We have no *such* issue reports novadays.

I get that, but people will submit other reports. Suppose the issue of overlapping elements has been solved: if I try sending an event to an element covered by another element, the covering element gets the event, which accurately replicates what happens when a user tries to click on the covered element. Great! I test my code with Selenium and everything works on FF, CH and IE 9-11. Two days later I get a bug report from a user using IE 9. For some strange reason the events that should be captured by my overlay are not captured. What is going on?? I fire up an IE session at Sauce Labs to load my application and try to reproduce the problem. For whatever reason when I try it manually I reproduce the bug I can reproduce it but when I try to reproduce the behavior in Selenium I cannot reproduce it. Why? Because now the problem is that Selenium is not reproducing how IE 9 actually handles the event. I've mentioned in my previous message that in FF and Chrome my overlay works fine even if I do not specify a background color but in IE 9 the same overlay won't work without a background color being specified. (I said IE without specifying a version in my previous message but it is in fact IE 9 that has this quirk. 10 and 11 are fine.) So unless Selenium's synthetic event code contains code that replicates the quirk of how IE 9 works, it won't be able to faithfully reproduce what happens when a user clicks on the overlay.

I've run into other quirks over the years, but (much to my regret now) I did not catalog them. I recall differences in the way keystroke events are generated. I also recall IE generating an input event when clicking an input element with a placeholder set, and I see that Selenium indeed generates this event but whereas Selenium with native events produces a sequence of events which faithfully replicates what happens when the user clicks (mousedown, focus, input, mouseup), with synthetic events the order is not one that is seen when a user interacts with a browser. (It is instead mousedown, mouseup, focus, input.)

The point here is that Selenium will have to chase browser quirks to faithfully reproduce what happens when a user interacts with the browser. Maybe that's an issue with native events too but I'd expect it to be less of an issue.

To be honest, so called "native" events are simulated too.

But they are simulated on the OS level, whereas so called "synthesized" events are simulated on the browser level.

Yes, I understood this.

 

And I hope we'll be able to fix all the remaining issues in the "synthesized" simulation to make it as close to real user as possible.

If I can use synthetic events and have them faithfully simulate what happens when a real user interacts with the browser, and if I don't have to have two code paths for my tests (one with native events and one with synthetic events), I'll be a happy camper. I've run a test suite that contains 131 scenarios. On FF, with native events 130 pass and 1 is skipped. With synthetic events 100 scenarios passed, 30 failed, 1 skipped. 

Thanks,

-- Louis

[Alexei Barantsev]

Hi,

On Thursday, September 17, 2015 at 11:09:11 PM UTC+3, Louis-Dominique Dubeau wrote:

You missed some:

https://github.com/SeleniumHQ/selenium/issues/813

https://github.com/SeleniumHQ/selenium/issues/825

https://github.com/SeleniumHQ/selenium/issues/862

I think I still have a case that I have not submitted yet.

FYI: these issues are fixed ;)