Event simulation for B2G, Eideticker, and responsiveness testing

[William Lachance]

(I just had a meeting with ctalbert and mdas about this, figured I might
as well type up some notes based on it for the benefit of others)

Here in auto-tools we have a number of projects in the works that
require more fine grained browser automation (especially on mobile) than
we're used to:

* Eideticker (https://wiki.mozilla.org/Project_Eideticker): Prototype
project to measure graphics performance in various situations by video
capture. I just started working on this, along with Mark Cote (who's
been heroically working on getting video capture working with various
Android devices).
* User responsiveness harness (https://github.com/ahal/peptest): Project
to measure responsiveness of browser by performing various user
interactions and measuring whether the event loop doesn't fire for a
specified period of time. Andrew Halberstadt is working on this.
* Boot2Gecko Automation (https://wiki.mozilla.org/B2G): Project to
perform automated integration testing of Boot to Gecko (where we won't
be able to start up firefox dynamically, because it will already be
running). Malini Das is going to be working on this.

All of these projects need to not only load web pages (as is the case
with Mochitest and Talos), but actually interact with both content and
browser chrome on a deep level. On the content level, we'd like to be
able to simulate clicking on various UI elements, touch events,
basically anything that a user might want to do with a webpage. On a
chrome level (where it exists), we'd like to be able to enter text into
the location/search bars, view/modify preferences, and a whole host of
other things. Especially for Eideticker and the user responsiveness
tests, we want the footprint of whatever automation to be minimal, so we
can measure user responsiveness and graphical performance in as close to
a real-world setting as possible.

If this sounds a bit like Mozmill
(https://github.com/mozautomation/mozmill) to you, you're right. :) As
you'll see below, we're planning to reuse some parts of Mozmill for the
work we're planning to do, but from what I understand from ctalbert the
Mozmill stuff is (1) kind of heavy and (2) can't easily be adapted to be
used in a context where the browser always running (like B2G). Thus the
need for a new project.

A number of different approaches to this are being considered, but the
one we're leaning towards implementing is basically something where
tests can be scripted via Selenium with some extensions for the non-B2G
case that allow interaction with the browser chrome (entering stuff in
the url bar, e.g.). This has a number of distinct advantages:

* Tests should be easy (maybe even fun!) to develop using a
well-documented framework. Automation developers can reuse their
hard-won test-writing skills across any of these three projects.
* For B2G, any work done by automation to improve Selenium integration
on Android/B2G will help third party app developers test their
applications on Mobile Firefox. Triple win!

There's the current minor problem is that we have no way of running
Selenium on either Android or Boot2Gecko. Clint's been evaluating
various approaches to this
(https://wiki.mozilla.org/Auto-tools/Projects/JSON_Protocol_Testing),
and thinks the way forward is likely to to be to extend the planned
remote debugging interface
(https://wiki.mozilla.org/Remote_Debugging_Protocol) to allow injecting
both chrome and browser based events into a running Firefox. This
interface is exposed via TCP and accepts commands written in the form of
JSON blobs. Event simulation in the browser was not the original
intention of the remote debugging protocol -- however, it seems silly to
create a new external browser automation interface when this is so close
to what we want.

This (currently hypothetical) protocol is not directly compatible with
Selenium (it's just raw JSON, no http headers). We need to write another
program to act as an HTTP webserver listening for Selenium REST calls
(http://code.google.com/p/selenium/wiki/JsonWireProtocol), translate
them into the JSON format expected by this debugging protocol, then send
them to the browser on the JS debugger socket. We don't really want to
pollute the current JS Debugger code with all sorts of automation stuff,
so as a first step we'll be wanting to create a dispatcher layer on top
of the JS Debugger code to send events to either JavaScript or our
automation stuff, as appropriate. We can then flesh out our set of nouns
and verbs for scripting browser interactions and proceed towards world
domination. :P

I believe Clint has started some work on extending the JS Debugger code
to do dispatching to the appropriate component (either the current JS
Debugging interface or our own event-handling code, as appropriate), but
in general this project is very much in its early stages. If you're
interested in diving in and helping out, or have any questions/comments
on the above, let us know (either by following up here or joining us on
irc.mozilla.org #ateam).

Will

[David Burns]

Hi Roy,

I can't talk for Eideticker but Marionette will eventually be what the
Selenium project uses. One thing that we should note is that one of the
good things about Eideticker is that it works with all browsers but
currently the Selenium project allows automating of WebViews and while
that is good enough for web testers this isn't good enough for
performance testing because webviews are not optimised like browsers.

The JSON Wire protocol is purely a transportation mechanism and nothing
that end users should ever know or even care about. For you, I wouldn't
worry about it!

When all browser vendors have implemented the Browser Automation Spec[1]
and especially the touch section that is currently being written then we
can think about it but unfortunately until then Eideticker is the best
tool we have to do the job!

David

[1] https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html

On 09/02/2013 20:52, roy...@gmail.com wrote:
> Hi Will,
>
> I notice the date on this post is a while ago - is there a workable solution in place yet for using Selenium with Marionette? I keep seeing people mentioning the JSON wireframe protocol, but I really need a code example of what to write in order to get using it (if it's been developed yet that is).
> I'm currently using python (since it's what comes bundled with Marionette) but if I could switch to Selenium (or even javascript) that would be fantastic!
> _______________________________________________
> dev-b2g mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-b2g