New: Use Marionette on Firefox for Android (Fennec)

[Maja Frydrychowicz]

As part of pushing forward with inter-operable web testing and the
WebDriver standard, we now run Marionette tests on Firefox for Android
(Fennec). This is the first instance of a WebDriver implementation being
used in production on a mobile browser.

In particular:
* Fennec support has been added to Marionette Test Runner [1] and mozrunner
[4].
* You can run Marionette tests locally on an emulator or device with |mach
marionette-test|. [2]
* Marionette is enabled in Fennec debug builds, so Marionette tests are now
running in automation and show up on Treeherder on mozilla-central as
tc[tier 3](Mn...) under Android 4.3 API15+ debug [3]. They will ride the
trains and we'll bump them up to Tier-1 once we're satisfied that the tests
are stable on m-c [5].
* You can trigger the tests with try syntax like |try: -b d -p
android-api-15 -u marionette -t none|


[1] Fennec in Marionette Runner - https://bugzilla.mozilla.org/s
how_bug.cgi?id=787203
[2] How To - https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionet
te/Running_Tests#For_Firefox_for_Android_(Fennec)
[3] TaskCluster configuration - https://bugzilla.mozilla.org/s
how_bug.cgi?id=1284874
[4] FennecEmulatorRunner - https://bugzilla.mozilla.org/s
how_bug.cgi?id=1278590
[5] Tests to green up - https://bugzilla.mozilla.org/show_bug.cgi?id=1297394

[James Willcox]

Nice work!

I guess there's not currently an easy way to enable Marionette for
non-debug builds? One thing I looked into before was using ChromeDriver and
Marionette to do head-to-head performance tests against Chrome on Android,
and it would be great if we could use Nightly or something.

James

On Fri, Aug 26, 2016 at 9:29 AM, Maja Frydrychowicz <ma...@mozilla.com>
wrote:

> _______________________________________________
> mobile-firefox-dev mailing list
> mobile-fi...@mozilla.org
> https://mail.mozilla.org/listinfo/mobile-firefox-dev
>
>

[Andreas Tolfsen]

James Willcox <jwil...@mozilla.com> writes:

> I guess there's not currently an easy way to enable Marionette for
> non-debug builds?

You can set the ENABLE_MARIONETTE output variable to something exciting
in your mozconfig to enable it in optimised builds.

> One thing I looked into before was using ChromeDriver and Marionette
> to do head-to-head performance tests against Chrome on Android, and it
> would be great if we could use Nightly or something.

One should be aware that no software instrumentation comes at zero cost.

WebDriver was not designed to be a performance measurement tool, and
cross-browser performance measurement is a particularly difficult area.
It does however provide ways to remote control the browser to _initiate_
the necessary measurements and _collect_ the data.

There are a lot of considerations to be taken with regards to variance
and outliers: you can’t test against live sites; you need to
forensically examine cached documents to ensure they don’t serve
different content based on the UA; the test environment needs to be
hermetic and network conditions reproducible; considerations must be
made with regards to cold-start vs. hot-start; population of various
caches could prove challenging &c. &c.

But given a sufficiently sophisticated test environment, there is
probably data available in Gecko that it would be interesting to harvest
and look at, especially if we could compare it to other browsers’.

The core issue preventing that is that vendors haven’t agreed
on _what_ categories of data it would make sense to expose in a
cross-browser API. The little discussion we’ve had so far about this
can be found here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1250290

As the Browser Tools- and Testing Working Group is finishing up the last
pieces of the puzzle in the base specification…

http://w3c.github.io/webdriver/webdriver-spec.html

… it will probably soon be time to start thinking about what
additional benefit we can garner from an out-of-process remote control
protocol like this in the future. We have already discussed permissions
management with the WebBluetooth WG. Logging- and performance APIs also
make a great deal of sense.

At the last WebDriver WG meeting we spoke with the Edge performance
PM Todd Reifsteck who is doing things similar to what you describe.
He had noted Firefox was slower at loading pages when run under
instrumentation, which can probably be attributed to the way Marionette
injects code on navigation. It would be good to track down and fix
that, but it proves the point that no automation comes for free.

For the record I don’t think that externally measured load time
comparisons are very interesting: there are likely less coarse metrics
to look at that has less variance, such as MOZ_PROFILER dumps or the
performance timeline.

[sajid....@cosmosoftware.io]

Hi Andreas,

Following is how my mozconfig looks like:

# My first mozilla config

# Build Firefox for Android:
ac_add_options --enable-application=mobile/android
ac_add_options --target=arm-linux-androideabi

# With the following Android SDK and NDK:
ac_add_options --with-android-sdk="/path/to/.mozbuild/android-sdk-macosx"
#ac_add_options --with-android-ndk="/path/to/.mozbuild/android-ndk-r15c"

# Enable artifact building:
ac_add_options --enable-artifact-builds

# Write build artifacts to:
mk_add_options MOZ_OBJDIR=./objdir-droid

ac_add_options --enable-debug
#ac_add_options --enable-debug-symbols
ac_add_options --enable-marionette
#ac_add_options --enable-geckodriver
ENABLE_MARIONETTE=1

The build was successful and I am able to run it using ./mach run. I have done adb forward tcp:2828 tcp:2828 and added marionette.defaultPrefs.enabled -> true & marionette.defaultPrefs.port -> 2828 but when I start fennec, I am not able to see any application (in the device) listening to 2828. Moreover client = Marionette(host='localhost', port=2828) and client.start_session() from python console simply times out.

Are you able to help me on that?