Is it time for mochitest-chrome on Android and B2G

[Jonathan Griffin]

Periodically, we field a request to add support for mochitest-chrome to
Android and B2G. To date, we've avoided this by pointing out ways that
mochitest-plain can be used for the same use case, which usually
involves SpecialPowers.

We have a new request for this, in the context of requestAutocomplete
(https://bugzilla.mozilla.org/show_bug.cgi?id=1021060#c16). The tests
for this, as well as some other features we've seen requests for, need
to be able to execute some setup code with chrome privileges, and
SpecialPowers isn't always flexible enough. As bholley points out,
SpecialPowers is a "best effort" and doesn't necessarily support
everything a test may wish to do.

Has the time come to bite the bullet and add mochitest-chrome support to
Android and B2G? This would be a non-trivial effort, and would need to
be done separately for Android and B2G. Adding support for this would
come at the expense of something else, possibly work related to Android
4.4 tests on emulators, work integrating our harnesses with structured
logging, and/or work on reducing our intermittent test failures on B2G.
These tasks are important and I wouldn't want to delay them without a
very clear need. Does that exist here?

Note that we are talking only about enabling harness support for
mochitest-chrome in Android and B2G in order to provide a framework for
tests that would otherwise be difficult to write. We are not talking
about taking the existing set of mochitest-chrome tests and getting them
to work in Android and B2G. Many of those tests don't apply to Android
or B2G, and for those that theoretically do, many of them won't work
because they rely on XUL files which aren't supported in B2G, and may
not be in Android (not sure on that point).

For more context about the history of mochitest-chrome on B2G, see
https://bugzilla.mozilla.org/show_bug.cgi?id=797164

Jonathan

[Bobby Holley]

I can't comment on the importance of the a-team's other priorities, or the
importance of running the tests in consideration on Android/B2G.

However, as the author of SpecialPowers.wrap, I can say that using it for
highly-involved chrome interaction should be considered Extremely Harmful.
These wrappers are designed as a convenience mechanism to access the
occasional privileged or cross-origin property, and have a lot of magic and
footguns that both (a) waste developer time and (b) reduce the value of our
test coverage (because it often isn't testing what the author thought it
was testing).

One middle-ground here is SpecialPowers.loadChromeScript(uri), which allows
arbitrary chrome scripts to be run (in their entirety) via the message
manager. I consider that to be a robust and valid approach, but I don't
have a good sense of whether the ergonomics there are acceptable for the
use-cases in question.

In general though, it seems like it would be pretty great if newly-added
mochitest-chrome tests would automatically run on B2G and Android if we
want those to be first class platforms. I myself write mochitest-chrome
tests all the time, and there's no reason that most of those couldn't also
run on mobile.

bholley


On Tue, Jun 17, 2014 at 4:32 PM, Jonathan Griffin <jgri...@mozilla.com>
wrote:

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Panos Astithas]

Support for mochitest-chrome on mobile is sorely needed in devtools. It
would increase our test coverage there by at least an order of magnitude.

Panos

[jmaher]

Could you give some examples of what tests we could run on mobile in chrome?

[Matthew N.]

On 6/17/14 4:32 PM, Jonathan Griffin wrote:
> Many of those tests don't apply to Android or B2G, and for those that
> theoretically do, many of them won't work because they rely on XUL
> files which aren't supported in B2G, and may not be in Android (not
> sure on that point).

I'm told Android uses XUL in some cases so it should work but I don't
know how well it's "supported" outside of what Android actually needs/uses.

On 6/18/14 11:25 AM, jmaher wrote:
> Could you give some examples of what tests we could run on mobile in chrome?

From a quick skim, I believe the majority of mochitest-chrome tests
could run on mobile.

Matthew

[Mark Finkle]

Fennec/Android does have the ability to run xpcshell-like tests. These tests are JS scripts loaded into a framework that simulates the xpcshell system. Technically it's not xpcshell though. The JS script is loaded into tab of the Fennec browser. If the JS script knows this, it can create a mochitest-chrome-like test too.

Here is an example:
http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/tests/testVideoDiscovery.js?force=1

The test framework is xpcshell-like but the first thing the test does is get the browser window and load a test HTML file into a browser tab.

----- Original Message -----

> Periodically, we field a request to add support for mochitest-chrome to
> Android and B2G. To date, we've avoided this by pointing out ways that
> mochitest-plain can be used for the same use case, which usually
> involves SpecialPowers.

> We have a new request for this, in the context of requestAutocomplete

> (https://bugzilla.mozilla.org/show_bug.cgi?id=1021060#c16). The tests

> for this, as well as some other features we've seen requests for, need
> to be able to execute some setup code with chrome privileges, and
> SpecialPowers isn't always flexible enough. As bholley points out,
> SpecialPowers is a "best effort" and doesn't necessarily support
> everything a test may wish to do.

> Has the time come to bite the bullet and add mochitest-chrome support to
> Android and B2G? This would be a non-trivial effort, and would need to
> be done separately for Android and B2G. Adding support for this would
> come at the expense of something else, possibly work related to Android
> 4.4 tests on emulators, work integrating our harnesses with structured
> logging, and/or work on reducing our intermittent test failures on B2G.
> These tasks are important and I wouldn't want to delay them without a
> very clear need. Does that exist here?

> Note that we are talking only about enabling harness support for
> mochitest-chrome in Android and B2G in order to provide a framework for
> tests that would otherwise be difficult to write. We are not talking
> about taking the existing set of mochitest-chrome tests and getting them

> to work in Android and B2G. Many of those tests don't apply to Android

> or B2G, and for those that theoretically do, many of them won't work
> because they rely on XUL files which aren't supported in B2G, and may
> not be in Android (not sure on that point).

[Gijs Kruitbosch]

Yeah, this caught me out yesterday. xpcshell tests on Android have, for
instance, a pre-existing default search engine, whereas xpcshell tests
running on desktop do not.

It would be nice if this were documented on MDN.

~ Gijs