Ensure the functionality of Marionette in B2G

[Bevis Tseng]

Hi all,

Marionette-webapi is heavily used by Telephony framework in B2G.
(The test cases are hidden currently in treehelder and we are working
aggressively to enable it in the treehelder by default once the criteria is
met.)
Testing suites are expected to be changed.
However, this test suite in B2G is broken frequently when the script is
changed.
See bug 1212608 as example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1212608#c29

Is there anyway to ensure the functionality of Mnw in B2G even it's hidden
currently?

This will be very helpful to reduce our time of debugging why mnw is broken
again.

Thanks & regards,
Bevis Tseng

[Henrik Skupin]

Bevis Tseng wrote on 11/10/2015 11:45 AM:

Hi Bevis,

> (The test cases are hidden currently in treehelder and we are working
> aggressively to enable it in the treehelder by default once the criteria is
> met.)
> Testing suites are expected to be changed.
> However, this test suite in B2G is broken frequently when the script is
> changed.
> See bug 1212608 as example:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1212608#c29
>
> Is there anyway to ensure the functionality of Mnw in B2G even it's hidden
> currently?

Sorry for that but there are some larger refactorings for marionette
runner happening those days. Looks like those test failures haven't been
seen. Maybe we should compile a list of external and/or internal test
suites which make use of marionette and are not reporting to Treeherder
as Tier 1 or Tier 2.

Otherwise some questions:

* Where are your tests located? Would it maybe make sense for you to
make use of a stable release of marionette instead? It can be downloaded
from PyPI or our internal mirror. It might not have to be forever but at
least until the results are shown as Tier-1 or Tier-2.

* Are your tests getting run on Try? If not, you might want to think
about to get them enabled there. If something is not being shown as
broken, developers will assume that all will be fine in getting a patch
landed. Or is this also the Tier-3 issue here?

Those are just my 2c and I assume some Marionette peers will also respond.

Best,

--
Henrik Skupin
Senior Test Engineer
Mozilla Corporation

[Andreas Tolfsen]

On 10 November 2015 at 10:45, Bevis Tseng <bts...@mozilla.com> wrote:
> Marionette-webapi is heavily used by Telephony framework in B2G.

> (The test cases are hidden currently in treehelder and we are working
> aggressively to enable it in the treehelder by default once the criteria is
> met.)

This is excellent news! I've broken marionette-webapi on a number of
occasions because it's not available in try, most recently in
https://bugzilla.mozilla.org/show_bug.cgi?id=1211503 (fixed in
https://bugzilla.mozilla.org/show_bug.cgi?id=1223028).

As we run into issues in consumer code, we always try to add a reduced
test case to the Marionette unit test suite. This is an effective
guard against the same problem happening again, later.

> However, this test suite in B2G is broken frequently when the script is
> changed.
>
> See bug 1212608 as example:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1212608#c29

This was caused by
https://bugzilla.mozilla.org/show_bug.cgi?id=1212608 and fixed
yesterday in https://bugzilla.mozilla.org/show_bug.cgi?id=1222388. The
fix landed on central this morning.

Unfortunately we see this type of mach-related breakage from time to
time because try isn't using mach to invoke the tests. I'm afraid
until there's a coordinated effort to fix this, there aren't many
options for preventing this specific type of breakage.

> Is there anyway to ensure the functionality of Mnw in B2G even it's hidden
> currently?

If the test suite is hidden and it has known problems I feel it's
unreasonably cumbersome to expect changes "not to break Mnw further".

I think the best option is to fix the test suite and have it
re-enabled as a Tier-1 job on try. If intermittents and bad tests are
preventing this, one should perhaps begin to question the usefulness
of these tests as they are not _currently_ contributing anything good
to the stability of Firefox OS.

> This will be very helpful to reduce our time of debugging why mnw is broken
> again.

This is a "help us help you" situation. I think the best course of
action is to get marionette-webapi tests running on try so that we can
tell upfront when Marionette patches break your harness.

When we do see breakage in consumers of Marionette we should be
vigilant and add unit tests for it to the Mn unit test suite as well,
as I did for https://bugzilla.mozilla.org/show_bug.cgi?id=1223028.

[Bevis Tseng]

Hi Henrik,

Thanks for your reply.
1. The marionette we use are the one inside gecko instead of the stable one.
2. The our test cases are available inside gecko as well in

gecko/dom/[icc|mobileconnection|mobilemessage|telephony]/tests/marionette/
3. The marionette for b2g can be run both in treehelder and locally with
b2g build.
For treehelder, just select *"b2g" in platforms* and *"marionette" in
unit-tests*.
Once the test suites are run, you can toggle the *"show/hide excluded
job"* in tree helder to see if marionette are executing as expected.
For locally build, you can refer the HOWTO in [1], and choose "emulator"
when executing "config.sh" to have local repo of the emulator build.

Currently, the simplest way for tools team to be aware that Mnw is run as
expected is to always have marionette in b2g be run in treehelder before
landing the fix.
(You don't have to check why the test case is failed but ensure that the
marionette is run properly from the logs.)

Is this suggestion ok for you?

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Building

Regards,
Bevis Tseng

On Tue, Nov 10, 2015 at 7:00 PM, Henrik Skupin <hsk...@mozilla.com> wrote:

> Bevis Tseng wrote on 11/10/2015 11:45 AM:
>
> Hi Bevis,
>

> > (The test cases are hidden currently in treehelder and we are working
> > aggressively to enable it in the treehelder by default once the criteria
> is
> > met.)

> > Testing suites are expected to be changed.

> > However, this test suite in B2G is broken frequently when the script is
> > changed.
> > See bug 1212608 as example:
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1212608#c29
> >

> > Is there anyway to ensure the functionality of Mnw in B2G even it's
> hidden
> > currently?
>

[Bevis Tseng]

On Tue, Nov 10, 2015 at 7:28 PM, Bevis Tseng <bts...@mozilla.com> wrote:

>
> Hi Henrik,
>
> Thanks for your reply.
> 1. The marionette we use are the one inside gecko instead of the stable
> one.
> 2. The our test cases are available inside gecko as well in
>
> gecko/dom/[icc|mobileconnection|mobilemessage|telephony]/tests/marionette/
> 3. The marionette for b2g can be run both in treehelder and locally with
> b2g build.
> For treehelder, just select *"b2g" in platforms* and *"marionette" in
> unit-tests*.
> Once the test suites are run, you can toggle the *"show/hide excluded
> job"* in tree helder to see if marionette are executing as expected.
> For locally build, you can refer the HOWTO in [1], and choose
> "emulator" when executing "config.sh" to have local repo of the emulator
> build.
>
> Currently, the simplest way for tools team to be aware that Mnw is run as
> expected is to always have marionette in b2g be run in treehelder before
> landing the fix.
> (You don't have to check why the test case is failed but ensure that the
> marionette is run properly from the logs.)
>
> Is this suggestion ok for you?
>
​

Oops.
I guess it's not okay per Andreas' comment that the mach is not used by
treehelder actually.