Replacing Mozmill driven tests for Firefox with Marionette

[Henrik Skupin]

Hi all,

I would like to finally announce that we are going to discontinue any
kind of development for the Mozmill framework and tests for Firefox. The
reason is that there is simply no man power to further develop this
tool. Also it's plainly only myself who has the knowledge of how it
works and how it grew over the past 6 years. It's simply not scalable
anymore.

Since end of last year we (members of the A-Team) were actively working
on replacing our tests. As you can imagine there is lots of work to do.
To being able to better spread work around we basically have chosen the
wider known Marionette framework. It's already in use by some products,
has the standardized Webdriver API, and has a better platform for
today's requirements compared to Mozmill.

So lots of new features and bug fixes went into Marionette during the
last quarter. All of those were necessary to support at least the most
important features of Mozmill like running different locales, exercising
restart tests, and others. As result we now have Marionette in a state
to get started in replacing Mozmill tests. We even have some tests
converted last quarter, so the important security and update tests could
be flipped over to Marionette now. The new test suite itself is named
Firefox UI Tests and can be accessed via:
https://github.com/mozilla/firefox-ui-tests.

Currently I'm working on replacing the Mozmill jobs in Mozmill CI with
the Marionette tests. So the first test results should be available
soon. As reporting platform we want to support treeherder, which will be
integrated over the next couple of weeks. So test results will be
broader visible.

One other big step forward is that we are now working on getting the
update tests for releases to run on RelEng infrastructure. Armen
Zambrano is helping us here. With that move the sign-offs from QA for
update tests won't be necessary anymore. RelEng will be informed
directly about update failures.

All the above should give you a quick overview and understanding what's
going on with Mozmill and Marionette. For the individual topics I might
follow-up with separate blog posts and a more detailed description of
work happening in those areas.

Please let us know if you have questions or want to participate in that
work. I'm always up for mentoring people from the community.

Best,

--
Henrik Skupin
Senior Test Engineer
Mozilla Corporation

[Eric Shepherd]

Henrik Skupin wrote:
> All the above should give you a quick overview and understanding what's
> going on with Mozmill and Marionette. For the individual topics I might
> follow-up with separate blog posts and a more detailed description of
> work happening in those areas.

Quick note: This obviously impacts the building/developing Mozilla
documentation, which has quite a bit of coverage of Mozmill and how to
use it for testing Firefox etc. If you can share any details, notes, etc
that would be relevant, I would appreciate it.

--

Eric Shepherd
Senior Technical Writer
Mozilla <https://www.mozilla.org/>
Blog: http://www.bitstampede.com/
Twitter: http://twitter.com/sheppy

[Philip Chee]

On 12/05/2015 20:23, Henrik Skupin wrote:
> Hi all,
>
> I would like to finally announce that we are going to discontinue
> any kind of development for the Mozmill framework and tests for
> Firefox. The reason is that there is simply no man power to further
> develop this tool. Also it's plainly only myself who has the
> knowledge of how it works and how it grew over the past 6 years. It's
> simply not scalable anymore.
>
> Since end of last year we (members of the A-Team) were actively
> working on replacing our tests. As you can imagine there is lots of
> work to do. To being able to better spread work around we basically
> have chosen the wider known Marionette framework. It's already in use
> by some products, has the standardized Webdriver API, and has a
> better platform for today's requirements compared to Mozmill.
>
> So lots of new features and bug fixes went into Marionette during
> the last quarter. All of those were necessary to support at least the
> most important features of Mozmill like running different locales,
> exercising restart tests, and others. As result we now have
> Marionette in a state to get started in replacing Mozmill tests. We
> even have some tests converted last quarter, so the important
> security and update tests could be flipped over to Marionette now.
> The new test suite itself is named Firefox UI Tests and can be
> accessed via: https://github.com/mozilla/firefox-ui-tests.

Thunderbird currently uses mozmill. Will Marionette[1] be available for
Thunderbird? Is there a porting guide?

Phil

--
Philip Chee <phi...@aleytys.pc.my>, <phili...@gmail.com>
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.

[1] Marionette is also a character from Micronauts

[Andreas Tolfsen]

On 13 May 2015, at 04:40, Philip Chee <phili...@gmail.com> wrote:
> On 12/05/2015 20:23, Henrik Skupin wrote:
>> To being able to better spread work around we basically
>> have chosen the wider known Marionette framework.
>

> Thunderbird currently uses mozmill. Will Marionette[1] be available for
> Thunderbird? Is there a porting guide?

I can’t think of any obvious reasons why Marionette couldn’t be made to work against Thunderbird’s chrome when it works against Firefox’.

[Andrew Halberstadt]

I don't believe marionette currently works on Thunderbird, but that's
most likely due to hardcoded assumptions than any deep technical
reasons. It could probably be made to work with a little bit of effort,
it's just that to date, no one has tried.

I thought thunderbird was using a very out-of-date version of mozmill
which was already unmaintained anyway? Or did you already upgrade to the
latest version?

-Andrew

[ISHIKAWA, Chiaki]

On 2015/05/13 23:24, Andrew Halberstadt wrote:> I don't believe

marionette currently works on Thunderbird, but that's
> most likely due to hardcoded assumptions than any deep technical
> reasons. It could probably be made to work with a little bit of effort,
> it's just that to date, no one has tried.
>
> I thought thunderbird was using a very out-of-date version of mozmill
> which was already unmaintained anyway? Or did you already upgrade to the
> latest version?
>
> -Andrew
>

At the end of my make -f client.mk (sorry, I used |make| on this
particular build), I see the following. Versions of moz* packages are
printed.
It is a pity that version of mozmill is not printed there although
versions of other moz* packages are clearly visible.
Maybe someone in the know can fix this:

--- begin quote ---
[...]
Collecting blessings>=1.3 (from mozlog==2.11)
Using cached blessings-1.6.tar.gz
Installing collected packages: mozrunner, mozdevice, mozprocess,
moznetwork, mozcrash, mozprofile, mozlog, blessings, mozinfo, mozfile,
manifestparser
Running setup.py install for mozrunner
Installing mozrunner script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Running setup.py install for mozdevice
Installing sutini script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Installing dm script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Running setup.py install for mozprocess
Running setup.py install for moznetwork
Installing moznetwork script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Running setup.py install for mozcrash
Running setup.py install for mozprofile
Installing mozprofile script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Installing diff-profiles script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Installing view-profile script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Running setup.py install for mozlog
Installing structlog script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Running setup.py install for blessings
Running setup.py install for mozinfo
Installing mozinfo script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Running setup.py install for mozfile
Running setup.py install for manifestparser
Installing manifestparser script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Successfully installed blessings-1.6 manifestparser-1.1 mozcrash-0.14
mozdevice-0.44 mozfile-1.1 mozinfo-0.7 mozlog-2.11 moznetwork-0.24
mozprocess-0.22 mozprofile-0.23 mozrunner-6.7
You are using pip version 6.0.6, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Processing ./jsbridge
Processing ./mozmill
Requirement already satisfied (use --upgrade to upgrade): mozrunner>=6.0
in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade):
manifestparser>=0.9 in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozmill==1.5.16)
Requirement already satisfied (use --upgrade to upgrade): mozcrash>=0.14
in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade):
mozdevice>=0.37 in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade): mozfile>=1.0
in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade): mozinfo>=0.7
in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade): mozlog>=1.5 in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade):
mozprocess>=0.17 in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade):
mozprofile>=0.18 in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade):
moznetwork>=0.24 in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozdevice>=0.37->mozrunner>=6.0->jsbridge==2.4.14)
Requirement already satisfied (use --upgrade to upgrade): blessings>=1.3
in
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/lib/python2.7/site-packages
(from mozlog>=1.5->mozrunner>=6.0->jsbridge==2.4.14)
Installing collected packages: mozmill, jsbridge
Running setup.py install for mozmill
Installing mozmill-thunderbird script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Installing mozmill-restart script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Installing mozmill script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Running setup.py install for jsbridge
Installing jsbridge script to
/REF-OBJ-DIR/objdir-tb3/_tests/mozmill-virtualenv/bin
Successfully installed jsbridge-2.4.14 mozmill-1.5.16
make[2]: Circular /REF-COMM-CENTRAL/comm-central/mozilla/CLOBBER <-
/REF-COMM-CENTRAL/comm-central/mozilla/CLOBBER dependency dropped.
Packaging dbgserver-40.0a1.en-US.xpi...
Packaging qui...@mozilla.org.xpi...

real 2m35.617s
user 1m51.424s
sys 0m17.268s

--- end quote ---

That said, I thought we moved to 2.0(???)

CI

[ISHIKAWA, Chiaki]

Ouch, there was a line near the end.

> Successfully installed jsbridge-2.4.14 mozmill-1.5.16

This is from a local C-C TB build.

CI

[Henrik Skupin]

Andrew Halberstadt wrote on 05/13/2015 04:24 PM:

> I thought thunderbird was using a very out-of-date version of mozmill
> which was already unmaintained anyway? Or did you already upgrade to the
> latest version?

Yes, its a very outdated version of Mozmill from the hotfix-1.5 branch,
which we do not support since more than a year. Given that I have to add
that we cannot maintain something which is not in use at all. It's sad
to say but if something is broken the TB devs will have to get it fixed
in their customized Mozmill version.

--
Henrik

[Yonggang Luo]

Maybe a generalize name is better?
Such as xul-ui-tests?

[Yonggang Luo]

在 2015年5月16日星期六 UTC+8下午4:46:35，Yonggang Luo写道：

Or gecko-ui-tests.

[Andrew Sutherland]

On Fri, May 15, 2015, at 02:26 PM, Henrik Skupin wrote:
> Yes, its a very outdated version of Mozmill from the hotfix-1.5 branch,
> which we do not support since more than a year. Given that I have to add
> that we cannot maintain something which is not in use at all. It's sad
> to say but if something is broken the TB devs will have to get it fixed
> in their customized Mozmill version.

This shouldn't be too big a problem. The Thunderbird mozmill tests
basically do the following:
- Spin up a Thunderbird instance with a controlled profile with the
mozmill extension in place that provides various testing code, and
listens so python can connect and tell jsbridge to do things.
- Use jsbridge to launch JS testing logic in-process that synchronously
(using nested event loops) pokes things and waits for things, with a
layer of helper functions wrapping the lower level mozmill
functionality.
- Detect crashes maybe, consume the results.

I would expect most of the brittleness to be related to:
- changes in how Gecko/Toolkit decides to use (or not use) extensions as
things are locked down to stop bad-actor-sneakily-installed extensions
- minor moz-specific DOM API changes that mozmill's helper logic
depended on
- Keeping up with Python 2.x releases?

It certainly doesn't seem worth it / feasible to convert to the
Gaia/JSMarionette style of operation where synchronous JS code is run in
a node.js process and remotes all checks. At least not at this time.
If/when Thunderbird ever goes more e10s, that would likely be the time
to overhaul the tests. Many of which could hopefully be salvaged by
just making modifications to the helper logic and leaving the tests
themselves largely unchanged.

Andrew