obsoleting automation.py in the near future

[Ted Mielczarek]

This came up in the platform meeting yesterday, and apparently there
were some concerns, so I'd like to talk a little more about what we're
doing and answer any questions that people have.

The A-Team is undertaking a project to move our test harnesses away from
using automation.py[1]. The primary goal here is maintainability. We've
long had to fix automation issues in multiple places because the code
that Talos and our unit test harnesses use was not shared. We've written
a set of modules ("mozbase"[2]) that encapsulate the important
functionality we've grown over the years in our various test harnesses.
Because they've been designed from the start as a set of Python modules,
they're easier to use in new projects, and also not as tied up with
specific harnesses. We've intended to make this transition for some time
now, but been held up by some blocking issues.

We're attacking this from multiple angles, since we have a myriad of
test harnesses. We're looking to first fix some of the simple
mini-harnesses like profileserver.py and leaktest.py that are only run
from object directories, as well as some of the B2G-specific harnesses,
since they have the fewest dependencies (mobile and B2G harnesses
inherit functionality from the desktop versions). We have a lot of work
to do, so we could certainly use some more help.

Our long-term goal will be to obsolete automation.py and have all
harnesses using mozbase modules directly. If we have necessary
functionality that's shared among harnesses that isn't currently
provided by mozbase we will add that functionality to mozbase as we
progress.

If you have any questions about the transition feel free to ask.

-Ted

1. https://bugzilla.mozilla.org/show_bug.cgi?id=775756
2. https://github.com/mozilla/mozbase/

[L. David Baron]

My main question is: what will continue to work and what will stop
working?

For example, which of these ways of running mochitests will still
work and which won't:
TEST_PATH=layout/style/test make mochitest-plain
_tests/testing/mochitest/runtests.py --test-path=layout/style --debugger=valgrind --debugger-args="..." --setenv=XPCOM_MEM_LOG_CLASSES=nsHTMLDocument
(other things that other people use should probably go here)

If particular things are going to stop working, I'd like to know
what they are; if you're planning to maintain compatibility with
existing ways to run and debug tests (and this is just a change in
what happens underneath), I'd also like to know that. It's not
clear to me from your message which sort of change this is.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla http://www.mozilla.org/ 𝄂

[Jeff Hammel]

The plan is to keep as much of the API/CLI initially the same. If there
will be planned/known breakages, then I will endeavor (and hope others
will do the same) to follow up to this list with the planned breakages.
This is intended as a refactor of the innards in stage one.

In subsequent stages, we are likely to change the way running e.g.
mochitests will work. Hopefully they will be simpler! I can't give any
answers as far as how they will change, as we (or at least I) am not
that far along enough yet. Suffice it to say, we will insist on not
dropping functionality (albeit one man's feature is another man's bug)
and will support as much of legacy methods as possible without having to
take huge steps backwards for them. But in this case also, we will
endeavor to communicate with dev-platform.

Jeff