Proposal: new modules for testing

[James Graham]

The current set of Testing modules under Core [1] have not been well
maintained, and substantial fraction of the people listed as owners or
peers are either no longer active project contributors, or no longer
actively in modules they own. At the same time, the breadth of the
testing code in the tree has increased, and the current structure in
which there are a couple of all-encompassing modules for everything test
related no longer reflects the de-facto ownership structure of the code.

In order to make the testing modules reflect reality, I'd like to make
the following proposal:

* Delete (or otherwise archive) the following Core modules:
- Test Harness
- Testing Infrastructure
- XPCShell Test Harness

* Create a new subdivision of the module hierarchy for testing modules.

* Add the modules listed below. The general structure of these, along
with the peer/owner assignments has been agreed with all the people
nominated as module owner or peer in the proposal. I've added some of
the required metadata (description, component, etc.) so any errors or
omissions are mine.

* Mochitest
- Description: Mochitest test framework
- Owner: ahal
- Peers: jmaher, gbrown
- Source dirs: testing/mochitest
- Bugzilla Component: Testing :: Mochitest
* Reftest (+ jsreftest + crashtest)
- Description: Reftest test framework
- Owner: dbaron
- Peers: jmaher, jwalden, kats
- Source dirs: layout/tools/reftest
- Bugzilla Component: Testing :: Reftest
* web-platform-tests infrastructure
- Description: web-platform-tests test framework
- Description: Infrastructure for running the cross-browser
web-platform-tests suite
- Owner: jgraham
- Peers: ato
- Source Dirs: testing/web-platform excluding
testing/web-platform/tests, testing/web-platform/meta and
testing/web-platform/mozilla, but including testing/web-platform/tests/tools
- Bugzilla Component: Testing :: web-platform-tests
* geckodriver + Marionette
- Description: WebDriver implementation
- Owner: ato
- Peers: dburns, whimboo, majazf, jgraham
- Source dirs: testing/marionette, testing/geckodriver
- Bugzilla Component: Testing :: Marionette and Testing ::
geckodriver
* Raptor
- Description: Raptor performance framework
- Owner: rwood
- Peers: davehunt
- Source dirs: testing/raptor
- Bugzilla Component: Testing :: Raptor
* Talos
- Description: Talos performance testing framework
- Owner: davehunt
- Peers: rwood, jmaher
- Source dirs: testing/talos
- Bugzilla Component: Testing :: Talos
* XPCShell
- Description: XPCShell test harness
- Owner: jmaher
- Peers: gbrown
- Source dirs: testing/xpcshell
- Bugzilla Component: Testing::XPCShell Harness
* gtest
- Description: GTest test harness
- Owner: gbrown
- Source dirs: testing/gtest
- Bugzilla Component: Testing :: GTest
* firefox-ui
- Description: Firefox UI test framework
- Owner: whimboo
- Peers: majazf
- Source dirs: testing/firefox-ui
- Bugzilla Component: Testing :: Firefox UI
* Mozbase
- Description: Base modules used for implementing test components
- Owner: <none>
- Peers: ahal, ato, bc, gbrown, jgraham, rwood, davehunt, whimboo
- Bugzilla Component: Testing :: Mozbase / Testing :: Mozbase Rust
* Tryselect
- Description: Frontend for selecting jobs on the try server
- Owner: ahal
- Peers: jgraham
- Source dirs: tools/tryselect
- Bugzilla Component: Firefox Build System :: Try
* mozharness
- Description: Configuration-driven script harness
- Owner: aki
- Peers: callek, tomprince
- Source dirs: testing/mozharness
- Bugzilla Component: Release Engineering :: Applications:
MozharnessCore

Note that mozbase is a little special in the above proposal in that it
doesn't have an overall owner. This is because there isn't any one
person who feels they have an understanding of the many subcomponents in
mozbase, but making each its own module in the module system seemed like
overkill.

I'm sure this proposal isn't perfect and doesn't cover all the code that
could be regarded as testing code. Nevertheless I think it's an
improvement over the status quo. Suggestions for improvement are
gratefully welcomed.

[1] https://wiki.mozilla.org/Modules/Core#Test_Harness

[Andreas Tolfsen]

Also sprach James Graham via governance:

> In order to make the testing modules reflect reality, I'd like to
> make the following proposal:

[…]

I’d just like to commend jgraham on running the long discussions
that led to this proposal overhauling the testing modules.

Its current state of disrepair has caused real day-to-day problems
related to acquiring commit access for new contributors, so I’d
like to see this proposal adopted sooner rather than later.

[Andrew McCreight]

On Thu, Jun 13, 2019 at 8:52 AM James Graham via governance <
gover...@lists.mozilla.org> wrote:

> Note that mozbase is a little special in the above proposal in that it
> doesn't have an overall owner. This is because there isn't any one
> person who feels they have an understanding of the many subcomponents in
> mozbase, but making each its own module in the module system seemed like
> overkill.
>

Modules should still have an owner. An owner doesn't need to understand
every piece of code they are responsible for. They just need to know who
knows about the different pieces, if anybody, and is willing to defer
decisions to those other people as needed.

Andrew

> I'm sure this proposal isn't perfect and doesn't cover all the code that
> could be regarded as testing code. Nevertheless I think it's an
> improvement over the status quo. Suggestions for improvement are
> gratefully welcomed.
>
> [1] https://wiki.mozilla.org/Modules/Core#Test_Harness

> _______________________________________________
> governance mailing list
> gover...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/governance
>

[James Graham]

On 19/06/2019 14:58, Andrew McCreight wrote:
>
>
> On Thu, Jun 13, 2019 at 8:52 AM James Graham via governance
> <gover...@lists.mozilla.org <mailto:gover...@lists.mozilla.org>> wrote:
>
> Note that mozbase is a little special in the above proposal in that it
> doesn't have an overall owner. This is because there isn't any one
> person who feels they have an understanding of the many
> subcomponents in
> mozbase, but making each its own module in the module system seemed
> like
> overkill.
>
>
> Modules should still have an owner. An owner doesn't need to understand
> every piece of code they are responsible for. They just need to know who
> knows about the different pieces, if anybody, and is willing to defer
> decisions to those other people as needed.
>

gbrown has agreed to take overall ownership of mozbase, so consider that
added to the proposal.

I'm not sure what else is needed to make progress here?

[Mike Connor]

With that open issue resolved, I think you're good to go. Thanks for all
of your hard work on sorting this out!

[ja...@hoppipolla.co.uk]

On Tuesday, 9 July 2019 15:30:30 UTC+1, Mike Connor wrote:
> With that open issue resolved, I think you're good to go. Thanks for all
> of your hard work on sorting this out!

I (finally) made the wiki changes to implement this. Please let me know if I missed anything or made any errors.

[wayne capper]