Further improving what builds get triggered on what changesets

[Gijs Kruitbosch]

Hi,

I've been noticing that the number of builds + tests we run on certain
changesets could still do with improving. I realize that we try to be
conservative because we don't want "magically appearing" orange, but I
still think that it should be possible to do better here. That seems
like it might also help with recent capacity problems (maybe?).

I don't know who's responsible for improving this, so I'm just posting
this here. The list is biased towards frontend desktop work because that
is what I do. Perhaps other people can provide further examples or
insight from android, b2g, platform, releng or sheriff perspectives into
what could/couldn't be improved here.


Some examples:

(1)
https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=0fd6c4c24dc9

This checkin only touched a single test file in
browser/components/preferences/in-content/tests/

It triggered browser builds on all platforms, but also all kinds of B2G
builds and tests, which are clearly useless, as well as running all
kinds of unrelated test suites for the browser builds.

Ideally, commits that only touch things in directories matching **test**
should only run the relevant test suite, if possible even with builds
from the previous rev.

(2)
https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=2b1d7ebb8325

This touched files in:
browser/base
browser/components
browser/locales
toolkit/obsolete/content

I don't think any of these are used outside Firefox for Desktop, and yet
android and b2g were built, too. The same happened with:

https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=80bd1ae9dd0a
which touched toolkit/content/widgets/ which AIUI is also desktop-only.

(3)
Checkins (couldn't find one offhand because summaries are little help in
identifying these) that only touch windows/linux/osx styles in browser/
and/or toolkit should only run those builds


~ Gijs

[Gregory Szorc]

The work I'm doing in bug 1132771 lays the groundwork for associating
arbitrary metadata with files. I'm also making changes to moz.build files
to enable them to be evaluated outside a build context. i.e. anyone with a
copy of the source tree can get metadata for any file without having to
install build system dependencies, run configure, etc. This means machines
can query metadata from source without having to build anything.

While that work is currently geared towards making the patch contribution
pathway easier (automatic selection of bug component and reviewers for a
patch/commit), using this "framework" for more intelligent job selection
(suggested trychooser syntax selection, limiting what jobs run in
automation, etc) is an eventual deliverable (likely Q2 unless someone else
picks it up sooner). This will help enable Release Engineering's 2015
organizational goal to more intelligently run jobs based on what changed.

There should be real progress on this front in the next few months. Next
steps are to wait patiently, then start populating metadata in moz.build
files whenever it is asked of you (expect an email from me once bug 1132771
lands, hopefully by the end of next week).

Gregory

On Sat, Feb 21, 2015 at 9:11 AM, Gijs Kruitbosch <gijskru...@gmail.com>
wrote:

> _______________________________________________
> firefox-dev mailing list
> firef...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>

[Xidorn Quan]

In addition, code change in widget/{android,cocoa,gonk,gtk,windows,...}
should only need build and test in the corresponding platforms.

- Xidorn

On Sun, Feb 22, 2015 at 4:11 AM, Gijs Kruitbosch <gijskru...@gmail.com>
wrote:

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Ryan VanderMeulen]

See also: bug 849385 and associated discussion.

[Kim Moir]

There is also the work Joel has done to analyze test coverage and what
tests can be run less often

https://elvis314.wordpress.com/2015/02/06/seta-search-for-extraneous-test-automation/

I'm currently investigating how this can be incorporated into our buildbot
schedulers to reduce test load.

https://bugzilla.mozilla.org/show_bug.cgi?id=1131269
https://bugzilla.mozilla.org/show_bug.cgi?id=1127527

Kim