Generic Task Cluster Tasks / File Based Task Scheduling

[Gregory Szorc]

Over in bug 1245953 I implemented some new features for in-tree Task
Cluster tasks:

* You can now define tasks that are neither "build" nor "test" tasks. This
mechanism is probably where you should place one-off tasks such as linting,
docs generation, code analysis, etc. See
https://hg.mozilla.org/integration/mozilla-inbound/rev/623765c2381e for
more on this feature.

* You can now only run tasks if certain files changed. This opens the door
for drastic reduction in automation load via more intelligent scheduling of
tasks via in-tree configurations. (For the curious, relevant commits/files
are determined from a combination of the Mercurial pushlog and phases, so
pushes to Try not pushing all changesets will still pull in changes from
draft changesets previously pushed to Try.) See
https://hg.mozilla.org/integration/mozilla-inbound/rev/eee2e3b43fc1 for
more on this feature.

As part of this, the eslint task has been converted to a generic task and
now only runs if certain files change. This means e.g. C++ only pushes
shouldn't trigger eslint tasks.

The new features are somewhat half baked. There are plans to drastically
overhaul the in-tree Task Cluster code, so these new features were all
about providing a stop-gap to enable simpler authoring of new, one-off
tasks until that work is done. Don't let that discourage you from
implementing new tasks, however! See e.g.
https://hg.mozilla.org/try/rev/ce8ac426fb1f for how simple it is to define
new tasks that run on checkin. It's literally a bunch of YAML, a Try push,
and your custom tasks show up on Tree Herder. It's somewhat magical.

[Ben Hearsum]

On 2016-02-17 01:59 PM, Gregory Szorc wrote:
> * You can now only run tasks if certain files changed.

This is *awesome*. Thank you!

[Andrew Halberstadt]

On 17/02/16 01:59 PM, Gregory Szorc wrote:
> * You can now only run tasks if certain files changed. This opens the door
> for drastic reduction in automation load via more intelligent scheduling of
> tasks via in-tree configurations. (For the curious, relevant commits/files
> are determined from a combination of the Mercurial pushlog and phases, so
> pushes to Try not pushing all changesets will still pull in changes from
> draft changesets previously pushed to Try.) See
> https://hg.mozilla.org/integration/mozilla-inbound/rev/eee2e3b43fc1 for
> more on this feature.
>
> As part of this, the eslint task has been converted to a generic task and
> now only runs if certain files change. This means e.g. C++ only pushes
> shouldn't trigger eslint tasks.

Somewhat related, we should also only lint the files that were touched
by a given push. This ties in to the mozlint proposal I made awhile
back. I should really try and polish that up and get something landed.

[Henrik Skupin]

Gregory Szorc wrote on 02/17/2016 07:59 PM:

> * You can now define tasks that are neither "build" nor "test" tasks. This
> mechanism is probably where you should place one-off tasks such as linting,
> docs generation, code analysis, etc. See
> https://hg.mozilla.org/integration/mozilla-inbound/rev/623765c2381e for
> more on this feature.

This is great. Something else which comes into my mind while reading
your email is if we could reduce the turnaround time e.g. for try builds
in case such a build only contains test changes and wouldn't need a new
binary to be produced.

Not sure if that would work but if we could grab an already existent
binary from the inbound changeset the try push is based off, a lot of
machine power could be saved (compile time vs. download time).

--
Henrik