Changing what TryServer jobs are run by default

[John O'Duinn]

(If you do not use TryServer, you can hit delete now!)

hi;

The new TryServer-as-a-branch runs everything that mozilla-central does:
all the builds, unittests and talos suites, on all the OS. All included,
this runs to over 65 hours of compute time per push.

When rolling this new TryServer into production, we set the default to
run everything for every patch sent to TryServer. This gave us a chance
to make sure everything worked properly; more importantly, this gave us
time for people to find out how much of the new functionality they
actually used most frequently.

Based on feedback we've received so far, we're proposing changing
TryServer to *not* run Talos *by default*. Of course, we still support
running Talos on TryServer; the only difference is that now, if you want
Talos results for your TryServer patch, you'll need to ping the RelEng
buildduty person to ask them to trigger your Talos jobs for you. If you
have any concerns with this change-of-default, or think we're missing
something, please comment in
https://bugzilla.mozilla.org/show_bug.cgi?id=579573.

Obviously, this is just an short term interim step until we have a
self-service interface for tryserver in production. At that point, users
will have much more granular control over their own jobs - be able to
select which specific suites to run, and being able to re-run those
suites on the same build if needed. That work is being tracked in
bug#473184 and bug#520226

Thanks
John.

PS: Already mentioned elsewhere, but worth repeating: if you want
TrySrver to skip builds, and unittests, for a given OS for your own
patch, you can use a custom mozconfig. Simply include this mozconfig as
part of your patch. Details in:
https://wiki.mozilla.org/Build:TryServerAsBranch#Disabling_specific_platforms_for_try_push

[Mike Beltzner]

On 2010-07-19, at 12:36 PM, John O'Duinn wrote:

> Based on feedback we've received so far, we're proposing changing
> TryServer to *not* run Talos *by default*. Of course, we still support
> running Talos on TryServer; the only difference is that now, if you want
> Talos results for your TryServer patch, you'll need to ping the RelEng
> buildduty person to ask them to trigger your Talos jobs for you. If you

Commented in the bug - this change seems virtuous, since most people are looking for compilation and test runs, but chasing down the buildduty sheriff is difficult and not a good workflow as opposed to being able to check a box on the tryserver page and have it happen automagically.

cheers,
mike

[Nicholas Nethercote]

Indeed. Which is why John's next paragraph said the following:

> Obviously, this is just an short term interim step until we have a
> self-service interface for tryserver in production. At that point, users
> will have much more granular control over their own jobs

Nick

ps: This is good stuff, RelEng folks, thanks for all your hard work!

[Mike Beltzner]

>> Commented in the bug - this change seems virtuous, since most
>> people are looking for compilation and test runs, but chasing down
>> the buildduty sheriff is difficult and not a good workflow as
>> opposed to being able to check a box on the tryserver page and have
>> it happen automagically.
>
> Indeed. Which is why John's next paragraph said the following:
>
>> Obviously, this is just an short term interim step until we have a
>> self-service interface for tryserver in production. At that point,
>> users will have much more granular control over their own jobs

I did not take that paragraph to mean the same thing, but maybe that's
because I define "self-service interface for tryserver" to be something
more than an additional checkbox. I haven't seen a good sketch of the
full plans for this eventual UI.

Even still, I don't think we can make this change without making it very
easy to say "I would like Talos runs with this build." I don't care if
the backend is that an email gets sent to all of releng and they handle
it; I care about making it simple to get performance data for tryserver
builds.

cheers,
mike

[Bobby Holley]

I'd imagine that the full self-service web interface will probably take some
work. What if, in the interim, we were to do talos runs if the commit
message of the new head revision contained the string "TALOS"? This would
make it dead simple to trigger perf runs with push-to-try.

Keep up the awesome work. Armen can attest to my being Mozilla RelEng's
biggest fan. <3

-bholley

On Tue, Jul 20, 2010 at 12:20 AM, Nicholas Nethercote <
n.neth...@gmail.com> wrote:

> On Tue, Jul 20, 2010 at 4:08 AM, Mike Beltzner <belt...@mozilla.com>
> wrote:
> > On 2010-07-19, at 12:36 PM, John O'Duinn wrote:
> >
> >> Based on feedback we've received so far, we're proposing changing
> >> TryServer to *not* run Talos *by default*. Of course, we still support
> >> running Talos on TryServer; the only difference is that now, if you want
> >> Talos results for your TryServer patch, you'll need to ping the RelEng
> >> buildduty person to ask them to trigger your Talos jobs for you. If you
> >

> > Commented in the bug - this change seems virtuous, since most people are
> looking for compilation and test runs, but chasing down the buildduty
> sheriff is difficult and not a good workflow as opposed to being able to
> check a box on the tryserver page and have it happen automagically.
>
> Indeed. Which is why John's next paragraph said the following:
>
> > Obviously, this is just an short term interim step until we have a
> > self-service interface for tryserver in production. At that point, users
> > will have much more granular control over their own jobs
>

> Nick
>
> ps: This is good stuff, RelEng folks, thanks for all your hard work!

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management
>

[Aki Sasaki]

On 7/19/10 10:13 PM, Mike Beltzner wrote:

> I did not take that paragraph to mean the same thing, but maybe that's
> because I define "self-service interface for tryserver" to be something
> more than an additional checkbox. I haven't seen a good sketch of the
> full plans for this eventual UI.

http://yfrog.com/0yriozwj

I really want it to look just like that. :)

[Aki Sasaki]

On 7/19/10 10:13 PM, Mike Beltzner wrote:

> I did not take that paragraph to mean the same thing, but maybe that's
> because I define "self-service interface for tryserver" to be something
> more than an additional checkbox. I haven't seen a good sketch of the
> full plans for this eventual UI.

http://yfrog.com/0yriozwj

[Lukas Blakk]

That's an initial brainstorm sketch and should not be confused with a
"full plan" for the eventual UI. The first steps I'll be working on in
a couple of weeks will be allowing for a commit message and/or a config
file checked in with your try push to dictate what gets built & tested.
That should help developers and releng which the UI comes together using
that functionality. Fixing that issue will allow for a more thorough
UI/Dashboard plan with sketches and more steps toward try improvement.

It was really great to chat about Try with many of you at the summit and
all your ideas are helping form the future of this dashboard. Keep the
feedback coming, and follow/add bugs to
https://bugzilla.mozilla.org/show_bug.cgi?id=572808

Lukas

[John O'Duinn]

hi;

The original topic of this thread (turning off talos by default) seems
to be fine with everyone, and we certainly can put the CPU cycles to
better use, so today we changed TryServer to not run talos jobs by default.

If you *do* need Talos run on your TryServer patch, please ping the
RelEng buildduty person, or file a bug in
mozilla.org:ReleaseEngineering, and we'll happily trigger it for you.

The rest of this thread has morphed into discussions of possible UI for
TryServer. All the suggestions (parsing hg push comments; a web UI; new
features in mercurial1.6, local config files, etc) that were proposed
here and at Summit are all great. Please follow along in bug#572808, and
if you have a suggestion thats not already covered in the bug, please
chime in there.

Thanks
John.
=====