moving Mozilla1.8 tinderboxes to Buildbot - perf impact
Robert Helmer
this likely do not watch m.d.builds only.
I've been working on moving the Mozilla1.8 branch tinderboxes to be
under Buildbot control, by merging nightly builders into the release
automation project http://wiki.mozilla.org/Build:Release_Automation
You can see the results of the staging machine:
http://tinderbox.mozilla.org/Mozilla1.8-Staging/
Currently there are 4 automatically forced builds per day (to keep the
boxes from falling off of the Tinderbox waterfall, and make sure we
get a timely nightly), and besides that it only builds on checkin. If
we can teach Tinderbox server not to drop these builders, then I think
the ideal would be once forced nightly release and besides that only
build on checkin.
This is all well and good, but the perf impact I mention in the title
is that the current build machines build as often as possible, and
perf machines pick up the results as soon as each build come out.
Builds generally outrun the perf results, but we get a lot of perf
runs as a side-effect of the constant build cycles.
I think only building on checkin plus one nightly release is the right
thing to do, as it reduces cycle time and makes machines available for
other tasks (releases, verification, etc). However, if we do this then
the perf boxes will sit idle until a checkin, and we get fewer perf
runs as a result.
If this is a problem, one idea I had was to just do as many perf runs
as possible on the same build, until a new one is available. However
the perf machines currently use the start time of the build they are
testing, this has the nice effect of matching up with checkins. The
problem we'll run into is overlap between the time a perf run started
and the time a build for it finished, we won't easily be able to
correlate checkins with perf runs.
if we want to do multiple perf run results per build, this is
something we'd need to teach tinderbox server and the graph server
about, as they currently don't support this. I think what would make
more sense is being able to declare a revision (branch/timestamp combo
is the equivalent for CVS) and also the time that the perf run
happened, right now there is no difference.
Any thoughts? If 4 times per day plus on checkin is not enough to get
reasonable perf numbers, we can go live with continuous build cycling
instead (this is the only issue I am blocked on!), just seems a shame
to do a new build when all we really need is a new perf run.
Schrep
> Cross-posting to m.d.performance this time, as people who care about
> this likely do not watch m.d.builds only.
>
> I've been working on moving the Mozilla1.8 branch tinderboxes to be
> under Buildbot control, by merging nightly builders into the release
Are you asking specifically for the 1.8 tree or assuming the same
infra will be used for 1.9 and Moz2? I'd say 4 builds per day as a
starter is just fine. It would be great to get the perf machines/
graph server to be able to submit multiple runs per build (to reduce
jitter amongst other things) but I wouldn't block on this. Assuming
we get a perf run after every checkin (as long as one isn't already
running) or a min of 4 per day sounds fine IMHO.
Rob Helmer
Same infrastructure, just rolling out to 1.8 branch first.
> I'd say 4 builds per day as a
> starter is just fine. It would be great to get the perf machines/
> graph server to be able to submit multiple runs per build (to reduce
> jitter amongst other things) but I wouldn't block on this. Assuming
> we get a perf run after every checkin (as long as one isn't already
> running) or a min of 4 per day sounds fine IMHO.
It would probably be ok, but I am starting to feel like we should get
the infra up and make sure it's stable before making visible changes
like this. We do not necessarily have to wait for the ability to
submit multiple runs per build, but I'd like a smooth transition
without worrying about side-effects from changing things like this.
A small number of people have expressed concerns to me directly about
this the performance aspect which is why I started the thread, and
I've told them that they need to weigh in here publicly, I'm not going
to make the perf or "our build system sometimes falls over" arguments
for them :)
I think we're probably be better off getting everything onto the new
infrastructure before we try to make any changes like this, if only to
separate the concern about this feature from the actual rollout (I
wouldn't want to have people calling for us to put the old system
back, for example). Once we're on buildbot, it's a simple switch in
the config file to turn BonsaiPoller on or off, so we can make that as
an isolated change, and be able to fall back to a system that we know
works.