Replacing tinderbox proposal
Axel Hecht
on the l10n buildbot, I'm running a web-display alternative to tinderbox
and the buildbot waterfall, and I'd like to get some opinions on whether
that's the right way for us.
Basic design:
The buildbot master feeds a database with its current status
information, and an independent webapp on top of that database is used
to visualize it.
Pros:
- The buildbot master is well protected from outside web traffic, and
only writes to the database server.
- The web interface can be developed and scaled independently of
buildbot. It's really just like any other webdev with a database backend.
- It ain't tinderbox ;-)
Cons:
- The database schema is rather tied to buildbot, apparently, and thus
it ties us harder to buildbot than before
- It's yet another system that nobody else uses, doesn't win us a lot
over tinderbox in a community aspect. At least at start.
- The database status hooks on the master are non-standard.
On to the details of the current implementation:
I'm using django as backend, mostly because that's in python which I can
hack and it was recommended by webdev. Other projects like verbatim are
on django, now, too.
You can see a test waterfall display on
http://l10n.mozilla.org/stage/builds/waterfall. It already supports a
few nice features. Say
http://l10n.mozilla.org/stage/builds/builds_for?change=25232, which
shows all builds for that change. It does show steps for builds
currently running, but only complete builds for done builds. If there
are parallel builds on a single builder, it automatically adds another
column, and only as many as required to be shown. You can query by build
property, like http://l10n.mozilla.org/stage/builds/waterfall?locale=el.
(Querying by buildername used to work, but is apparently regressed.)
I have some tests already, though (read regression above) really only
simplistic ones, using the django testing infrastructure. Hooking up
windmill for testing should work, I got a test starting up fx already,
but I didn't actually code a test yet.
Limitations so far:
- I'm falling back to the buildbot status display for log displays for
now, probably requires some upstream buildbot changes to fix this properly.
- No time column (I do have a blame colum, though, and there are hovers
with start time)
- the table cells in the waterfall are out of sync at times
- no "Trees", but I have a concept in mind to alias a Tree with a
default query
-- and a tree rules message and open/closed state
- no build comments yet. Should those go by build or by step? Should
they require auth or just captcha?
As you can tell, my webdev skills aren't the best, but improving this
system should be fairly easy. I'm making good progress in creating
fixtures to boot test databases with sample data, on which webdev or
others could actually give us all their skills.
What do others think? Shaver was looking in particular at :bs, ted, bz,
ctalbert, vlad, at least.
Questions, too?
Axel
PS: Code is at http://hg.mozilla.org/users/axel_mozilla.com/django-site/
Robert Kaiser
> Basic design:
>
> The buildbot master feeds a database with its current status
> information, and an independent webapp on top of that database is used
> to visualize it.
So the same as tinderbox, just completely tied to buildbot (bye, bye,
1.8 tinderboxes that still run for SeaMonkey and Thunderbird) and
without anyone else using it, and completely rewritten with completeley
new bugs and probably in an even worse language (say, python)? Phew, big
win. :-/
Robert Kaiser
Ben Hearsum
> Hi all,
>
> on the l10n buildbot, I'm running a web-display alternative to tinderbox
> and the buildbot waterfall, and I'd like to get some opinions on whether
> that's the right way for us.
>
> Basic design:
>
> The buildbot master feeds a database with its current status
> information, and an independent webapp on top of that database is used
> to visualize it.
>
I really like the idea of splitting out the frontend from the Buildbot
backend.
One of my big requirements here is that we do *not* implement in a
deeply invasive way. I'm vaguely familiar with your buildbot/django
mashup and IMHO, it's not the way to go. One of the main reasons we
switched to Buildbot was that it is an upstream project which we can not
only contribute to, but receive bugfixes and enhancements from upstream.
IMHO, things that touch core Buildbot code make this much more difficult.
I've discussed the idea of having a Buildbot Status plugin which dumps
information into a database with a few people. This is a much more
favourable way to go in the short term, IMHO. I think that this could be
written in a generic way, too, and be upstreamable. In the long term
we'd probably want to use whatever database backend Buildbot one-oh uses.
> Pros:
>
> - The buildbot master is well protected from outside web traffic, and
> only writes to the database server.
> - The web interface can be developed and scaled independently of
> buildbot. It's really just like any other webdev with a database backend.
> - It ain't tinderbox ;-)
>
+++++
> Cons:
>
> - The database schema is rather tied to buildbot, apparently, and thus
> it ties us harder to buildbot than before
> - It's yet another system that nobody else uses, doesn't win us a lot
> over tinderbox in a community aspect. At least at start.
>
As mentioned above I don't think this is necessarily the case. More
importantly, I don't think the maintenance of *only* a frontend is as
big of a deal.
Axel Hecht
> On 1/8/09 11:54 AM, Axel Hecht wrote:
>> Hi all,
>>
>> on the l10n buildbot, I'm running a web-display alternative to tinderbox
>> and the buildbot waterfall, and I'd like to get some opinions on whether
>> that's the right way for us.
>>
>> Basic design:
>>
>> The buildbot master feeds a database with its current status
>> information, and an independent webapp on top of that database is used
>> to visualize it.
>>
>
> I really like the idea of splitting out the frontend from the Buildbot
> backend.
>
> One of my big requirements here is that we do *not* implement in a
> deeply invasive way. I'm vaguely familiar with your buildbot/django
> mashup and IMHO, it's not the way to go. One of the main reasons we
> switched to Buildbot was that it is an upstream project which we can not
> only contribute to, but receive bugfixes and enhancements from upstream.
> IMHO, things that touch core Buildbot code make this much more difficult.
>
> I've discussed the idea of having a Buildbot Status plugin which dumps
> information into a database with a few people. This is a much more
> favourable way to go in the short term, IMHO. I think that this could be
> written in a generic way, too, and be upstreamable. In the long term
> we'd probably want to use whatever database backend Buildbot one-oh uses.
>
"Deeply invasive" is not what I'm doing.
http://hg.mozilla.org/users/axel_mozilla.com/django-site/file/tip/bb2mbdb/status.py
is a combination of a standard scheduler and a standard set of status
observers.
The issues are more about the database schema and me using django to
create it.
The talks on the buildbot list about using a sqlite backbone don't seem
to scale to me, nor being independent enough of the database backend to
then work on mysql. I at least found a series of problem when moving
over from sqlite to mysql on django, too. And then another set of
problems when testing again on sqlite.
Upstream would be nice here, but it's really not a huge piece of code.
And upstream might make the webdev piece tough to adapt to that other
database scheme.
Axel
Vladimir Vukicevic
> What do others think? Shaver was looking in particular at :bs, ted, bz,
> ctalbert, vlad, at least.
It's interesting, but I don't think that a waterfall is all that useful
for us any more. We now have things that are directly keyed by
changeset ID, and we have multiple chunks of data that all refer to the
same changeset ID -- the time is largely irrelevant (and is still
available for things like tracking down regressions).
If we're building a new system, I'd much prefer something based on that
-- for each changeset that's built by any machine, show the build status
for each machine (including "in progress"), the test results, etc. all
tied directly to that build ID. A separate display on the same page
would basically show just the pushlog, to correlate people's commits to
each build ID (can't really display that as part of the build piece,
because different machines might pick up different commit ids if a few
people push in a short amount of time).
That's the data that people really want to look at -- for a particular
commit, what are the results of that commit across all platforms and
tests. So let's build a display based around that, if we're building
something new.
- Vlad
Justin Dolske
> It's interesting, but I don't think that a waterfall is all that useful
> for us any more.
...
> If we're building a new system, I'd much prefer something based on that
> -- for each changeset that's built by any machine, show the build status
> for each machine (including "in progress"), the test results, etc. all
> tied directly to that build ID.
Yeah, that's exactly what I've been thinking. Waterfalls are unwieldy
with lots of columns, and they don't directly show the information most
developers are interested in. You can get at it, but it involves lots of
scrolling, log checking, crossreferencing, and taking notes to keep
things straight.
I'm not sure if anyone has made a list of the different use-cases, but
for me the most common are:
1) Is is ok to push? A simple page with current tree status would
suffice, perhaps with some info about what the last fully tested
changeset was.
2) Was my changeset ok? No build/test failures, perf regressions, etc? I
want a simple per-changeset page that I can glance at to see "in
progress", "all good", or "uh-oh."
No doubt sheriffing has other use cases, as do build and IT.
Justin
Axel Hecht
http://l10n.mozilla.org/stage/builds/builds_for?change=25258 looks to me
like it'd be exactly that. It's not directly related to a changeset ID,
as those aren't really unique in a system with multiple repositories,
though, maybe they are. We do have different l10n repos with the same
revisions on l10n-central and releases/l10n-mozilla-1.9.1, so not
strictly, but maybe that's subject to interpretation.
Looking at it, mapping the same view to a query parameter for hg
'branch' and revision would be hard to do. Let me ponder about this a
bit, I'm getting ideas while a write this.
Axel
Ted Mielczarek
interesting part is getting more and useful data into a more hackable
webapp, that would let us get to more useful representations of that
data. As it stands, Tinderbox server is not easily hackable, and just
doesn't have lots of the data we could use.
That being said, I do like the Tinderbox architecture (broadly
speaking) of loose coupling between the clients and server, having
clients communicate only with a somewhat-defined API in email. If
we're going to replace Tinderbox, I think we should preserve this
loose coupling. Kairo's right that we do have some legacy Tinderbox
clients (like Firefox 3.0.x) and the loose coupling also makes it easy
for external people to point boxes at our Tinderbox (like static-
analysis-bsmedberg). Granted I don't care much if the interface is the
existing email one or some sort of Web API, but having a defined
public interface instead of simply having one side put data into a
database, and the webapp read it out is a big benefit. Preserving the
existing email interface would mean minimal disruption of legacy
clients. (And we already have TinderboxMailNotifier for buildbot, so
that's nice).
I don't think Kairo's point holds much water (no offense), in that
AFICT we're the only ones who care about Tinderbox server at this
point anyway, and the code is old and dusty and nobody wants to hack
it. (And when they do, it's difficult.) Points about language aren't a
great argument. I wouldn't care if it was Perl if it was well-written
and easy to setup, but it happens to be neither. I will say that the
biggest problem with a rewrite is that you have to work just to get
back to feature parity. (Which we still don't have when comparing
hgweb/pushlog to bonsai.)
-Ted
Clint Talbert
> The actual interface here is not the interesting part to me, the
> interesting part is getting more and useful data into a more hackable
> webapp, that would let us get to more useful representations of that
> data. As it stands, Tinderbox server is not easily hackable, and just
> doesn't have lots of the data we could use.
>
> That being said, I do like the Tinderbox architecture (broadly
> speaking) of loose coupling between the clients and server, having
> clients communicate only with a somewhat-defined API in email. If
> we're going to replace Tinderbox, I think we should preserve this
> loose coupling.
I'm really in agreement with Ted here. I think having a more-easily
hackable backend that preserves the existing interfaces will help us to
get back to feature parity with Tinderbox the fastest, and will give us
the most bang for the buck.
Now having said that...
I've been around the project for long enough to see several attempts at
"tinderbox redesign/tweaks", but not long enough to know all the tribal
knowledge about what tinderbox.next should look like. Given the
responses on this thread, it seems that we ought to clearly define what
it is we want a redesigned tinderbox to offer us before we start hacking
away at it.
So, unless I'm the only one who's unclear about what changes need to be
made, perhaps we should start putting down concrete requirements, and
analyze Axel's solution from that list of requirements. I imagine there
are things to learn from his work, but without knowing where it is we
ultimately want to end up, I can only speculate about what's good and
bad about this proposal.
I'm happy to start a new thread about this if there's interest, so we
don't hijack Axel's.
Clint
Axel Hecht
Actually, I doubt that a mail thread is good for feature req lists, how
about some wiki pages? As yes, I'm not sure either.
Axel
Axel Hecht
> The actual interface here is not the interesting part to me, the
> interesting part is getting more and useful data into a more hackable
> webapp, that would let us get to more useful representations of that
> data. As it stands, Tinderbox server is not easily hackable, and just
> doesn't have lots of the data we could use.
>
> That being said, I do like the Tinderbox architecture (broadly
> speaking) of loose coupling between the clients and server, having
> clients communicate only with a somewhat-defined API in email. If
> we're going to replace Tinderbox, I think we should preserve this
> loose coupling. Kairo's right that we do have some legacy Tinderbox
> clients (like Firefox 3.0.x) and the loose coupling also makes it easy
> for external people to point boxes at our Tinderbox (like static-
> analysis-bsmedberg). Granted I don't care much if the interface is the
> existing email one or some sort of Web API, but having a defined
> public interface instead of simply having one side put data into a
> database, and the webapp read it out is a big benefit. Preserving the
> existing email interface would mean minimal disruption of legacy
> clients. (And we already have TinderboxMailNotifier for buildbot, so
> that's nice).
I can't see backwards compat and significant improvements in UE in one
box. Like, the first level of improvement I see would be to get from
monolithic reporting to reporting by step. I know I'm extremist, but
here's my math: 70 locales * 3 platforms * 7 steps, if we'd do that like
tinderbox, 2 mails per step (start and stop/log). Hrm. The information
that's required for views like vlad proposed goes further still, either
into associating changes or build properties.
If we have to import legacy builds, maybe we can fake something, though
I wouldn't make any compromises for old releases running tinderbox
clients still.
The mail thing does come with the advantage that a downtime of the
tinderbox server doesn't matter much, but I'm not sure how much we
should sacrifice for the data in that time window.
Axel
Robert Kaiser
> That being said, I do like the Tinderbox architecture (broadly
> speaking) of loose coupling between the clients and server, having
> clients communicate only with a somewhat-defined API in email.
I also think we probably should think about at least supporting that
tinderbox email API, perhaps along with additional but easily
interoperable APIs (XHR or such?) that could allow us e.g. to push
intermediate info about progress while the build is still running.
The current Tinderbox mail API has the upside of being very simple, so
that it's easy to plug another client into it, and we probably should
keep it simple and transparent.
> I don't think Kairo's point holds much water (no offense)
I agree, this was a bit more tongue-in-cheek, actually.
Robert Kaiser
Axel Hecht
The first round of feedback was:
- killing tinderbox isn't a problem per-se
- there are a variety of requests on how to visualize build results
- using the buildbot built-in display doesn't do it
Which views are important and which features need to be is mostly
undisputed, too. I won't focus on that for now, as there are actually
disputed parts for the back-end.
There are two different opinions:
- use buildbot status plugin to update database (me, bhearsum up to
django, see below)
- use a looser coupling over some RPC API, REST or mail
I've chatted with bhearsum on the phone for a bit, and his main concerns
are less about the actual design but more about "who's going to own
that". Which is the reason he's skeptical about django, as that's a new
codebase for him. We do agree that there's a benefit in using the same
database abstraction on both the build and the presentation layer, though.
I'd like to summarize at where I see the key points when talking about
the close or loose coupling:
- binding the presentation layer to buildbot
- integration of legacy systems (fx3)
- integration of non-releng systems (bs' analytics)
- Binding to buildbot:
The question I see behind this one is mostly which data model to use.
The closer we are to buildbot, the more impact changes over there can
have. The further we are away from buildbot and thus, the data we
actually have, the more likely it is that we could present a view of
that data, if we just not had it thrown away before.
I really don't think it's worth the risk. And I'm not too thrilled about
creating an independent data model, and to maintain the additional code
for it.
- Integration of legacy systems still running on tinderbox:
I'm happy to just keep them reporting to tinderbox. Ok, no improvements
on the web front for them, but those are phasing out anyway. And they
don't suffer from the regressions in presentation that the shift to hg
and, to some extent, buildbot brought.
- Integration of non-releng systems (bs' analytics)
I think we should be more aggressive on bringing those systems upstream.
Like, the reason I started to work on this is that I have tasks in l10n
that I can only hardly get done because my systems don't integrate with
the main build system. If we decide a system is important enough to get
on the main page, it should be on a path to move to shared
infrastructure. That doesn't imply releng owning those, we're currently
gaining some experience in that while getting the l10n infrastructure on
sustained ground. For bs' system in particular, I could see the slave
being special for Benjamin, possibly at his home office, while the
master is in the colo. Then has the db locally (I assume here), and the
reporting on the build is done centrally.
While talking to bhearsum, it became obvious that we're not just talking
about the technical pieces, but also about project ownership and
resources. We're all scared of saying "yes" and then owning it, like we
usually do in Mozilla. That's OK. We'll survive that.
The bigger issue that I'm having in my head these days are the more or
less random failures we see in our infrastructure, and we should make
sure that buildbot is more part of the answer than part of the problem.
There's no good reason in fixing the reporting infrastructure when the
actual results are throwing dices.
Axel
Axel Hecht
<...>
>
> - Integration of legacy systems still running on tinderbox:
> I'm happy to just keep them reporting to tinderbox. Ok, no improvements
> on the web front for them, but those are phasing out anyway. And they
> don't suffer from the regressions in presentation that the shift to hg
> and, to some extent, buildbot brought.
>
> - Integration of non-releng systems (bs' analytics)
> I think we should be more aggressive on bringing those systems upstream.
> Like, the reason I started to work on this is that I have tasks in l10n
> that I can only hardly get done because my systems don't integrate with
> the main build system. If we decide a system is important enough to get
> on the main page, it should be on a path to move to shared
> infrastructure. That doesn't imply releng owning those, we're currently
> gaining some experience in that while getting the l10n infrastructure on
> sustained ground. For bs' system in particular, I could see the slave
> being special for Benjamin, possibly at his home office, while the
> master is in the colo. Then has the db locally (I assume here), and the
> reporting on the build is done centrally.
PS:
Technically, one could write a tinderbox mail parser and make it feed
the build db with as much as it knows about the build, and claim that it
was a single build step. I don't see that as a reason to make the
mainstream builds go through the slow and backwards-compat path, though.
Axel
Ted Mielczarek
> Technically, one could write a tinderbox mail parser and make it feed
> the build db with as much as it knows about the build, and claim that it
> was a single build step. I don't see that as a reason to make the
> mainstream builds go through the slow and backwards-compat path, though.
You know, I think I'm mostly concerned with not starting from scratch
and winding up in a situation like we are with the pushlog, where it's
a replacement for bonsai, except it's missing a bunch of features that
developers need. I agree that we can write off the 1.9.0 builds since
they'll eventually be unsupported. For external builds, I guess
there's no compelling reason to continue to use tinderbox's mail
format, so as long as we provide some sort of API for reporting, I'd
be happy. (Even if it was as simple as Tinderbox allows right, now
with "build started" / "build finished {success,testfail,burning}") If
we did a HTTP-based API it ought to be simple to write a buildbot
notifier just like TinderboxMailNotifier and just swap it in to let us
continue to support these scenarios.
-Ted