What do we want to do with the Valgrind builds?
Phil Ringnalda
there happen to be 5 slaves free after the once a day trigger. In
theory, I think, we have them on both Linux and Linux64, though
apparently there haven't been 5 free Linux64 slaves since forever, or
maybe those are turned off.
They are likely to be red a fair amount. We don't have it running on
Try, it's not currently something that everyone does themselves before
pushing, we've already had one transient failure in bug 621054
(apparently in a non-WebGL path, so it went away when that was enabled),
and bug 620828 wants to make both our existing leaks and future leaks
turn it red.
If I r+ bug 617431, then they'll be visible on tbpl, and then some
random push at 8am on a slow day, or more likely at 10pm or 1am after a
busy day, will have a red build against it from something that it did
not cause, and nobody will know what to do with it, and if they don't
hide it in tinderbox by the time I get up then I will.
Do we want Valgrind, badly enough to do what we need to do, have it
running on Try and running on every push, rather than as this idle-time
nightly?
The only way I can see to make the current once-a-day thing acceptable
would be to hide it, at which point the only way that I've been
realizing it ran, by seeing firebot say "Firefox: Build 'valgrind-linux'
added to tinderbox. (Status: Burning)," will go away, and then nobody
will ever notice anything it does.
L. David Baron
> Do we want Valgrind, badly enough to do what we need to do, have it
> running on Try and running on every push, rather than as this
> idle-time nightly?
I think there's value in having tests that run once a day. If we
look at the results, it can be quite useful, even though it doesn't
run for every push. Running more often is obviously better, but
there may be more value in using our limited resources to run more
tests less often rather than run fewer tests more often.
However, I think we need way to present the results of tests that
run once a day so that we see them, but that they aren't "blamed" on
one push and they don't cause immediate tree closure when they fail.
Tinderbox / TBPL clearly doesn't do that. I don't know what the
right way is, though if the tests have clear failure/success
conditions, then email to dev.tree-management might be. I'd love to
hear better ideas, though.
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
Axel Hecht
FWIW, if the schedulers are done to do so, those builds would be
associated with all changes since the last build, and if the build
status exported that, it'd show up.
The other question is if there are implications for the tree policies
when a test is run occasionally. Like, it's tough to back out a change
that's almost 24 hours old.
Axel
Nicholas Nethercote
>
> They are likely to be red a fair amount. We don't have it running on
> Try, it's not currently something that everyone does themselves before
> pushing, we've already had one transient failure in bug 621054
> (apparently in a non-WebGL path, so it went away when that was enabled),
> and bug 620828 wants to make both our existing leaks and future leaks
> turn it red.
Until we have experience, that's just speculation about how often
it'll be red.
Besides, if it's not automated, we'll just end up in the current
situation where someone
(e.g. jseward) will run it manually every so often and report the
problem, and someone will have to fix it. Because Valgrind's usually
right, and the things it complains about (memory leaks, bad memory
accesses, uses of undefined values, etc.) almost always warrant fixing
quickly.
That avoids the awkwardness of an obviously red/orange tree by hiding
the problem. That doesn't sound good to me. It also leads to less
useful regression ranges than automated daily testing does.
How slow is the Valgrind test run? From the way it's scheduled it
sounds like it takes a long time. But I thought the idea was for it
to run a tiny fraction of the usual test suite, precisely because it
does cause a big slowdown.
Nick
Mark Banner
> However, I think we need way to present the results of tests that
> run once a day so that we see them, but that they aren't "blamed" on
> one push and they don't cause immediate tree closure when they fail.
> Tinderbox / TBPL clearly doesn't do that. I don't know what the
> right way is, though if the tests have clear failure/success
> conditions, then email to dev.tree-management might be. I'd love to
> hear better ideas, though.
From a tinderbox perspective, I think you either send the results to a
different tree, or classify the builders (somehow) as non-tree-closing.
From a TBPL persepective, I think it could be extended to integrate
these builders into a seperate column/location of one-off builds. This
would also mean it would make better sense for nightly builds.
Standard8
Chris AtLee
> How slow is the Valgrind test run? From the way it's scheduled it
> sounds like it takes a long time. But I thought the idea was for it
> to run a tiny fraction of the usual test suite, precisely because it
> does cause a big slowdown.
Total time is anywhere from 20 minutes to 2 hours. The bulk of the time
is actually doing the special valgrind build, not running the tests, so
if we already have an object dir handy, the recompile is usually quick.
Mike Shaver
> On 03/01/11 07:23 PM, Nicholas Nethercote wrote:
>>
>> sounds like it takes a long time. But I thought the idea was for it
>> to run a tiny fraction of the usual test suite, precisely because it
>> does cause a big slowdown.
>
> actually doing the special valgrind build, not running the tests, so if we
> already have an object dir handy, the recompile is usually quick.
Maybe we should look at making valgrind the default config -- IIRC it
doesn't have a lot of overhead, and we can include the valgrind
headers easily.
Mike