Worth a downtime to clear the Talos timeouts oranges?

[Johnathan Nightingale]

So, everyone who's watched the tree lately knows about bug 478603 -
it's behind the frequent Talos oranges with "failed to initialize
browser". Basically, sometimes we take longer to shutdown than talos
expects, so that when the next test starts we still have the old
process lying around, and everything explodes. This is bad. During a
code freeze, it's particularly bad to have trees going orange, and we
don't want to teach people to ignore them, either.

Over in bug 480413, Alice has implemented a Tshutdown test so that we
can know about things which make shutdown terrible, and a happy side
effect of this is that we would no longer be counting on a timer to
guess about shutdown, we'd actually be watching and measuring it, so
the orange should go away.

That patch is reviewed and ready, but it means scheduled downtime to
push it to production. Obviously no one wants scheduled downtime
during a freeze. But no one wants almost perma-orange trees either.

Right now, comments in the bug suggest Alice and John are waiting
until post-freeze to push it, which is probably prudent. But I wanted
to bring it up here in case anyone wanted to make an impassioned plea
to have it happen sooner. Because I sort of want it to, but can grit
my teeth through a week more orange if that's going to be safest for
the tree.

How does it make YOU feel?

Johnathan

---
Johnathan Nightingale
Human Shield
joh...@mozilla.com

[Mike Beltzner]

Would it be hard to do this the Sunday night before the all-hands, or
early that Monday morning? I can guarantee there will be a distinct
lull in development during those times.

cheers,
mike

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management

[Shawn Wilsher]

On 4/17/09 12:47 PM, Mike Beltzner wrote:
> Would it be hard to do this the Sunday night before the all-hands, or
> early that Monday morning? I can guarantee there will be a distinct lull
> in development during those times.

That doesn't help us with our current problem of having lots of orange
talos that people are essentially ignoring. Because of our situation,
I'd support a downtime assuming it only takes a few hours.

Cheers,

Shawn

[alice nodelman]

I would be a supporter of the roll out - having the vista boxes
constantly orange is a pretty big interrupt in my day.

I also hacked into the Tshutdown patch that if a single test fails it
should continue on with the rest of the tests. So if the Ts failure
persists (and I hope that it doesn't) we would still get Tp/Tdhtml/etc
test results from that build.

I would expect this to take on the order or 3-4 hours to land, just to
be sure that everything is looking good and we haven't introduced any
regressions.

alice.

[Mike Shaver]

On Fri, Apr 17, 2009 at 5:26 PM, alice nodelman <anod...@mozilla.com> wrote:
> I would be a supporter of the roll out - having the vista boxes constantly
> orange is a pretty big interrupt in my day.

Yeah, I'm on board here too. Let's do it Monday, as early as we can?

Mike