Firefox Build Machine Downtime - 2008/11/25, 5am - 6am PST

[Ben Hearsum]

During the downtime today we discovered that one of our critical
machines is unpowered - causing slowness and occasional slave
disconnets, resulting in a burning tree. In order to maintain
reliability of service we will need to bring it offline for a short
period of time in order to upgrade it. At approx 5am PST tomorrow we
will be shutting off the following services:
* mozilla-central, mozilla-1.9.1, Firefox 3.0 builds, leak tests, and
unit tests. Talos machines will be up, but with nothing to test.

This closes all trees except Mozilla1.8.

We expect this routine upgrade to be brief and problem free.

If there is any reason why we shouldn't go ahead and do this please
respond to this post or e-mail rel...@mozilla.com

Thanks for your cooperation and apologies for the short notice.

- Ben

[Mike Beltzner]

On 24-Nov-08, at 3:03 PM, Ben Hearsum wrote:

> If there is any reason why we shouldn't go ahead and do this please
> respond to this post or e-mail rel...@mozilla.com

As long as that doesn't interfere with the planned creation of Beta 2
builds, I see no reason why not. :)

cheers,
mike

[John O'Duinn]

Ben Hearsum wrote:
> During the downtime today we discovered that one of our critical
> machines is unpowered -

errr... that should be "underpowered", not "unpowered"!

(The VM is up and running, we're just adding more RAM to the VM, and
upgrading the VMware tools.)

> causing slowness and occasional slave
> disconnets, resulting in a burning tree. In order to maintain
> reliability of service we will need to bring it offline for a short
> period of time in order to upgrade it. At approx 5am PST tomorrow we
> will be shutting off the following services:
> * mozilla-central, mozilla-1.9.1, Firefox 3.0 builds, leak tests, and
> unit tests. Talos machines will be up, but with nothing to test.
>
> This closes all trees except Mozilla1.8.
>
> We expect this routine upgrade to be brief and problem free.
>
> If there is any reason why we shouldn't go ahead and do this please
> respond to this post or e-mail rel...@mozilla.com

Note: this schedule is assuming we've been given a "go" for beta2
already. If FF3.1beta2 changes are still being landed tomorrow morning,
we'll hold off and wait until the formal "go". The important point is
that we want to fix this problem before we start the FF3.1beta2 builds...

> Thanks for your cooperation and apologies for the short notice.
> - Ben

tc
John.

[Ben Hearsum]

We're going to get a bit of a late start on this while waiting for the
nightlies to finish. We still expect to finish on time.

[Ben Hearsum]

This downtime was completed successfully. Builds, unit tests, and talos
should all be coming back up now.