[Fwd: Caution Flag during EqualLogic downtime tomorrow]
John O'Duinn
Firmware upgrade work is all done and went well. All RelEng VMs are now
back online, accepting jobs and so far look just like the slaves are
building / unittesting just fine after the firmware update.
We now declare all these systems to be back online and working normally,
and end the "Caution Flag" time.
If you see any weirdness or problems that might be related to this
downtime, please go to Bugzilla and file a blocker bug in
mozilla.org:ReleaseEngineering.
take care
John.
-------- Original Message --------
Subject: Caution Flag during EqualLogic downtime tomorrow
Date: Mon, 11 May 2009 13:50:44 -0700
From: John O'Duinn <jod...@mozilla.com>
Reply-To: jod...@mozilla.com
To: dev. planning <dev-pl...@lists.mozilla.org>, dev-tree-management
<dev-tree-...@lists.mozilla.org>
Background: we need to upgrade firmware to handle high-load situations
on EqualLogic arrays. We held off doing this until after FF3.5b4
shipped, but can finally do this. Applying this firmware patch requires
powering off all EqualLogic arrays, and all VMs that are running on them.
What does this mean, for each of the different trees?
Thunderbird2.0
Firefox3.0
==========
* Trees will be closed from 7pm PST Tuesday 12th May until 7am PST
Wednesday 13th May.
mozilla-191
mozilla-central
tracemonkey
TryServer
=========
* will remain open, despite powering off 76 VMs
** all build/unittest/talos functionality will run as usual.
** expect delays, long queues from 7pm PST Tuesday 12th May until 7am
PST Wednesday 13th May.
** linux64 will be off.
* we want to avoid risky/large landings in this window, so we ask that
these landings be postponed until after all systems are back online again.
* all checkins to these trees must be approved by the sheriff on duty,
no exceptions.
* please use TryServer sparingly.
Hope all that makes sense - let me know if you have any questions/concerns.
take care
John.