Short notice tree-closing downtime Tuesday night 6pm-9pm PDT 19jul2011

[John O'Duinn]

hi;

Sorry for the short notice. I'm just out of a meeting with ravi@IT.

IT needs a tree-closing downtime tomorrow evening 6pm-9pm PDT. This will
impact all trees. This downtime is to upgrade and reboot routers in
SCL1. This is following-on after last Tuesday's downtime, details in
bug#662071.

This work is different to, and in addition to, the non-tree-closing
downtime that was already scheduled for tomorrow (Tuesday) night
http://blog.mozilla.com/it/2011/07/11/mozilla-scheduled-maintenance-712-tree-closure-14-192011-1800-pst-0100-utc/

As the routers are currently running traffic fine, its not a
full blown emergency outage. However, because all the config/admin
interfaces are broken, the routers are running in an unsafe, and
erroring, state. Now that IT have info on how to proceed, they would
like to proceed as quickly as possible.

Doing this tomorrow (Tuesday) means we avoid disrupting the 6.0b3 builds
on Wednesday. If we do not do this Tues evening, its unclear when IT can
next do this, or how long the routers will continue running in their
current state.

Further details in
https://bugzilla.mozilla.org/show_bug.cgi?id=671366#c6

(Sorry for the short notice on this; its news to all of us here.)

John.

[Nick Thomas]

The work affecting build systems is now complete and the trees are now
reopened. IT is still working on the connection between SJC and PHX, so
not all websites/services may not be back online yet.

Cheers,
Nick Thomas