Emergency Mozilla Downtime - 03/05/2009, 8pm - 11pm PST (0400 - 0700 03/06/2009 UTC)

[matthew zeier]

(Apologies for the short notice. We generally want to stick to our
normal advertised maintenance windows but sometimes technology works
against us.)

We will have an emergency maintenance window tonight from 8pm to
11:00pm PST to fix router configurations after some instability
earlier today.

The following work will take place:

* 8:00pm PST (0400 UTC) Turn up border1 BGP session to Mzima & Any2.
See bug 481755 for details. In order to reduce some router CPU load
and instability caused earlier this morning we shed several BGP peers.
During this window we’ll re-establish peering with upstream transit
providers in a controlled environment. No downtime expected.
Let us know if there are any objections as soon as possible.

[matthew zeier]

Sorry, in all that was going on today I forgot today is Thursday and
while Thursday's are normal maintenance nights, Thursdays during
release are not.

That said, we'll be tacking on the following two tasks:

* 7:00pm PST (0300 UTC) mrapp-stage02 reboot. See bug 480415 for
details. We'll be rebooting mrapp-stage02 to pick up changes to
ethernet bonding configuration. Duration 15 minutes.

* 7:00pm PST (0300 UTC) mozilla.org MX changes. We'll be changing DNS
MX records for mozilla.org. No downtime expected.

[matthew zeier]

We will have a scheduled maintenance window tonight from 7:00pm to
11:00pm PDT. The following changes will take place:

* 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up code updates (bug 482605). No downtime
expected.

* 7:00pm PDT (0200 UTC) intranet.mozilla.org memory upgrade. We’ll be
increasing the memory on the server behind intranet.mozilla.org (bug
485058). Duration 15 minutes.

* 7:00pm PDT (0200 UTC) IRC server upgrades. We’ll be upgrading the
IRC server daemon on our IRC servers this week. Tonight,
sand.mozilla.org will be upgraded. On Thursday, gravel.mozilla.org
will be upgraded. If you happen to be using a server that is taken
down for an upgrade, as long as your client is using irc.mozilla.org
for connecting, you will be automatically changed to the other server.
No downtime expected

* 7:00pm PDT (0300 UTC) wiki.mozilla.org & intranet wiki upgrades.
We’ll be upgrading the MediaWiki software on these sites. Duration 3
hours.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.

Feel free to comment directly in those bugs if you see issues past the
planned downtime.

[matthew zeier]

(Apologies for the short notice. We generally want to stick to our
normal advertised maintenance windows but sometimes technology works
against us.)

We will have an emergency maintenance window tonight from 10:15pm to
12:15am PDT.

The following work will take place:

* 10:15pm PDT (0515 UTC) addons.mozilla.org database replication re-
sync. We’ll be taking down the master MySQL database server to restart
slave replication after a catastrophic replication failure. Duration 2
hours.

As always, we aim to keep downtime to as little as possible, but
unexpected complications can arise causing longer downtime periods
than expected. All systems should be operational by the end of the
maintenance window.

Feel free to comment directly if you see issues past the planned
downtime.