Mozilla Scheduled Downtime - 02/10/2009, 7pm - 11pm PST (0300 - 0700 02/11/2009 UTC)
matthew zeier
11:00pm PST. The following changes will take place:
* 7:00pm PDT (0230 UTC) mail.mozilla.com / Zimbra Upgrade. We’ll be
upgrading Zimbra to 5.0.13. Duration 3-4 hours.
* 7:00pm PST (0300 UTC) services.addons.mozilla.org load balancer
change. We’ll be moving services.addons.mozilla.org (used in the Add-
ons Manager) behind the production Cisco ACE/Zeus ZXTM cluster. For
background, see this blog post. No downtime is expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment if you see issues past the planned downtime.
matthew zeier
11:00pm PST. The following changes will take place:
* 7:00pm PST (0300 UTC) OSPF change. We’ll be deploying a
configuration change on how the default route from BGP speaking
routers is injected into OSPF. See bug 466111 for more detail. No
downtime expected.
* 9:00pm PST (0500 UTC) addons.mozilla.org Cisco ACE/Zeus ZXTM
performance testing. We will be running a performance test of
addons.mozilla.org behind the production ACE/Zeus ZXTM cluster. See
bug 477130 for more information. This is the second load test similar
to bug 467502 but on the new production hardware. No downtime is
expected, however the test period is expected to run for eighteen (18)
hours.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bugs if you see issues past the
planned downtime.
matthew zeier
11:00pm PST. The following changes will take place:
* 7:00pm PST (0300 UTC) mpt-vpn.mozilla.com VPN configuration change.
We’ll be restarting OpenVPN to pick up some configuration changes. Any
existing VPN connections will be dropped. Duration 5 minutes.
* 7:00pm PST (0300 UTC) Level3 BGP turnup. We’ll be turning up transit
peering with Level3. No downtime expected.
* 7:00pm PST (0300 UTC) MySQL slave database configuration change.
We’ll be making a runtime change to all MySQL slave databases to
ensure they are read-only. See bug 477099 for details. No downtime is
expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bugs (or this blog) if you see
matthew zeier
11:00pm PST. The following changes will take place:
* 7:00pm PST (0300 UTC) addons.mozilla.org updates. We’ll be updating
addons.mozilla.org to pick up code updates (bug 478960) and
configuation changes (bug 475371). No downtime expected.
* 7:30pm PST (0330 UTC) Kernel upgrades. We’ll be doing kernel
security updates on various machines affecting the following services
for approximately 10 minutes each:
* mradm01.mozilla.org
* gravel.mozilla.org
* build graphs / pageload tests
* CVS, cvs-mirror, SVN, Hg
* LXR/MXR
* Despot, Doctor
* ftp.mozilla.org
* quality.mozilla.org
* videos.mozilla.org
* stage.mozilla.org
* people.mozilla.com
* Buildbot Try Server
* MDC Search
* Internet-facing mail relays, list servers, and outbound mail (mail
will be queued)
* Any sites using LDAP for authentication
* DNS/DHCP/wifi/telephone services in all Mozilla offices (except
Beijing)
* support.mozilla.com chat server
* irc.mozilla.org
* All database-backed websites and services, including but not
limited to: Bugzilla, Breakpad, Bonsai, SUMO, SpreadFirefox,
Addons, bouncer/download, Blogs, QMO, Litmus
* 9:00pm PST (0500 UTC) l10n.mozilla.org CPU upgrade. We’ll be adding
a second virtual CPU to the l10n.mozilla.org VM. Duration 10 minutes.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bugs if you see issues past the
planned downtime.
matthew zeier
11:00pm PST. The following changes will take place:
* 7:00pm PST (0300 UTC) support.mozilla.com / SUMO updates. We’ll be
updating support.mozilla.com to pick up code updates (bug 480342). No
downtime expected.
* 7:00pm PST (0300 UTC) crash-stats.mozilla.com (Socorro) database
maintenance & content push. We’ll be replacing a faulty DIMM module
(bug 479793) and picking up code updates (bug 480418). During this
time, crash-stats.mozilla.com will be offline. crash-
reports.mozilla.com will be up and accept crashes. They will be queued
and processed later. Duration 20 minutes.
matthew zeier
11:00pm PST. The following changes will take place:
* 7:00pm PST (0200 UTC) Netscaler OS upgrade. We’ll be upgrading the
Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up
code fixes. No downtime is expected.
matthew zeier
tonight. http://blog.mozilla.com/it/ has been updated to reflect the
changes.
- mz
* 7:00pm PDT (0200 UTC) Bugzilla Upgrade. We’ll be updating
bugzilla.mozilla.org to pick up upstream changes from the Bugzilla 3.2
branch. No downtime is expected.
* 8:00pm PDT (0300 UTC) developer.mozilla.org database maintenance.
We’ll be applying a bugfix from Mindtouch to Dekiwiki’s stored
procedures in the database, and then taking a snapshot of the master
database to use for resyncing the slaves. 30 minutes of downtime is
expected.
matthew zeier
* 7:00pm PDT (0200 UTC) addons.mozilla.org updates. We’ll be updating
addons.mozilla.org to pick up code updates (bug 482605). No downtime
expected.
* 7:00pm PDT (0200 UTC) tinderbox.mozilla.org updates. We’ll be
updating tinderbox.mozilla.org to pick up code updates (bug 390349).
No downtime expected.
* 8:00pm PDT (0300 UTC) Mailman updates. We’ll be updating Mailman on
both lists.mozilla.org and mail.mozilla.org (bugs 483050/483051). We
expect intermittent downtime but this won’t cause any lost mail or
postings. Duration 2 hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com updates and database
upgrade. We’ll be updating support.mozilla.com to pick up code updates
(bug 485743) and moving the backend database to new hardware (bug
484891). Duration 30 minutes.
* 7:00pm PDT (0200 UTC) chat-support.mozilla.com DNS change. We’ll be
moving chat-support.mozilla.com to a different load balancer and will
need to change DNS to do so (bug 484581). Users with a hard-coded IP
address should switch to the hostname. No downtime expected.
* 7:00pm PDT (0200 UTC) Deploy Internap Flow Control Platform. We’ll
be flipping the switch on the FCP to allow it to make live routing
decisions. No downtime expected.
* 7:00pm PDT (0200 UTC) IRC server upgrades. We’ll be upgrading the
IRC server daemon on our IRC servers this week. Tonight,
gravel.mozilla.org will be upgraded. If you happen to be using a
server that is taken down for an upgrade, as long as your client is
using irc.mozilla.org for connecting, you will be automatically
changed to the other server. No downtime expected
* 8:00pm PDT (0200 UTC) DHCP infrastructure changes. We’ll be
implementing an active/standby DHCP server setup which requires some
reconfiguration on the current DHCP server. No downtime expected.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com 1.0.1 updates. We’ll be
updating support.mozilla.com to pick up code updates (bug 487210). No
downtime expected.
* 7:00pm PDT (0200 UTC) www.spreadfirefox.com updates. We’ll be
updating www.spreadfirefox.com to pick up code updates (bug 481632).
Duration 3 hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) addons.mozilla.org updates. We’ll be updating
addons.mozilla.org to pick up code updates (bug 487217). No downtime
expected.
* 7:00pm PDT (0200 UTC) www.spreadfirefox.com updates. We’ll be
updating www.spreadfirefox.com to pick up code updates (bug 481632).
Duration 4 hours.
* 7:00pm PDT (0200 UTC) crash-stats.mozilla.com (Socorro) database
migration tests. We’ll testing the process of upgrading the production
database server from 32bit RHEL5 to 64bit RHEL5. During this time,
crash-stats.mozilla.com will be offline. crash-reports.mozilla.com
will be up and accept crashes. They will be queued and processed
later. Duration 4 hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) addons.mozilla.org updates. We’ll be updating
addons.mozilla.org to pick up code updates (bug 490857). No downtime
expected.
* 7:00pm PDT (0200 UTC) www.spreadfirefox.com updates. We’ll be
updating www.spreadfirefox.com to pick up code updates (bug 481632).
Duration 4 hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) TryServer maintenance. Release Engineering
will be resetting the hg repo being used under TryServer. See bug
491092 for details. Duration 15 minutes.
* 7:00pm PDT (0200 UTC) VM datastore migrations. In preparation for an
upcoming EqualLogic firmware upgrade, we’ll be moving several VM’s
underlying datastore to another iSCSI array. See bug 491767 for
details. No downtime is expected but we’re doing this during this
window in case there are issues with storage vmotion. The list of VMs/
hosts that will be moved are:
* production-master
* try-master
* try-win32-slave01
* try-win32-slave02
* try-win32-slave03
* graphs.mozilla.org / dm-graphs01
* build.mozilla.org / dm-wwwbuild01
* geodns01.sj.mozilla.com
* pm-ns03 (LDAP slave)
matthew zeier
* 7:00pm PDT (0200 UTC) developer.mozilla.org / MDC database
migration. We’ll be migrating the database for developer.mozilla.org
tonight. See bug 493053 for details. Duration 30 minutes.
* 7:00pm PST (0200 UTC) PHP 5.2 upgrades. We’ll be rolling out PHP 5.2
upgrades on the production web servers (bug 492430). We don’t expect
any user-facing downtime. Duration three hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com update. We’ll be updating
support.mozilla.com to pick up code updates (bug 495254). No downtime
expected.
* 10:00pm PDT (0500 UTC) developer.mozilla.org / MDC upgrade. We’ll be
updating the software for developer.mozilla.org tonight. See bug
493053 for details. Duration 60 minutes.
* 10:00pm PST (0500 UTC) LDAP upgrades. We’ll be updating OpenLDAP and
implementing an improved password policy module. All services that use
LDAP for authentication will be impacted. Duration 10 minutes.
matthew zeier
* 9:00pm PDT (0400 UTC) addons.mozilla.org update. We’ll be updating
addons.mozilla.org to pick up code updates (bug 496907). No downtime
expected.
* 9:30pm PDT (0430 UTC) support.mozilla.com update. We’ll be updating
support.mozilla.com to pick up code updates (bug 495789). No downtime
expected.
* 10:00pm PDT (0500 UTC) Live Chat (Openfire) restart. We’ll be
upgrading the LIve Chat plugins and restarting the Openfire service
(bug 495743). Duration 30 minutes..
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) svn.mozilla.org & viewvc.svn.mozilla.org
maintenance. We’ll be performing some server and software maintenance
on svn.mozilla.org. Duration 4 hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) addons.mozilla.org app cluster maintenance.
We’ll be performing some server and software maintenance on
addons.mozilla.org web server cluster. No downtime expected however,
we’re planning for this work to take 4 hours.
* 8:00pm PDT (0200 UTC) Database Server maintenance. We’ll be
performing kernel updates RAID hardware upgrades on most of our
database servers affecting the following services for approximately 15
minutes each:
* Various Mozilla blogs (blog.mozilla.com)
* All Mozilla intranet services
* Kubla
* addons.mozilla.org
* support.mozilla.com (both the site and chat services)
* pastebin.mozilla.org
* quality.mozilla.org
* reporter.mozilla.org
* www.spreadfirefox.com
* All wikis (external and internal)
* download.mozilla.org (bouncer)
* bugzilla.mozilla.org
* bonsai.mozilla.org
* litmus.mozilla.org
* graphs.mozilla.org
matthew zeier
* 5:00pm PDT (0000 UTC) cg-vmware01 ESX Upgrade. We’ll be updating ESX
on cg-vmware01 which will affect the following VMs for 2 hours:
cg-ecmascript01
landfill
tinderbox.bugzilla
cg-centos01
windows.bugzilla
oracle.bugzilla
* 7:00pm PDT (0200 UTC) developer.mozilla.org/MDC update. We’ll be
updating developer.mozilla.org/MDC to pick up code updates (bug
505301). Duration 30 minutes.
* 7:00pm PDT (0200 UTC) addons.mozilla.org update. We’ll be updating
addons.mozilla.org to pick up code updates (bug 508718). Duration 30
minutes.
* 7:00pm PDT (0200 UTC) www.spreadfirefox.com update. We’ll be
updating www.spreadfirefox.com to pick up code updates (bug 508852).
Duration 30 minutes.
* 10:30pm PDT (0530 UTC) DHCP infrastructure changes. We’ll be
implementing an active/standby DHCP server setup on the build network
which requires some reconfiguration on the current DHCP server. No
downtime expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned
downtime.
matthew zeier
* 9:00pm PDT addons.mozilla.org EV SSL certificate deployment. We’ll
be replacing the wild card certificate with a GeoTrust EV SSL
certificate tonight. No downtime expected.
matthew zeier
* 9:00pm PDT (0400 UTC) addons.mozilla.org update. We’ll be updating
addons.mozilla.org to pick up code updates (bug 512892). Duration 30
minutes.
* 9:00pm PDT (0400 UTC) GeoDNS update. We’ll be updating pushing out
code updates to our GeoDNS platform. These changes came from Jia Jun
from the Google Summer of Code project he worked on. Duration 30
minutes.
matthew zeier
* 8:00pm PDT (0300 UTC) LDAP infrastructure work. We’ll be refactoring
parts of our LDAP infrastructure to improve reliability. During this
time, sites that rely on LDAP authentication may be impacted. Duration
one hour.
matthew zeier
performing scheduled maintenance Monday from 10:00 am to 4:00pm PDT.
* 10:00am PDT (0300 UTC) mail.mozilla.org / Mailman hardware upgrade.
We’ll be upgrading the server that runs mail.mozilla.org and need to
copy over the Mailman list archives (all 89GB of it!). During this
time, lists behind this server will have their emails delayed and
queued up on on mail relays. See bug 501077 for more details. Duration
8 hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 6:00pm PDT (0100 UTC) developer.mozilla.org / MDC upgrade. We’ll be
updating the software for developer.mozilla.org tonight. See bug
511852 for details. Duration 60 minutes.
* 6:00pm PDT (0100 UTC) SiteSpect deployment. We’ll be implementing
multivariate testing with SiteSpect on www.mozilla.com. No downtime
expected.