Mozilla Scheduled Maintenance - July 29, 2008, 7pm - 11:00pm PDT (0200 - 0600 UTC Wednesday)
matthew zeier
PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up recent security and code updates. No
downtime is expected.
Please let me know if you have any reason why we should not proceed with
the planned maintenance. As always, we aim to keep downtime to as little
as possible, but unexpected complications can arise causing longer
downtime periods than expected. All systems should be operational by the
end of the maintenance window.
matthew zeier
will take place:
* 7:00pm PDT (0200 UTC) wiki.mozilla.org updates. We’ll be moving
wiki.mozilla.org completely behind SSL. No downtime is expected.
* 7:00pm PDT (0200 UTC) www.mozilla.org DNS change. We’ll be offloading
www.mozilla.org to a CDN and pushing a DNS change to make that active.
No downtime is expected.
* 8:00pm PDT (0300 UTC) bugzilla.mozilla.org database work. Bugzilla
will be down or slow while we pull out some data to fix an issue related
to the Bugzilla reorganization. Duration 30 minutes.
Please let me know if you have any reason why we should not proceed with
the planned maintenance. As always, we aim to keep downtime to as little
as possible, but unexpected complications can arise causing longer
downtime periods than expected. All systems should be operational by the
end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 449397. Feel
free to comment directly in that bug if you see issues past the planned
downtime.
matthew zeier
* 7:00pm PDT (0200 UTC) www.mozilla.org DNS change. We’ll be offloading
www.mozilla.org to a CDN and pushing a DNS change to make that active.
This is a retry from lastweek’s attempt. No downtime is expected.
matthew zeier
> o SVN, Hg
> o MXR
> o build graphs / pageload tests
> o stage.mozilla.org
> o Any sites using LDAP for authentication
Sorry for any heartache.
matthew zeier wrote:
>
>
> We will have a scheduled maintenance window tonight from 7pm to 11:00pm
> PDT. This is being tracked in master bug 451221. The following changes
> will take place:
>
> * 7:00pm PDT (0200 UTC) developer.mozilla.org upgrade. MDC will be
> relaunched on MindTouch Deki. Duration 10 minutes. See Eric Shepherd’s
> blog for more information.
>
> * 7:00pm PDT (0200 UTC) duration 4 hours: Kernel upgrades. We’ll be
> doing kernel security updates on various machines affecting the
> following services for approximately 10 minutes each:
> o build graphs / pageload tests
> o CVS, cvs-mirror, SVN, Hg
> o LXR/MXR
> o doctor
> o despot
> o videos.mozilla.org
> o stage.mozilla.org
> o Buildbot Try Server
> o Any sites using LDAP for authentication
> o jumphost for community build lan
> o internal DNS and DHCP in:
> + MPT colo
> + Mountain View office
> + Toronto office
> o Telephone service in the above locations
>
> * 7:15pm PDT (0200 UTC) blog.mozilla.com upgrade. We’ll be picking up a
> Wordpress MU update. Duration ~ 5 minutes.
>
> * 7:15pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating
> support.mozilla.com to pick up code updates. No downtime is expected.
>
> Please let me know if you have any reason why we should not proceed with
> the planned maintenance. As always, we aim to keep downtime to as little
> as possible, but unexpected complications can arise causing longer
> downtime periods than expected. All systems should be operational by the
> end of the maintenance window.
>
> As mentioned, this maintenance window is tracked in bug 451221. Feel
> free to comment directly in that bug if you see issues past the planned
> downtime.
>
>
--
matthew zeier | Mgr. IT Operations | Mozilla Corp. | (650)903-0800 x219
matthew zeier
matthew zeier
will take place:
* 7:00pm PDT (0200 UTC) www.mozilla.com DNS change. As part of our CDN
trial/test, we’ll be offloading www.mozilla.com to CDNetworks and
pushing a DNS change to make that active. See bug 450744 for details. No
downtime is expected.
* 7:00pm PDT (0200 UTC) addons.mozilla.org updates. We’ll be updating
addons.mozilla.org to pick up code updates. No downtime is expected.
* 7:00pm PDT (0200 UTC) duration 4 hours: Kernel upgrades. We’ll be
doing kernel security updates on various machines. The full list of
affected machines and services is listed below.
* 8:00pm PDT (0300) UTC) bugzilla.mozilla.org database & app server
upgrades. Both the databases and app servers will be getting memory
upgrades to improve performance. Duration 30 minutes.
Tonight’s kernel upgrades will affect the following services for
approximately 10 minutes each:
* All database backed websites, including:
addons.mozilla.org
spreadfirefox.com
quality.mozilla.org
download.mozilla.org
Bugzilla
Bonsai
many others
* All sites that use LDAP for authentication
* Internal DNS services in the MPT colo
* people.mozilla.com
* SVN, Hg
* MXR
* half of irc.mozilla.org (gravel users will be failed over to sand)
* developer.mozilla.org
* mozilla-europe content management
* ftp.mozilla.org
* quality.mozilla.org
* External geodns nameservice
* web interface for bugzilla.org, mozilla.org and lists.mozilla.org
mailing lists
* search services for www.mozilla.com
* build graphs (old and new)
* pageload test server
* tinderbox symbol upload
* try build server
* stage.mozilla.org
* breakpad
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 451495. Feel
free to comment directly in that bug (or email me) if you see issues
past the planned downtime.
matthew zeier
We will have a scheduled maintenance window tonight from 8pm to 1:00am
PDT. The following changes will take place:
* 8pm PDT (0200 UTC) - mozilla.com mail upgrade. We’ll be upgrading
Zimbra to 5.0.9. Duration 3 hours.
* 9pm PDT (0400 UTC) - Bugzilla upgrade. We’ll be upgrading Bugzilla to
version 3.2. Duration 4 hours.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
Feel free to let me know if you see issues past the planned downtime.
matthew zeier
We will have a scheduled maintenance window tonight from 7pm to 11:00pm
will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up code updates. No downtime is expected.
* 7:00pm PDT (0200 UTC) tinderbox.mozilla.org updates. We’ll be updating
tinderbox.mozilla.org to pick up the latest code from CVS. Duration 2
hours. See bug 446547 for more details.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 453325. Feel
free to comment directly in that bug (or http://blog.mozilla.com/it/) if
Jonas Sicking
I'm keeping the tree closed since we needed to close it for tracemonkey
landing anyway. Please reopen and update
https://wiki.mozilla.org/Tree_Closures#Recent_Closures
When the upgrade is done.
/ Jonas
matthew zeier
* 7:00pm PDT (0200 UTC) addons.mozilla.org DNS change. As part of our
CDN trial/test, we’ll be offloading addons.mozilla.org to CDNetworks and
pushing a DNS change to make that active. See bug 450745 for details. No
downtime is expected.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned downtime.
matthew zeier
us a little more time to verify the logging format.
matthew zeier
PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) duration 4 hours: Red Hat 5 upgrades. We’ll be
picking up OS upgrades on several machines tonight. The following
services/hosts will be affected:
o cvs-mirror.nl.mozilla.com
o doctor.mozilla.org
o despot.mozilla.org
matthew zeier
PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) addons.mozilla.org DNS change. As part of our
CDN trial/test, we’ll be offloading addons.mozilla.org to CDNetworks and
pushing a DNS change to make that active. See bug 450745 for details. No
downtime is expected.
Please let me know if you have any reason why we should not proceed with
matthew zeier
We will have a scheduled maintenance window tonight from 7pm to 11:00pm
PDT. This is being tracked in master bug 455530. The following changes
will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up code updates. No downtime is expected.
* 7:00pm PDT (0200 UTC) addons.mozilla.org database work.
addons.mozilla.org will be offline to move the master database to a new
MySQL 5 cluster. Duration 15 minutes.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 455530. Feel
free to comment directly in that bug (or this blog) if you see issues
past the planned downtime.
matthew zeier
will take place:
* 7:00pm PDT (0200 UTC) addons.mozilla.org updates. We’ll be updating
addons.mozilla.org to pick up code updates. No downtime is expected.
* 7:00pm PDT (0200 UTC) MDC / developer.mozilla.org updates. We’ll be
updating developer.mozilla.org to pick up code updates. No downtime is
expected.
* 7:00pm PDT (0200 UTC) Labs Forum upgrade. The Labs Forum software will
be switched to Vanilla. Duration 2 hours.
* 7:00pm PDT (0200 UTC) blog/wiki database restart. We need to restart
MySQL on one of our database masters to change some configuration
settings. This will affect all of the sites listed below. Duration 5
minutes.
o www.spreadfirefox.com
o All Blogs
o All Wikis
o All Intranet Sites
o All Forums
o support.mozilla.org
o reporter.mozilla.org
o pastebin.mozilla.org
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 455896. Feel
matthew zeier
will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up code updates. No downtime is expected.
* 7:00pm PDT (0200 UTC) spreadfirefox.com updates. We’ll be updating
spreadfirefox.com to pick up code updates. No downtime is expected.
* 7:00pm PDT (0200 UTC) Wiki upgrades. We’ll be upgrading all Intranet
wikis hosted by Mozilla. This will affect anything under
intranet.mozilla.org. Duration 3 hours.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 456442. Feel
matthew zeier
will take place:
* 7:00pm PDT (0200 UTC) addons.mozilla.org/*.addons.mozilla.org database
restart and code updates. We’ll be updating addons.mozilla.org to pick
up code updates. We’ll also be restarting the master database to change
some configuration settings. See bug 458110 for more details. This will
impact addons.mozilla.org and all related sub-domains. Duration 15 minutes.
* 7:00pm PDT (0200 UTC) Final MySQL 5 migration. We will be completing
our last MySQL 4 master to MySQL 5. The following services will go
offline for up to 30 minutes while they are moved to a new master database:
o Bonsai
o Despot
o Graph Server
o Litmus
* 9:00pm PDT (0200 UTC) San Jose Netscaler software upgrade. We’ll be
doing a vendor recommended upgrade on the San Jose Netscalers. No
downtime expected.
* 10:00pm PDT (0200 UTC) Cisco IOS upgrade. We’ll be upgrading IOS on
two of the embedded HP BladeSystem Cisco switches to pick up a bug fix
(BugID CSCsf17036). The systems attached to these switches are
multi-homed. No downtime expected.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 458901. Feel
matthew zeier
will take place:
* 7:00pm PDT (0200 UTC) Thirdlane PBX upgrades. We will be upgrading the
configuration GUI and its underlying Asterisk add-on features for our
phone system. The phone system itself in Mountain View will probably be
down less than 5 minutes. Specific features (conference call rooms,
provisioning of new telephones, calls to phones in Toronto) may be
offline for up to 6 hours. Duration 6 hours
* 7:00pm PDT (0200 UTC) Mailman / mailing list upgrades. We’ll be
upgrading Mailman to 2.1.11 and upgrading the host OS to RHEL5. This
will affect all mailing lists and all email services under
lists.mozilla.org, mail.mozilla.org, mozilla.org and bugzilla.org.
Duration 3 hours.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 459262. Feel
guza...@hotmail.com
--------------------------------------------------
From: "matthew zeier" <m...@mozilla.com>
Sent: Friday, October 10, 2008 5:11 AM
To: "matthew zeier" <m...@mozilla.com>
Cc: <dev-pl...@lists.mozilla.org>; <dev-moz...@lists.mozilla.org>;
<gen...@lists.mozilla.org>
Subject: Mozilla Scheduled Downtime - 10/09/2008, 7pm - 11pm PDT (0200 -
060010/10/2008 UTC)
> _______________________________________________
> general mailing list
> gen...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/general
>
matthew zeier
will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up code updates. No downtime is expected.
* 7:00pm PDT (0200 UTC) Mailman / mailing list upgrades. We’ll be
upgrading Mailman to 2.1.11 and upgrading the host OS to RHEL5. This
will affect all mailing lists and all email services under
mail.mozilla.org, mozilla.org and bugzilla.org. Duration 3 hours.
* 7:30pm PDT (0230 UTC) mail.mozilla.com / Zimbra Upgrade. We’ll be
upgrading Zimbra to 5.0.10. Duration 3-4 hours.
* 8:00pm PDT (0300 UTC) ) MDC / developer.mozilla.org updates. We’ll be
moving developer.mozilla.org completely behind SSL. No downtime is expected.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 460825. Feel
matthew zeier
will take place:
* 8:00pm PDT (0300 UTC) ) MDC / developer.mozilla.org updates. We’ll be
updating developer.mozilla.org to pick up code updates. No downtime is
expected.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 460826. Feel
matthew zeier
normal advertised maintenance windows but sometimes technology works
against us.)
We will have an emergency maintenance window tonight from 6pm to 11:00pm
PST.
The following work will take place:
* 6:00pm PST (0200 UTC) ) EqualLogic storage array firmware upgrade.
See bug 464458 for details. We’ll be locking the tree to all check-ins
at 5:00 PST tonight to start the maintenance window and will unlock it
once the machines have come back up and gone green for a cycle.
Last night one of our network attached storage devices experienced an
error and didn’t fail over properly. The effect was such that we had to
close the tree as Tinderbox wasn’t getting proper data for updates, and
we couldn’t tell what was happening with our builds. While we were able
to restore that functionality, there is a high chance that any of our
VMs attached to this storage device may experience further problems.
To fix this, we’ll need to pause all of our VM systems (most notably
build machines & unit test machines, but also litmus and some others)
for a few hours. The affected systems would be:
fxdbug-win32-tbox
fx-win32-1.9-slave2
moz2-linux-slave1
patrocles
fx-linux-1.9-slave09
try-master
try-unit-linux-01
xr-linux-tbox
egg
fxdbug-linux-tbox
fx-linux-1.9-slave2
moz2-linux64-slave01
l10n-linux-tbox
prometheus-vm
tb-linux-tbox
try1-win32-slave
try-unit-linux-02
bm-l10n-win2k3-01
dm-bugs-test-app01
dm-graphs-stage01
dm-litmus02
dm-mailman01
fx-win32-1.9-slave08
fx-win32-1.9-slave09
moz2-linux-slave04
moz2-win32-slave04
pm-app01
sand
sm-summit01
moz2-linux-slave09
moz2-linux-slave10
moz2-linux-slave11
moz2-linux-slave12
moz2-win32-slave09
moz2-win32-slave10
moz2-win32-slave11
moz2-win32-slave12
moz2-win32-slave13
test-linslave
test-mgmt
moz2-linux-experimental1
moz2-linux-slave01
moz2-linux-slave13
moz2-linux-slave14
moz2-linux-slave15
moz2-linux-slave16
moz2-win32-slave14
moz2-win32-slave15
moz2-win32-slave16
moz2-win32-slave17
moz2-win32-slave18
try-unit-win32-01
test-winslave
try-linux-slave03
try-win32-slave03
bm-buildgraph01
bm-symbolfetch01
dm-ausstage01
dm-chat01
geodns01.sj
dm-graphs01
dm-webtools01
im-bes01
mrapp-intranet01
mrapp-stage01
pm-ns03
production-1.9-master
qm-evtest01
tm-amo01-webdev01
After discussing this with the sheriff, product leads and release
engineering team, we’ve decided it would be best to do this as soon as
possible. We’ll be locking the tree to all checkins at 5:00pm PST
tonight to start the maintenance window, and we’ll unlock it once the
machines have come back up and gone green for a cycle. Vlad will try to
shepherd the remaining patches for beta 2 blockers that are in the
checkin queue before 5pm, but at that time we’ll stop all work and pick
it up tomorrow morning.
Let us know if there are any objections as soon as possible.
Dave Miller
build resources:
build graphs (varying domains)
Litmus
mail.mozilla.org, mozilla.org, and bugzilla.org email
mailing list web interfaces
search services for www.mozilla.com and developer.mozilla.org
one of the two irc servers (sand/san jose)
support.mozilla.com live chat services
LDAP services for the community build network
intranet.mozilla.org
In article
<mailman.283.12265327...@lists.mozilla.org>, matthew
zeier <m...@mozilla.com> wrote:
> (Apologies for the short notice. We generally want to stick to our
> normal advertised maintenance windows but sometimes technology works
> against us.)
>
> We will have an emergency maintenance window tonight from 6pm to 11:00pm
> PST.
>
> The following work will take place:
>
> * 6:00pm PST (0200 UTC) ) EqualLogic storage array firmware upgrade.
> See bug 464458 for details. Wešll be locking the tree to all check-ins
> at 5:00 PST tonight to start the maintenance window and will unlock it
> once the machines have come back up and gone green for a cycle.
>
> Last night one of our network attached storage devices experienced an
> error and didnšt fail over properly. The effect was such that we had to
> close the tree as Tinderbox wasnšt getting proper data for updates, and
> we couldnšt tell what was happening with our builds. While we were able
> to restore that functionality, there is a high chance that any of our
> VMs attached to this storage device may experience further problems.
>
> To fix this, wešll need to pause all of our VM systems (most notably
> engineering team, wešve decided it would be best to do this as soon as
> possible. Wešll be locking the tree to all checkins at 5:00pm PST
> tonight to start the maintenance window, and wešll unlock it once the
> machines have come back up and gone green for a cycle. Vlad will try to
> shepherd the remaining patches for beta 2 blockers that are in the
> checkin queue before 5pm, but at that time wešll stop all work and pick
> it up tomorrow morning.
>
> Let us know if there are any objections as soon as possible.
--
Dave Miller http://www.justdave.net/
System Administrator, Mozilla Corporation http://www.mozilla.com/
matthew zeier
11:00pm PST. This is being tracked in master bug 464743. The following
changes will take place:
* 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up code updates. No downtime is expected.
* 7:00pm PST (0300 UTC) Root-level DNS changes. We will be updating
root-level DNS for mozilla.org, mozilla.com, and spreadfirefox.com to
Mozilla-operated nameservers. No downtime is expected.
* 7:00pm PST (0300 UTC) mozilla.com MX change. We’ll be updating
mozilla.com MX records as per bug 464745. No downtime is expected.
* 7:00pm PST (0300 UTC) Netscaler OS upgrade. We’ll be upgrading the
Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up
code fixes. No downtime is expected.
* 10:00pm PST (0600 UTC) Cisco FWSM OS upgrade. We’ll be updating the
Cisco firewalls to resolve bug CSCsd67334. The two FWSMs run as an HA
pair and this upgrade should be hitless. No downtime is expected.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 464743. Feel
matthew zeier
We will have a scheduled maintenance window tonight from 7:00pm to
11:00pm PST. This is being tracked in master bug 465599. The following
changes will take place:
* 6:00pm PST (0200 UTC) bugzilla.mozilla.org updates. We’ll be updating
bugzilla.mozilla.org to pick up code updates. See bug 464713 for more
details. Duration 5 minutes.
* 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating
support.mozilla.com to pick up code updates. No downtime is expected.
* 10:00pm PST (0600 UTC) border2 Sup720-3BXL upgrade. See
http://blog.mozilla.com/mrz/ for more details. During this time all all
BGP sessions on border2 will be shutdown and traffic will continue to
route through border1. No user-facing downtime is expected.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 465599. Feel
matthew zeier
12:00pm PST. The following changes will take place:
* 10:00pm PST (0600 UTC) border1 Sup720-3BXL upgrade. See
http://blog.mozilla.com/mrz/ for more details (this is the second half
of Tuesday’s upgrade). During this time all all BGP sessions on border1
will be shutdown and traffic will continue to route through border2. No
user-facing downtime is expected.
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
Feel free to comment directly in that bug (or this blog) if you see
matthew zeier
* 9:00pm PST (0500 UTC) addons.mozilla.org/Zeus ZXTM performance
testing. We will be running a performance test of addons.mozilla.org
behind a Zeus ZXTM cluster. See bug 467502 for more information. No
downtime is expected, however the test period is expected to run for
eighteen (18) hours.
matthew zeier
* 7:00pm PST (0300 UTC) addons.mozilla.org updates. We’ll be updating
addons.mozilla.org to pick up code updates. See bug 469210 for more
details. No downtime expected.
* 7:00pm PST (0300 UTC) Bouncer/download.mozilla.org database upgrade.
We’ll be upgrading the hardware the Bouncer/download.mozilla.org
database runs on. See bug 464748 for more details. No downtime expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bug (or this blog) if you see
matthew zeier
maintenance tonight from 7:00pm to 11:00pm PST. The following changes
will take place:
* 7:00pm PST (0300 UTC) mozilla.org MX change. We’ll be updating
mozilla.org MX records. No downtime is expected.
* 7:00pm PST (0300 UTC) duration 4 hours: Kernel upgrades. We’ll be
doing kernel security updates on various machines affecting the
following services for approximately 10 minutes each:
- build graphs / pageload tests
- CVS, cvs-mirror, SVN, Hg
- LXR/MXR
- Despot, Doctor
- ftp.mozilla.org
- quality.mozilla.org
- videos.mozilla.org
- stage.mozilla.org
- people.mozilla.com
- Buildbot Try Server
- MDC Search
- internet-facing mail relays, list servers, and outbound mail (mail
will be queued)
- Any sites using LDAP for authentication
- DNS/DHCP/wifi/telephone services in all Mozilla offices (except
Beijing)
- support.mozilla.com chat server
- irc.mozilla.org
- All database-backed websites and services, including but not
limited to: Bugzilla, Breakpad, Bonsai, SUMO, SpreadFirefox, Addons,
bouncer/download, Blogs, QMO, Litmus
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in this blog if you see issues past the
planned downtime.
matthew zeier
to 11:00pm PST. The following change will take place:
* 8:00pm PST (0500 UTC) Cisco IOS upgrades. We’ll be upgrading IOS on
the two core switches to pickup feature enhancements. See bug 473084
for more details. The systems attached to these switches are multi-
homed. No user-facing downtime is expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in the bug if you see issues past the
planned downtime.
matthew zeier
* 7:00pm PST (0300 UTC) labs.mozilla.com updates. We’ll be updating
labs.mozilla.com to pick up code updates (bug 469210). No downtime
expected.
* 7:30pm PDT (0300 UTC) ) MDC / developer.mozilla.org updates. We’ll
be updating developer.mozilla.org to pick up code updates (bug
471945). No downtime is expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bugs if you see issues past the
planned downtime.
matthew zeier
changes will take place:
* 7:00pm PST (0300 UTC) labs.mozilla.com blog updates. We’ll be
updating labs.mozilla.com’s blog to pick up code updates. No downtime
is expected.
* 7:00pm PST (0300 UTC) developer.mozilla.org / MDC &
library.mozilla.org database maintenance. Both developer.mozilla.org
and library.mozilla.org will be offline to re-sync database slaves
(bug 475477). Duration 30 minutes..
* 7:00pm PST (0300 UTC) versioncheck.addons.mozilla.org &
fxfeeds.mozilla.com load balancer change. We’ll be moving both
versioncheck.addons.mozilla.org & fxfeeds.mozilla.com behind the
production Cisco ACE/Zeus ZXTM cluster. For background, see this blog
post[1]. No downtime is expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
As mentioned, this maintenance window is tracked in bug 475602. Feel
free to comment directly in that bug if you see issues past the
planned downtime.
matthew zeier
PST. The following changes will take place:
* 7:00pm PDT (0300 UTC) crash-stats.mozilla.com (Socorro) database
maintenance. We’ll be partitioning the production Socorro database
(refer to bug 475815 for details). During this time, crash-
stats.mozilla.com will be offline. crash-reports.mozilla.com will be
up and accept crashes. They will be queued and processed later.
Downtime is expected to last into Saturday but total time is unknown
and could run throughout the weekend.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bugs if you see issues past the
planned downtime.
matthew zeier
* 7:00pm PDT (0200 UTC): EqualLogic firmware upgrade. We will be
upgrading the firmware on the EqualLogic storage arrays. We’ve made
every effort to minimize user-facing downtime by shifting SAN volumes
around, however, we are planning for the following systems/services to
be offline for approximately four hours.
o Build & Release Infrastructure will be degraded.
Please refer to John’s post[1] for more information regarding checkins.
o crash-stats.mozilla.com (we’ll still collect reports,
however, reporting and processing will be affected.)
o support.mozilla.com chat services
o mail.mozilla.org (mailman mailing lists)
o air.mozilla.com
o www.mozilla.com search
o im-bes01 / Blackberry Enterprise Server
o litmus.mozilla.org
o graphs stage and stage2 (bm-buildgraph01)
o aus stage - dm-ausstage01
o build-graphs.mozilla.org
o icecast.mozilla.org
o videos.mozilla.org
* 7:00pm PDT (0200 UTC): Cisco FWSM OS upgrade. We’ll be updating the
Cisco firewalls. The two FWSMs run as an HA pair. No downtime is
expected.
* 7:00pm PDT (0200 UTC) support.mozilla.com / SUMO updates. We’ll be
updating support.mozilla.com to pick up code updates (bug 492030). No
downtime expected.
* 7:00pm PDT (0200 UTC) services.addons.mozilla.org / SAMO updates.
We’ll be updating services.addons.mozilla.org to pick up code updates
(bug 492389). No downtime expected.
* 7:00pm PDT (0200 UTC): C01 Database Cluster kernel upgrades. We’ll
be doing kernel security updates on the C01 database cluster. The
following database-driven services will be affected for about 10
minutes:
o blog.lizardwrangler.com
o blog.mozilla.com
o blog.mozillafoundation.org
o communitystore.mozilla.org
o contribute.mozilla.org
o firefoxflicks.com
o getfirebug.com
o impact.mozilla.com
o kubla.mozilla.org
o labs.mozilla.com blogs and forums
o livetitles.com
o mozilla-europe.org
o mozillaparty.com
o pastebin.mozilla.org
o priorartshare.com
o quality.mozilla.org
o reporter.mozilla.org
o spreadfirefox.com
o survey.mozilla.com
o survey.mozilla-europe.org
o developer.mozilla.org
o wiki.mozilla.org
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bugs if you see issues past the
planned downtime.
matthew zeier
11:00pm PDT. The following changes will take place:
* 6:00pm PDT (0100 UTC): crash-stats.mozilla.com database migration.
We’ll be migrating the current production breakpad database to new
hardware. Crash reports will still be collected but processing and
reporting will be offline. Duration 5 hours.
* 7:00pm PDT (0200 UTC): Cisco FWSM OS upgrade. We’ll be updating the
standby Cisco firewall. No downtime is expected.
* 7:00pm PDT (0200 UTC) support.mozilla.com / SUMO updates. We’ll be
updating support.mozilla.com to pick up code updates (bug 492030). No
downtime expected.
* 7:30pm PST (0330 UTC) Kernel upgrades. We’ll be doing kernel
security updates on various machines affecting the following services
for approximately 10 minutes each:
o mradm01.mozilla.org
o gravel.mozilla.org
o build graphs / pageload tests
o CVS, cvs-mirror, SVN, Hg
o LXR/MXR
o Despot, Doctor
o ftp.mozilla.org
o quality.mozilla.org
o videos.mozilla.org
o stage.mozilla.org
o people.mozilla.com
o MDC Search
o Internet-facing mail relays, list servers, and
outbound mail (mail will be queued)
o Any sites using LDAP for authentication
o DNS/DHCP/wifi/telephone services in all Mozilla
offices (except Beijing)
o support.mozilla.com chat server
o irc.mozilla.org
o All database-backed websites and services, including
but not
limited to: Bugzilla, Breakpad, Bonsai, SUMO,
SpreadFirefox, Addons, bouncer/download, Blogs, QMO, Litmus
Please let me know if you have any reason why we should not proceed
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) support.mozilla.com update. We’ll be updating
support.mozilla.com to pick up code updates (bug 499964). Duration 4
hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) graphs.mozilla.org update. We’ll be updating
graphs.mozilla.org to pick up code updates (bugs 501432 and 498802).
Duration 4 hours.
* 8:00pm PDT (0300 UTC) DHCP infrastructure changes. We’ll be
implementing an active/standby DHCP server setup which requires some
reconfiguration on the current DHCP server. No downtime expected.
* 10:00pm PDT (0500) UTC hg repo maintenance. We’ll be resetting the
hg repo (bug 500246). Duration 30 minutes.
matthew zeier
* 6:00pm PDT (0100 UTC) crash-stats.mozilla.com database maintenance.
We’ll be switching crash-stats.mozilla.com to use pgbouncer instead of
accessing the postgres db directly. This will impact the crash-
stats.mozilla.com website and the ability to process priority reports.
Duration 15 minutes.
* 8:00pm PDT (0300 UTC) Bugzilla upgrade and network changes. We’ll
be making some changes to our internal network that will affect the
Bugzilla application servers. We’ll also use this time to upgrade to
version 3.2.4. Downtime shouldn’t be more than 15 minutes expected,
however we’re planning for an hour.
* 11:00pm PDT (0600 UTC) svn/hg maintenance. We’ll be updating server
software. Both svn.mozilla.org and hg.mozilla.org will be impacted.
Duration 15 minutes.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned
downtime.
matthew zeier
11:00pm PDT. The following changes will take place:
* 6:00pm PDT (0200 UTC) blog.mozilla.com upgrade. We’ll be picking up
a Wordpress MU update. Duration 5 minutes.
* 6:00pm PDT (0200 UTC) support.mozilla.com update. We’ll be updating
support.mozilla.com to pick up code updates (bug 504148). No downtime
expected.
* 7:00pm PDT (0200 UTC) Labs VMware update. We will be upgrading the
Labs VMware ESX cluster. This will affect most labs projects,
including Bespin, Jetpack, Ubiquity and Prism. Weave and Personas
should not be affected. Duration 2 hours.
matthew zeier
11:00pm PDT. The following changes will take place:
* 9:00pm PDT (0400 UTC) aus2.mozilla.org/download.mozilla.org
maintenance. We’ll be making some changes to our internal network that
could affect the servers that run aus2.mozilla.org &
download.mozilla.org. Downtime shouldn’t be more than 15 minutes
expected, however we’re planning for an hour .
matthew zeier
11:00pm PDT. The following changes will take place:
* 6:00pm PST (0100 UTC) Kernel upgrades. We’ll be doing kernel
security updates on various machines affecting the following services
for approximately 10 minutes each:
* sand.mozilla.org (part of irc.mozilla.org)
* All RHEL-based development machines (hostnames beginning with sm-
* or cm-*)
* build graphs / pageload tests
* CVS, cvs-mirror, SVN, Hg
* LXR/MXR
* Despot, Doctor
* ftp.mozilla.org
* quality.mozilla.org
* videos.mozilla.org
* stage.mozilla.org
* people.mozilla.com
* MDC Search
* Internet-facing mail relays, list servers, and outbound mail
(mail will be queued)
* Any sites using LDAP for authentication
* DNS/DHCP/wifi/telephone services in all Mozilla offices (except
Beijing)
* support.mozilla.com chat server
* All database-backed websites and services, including but
notlimited to: Bugzilla, Breakpad, Bonsai, SUMO, SpreadFirefox,
Addons, bouncer/download, Blogs, QMO, Litmus
* 8:00pm PDT (0300 UTC): LDAP server maintenance. We’ll be testing out
LDAP failover and redundancy setup. Resources that depend on LDAP
authentication may experience disruption during this time. Duration 1
hours.
* 9:00pm PDT (0400 UTC) addons.mozilla.org maintenance. We’ll be
making some changes to our internal network that could affect the
servers that run addons.mozilla.org. Downtime shouldn’t be more than
15 minutes expected, however we’re planning for an hour .
* 10:00pm PDT (0500 UTC): Bugzilla component renames. We’ll be
renaming three components in Bugzilla to complete leftover items that
got missed in the last component reorganization. See bug 461546 for
details (step 4 on the first attachment). Duration 10 minutes.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly in those bugs if you see issues past the
planned downtime.
matthew zeier
* 7:00pm PDT (0200 UTC) The Amsterdam Reboot. We’ll be turning down
services in Amsterdam and pulling production traffic back to San Jose
in preparation for The Amsterdam Reboot next week. No user facing
downtime is expected.
* 8:00pm PDT (0300 UTC) Layer42 BGP turnup. We’ll be turning up BGP
peering with Layer42 and pushing production traffic through Layer42.
No downtime expected.
* 9:00pm PDT (0400 UTC) www.mozilla.org relaunch. We’ll be pushing out
a new version of www.mozilla.org. See bug 510267 for details.
Duration 30 minutes.
* 9:00pm PDT (0400 UTC) PHP upgrade. We be upgrading PHP on
addons.mozilla.org. See bug 506703 for details. No downtime expected.
* 9:00pm PDT (0400 UTC) addons.mozilla.org EV SSL certificate
deployment. We’ll be replacing the wild card certificate with a
VeriSign EV SSL certificate tonight. No downtime expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned
downtime.
matthew zeier
* 8:00pm PDT (0300 UTC) crash-stats.mozilla.com content push. We’ll be
picking up code updates (bug 515739). During this time, crash-
stats.mozilla.com should be up but some aggregated reports may not be
available. crash-reports.mozilla.com will be up and accept crashes.
Duration two hours.
* 8:00pm PDT (0300 UTC) graphs.mozilla.org update. We’ll be updating
graphs.mozilla.org to pick up code updates (bug 515461). Duration one
hour.
* 9:00pm PDT (0300 UTC) The Amsterdam Reboot, Part II. We’ll be
rolling changes to push several production websites out of Amsterdam.
This involves changing DNS over to our Zeus GLB servers. No downtime
expected. The following sites will be moved tonight:
* addons.mozilla.org (bug 514994)
* developer.mozilla.org (bug 484807)
* 9:00pm PDT (0300 UTC) EdgeCast CDN trials. In our continuing effort
to bring content to closer to end users, we’ll be starting a trial
with EdgeCast. We’ll be flipping DNS for www.mozilla.com over to
EdgeCast. No downtime expected.
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:30pm PDT (0230 UTC): Cisco FWSM OS upgrade. We’ll be updating the
Cisco firewall module. No downtime is expected.
* 8:00pm PDT (0300 UTC) support.mozilla.com / SUMO updates. We’ll be
updating support.mozilla.com to pick up code updates (bug 520042). No
matthew zeier
* 9:00pm PDT (0400 UTC) addons.mozilla.org update. We’ll be updating
addons.mozilla.org to pick up code updates (bug 523428). Duration 30
minutes.
* 9:00pm PDT (0400 UTC) We’ll be rolling changes to push several
production websites out of Amsterdam. This involves changing DNS over
to our Zeus GLB servers. No downtime expected.
The following sites will be moved tonight:
* www.spreadfirefox.com
* support.mozilla.com
Dave Miller
8:00pm PDT. The following changes will take place:
* 5:00pm PDT (0000 UTC) addons.mozilla.org database hardware upgrade.
We’ll be adding additional RAM to the master database server for
addons.mozilla.org. (bug 522112). Duration 30 minutes (although the site
may stay down pending completion of the file moves, see below)
* 5:00pm PDT (0000 UTC) We'll be taking advantage of the database outage
to remap part of the back-end filesystem on addons.mozilla.org. (bug
517015). Duration 90 minutes.
* 7:00pm PDT (0200 UTC) We'll be making DNS changes to move the
support.mozilla.org domain to our Zeus GLB servers. (bug 522923) No
downtime expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned downtime.
--
Dave Miller http://www.justdave.net/
System Administrator, Mozilla Corporation http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System http://www.bugzilla.org/
Dave Miller
fit in tonight:
* 7:00pm PDT (0200 UTC) We'll be deploying a software upgrade to the
support.mozilla.com application (upgrading to version 1.4.1). (bug
523264). Duration 30 minutes.
Dave Miller wrote on 10/22/09 7:15 PM:
matthew zeier
11:00pm PDT. The following changes will take place:
* 7:00pm PDT (0200 UTC) labs.mozilla.com migration. We’ll be migrating
labs.mozilla.com to mozillalabs.com (bug 489638). Duration 60 minutes.
* 9:00pm PDT (0400 UTC) addons.mozilla.org backend server maintenance.
We’ll be making changes to the backend storage (bug 517015) and
database servers (bug 524011). Duration 90 minutes.
matthew zeier
* 7:00pm PST (0300 UTC) support.mozilla.com update. We’ll be updating
support.mozilla.com to pick up code updates (bug 526284) . Duration 60
minutes.
* 7:00pm PST (0300 UTC) getpersonas.com update and database upgrade.
We’ll be updating getpersonas.com to pick up code updates (bug 526291)
and moving its database to a different database cluster (bug 521597).
Duration 10 minutes.
* 7:00pm PST (0300 UTC) smtp.mozilla.org configuration change. We’ll
be removing mozilla.com and mozilla.org from the authorized recipients
list on the inbound mail servers, since they are no longer used as the
MX for those domains (bug 524804) No downtime expected.
* 11:00pm PST (0700 UTC) www.mozilla-europe.org GLB DNS change. We’ll
be making DNS changes to move www.mozilla-europe.org to our Zeus GLB
servers (bug 523711). No downtime expected.
matthew zeier
* 6:00pm PST (0200 UTC) crash-stats.mozilla.com database maintenance.
We’ll be working on the database and will impact the materialized
views on crash-stats.mozilla.com. All of the cron generated reports
will be down for the duration of this window. See bug 526624 for
details.Duration 7 hours.
matthew zeier
suffered an approximately 13 minute outage.
Mozilla uses Cisco Firewall Service Modules in the core Cisco 6509
switches. At about 5:07am the primary unit failed and appears to have
crashed within the failover code. This caused an incomplete failover
and the standby FWSM never assumed full control.
At 5:19am PST the primary FWSM was manually reset which allowed the
standby to complete its failover.
We have grabbed crashinfo and are working with Cisco TAC to diagnose.
Specific details will be tracked in bug 528186.
We apologize for any inconvenience this may have caused and will
continue to work with Cisco on a remedy.
matthew zeier
* 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating
addons.mozilla.org to pick up code updates (bug 528272). Duration 30
minutes.
* 7:00pm PST (0300 UTC) getpersonas.com update. We’ll be updating
getpersonas.com to pick up code updates (bug 527792). Duration 10
minutes.
* 7:00pm PST (0300 UTC) crash-stats.mozilla.com database maintenance.
We’ll be working on the database and will impact the materialized
views on crash-stats.mozilla.com. All of the cron generated reports
will be down for the duration of this window. See bug 526624 for
details. Duration 7 hours.
* 9:00pm PST (0500 UTC) dev-vmware01 ESX server upgrade. We’ll be
upgrading the server hardware for dev-vmware01. During this window
the following VMs will be offline for 2 hours:
* mrapp-stage03
* sm-ctc01
* sm-chat01
* sm-cms01
* sm-crm01
* sm-dekistage01
* 9:00pm PST (0500 UTC) Netscaler OS upgrade. We’ll be upgrading the
Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up
code fixes. No downtime is expected.
Dave Miller
7:30pm PST. The following changes will take place:
* 6:00pm PST (0200 UTC) bugzilla.mozilla.org update. We'll be
upgrading Bugzilla from version 3.2.4 to 3.4.3 (bug 503055). Duration
90 minutes.
Feel free to comment directly on the bug or on #bmo on irc.mozilla.org
if you see issues past the planned downtime.
--
Dave Miller
8:00am PST. The following changes will take place:
* 6:00am PST (1400 UTC) stage.mozilla.org - core operating system will
be upgraded from RHEL4 to RHEL5. Expected downtime 60 minutes, things
may not behave quite right for an hour or two after the upgrade as the
loose ends get cleaned up. This will affect any build systems that
upload to stage.mozilla.org. (bug 524794)
Feel free to comment directly on the bug or on #build on irc.mozilla.org
matthew zeier
11:00pm PST. The following changes will take place:
* 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating
addons.mozilla.org to pick up code updates (bug 529600). Duration 30
minutes.
* 9:00pm PST (0500 UTC) getpersonas.com GLB DNS change. We’ll be
making DNS changes to move getpersonas.com to our Zeus GLB servers
(bug 525908). No downtime expected.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned
downtime.
matthew zeier
* 6:00pm PST (0100 UTC) Breakpad upgrade. We will be upgrading the
breakpad environment to the next release ([1]). In addition, we will
also be making changes to the storage layout to help back-end storage
scaling. The crash collector should not be impacted during the
upgrade. The reporter however may be unavailable during parts of the
upgrade. Please contact us in #breakpad if you notice issues past 9pm.
Duration 3 hours.
* 6:00pm PST (0100 UTC) support.mozilla.com update. We’ll be updating
support.mozilla.com to pick up code updates (bug 532232). Duration 60
minutes.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
matthew zeier
* 7:00pm PST (0300 UTC) Geo-location support to bouncer/
download.mozilla.org. We’ll be turning on geo-location support to
download.mozilla.org tonight. This should improve the download
experience by directing users to mirrors that are geographically close
to their origin IP. See bug 459919 for more details. No downtime
expected.
* 7:00pm PST (0300 UTC) PHP 5.2 upgrades. We’ll be rolling out PHP 5.2
upgrades on the production web servers (bug 533486). We don’t expect
any user-facing downtime. Duration 3 hours.
Please let me know if you have any reason why we should not proceed
with this planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned
downtime.
matthew zeier
1.0rc1 / pre-Firefox 3.6 releases to perform maintenance on some
Production & Release Engineering infrastructure.
During this window, all check-in trees will be closed.
We’ll be performing maintenance Sunday from 12:00pm to 6:00pm PST .
The following changes will take place:
* 12:00pm PST (2000 GMT) Duration 2 hours: NetApp OS upgrade. We’ll be
updating the two Network Appliance file servers that handle a number
of Release Engineering VMs. See bug 531233 for more details.
* 12:00pm PST (2000 GMT) Duration 2 hours: b01 database cluster
maintenance. We’ll be upgrading the firmware on the b01 database
cluster (bug 535859) and performing maintenance to clear up disk space
(bug 528573). This work will affect the following services:
• developer.mozilla.org / MDC
• litmus.mozilla.org
• bonsai.mozilla.org
• ViewVC for SVN
* 1:00pm PST (2100 GMT) Duration 4 hours: ESX hardware upgrade. We’ll
be upgrading the production ESX cluster from old dual CPU HP DL385
servers to our standard 8-core/32GB RAM HP BL460 platform. Since this
upgrade involves switching CPU platforms (AMD to Intel), it requires
each VM to be shutdown, migrated and powered up. Each VM’s downtime
will be not unlike a normal reboot.The following list includes notable
VMs & services that will be briefly affected:
• mpt-vpn.mozilla.com
• graphs-stage.mozilla.org
• support.mozilla.com Live Chat
• hacks.mozilla.org
• quality.mozilla.org
• build-graphs.mozilla.org
• All Build, unittest, Talos and TryServer systems
• irc.mozilla.org / sand.mozilla.org
John O'Duinn
matthew zeier wrote:
> We�ll be taking advantage of the end-of-holiday lull and post-Fennec
> 1.0rc1 / pre-Firefox 3.6 releases to perform maintenance on some
> Production & Release Engineering infrastructure.
>
> During this window, all check-in trees will be closed.
>
> We�ll be performing maintenance Sunday from 12:00pm to 6:00pm PST . The
> following changes will take place:
>
> * 12:00pm PST (2000 GMT) Duration 2 hours: NetApp OS upgrade. We�ll be
> updating the two Network Appliance file servers that handle a number of
> Release Engineering VMs. See bug 531233 for more details.
>
> * 12:00pm PST (2000 GMT) Duration 2 hours: b01 database cluster
> maintenance. We�ll be upgrading the firmware on the b01 database cluster
> (bug 535859) and performing maintenance to clear up disk space (bug
> 528573). This work will affect the following services:
> � developer.mozilla.org / MDC
> � litmus.mozilla.org
> � bonsai.mozilla.org
> � ViewVC for SVN
>
> * 1:00pm PST (2100 GMT) Duration 4 hours: ESX hardware upgrade. We�ll be
> upgrading the production ESX cluster from old dual CPU HP DL385 servers
> to our standard 8-core/32GB RAM HP BL460 platform. Since this upgrade
> involves switching CPU platforms (AMD to Intel), it requires each VM to
> be shutdown, migrated and powered up. Each VM�s downtime will be not
> unlike a normal reboot.The following list includes notable VMs &
> services that will be briefly affected:
> � mpt-vpn.mozilla.com
> � graphs-stage.mozilla.org
> � support.mozilla.com Live Chat
> � hacks.mozilla.org
> � quality.mozilla.org
> � build-graphs.mozilla.org
> � All Build, unittest, Talos and TryServer systems
> � irc.mozilla.org / sand.mozilla.org
While the ESX work is from 1-5, note that the downtime is from 12-6.
This is to give us time to power down the all the RelEng systems, as
well as spin them all back up and verify they are running "green" before
reopening the trees.
tc
John.
=====
matthew zeier
11:00pm PST. The following changes will take place:
* 9:00pm PST (0500 UTC) getpersonas.com update. We’ll be updating
getpersonas.com to pick up code updates (bug 537956). Duration 30
minutes.
matthew zeier
* 7:00pm PDT (0300 UTC) crash-stats.mozilla.com & crash-
reports.mozilla.com content push. We’ll be picking up code updates
(bug 538261). Duration one hour.
matthew zeier
01/13/2010 UTC)
We will have a scheduled maintenance window tonight from 5:00pm to
* 9:00pm PST (0500 UTC) addons.mozilla.org update. We’ll be updating
addons.mozilla.org to pick up code updates (bug 539074). Duration 30
minutes.
* 9:00pm PST (0500 UTC) blog.mozilla.com upgrade. We’ll be picking up
a Wordpress MU update. No downtime expected..
Dave Miller
* 7:00pm Mozilla Developer Center upgrade – We will be upgrading
developer.mozilla.org to Mindtouch 9.12. 30 minutes downtime
expected. (bug 531977)
* 7:00pm Getfirebug.com upgrade – We will be deploying a site
redesign for the getfirebug.com website. 30 minutes downtime
expected. (bug 538320)
* 7:00pm jetpackgallery.mozillalabs.com – We will be upgrading the
site to version 1.2 of Jetpack Gallery. 15 minutes downtime
expected. (bug 542083)
* 7:30pm Kernel upgrades. We’ll be doing kernel security updates on
various machines affecting the following services for
approximately 10 minutes each:
o breakpad.mozilla.org
o crash-reports.mozilla.org
Some services may have more than one outage as various back-end
components each get their upgrades (for example, a web
application will go down once when the web servers are upgraded
and again when its database server is upgraded).
* 8:00pm crash-reports.mozilla.org – We will be performing database
maintenance to improve search capability on
crash-reports.mozilla.org. No downtime is expected, but site
performance may be degraded for up to 3 hours. (bug 538313)
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
--
Dave Miller
* 7:00pm - Breakpad maintenance - We will be upgrading
crash-stats.mozilla.org and doing maintenance on the Breakpad
database to improve future performance. crash-stats.mozilla.org
will be down for around 4 hours. No crash reports will be
processed during this window. We will still be accepting and
storing incoming crashes. (bug 542390)
Dave Miller
* 7:00pm – Upgrade the server ‘mradm01′ from RHEL4 to RHEL5 (bug
542454). Actual downtime is expected to be 30 minutes or less, but it
may take through the end of the window to get all services completely
functional again afterwards. This server hosts ns1.mozilla.org (DNS
server) and the nagios slave (facility-wide system monitoring) for the
MPT colo facility. DNS services are redundant (ns2 and ns3 will pick up
the slack). We will be without monitoring in the MPT colo facility
while the server is being upgraded, however. This means any problems
that are normally automatically detected and cause the oncall sysadmin
to get paged will not be detected or paged about during the outage. We
ask that if anyone observes any such problems during the window that you
report them in the #bmo channel on irc.mozilla.org (click the following
link to go there via your browser:
http://mibbit.com/chat/?server=irc.mozilla.org&channel=%23bmo&forcePrompt=true
)
Please let me know if you have any reason why we should not proceed with
this planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be operational
by the end of the maintenance window.
Feel free to reply to me directly if you see issues past the planned
downtime.
matthew zeier
* 7:00pm PST (0300 UTC) support.mozilla.com update. We’ll be updating support.mozilla.com to pick up code updates (bug 546712) . Duration 60 minutes.
* 9:00pm PDT (0300 UTC) Phoenix Datacenter, Part I. We’ll be rolling changes to push several production websites out of Phoenix. This involves the following work:
* Adding ns1-glb.phx.mozilla.net as an authoritative GLB nameserver (NS RR).
* Changing DNS for www.mozilla.com
No downtime expected.
Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned downtime.
matthew zeier
* 7:00pm PST (0300 UTC) getpersonas.com update. We’ll be updating getpersonas.com to pick up code updates (bug 547869). Duration 30 minutes.
* 7:00pm PST (0300 UTC) graphs.mozilla.org update. We’ll be updating graphs.mozilla.org to pick up code updates (bug 547899) . Duration 60 minutes.
* 8:00pm PST (0400 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 547917). Duration 60 minutes.
matthew zeier
We will have a scheduled maintenance window tonight from 7:00pm to 11:00pm PDT. The following changes will take place:
* 7:00pm PST (0300 UTC) Breakpad maintenance. We’ll be picking upi code updates and re-factoring the backend storage system. See bugs 548593 and 548566 for more details.Duration 3 hours.
* 7:00pm PST (0300 UTC) viewvc.svn.mozilla.org upgrades. We’ll be updating ViewVC on the servers behind viewvc.svn.mozilla.org. See bug 545296 for details.
* 9:00pm PST (0500 UTC) blog.mozilla.com upgrade. We’ll be picking up a Wordpress MU update. Blogs hosted behind this site may be unavailable. Duration 60 minutes.
matthew zeier
* 7:00pm PST (0300 UTC) Zeus GLB code upgrades. We’ll be picking up a hotfix for the Zeus GLB to fix an issue discovered while turning up production traffic in Phoenix. No downtime expected.
* 7:00pm PST (0300 UTC) mozillalabs.com update. We’ll be updating mozillalabs.com to pick up new content (bug 549401). No downtime expected.
* 9:00pm PST (0500 UTC) support.mozilla.com load balancer change. We’ll be moving support.mozilla.com behind the production Cisco ACE/Zeus ZXTM cluster. No downtime is expected.
* 9:00pm PST (0500 UTC) SiteSpect deployment for addons.mozilla.org. We’ll be implementing multivariate testing with SiteSpect on addons.mozilla.org. No downtime expected.
matthew zeier
* 7:00pm PST (0300 UTC) graphs.mozilla.org memory upgrade. We’ll be adding memory to the graphs.mozilla.org VM tonight (bug 548371). Even those this amounts to a short reboot, it will require tree closure. Duration 1 hour.
* 7:00pm PST (0300 UTC) getpersonas.com update. We’ll be updating getpersonas.com to pick up new content (bug 550084). No downtime expected.
* 8:00pm PST (0300 UTC) Zeus GLB code upgrades. We’ll be picking up a hotfix (again… Tuesday’s hotfix didn’t work) for the Zeus GLB to fix an issue discovered while turning up production traffic in Phoenix. No downtime expected.
* 9:00pm PST (0500 UTC) SiteSpect deployment for support.mozilla.com. We’ll be implementing multivariate testing with SiteSpect on support.mozilla.com. No downtime expected.
matthew zeier
During this window, all check-in trees will be closed.
We’ll be performing maintenance Saturday from 3:00pm to 4:00pm PDT . The following changes will take place:
* 3:00pm PDT (2200 UTC) Duration 1 hours: ftp.mozilla.org memory upgrade. We’ll be adding additional memory to ftp.mozilla.org. This will impact the following services:
* stage.mozilla.org
* archive.mozilla.org
* ftp.mozilla.org
matthew zeier
We'll be doing OS & firmware upgrades on our NetApp cluster that requires physical reboots of the disk array shelves. Unlike most NetApp OS upgrades, this will take a number of services offline.
We plan to manually failover a number of production services to the Phoenix data center during the week of April 12, however, this work will close all check-in trees.
I'll post a reminder as we get close to the date.
I've blogged with more details @
http://blog.mozilla.com/mrz/2010/03/29/upcoming-downtime-april-17-san-josempt/
- mz
matthew zeier
* 7:00 PDT (0200 UTC) l10n.mozilla.org maintenance. In preparation for upcoming ESX upgrades in Amsterdam, we’ll be migrating the l10n.mozilla.org VM to Phoenix. This host will be offline while we copy it. Duration 6 hours.
* 7:00 PDT (0200 UTC) GeoDNS upgrade. We’ll be adding geodns04.phx.mozilla.com as an additional nameserver for geo.mozilla.com. No downtime expected.
* 8:00pm PDT (0300 UTC) support.mozilla.com update. We’ll be updating support.mozilla.com to pick up code updates (bug 555003) . Duration 60 minutes.
* 9:00pm PST (0500 UTC) Bouncer / download.mozilla.org upgrade. We’ll be picking up updates to our download redirection service (bug 550510). Since it requires database schema changes, both San Jose and Phoenix may be down for a brief period of time. Duration 15 minutes.
* 9:30pm PST (0530 UTC) www.spreadfirefox.com load balancer change. We’ll be moving www.spreadfirefox.com behind the production Cisco ACE/Zeus ZXTM cluster (bug 497394). No downtime is expected.
matthew zeier
* 6:00pm PST (0100 UTC) crash-stats.mozilla.com (Socorro) content push & backend processors changes . We’ll be picking up code updates (bug 556535, full list of changes at http://tinyurl.com/socorro-1-6). crash-reports.mozilla.com will be up and will accept crashes but processing may be slightly delayed. This is in preparation for some larger work this weekend. Duration 3 hours.
* 9:30pm PST (0530 UTC) www.spreadfirefox.com load balancer change. We’ll be moving www.spreadfirefox.com behind the production Cisco ACE/Zeus ZXTM cluster (bug 497394). The is a re-attempt from Tuesday’s window. No downtime is expected.
* 10:00 PDT (0600 UTC) GeoDNS upgrade. We’ll be adding geodns04.phx.mozilla.com as an additional nameserver for geo.mozilla.com. The is a re-attempt from Tuesday’s window. No downtime expected.
matthew zeier
We will have a scheduled maintenance window tonight from 9:00pm to 11:00pm PDT. The following changes will take place:
* 9:00pm PST (0400 UTC) addons.mozilla.org primary site move. In preparation for Saturday’s scheduled downtime, we’ll be manually failing addons.mozilla.org to Phoenix. We’ll need to take the site offline for a bit to do a final content sync. Duration 30 minutes.
Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.
Feel free to comment directly if you see issues past the planned downtime.
[1] http://blog.mozilla.com/mrz/2010/03/29/upcoming-downtime-april-17-san-josempt/