Fwd: [TCW] Scheduled Tree Closing Maintenance Window 2016-10-08 06:00a PDT 9 hours

[Hal Wine]

Our usual Tree Closing Window is Saturday, Oct 8. This time, there is some
major work, so trees will be closed hard at the start at 0600 PT.

We hope to be able to switch to "soft close" around 1100 PT. "Soft Close"
means the trees are open, but there may be possible hickups, and it is up
to devs to sheriff their own jobs and retry as appropriate.

See below for further details.

---------- Forwarded message ----------
From: m...@mozilla.com <m...@mozilla.com>
Date: Fri, Sep 30, 2016 at 10:31 AM
Subject: [TCW] Scheduled Tree Closing Maintenance Window 2016-10-08 06:00a
PDT 9 hours
To:


bcc:all

Start Date/Time: 2016-10-08 06:00a PDT
End date/time: 2016-10-08 15:00p PDT

*Impact to Users:*

- Trees will be in hard close state for the beginning of the outage; A
decision will be made after DB patching is done to open the trees to a
soft close state

-

Possible brief interruption to services in SCL3 06:00-06:15a
-

Many systems will be patched (both the database and application
servers) - there will be very brief outages to mana, bugzilla, buildbot,
single home internal sites, etc between at 08:00a and 13:00. These are
expected and should last < 10 min per site. MOC will validate
functionality of all sites.


Tracker Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1302784

List of work to be completed:


NA
need to reset the optic on border2.scl3 xe-1/1/0
NA
upgrade core1.releng to fix ssh problem
https://bugzilla.mozilla.org/show_bug.cgi?id=925238 Enable IPv6 for
proxy[1234].dmz.scl3 and proxy[12].dmz.phx1

Patch releng dns / dc proxies
https://bugzilla.mozilla.org/show_bug.cgi?id=1297935 purposely failover
fw1.releng.scl3 to collect data for Juniper
NA
shut down ISP link to Telia, to investigate the impact of reachability to
Mozilla's main AS
NA
Need to kernel patch and db upgrade mana, bugzilla, reviewboard
https://bugzilla.mozilla.org/show_bug.cgi?id=1292084 Need to kernel patch
generic, devtools, buildbot and treeherder clustsers
https://bugzil.la/1290275 Update and reboot 51 machines for Webops during
the TCW
https://bugzilla.mozilla.org/show_bug.cgi?id=1231186 Briefly enable service
checks on testing Nagios 4 host
https://bugzilla.mozilla.org/show_bug.cgi?id=1305855 Patch and reboot
admin1[ab].private.scl3