Re: Fwd: [TCW] Scheduled Tree Closing Maintenance Window 2016-10-08 06:00a PDT 9 hours

[Axel Hecht]

This will also affect the automation on the l10n dashboard, AFAICT.

Thus I'll switch it off at some point on Saturday (or even Friday
night), and switch it back on once things are back up.

We won't be loosing data, it's just that the data won't be generated in
a timely fashion.

Axel

On 03/10/16 18:58, Hal Wine wrote:
> Our usual Tree Closing Window is Saturday, Oct 8. This time, there is some
> major work, so trees will be closed hard at the start at 0600 PT.
>
> We hope to be able to switch to "soft close" around 1100 PT. "Soft Close"
> means the trees are open, but there may be possible hickups, and it is up
> to devs to sheriff their own jobs and retry as appropriate.
>
> See below for further details.
>
> ---------- Forwarded message ----------
> From: m...@mozilla.com <m...@mozilla.com>
> Date: Fri, Sep 30, 2016 at 10:31 AM
> Subject: [TCW] Scheduled Tree Closing Maintenance Window 2016-10-08 06:00a
> PDT 9 hours
> To:
>
>
> bcc:all
>
> Start Date/Time: 2016-10-08 06:00a PDT
> End date/time: 2016-10-08 15:00p PDT
>
> *Impact to Users:*
>
> - Trees will be in hard close state for the beginning of the outage; A
> decision will be made after DB patching is done to open the trees to a
> soft close state
>
> -
>
> Possible brief interruption to services in SCL3 06:00-06:15a
> -
>
> Many systems will be patched (both the database and application
> servers) - there will be very brief outages to mana, bugzilla, buildbot,
> single home internal sites, etc between at 08:00a and 13:00. These are
> expected and should last < 10 min per site. MOC will validate
> functionality of all sites.
>
>
> Tracker Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1302784
>
> List of work to be completed:
>
>
> NA
> need to reset the optic on border2.scl3 xe-1/1/0
> NA
> upgrade core1.releng to fix ssh problem
> https://bugzilla.mozilla.org/show_bug.cgi?id=925238 Enable IPv6 for
> proxy[1234].dmz.scl3 and proxy[12].dmz.phx1
>
> Patch releng dns / dc proxies
> https://bugzilla.mozilla.org/show_bug.cgi?id=1297935 purposely failover
> fw1.releng.scl3 to collect data for Juniper
> NA
> shut down ISP link to Telia, to investigate the impact of reachability to
> Mozilla's main AS
> NA
> Need to kernel patch and db upgrade mana, bugzilla, reviewboard
> https://bugzilla.mozilla.org/show_bug.cgi?id=1292084 Need to kernel patch
> generic, devtools, buildbot and treeherder clustsers
> https://bugzil.la/1290275 Update and reboot 51 machines for Webops during
> the TCW
> https://bugzilla.mozilla.org/show_bug.cgi?id=1231186 Briefly enable service
> checks on testing Nagios 4 host
> https://bugzilla.mozilla.org/show_bug.cgi?id=1305855 Patch and reboot
> admin1[ab].private.scl3
>

[Axel Hecht]

Things are back online and caught up.

Axel