Planned Tree Closure Window - June 28th

[Andrew Halberstadt]

Hey everyone,

We're planning a tree closure to migrate Firefox CI's artifact storage backend from Amazon S3 to Google Cloud Storage:

When: June 28th @ 12:00 EST

Duration: 2hrs (or longer if we need to rollback)

What to expect:

Try will remain open. While we anticipate the transition should be mostly seamless, you may notice some recent artifacts will be missing for a short period of time following the cutover (but they will be synced over soon) and some unlucky tasks may fail with an exception and need to be rerun.

We'll also be migrating the Community Taskcluster instance tomorrow in order to detect any issues that arise beforehand. If you notice missing artifacts or other issues, please reach out on #firefox-ci (for issues with the Firefox CI instance) or #taskcluster (for issues with the Community instance).

Big thanks to Eric Maydeck for all the prerequisite work to make the migration possible!

Please reach out if you have any questions or concerns.

Cheers,

Andrew

--
You received this message because you are subscribed to the Google Groups "firef...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firefox-dev...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAAJAz%2B7XNBBUN5QcZJY2-i%3DByNu3vbdvtD3u6y4bvv_M1iDnaA%40mail.gmail.com.

[Andrew Halberstadt]

Update: Firefox CI artifact migration and planned tree closure is moved to July 12th @ 12:00 EST.

The community migration was completed successfully, however a problem related to how S3 and GCS handle gzip compressed files came up which will likely cause some issues in Firefox CI (read below for details). Since June 28th is preceding a long weekend followed by the Firefox 115 release, we've decided to delay the migration until July 12th, when we'll be under no time pressure to fix fallout.

Issue Details

It turns out that S3 serves the file with whatever encoding it was uploaded with no matter what. So if you request a gzip compressed artifact, S3 will serve you that gzip compressed artifact. GCS on the other hand, pays attention to the Accept-Encoding header. So if you request a gzip compressed artifact, GCS will serve you the gzip compressed artifact if you specify `Accept-Encoding: gzip`, but otherwise helpfully decompress the artifact (as the lack of the `Accept-Encoding` header indicates you can't handle gzipped files). This means that after the transition, places that were expecting downloaded artifacts to be gzipped, may instead be decompressed.

We'll use the extra time to perform extra testing and validation to try and root out issues ahead of time. However, it is not possible to find every place that downloads artifacts from Taskcluster. If you maintain any tasks or tools that download gzipped Taskcluster artifacts and want to avoid any disruption, please ensure you are passing the `Accept-Encoding: gzip` header in your GET requests.

- Andrew

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAAJAz%2B4GLwakiELwb3%3DAqZefnAjLUC%3D5BzxoCV_QY0%2BEGQYnvQ%40mail.gmail.com.

[Andrew Halberstadt]

Quick reminder that the planned tree closure is scheduled for 12:00 EST (one hour from now) and should last approximately two hours. Try will remain open. Updates will be posted to #firefox-ci on Matrix and I'll follow-up here when the migration is finished as well.

Please be on the lookout for suspicious failures, especially around:

1. Missing artifacts.

2. Failure to decompress artifacts.

Please ping the #firefox-ci channel on Matrix if you find any issues.

Thanks,

Andrew

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAAJAz%2B5cF9eiCKqXVSZy2VXTyhHdv2GSGewfTHKURTfH_3hd4g%40mail.gmail.com.

[Andrew Halberstadt]

Just a quick status update:

The artifact storage backend has been cut over, but a final sync task is still ongoing to sync the last delta of artifacts from S3 -> GCS. So if you see any task failures around missing artifacts, this is expected. Once the sync task wraps up in the next 30min or so, the artifacts should be available again. At which point rerunning your tasks should make them green.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAAJAz%2B7d1QCDMxFAsTecTtN94WbRPSNfs%2BtzbGmSzKR6y9gbxA%40mail.gmail.com.

[Andrew Halberstadt]

The migration is now complete and trees have been re-opened. Big thanks to Eric Maydeck for doing the hard bits and making this migration so smooth!

I reran any failures I noticed on try and they are all coming back green so far. If I missed any tasks, a rerun / retrigger should be all you need.

Thanks for your patience!

Andrew

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAAJAz%2B52EZ0JfwynZaX7TKFBuw5OGWJcm2A0FUfCcso6%2Bsf_7A%40mail.gmail.com.