Re: Intent to uplift to 60: Taskgraph and Release automation changes

[Tom Prince]

I have prepared a branch with the changes I intend to uplift. It is
available at https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1].
I intend to push these changes to mozilla-esr60 Thursday morning, unless I
head objections before then. In the meantime, I am going to do staging
releases with these changes, to verify they work.

-- Tom

[1] This repository has changeset evolution enabled. I intened to keep the
state of the repo in-sync with what I am going to uplift.

On Fri, Jun 29, 2018 at 2:31 PM Tom Prince <moz...@hocat.ca> wrote:

> tl;dr To avoid having our release automation diverging we should default
> to uplifting changes to code involved in release automation that doesn't
> affect the bits being shipped.
>
> Over the lifetime of ESR52, the release process and surrounding automation
> for non-ESR releases changed significantly (due to the taskcluster
> migration). This required releaseduty to maintain very divergent
> instructions for ESR releases ([1] vs [2]). Given the large amount of
> changes involved in the final migration of buildbot, this was probably
> unavoidable. While we don't anticipate major changes there are a number of
> improvements in the pipeline[3][4][5] that will cause divergence.
>
> We've also ran into issues (for example [1]) due to the build and release
> automation not being kept up-to-date. We also want to keep our automation
> tooling up-to-date with security fixes like [7].
>
> We could just up-lift patches that have direct impact on the release
> process, but as the ESR cycles progresses, it will become more difficult to
> uplift changes. Instead I propose that we adopt a policy of uplifting
> relevant changes to ESR60:
>
> - taskgraph code (`taskcluster/taskgraph`)
>
> - release automation tasks (`taskcluster/ci/release-*` mostly)
>
> - testing/mozharness cleanups
>
> As none of the changes impact the built artifacts, they should be low
> risk, and make any changes that are critical to uplift less risky to
> uplift. I'm excluding from this proposal changes to
>
> - toolchain version updates
>
> - test configuration
>
> - new build configurations
>
> My initial plan is as follows. It open to suggestions and to change as we
> get experience with the process.
>
> 1. I will prepare a branch with changes to bring ESR60 up-to-date and post
> that review.
>
> 2. I will run a staging release with the changes to verify that the
> changes won’t cause issues when we go to do a release.
>
> 3. After each release cycle, I will repeat the above steps to pick up any
> changes from the most recent release cycle. (After the initial uplift, this
> should be fairly smooth)
>
> There may be patches that require uplifting in a more timely manner
> (security fixes or things that require coordination with out-of-tree
> changes); they can continue to be handled in an ad-hoc manner. After a few
> cycles, once this process has worked smoothly, we’ll look at migrating the
> uplifts to be handled by releaseduty.
>
> I don't have an exhaustive list of changes, but the following is a
> sampling of changes (some of which have been uplifted, or there are already
> plans to uplift).
>
> - Mercurial Updates (Bug 1448438)
>
> - Release Notification fixes (Bugs 1459185 and 1459116) [2]
>
> - Converting actions to use hooks (Bug 1415868)
>
> - Pretty release platforms (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1456234)
>
> - Fetch tasks (https://bugzilla.mozilla.org/show_bug.cgi?id=1460777)
>
> - Publishing buildhub data (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1442306)
>
> - Declarative artifacts (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1462393)
>
> Additionally, I plan to forward port some fixes that were made directly to
> esr60 in release automation to make this process easier.
>
>
> -- Tom
>
>
> [1]
> https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto_58cycle_and_52esr_only.md
>
> [2]
> https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/release-promotion/desktop/howto.md
>
> [3] https://bugzilla.mozilla.org/show_bug.cgi?id=1415868
>
> [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1462393
>
> [5] https://bugzilla.mozilla.org/show_bug.cgi?id=shipit-v2
>
> [6] https://bugzilla.mozilla.org/show_bug.cgi?id=1449350
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1459391
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1446296
>
> [7] https://bugzilla.mozilla.org/show_bug.cgi?id=1385978
>

[Tom Prince]

I've prepared a second round of uplifts that include changes between
FIREFOX_NIGHTLY_62_END and FIREFOX_NIGHTLY_64_END, available at

https://hg.mozilla.org/users/mozilla_hocat.ca/esr60-stage [1]. I intend to

push these changes to mozilla-esr60 Friday, unless I head objections before
then.

[Ryan VanderMeulen]

Thanks so much for doing this, Tom. Having consistent CI scheduling across
our supported branches will make our lives much easier as ESR60 ages over
the next year.