Proposal Shippable Build (Re-communique)

[Justin Wood]

[please keep the release-engineering list CC'd on any replies, to try and
cut down on thread fragmentation]


Tl;dr Releng is starting to implement “Shippable” builds [1], and in doing
so will shift some testing, such as talos performance measures, over to
this new build type. This will entail disabling “pgo”, “nightly”, and
“release” builds in favor of our new "shippable" type. - For any teams who
may be impacted I’m seeking any concerns that could force us to revise our
plan here, or any sort of unknown hard blockers.

So, back in March (of 2017) aki sent along a proposal,
https://groups.google.com/forum/#!msg/mozilla.release.engineering/MRfHnhaVI48/l3kew8UmAAAJ
that we are now in a position to begin implementing, with the intent to
consider ourselves done in H2 2018.

I’ll avoid rehashing too much context from that post here, though here are
the highlights:

== (More) Background ==

We have a dizzying array of build configuration types we support and it’s
not always clear which type is most important on a given branch or where we
should run which tests. The mozilla-beta repo runs “nightly” builds per
push, but still wants testing; while on mozilla-central “nightly” only
happen periodically, when we’re ready to ship.

== Proposal ==

* We will keep (what is currently known as) depend opt and debug builds

* PGO, Nightly, and Release build types will exist as newly minted
“shippable” builds
** These “shippable” builds will be able to be promoted to an actual
release when built on a release branch
** Shippable builds may run periodically on integration branches

*** If periodically they will be backfillable.

* Release Branches should, where possible, only build shippable builds,
leaving the “current” depend opt builds out of consideration. Note the
mozilla-central repo is a release branch in this model.

== Not In Scope ==

* We will not consider “all beta like tasks, all esr like tasks and all
central like tasks buildable on the same push” as in scope, that has a lot
of cost/benefit analysis to be had and is another project entirely.
* We will not consider name bikeshedding at this stage.

** Dep => CI was one such brought up in the past thread, and while likely a
good idea, I don’t want to derail the bigger pieces for that.


== Questions? Concerns? ==

Does anyone have any issues with the above plan? Or any concerns overall
with the implementation timeline?


Thank You,

-Justin Wood (Callek), Release Engineer

[1] - https://bugzilla.mozilla.org/show_bug.cgi?id=1352113

[Aki Sasaki]

Yes, that's the plan. Initially we might be doing both shippable and opt
dep on-push on integration branches; later we might make shippable builds
on integration branches run periodically, with the ability to backfill.

On Thu, Aug 9, 2018 at 5:31 PM, Boris Zbarsky <bzba...@mit.edu> wrote:

> On 8/9/18 4:29 PM, Aki Sasaki wrote:
>
>> Under the current proposal, we will no longer be doing non-pgo opt builds
>> on m-c, for platforms where we ship pgo.
>> We will keep debug non-pgo, and I think asan, and other configurations
>> that are useful on m-c. If there are compelling reasons to keep opt non-pgo
>> dep builds on m-c, we can do that as well, but the plan is to not need them.
>>
>> Do you have concerns about this?
>>
>
> Are we still going to be doing both PGO and non-PGO opt builds on the
> various integration branches?
>
> Mostly I'm interested in having a way to notice when there are
> PGO-specific bugs.
>
> -Boris
>
> P.S. Re-adding dev-planning to cc; it got dropped for some reason.
>

[Boris Zbarsky]

[Ben Hearsum]

One small clarification inline below:

On Thu, Aug 09, 2018 at 03:18:18PM -0400, Justin Wood wrote:
> == Proposal ==
>
> * We will keep (what is currently known as) depend opt and debug builds

So opt is being renamed....

>
> * PGO, Nightly, and Release build types will exist as newly minted
> “shippable” builds

...and these three build types are getting a new name...

Does that mean that what is currently known as opt will continue to exist independent of shippable? Or all they all collapsing into shippable builds? If they're going to exist separately, what will their new name be?

- Ben

[Justin Wood]

The "Depend Opt" builds will continue to be around on at least integration
branches. It will likely go away on release branches like mozilla-central,
and mozilla-beta.

Naming is a concern but I wouldn't worry too much if the naming stays the
same, though the previous thread called for "CI" as a name suggestion,
though there was still some potential confusion there.

The naming I consider a bikeshed point and I want to tease out a bit any
concerns outside of this jobs naming.

~Justin Wood (Callek)

[Justin Wood]

...reply inline...

On Tue, Aug 14, 2018 at 9:54 PM, Gregory Szorc <g...@mozilla.com> wrote:

> On Thu, Aug 9, 2018 at 12:18 PM, Justin Wood <jw...@mozilla.com> wrote:
>
>> [please keep the release-engineering list CC'd on any replies, to try and
>> cut down on thread fragmentation]
>>
>>
>> Tl;dr Releng is starting to implement “Shippable” builds [1], and in doing
>> so will shift some testing, such as talos performance measures, over to
>> this new build type. This will entail disabling “pgo”, “nightly”, and
>> “release” builds in favor of our new "shippable" type. - For any teams who
>> may be impacted I’m seeking any concerns that could force us to revise our
>> plan here, or any sort of unknown hard blockers.
>>
>> So, back in March (of 2017) aki sent along a proposal,
>> https://groups.google.com/forum/#!msg/mozilla.release.engine
>> ering/MRfHnhaVI48/l3kew8UmAAAJ
>> that we are now in a position to begin implementing, with the intent to
>> consider ourselves done in H2 2018.
>>
>> I’ll avoid rehashing too much context from that post here, though here are
>> the highlights:
>>
>> == (More) Background ==
>>
>> We have a dizzying array of build configuration types we support and it’s
>> not always clear which type is most important on a given branch or where
>> we
>> should run which tests. The mozilla-beta repo runs “nightly” builds per
>> push, but still wants testing; while on mozilla-central “nightly” only
>> happen periodically, when we’re ready to ship.
>>

>> == Proposal ==
>>
>> * We will keep (what is currently known as) depend opt and debug builds
>>

>> * PGO, Nightly, and Release build types will exist as newly minted
>> “shippable” builds

>> ** These “shippable” builds will be able to be promoted to an actual
>> release when built on a release branch
>> ** Shippable builds may run periodically on integration branches
>>
>> *** If periodically they will be backfillable.
>>
>> * Release Branches should, where possible, only build shippable builds,
>> leaving the “current” depend opt builds out of consideration. Note the
>> mozilla-central repo is a release branch in this model.
>>
>> == Not In Scope ==
>>
>> * We will not consider “all beta like tasks, all esr like tasks and all
>> central like tasks buildable on the same push” as in scope, that has a lot
>> of cost/benefit analysis to be had and is another project entirely.
>> * We will not consider name bikeshedding at this stage.
>>
>> ** Dep => CI was one such brought up in the past thread, and while likely
>> a
>> good idea, I don’t want to derail the bigger pieces for that.
>>
>>
>> == Questions? Concerns? ==
>>
>> Does anyone have any issues with the above plan? Or any concerns overall
>> with the implementation timeline?
>>
>

> Overall this seems very reasonable and a well put together plan!
>
> I think we all agree that running tests on a different build that we ship
> to users is not ideal. Promoting normal CI builds to builds we ship seems
> like how we would do things if designing the system from scratch. And this
> proposal gets us closer to that ideal. So yay.
>
> I have a hopefully minor concern about potential negative impact on
> end-to-end build and test times. The builds we ship to users have
> historically been performed in a fresh environment without caching. This
> helps ensure reproducibility, minimizes build system risks from files left
> over from a previous build, rules out various cache poisoning attacks, etc.
> Regular builds in CI are allowed to be non-clobber / incremental builds and
> can aggressively use various caching. If I'm reading this proposal
> correctly, the new world will simply have "shippable" builds and not e.g.
> "PGO" builds. I'm assuming this means we will need to apply our "clean
> room" approach to all "shippable" builds. And it follows that these builds
> will be slower since they can't take advantage of various speedups.
>
> Is my understanding correct? If so, are we concerned about the impact of
> longer build times for "shippable" builds? Do we have numbers to quantify
> the expected build time difference? Will "shippable" builds on Try also
> have speedups disabled? Or can we (optionally) cheat there so developers
> aren't inconvenienced as much?
>

Close on your understanding. We intend to keep what is currently "Depend
Opt" on integration branches including try, the "shippable" will exist on
release branches (mozilla-central, beta, etc.).

Shippable will be akin to what is "Nightly" or "PGO" builds today, the
intent being that "what we expect to ship to users is how we build
shippable builds"


>
> FWIW, some of our choices around "clean room" builds are debatable. For
> example, once we have a build system whose dependencies we trust and can
> automatically purge orphaned outputs, we don't need to worry about clobber
> versus incremental. (Tup will enable this.) And with a bit more work to
> shore up security, we could probably get sccache to a point where we trust
> using it on "shippable" builds. (Also, ignore the fact that we stopped
> doing incremental builds when we moved to Taskcluster. It was an unintended
> regression and we should go back to incremental builds at some time if they
> lower end-to-end times without causing too many other problems.)
>

Yea, in my mind "Clean Room" builds will be what shippable is going to be
today, but does not preclude the concept that in the future we may choose
not to clean room them. The bottom line is "Shippable is the style of
building we are comfortable shipping to users" - If that stays PGO then
great, if we in the future deem PGO isn't needed Shippable can stay and we
just stop doing PGO. Should we in the future decide we want to enable
sccache or use a pre-populated object directory, that too is doable.

I don't know how far we will be swinging on that design and specifically it
is out of scope in my mind, since the intent is to consolidate and make
sure that "Ships to users" is the main metric we focus on here. Changes
along that axis are possible but not required, similarly we want to keep
"Dep Opt" (or whatever we call them in the new world) for faster turnaround
time on try and integration branches so sheriffs and developers alike can
see their results as fast as possible.

~Justin Wood (Callek)