On Tue, Aug 14, 2018 at 9:54 PM, Gregory Szorc <g...@mozilla.com
> On Thu, Aug 9, 2018 at 12:18 PM, Justin Wood <jw...@mozilla.com
>> [please keep the release-engineering list CC'd on any replies, to try and
>> cut down on thread fragmentation]
>> Tl;dr Releng is starting to implement “Shippable” builds , and in doing
>> so will shift some testing, such as talos performance measures, over to
>> this new build type. This will entail disabling “pgo”, “nightly”, and
>> “release” builds in favor of our new "shippable" type. - For any teams who
>> may be impacted I’m seeking any concerns that could force us to revise our
>> plan here, or any sort of unknown hard blockers.
>> So, back in March (of 2017) aki sent along a proposal,
>> that we are now in a position to begin implementing, with the intent to
>> consider ourselves done in H2 2018.
>> I’ll avoid rehashing too much context from that post here, though here are
>> the highlights:
>> == (More) Background ==
>> We have a dizzying array of build configuration types we support and it’s
>> not always clear which type is most important on a given branch or where
>> should run which tests. The mozilla-beta repo runs “nightly” builds per
>> push, but still wants testing; while on mozilla-central “nightly” only
>> happen periodically, when we’re ready to ship.
>> == Proposal ==
>> * We will keep (what is currently known as) depend opt and debug builds
>> * PGO, Nightly, and Release build types will exist as newly minted
>> “shippable” builds
>> ** These “shippable” builds will be able to be promoted to an actual
>> release when built on a release branch
>> ** Shippable builds may run periodically on integration branches
>> *** If periodically they will be backfillable.
>> * Release Branches should, where possible, only build shippable builds,
>> leaving the “current” depend opt builds out of consideration. Note the
>> mozilla-central repo is a release branch in this model.
>> == Not In Scope ==
>> * We will not consider “all beta like tasks, all esr like tasks and all
>> central like tasks buildable on the same push” as in scope, that has a lot
>> of cost/benefit analysis to be had and is another project entirely.
>> * We will not consider name bikeshedding at this stage.
>> ** Dep => CI was one such brought up in the past thread, and while likely
>> good idea, I don’t want to derail the bigger pieces for that.
>> == Questions? Concerns? ==
>> Does anyone have any issues with the above plan? Or any concerns overall
>> with the implementation timeline?
> Overall this seems very reasonable and a well put together plan!
> I think we all agree that running tests on a different build that we ship
> to users is not ideal. Promoting normal CI builds to builds we ship seems
> like how we would do things if designing the system from scratch. And this
> proposal gets us closer to that ideal. So yay.
> I have a hopefully minor concern about potential negative impact on
> end-to-end build and test times. The builds we ship to users have
> historically been performed in a fresh environment without caching. This
> helps ensure reproducibility, minimizes build system risks from files left
> over from a previous build, rules out various cache poisoning attacks, etc.
> Regular builds in CI are allowed to be non-clobber / incremental builds and
> can aggressively use various caching. If I'm reading this proposal
> correctly, the new world will simply have "shippable" builds and not e.g.
> "PGO" builds. I'm assuming this means we will need to apply our "clean
> room" approach to all "shippable" builds. And it follows that these builds
> will be slower since they can't take advantage of various speedups.
> Is my understanding correct? If so, are we concerned about the impact of
> longer build times for "shippable" builds? Do we have numbers to quantify
> the expected build time difference? Will "shippable" builds on Try also
> have speedups disabled? Or can we (optionally) cheat there so developers
> aren't inconvenienced as much?
Close on your understanding. We intend to keep what is currently "Depend
Opt" on integration branches including try, the "shippable" will exist on
release branches (mozilla-central, beta, etc.).
Shippable will be akin to what is "Nightly" or "PGO" builds today, the
intent being that "what we expect to ship to users is how we build
> FWIW, some of our choices around "clean room" builds are debatable. For
> example, once we have a build system whose dependencies we trust and can
> automatically purge orphaned outputs, we don't need to worry about clobber
> versus incremental. (Tup will enable this.) And with a bit more work to
> shore up security, we could probably get sccache to a point where we trust
> using it on "shippable" builds. (Also, ignore the fact that we stopped
> doing incremental builds when we moved to Taskcluster. It was an unintended
> regression and we should go back to incremental builds at some time if they
> lower end-to-end times without causing too many other problems.)
Yea, in my mind "Clean Room" builds will be what shippable is going to be
today, but does not preclude the concept that in the future we may choose
not to clean room them. The bottom line is "Shippable is the style of
building we are comfortable shipping to users" - If that stays PGO then
great, if we in the future deem PGO isn't needed Shippable can stay and we
just stop doing PGO. Should we in the future decide we want to enable
sccache or use a pre-populated object directory, that too is doable.
I don't know how far we will be swinging on that design and specifically it
is out of scope in my mind, since the intent is to consolidate and make
sure that "Ships to users" is the main metric we focus on here. Changes
along that axis are possible but not required, similarly we want to keep
"Dep Opt" (or whatever we call them in the new world) for faster turnaround
time on try and integration branches so sheriffs and developers alike can
see their results as fast as possible.
~Justin Wood (Callek)