Lack of happenings in core build system land

[Gregory Szorc]

I wanted to make people aware that from my perspective as build config
module owner, the build config module and the Firefox build system are
under water. There are bugs coming in that aren't being addressed
because there aren't resources available to address them. Forward
progress has stalled to a snail's pace. We are effectively only triaging
the most important issues or performing reviews if someone writes a patch.

The severity of the problem depends on who you are. By some accounts,
the build system is staying afloat. We are fixing major issues
(especially if they impact automation). And, people are still scratching
itches here and there. People are still able to build Firefox. So, the
problem is easy to glance over. But, some people have expressed
complaints directly to me. Who knows how many more are silently
resenting the current status.

From my perspective as module owner, bugs that are impacting
productivity are falling through the cracks and a significant number of
people are impacted by this. Furthermore, forward progress in the build
system has stalled for the past 3 months or so. Our trajectory towards
faster builds has been suspended. This is really unfortunate, as we are
very close to addressing some of the big-ticket items that will enable
much faster "light builds" (such as optimizing linking, being smarter
about XPCOM manifest generation, and optimizing install manifests).

The root cause of all of this is that the main contributors to the build
system are currently all tasked with other things. I'm off doing Firefox
data metrics work. Mike Hommey and Mike Shal are busy making automation
better (which I consider work outside of the core build system even
though it's under the same "build" umbrella). All other peers are
non-active or only scratch itches when they need to. All other
contributors basically scratch itches as needed. There is nobody to tend
to the day-to-day bug squashing much less big-ticket features. From what
I know about goals and projects of build module peers, don't expect
anything to change for at least a few months.

To be clear, I'm not arguing that the core build system is more
important than whatever the previously-active build system contributors
are currently working on. What I am saying is that nobody is currently
working on things that I feel are needed for the build module to remain
healthy (notably bug triage and investment in the future). I also don't
mean to diminish the contributions of non-peers and non-active peers.
Your work is pretty much the only thing that's giving the build module
signs of life. Keep it up!

I regret having to write this post. Hopefully greater awareness will
bring change. Until then, I apologize for any discomfort an unattended
build system/module is bringing.

Gregory

[Michael Shal]

----- Original Message -----
> From: "Gregory Szorc" <g...@mozilla.com>
>
> The severity of the problem depends on who you are. By some accounts,
> the build system is staying afloat. We are fixing major issues
> (especially if they impact automation). And, people are still scratching
> itches here and there. People are still able to build Firefox. So, the
> problem is easy to glance over. But, some people have expressed
> complaints directly to me. Who knows how many more are silently
> resenting the current status.

This part always seemed quite strange to me - build config essentially has two consumers: developers and automation. While improvements to the core build system can benefit both, organizationally we only have a team of people dedicated to the automation side of things. Essentially this means we only make significant progress in the dev side when there's free time, or when frustration with the current state of the build system boils over and someone decides to put their real work aside for a while. This makes long term progress much harder to achieve. IMO there's no reason we can't deliver a 100% reliable, 0%-clobber, near-instantaneous turnaround build system experience to developers. We just love machines more, I guess.

-Mike

[gregor...@gmail.com]

I think you hit the nail on the head: we have staff whose responsibility is automation but we have no staff whose primary responsibility is developers (in terms of build system at least). The work will happen where there is management accountability and there is currently nobody accountable for the core build system.

I don't agree with that decision (I wish the core build system had at least 1 full-time staff person focusing on developer ergonomics and who wouldn't get distracted by automation foo). However, I can rationalize it a bit. There are concrete costs attached to automation, such as our EC2 bill. Improvements and efficiency in automation can be measured. The core build system doesn't have the luxury of direct costs. Therefore, estimating the cost of an inefficient build system and the impact of improvements is difficult. Compound this with the fact the build system has been inefficient for a decade and I think a lot of old-time engineers and managers have the perspective that we've been dealing with it for years without too much of a problem, it therefore isn't a problem worth fixing.

I have little doubt that a team of 3 to 5 people working on the core build system full-time could pump out enough efficiency/productivity wins for 12-18 months before we even questioned the financial investment of those staff. The way I look at it, if one person raises the productivity of 100 people by 1%, that person's salary is paid for. I think a 1% productivity boost through a better build system is easily attainable. You figure Mozilla has a few hundred engineers that would benefit, and you've justified multiple staff working on the build system! I just wish someone with the power of the purse would agree with me and we'd start staffing the core build system.