Fwd: Comm-central build system changes

[Philip Chee]

-------- Original Message --------
Date: Tue, 19 Mar 2013 23:49:37 -0500
From: Joshua Cranmer 🐧 <Pidg...@gmail.com>
Subject: Comm-central build system changes

At present, a few of us have been working on making the comm-central
build system work better with mozilla-central. Since we're close to
achieving a major milestone, I felt it's best to post a status report on
the current effort.

The short version:
* comm-central/configure.in is on the verge of going away (patches
requesting review this week)
* comm-central/config/* and build/* will be going away soon (next month
or so)
* The way you build Thunderbird will probably changing as well
* Some requests for discussion on going forward are included

Why are making these changes? Some history will explain this.

In the old days, everything was in CVS and everyone used the same build
system without caring about anything since CVS lets you checkout an
arbitrary subset of files. When Firefox shifted development to Mercurial
in 2008, the CVS import didn't include all the files in CVS but rather
the subset that were deemed necessary for Mozilla 2 [1].

When comm-central was created [2], there were two feasible choices for
handling mozilla-central: build mozilla-central under comm-central or
vice versa; the first was the option that was taken. At the time, the
build system was still very similar to the original build system, so
there were no major "hacks" that were necessary to get this working. Oh,
how times have changed.

The change that is probably most responsible for making the original
comm-central build choice regrettable is the change that made
comm-central code go inside of libxul, which can be called a mild hack.
And, going back and reading the original bugs where these things were
made, this change was probably originally intended to be temporary [3].

Now, with the fast pace of the moz.build changes and impending
rewrite/obliteration of the mozilla-central build system, we stand a
good chance of suffering extensive outages as our house of cards
crumbles. And thus we need to take major action to reduce the amount of
code we need to port from the mozilla-central build system.

I mentally divide this work into two steps. The first step is the
emasculation of configure--bug 846540. This set of patches harmonizes
the configuration of comm-central with mozilla-central by injecting all
of our build variables into the mozilla-central build system.
Incidentally, this work would have eliminated the bustage from last week
when the XPIDLSRCS changes landed [4]. This is the major milestone of
which I speak: the emasculated configure works on Linux platforms, and I
am currently tracking down issues related to OS X universal builds and
Windows configure errors. When these patches land, the only thing
different between the two build system changes are the un-backported
comm-central build changes [5] and path issues.

The second step of changes is the step that eliminates the entire build
system from comm-central. There are a few options to go about this.
Presently, to make minimal adjustments to paths, I introduce lots of
symlinks to make comm-central appear to be both a subdirectory of and an
overlay on top of mozilla-central. Options we could use are:

1. Move comm-central to mozilla-central/comm. This is kind of the
default assumption I'm working towards.
2. Keep mozilla-central in comm-central/mozilla, but add magic to the
build system to make everything work. This might be possible, but it
would require some gross hacks and is probably an unstable solution: I'm
already finding subtle bugs in this arrangement just from emasculating
configure [6].
3. Option #1, but allow a build under the current configuration by
injecting enough symlinks to make it work.
4. Keep mozilla-central in comm-central/mozilla, but inject symlinks to
make comm-central appear to overlay mozilla-central.
5. Hold the final change hostage to merging comm-central into
mozilla-central.

Most of these options require making changes to our release engineering
systems. Using symlinks can avoid the need for changes, but comes with
increased fragility and would completely rule out builds on FAT
systems--I doubt they are tenable long-term options. Opinions on the way
to go forward here are appreciated.

One additional comment I hinted at in my list of options is as follows:
once the comm-central build system is eliminated, there would remain no
technical barrier to actually merging comm-central and mozilla-central
into the same repository. This is not a change that can be made without
wider project discussion, but I think the imminent prospect of
eliminating the build system implies that this discussion should be
started soon. Like the final build system-elimination phase, this would
require changes to release engineering infrastructure. Benefits of the
change include making tree-wide changes (like the recent nsITreeView
mega-change or the constant moz.build changes) much easier to commit, as
well as follow-on effects resulting from infrastructure coping with the
fact that mozilla-central builds more than one product. I mention this
now because unresolved technical aspects that could go with this change
(e.g., whether to merge comm-central "flat" or all in a comm/
subdirectory) could have some influence on decisions taken while
handling the intermediate structure of the build system.

Those of you who are interested in seeing the current progress on this
front are invited to peruse all of my submissions to try-comm-central.

As always, comments, questions, opinions, and thoughts are welcome.

Footnotes:
[1] History trivia: Mozilla 2 was the codename for the version of
Mozilla that would contain large sweeping changes to eradicate technical
debt including such things as automatic garbage collection instead of
reference counting, using C++ exceptions instead of rv-everywhere, and
"outparamdel" (moving return values to be just that instead of the out
parameter). Instead, all that we got for the real Mozilla 2 (Firefox 4)
was a rewrite of the component manager and the abolition of the notion
of frozen interfaces.
[2] More trivia: the original codename for comm-central was
"CaMaSutra"-- Calendar, Mail, Suite. Actually, this might work as an
internal name for mailnews non-Gecko internals in lieu of rkent's Skink
name.
[3] Interesting retrospective: it seems that mozilla-central build
system changes are the best way to convert a lackadaisical
make-the-comm-central-build-system-better approach into a
get-it-committed-now-I-don't-care-how approach.
[4] Technical details: when the first set of moz.build files landed,
Gregory was nice enough to make changes to the mozbuild infrastructure
to properly support building comm-central. Unfortunately, what he and
all reviewers missed was the fact that the changes only munged the bases
of relative paths, executing all the files with the same config.status
file. The largest difference between the two build systems was
MOZ_LDAP_XPCOM, which wasn't used to select files to build until the
XPIDL flags landed. When I saw the bustage, I tested the fix locally to
make sure it worked... on the tree that already had all these changes.
Lesson: don't test emergency bustage fixes for a build system on top of
a tree that partially rewrites the build system.
[5] Yes, we actually have useful features in the comm-central build
system that mozilla-central doesn't have. The one I know of offhand is
$(DIR_INSTALL).
[6] The really subtle one: autoconf works out the wrong relative path
for ../ldap/sdks/c-sdk, which is normally innocuous, but ends up causing
the second architecture build on an OS X universal build to pick up the
wrong cache config file.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist