Moving Away from Makefile's

[Gregory Szorc]

tl;dr We're proposing moving away from Makefile's as the sole source of
the build system definition. This will lead to faster build times.
Bikeshedding^wFeedback on the file format is requested.

The existing build system is defined by Makefile.in's scattered around
the source tree (typically one Makefile.in per directory). At configure
time, these Makefile.in's get preprocessed into Makefile's using simple
variable substitution. Then make/pymake is let loose on the result. It
is a very traditional model.

We are attempting to move to a model where the build definition is
generic and data-driven. By treating the build definition as data
(rather than a glorified shell script that is Makefiles), this will
allow us to take that data and convert it into formats understood by
other, better/faster build backends, such as non-recursive make files,
Tup, Ninja, or even Visual Studio.

Up until now, the focus has been on making Makefile.in's themselves
generic and data-driven [1]. We would use pymake's API to parse, load,
and extract data from Makefile.in's to construct the build definition.
In the long run, we'd realize that using make files for data definition
was silly (and a large foot gun) and thus we would switch to something else.

After a long IRC conversation, Mike Hommey and I concluded that we want
to begin the transition away from Makefile.in's ASAP.

Essentially, the proposal is to move (not duplicate) some data from
Makefile.in's into new files. Initially, this would include things like
subdirectories to descend into and files to copy/preprocess. Simple
stuff to start with. Eventually, scope would likely increase to cover
the entirety of the build system definition (like compiling), rendering
Makefile.in's obsolete. But, it will take a *long* time before we get there.

In the new world, the source of truth for the build system is jointly
defined by existing Makefile.in's and whatever these new files are that
we create. I'll call these not-yet-existing files "build manifest
files." Somewhere in the build process we read in the build manifest
files and generate output for the build backend of choice.

Our existing non-recursive make backend should integrate with this
seamlessly. Instead of a dumb variable substitution phase for
configuring the build backend, we'll have some additional logic to write
out new make files derived from the contents of the build manifest
files. This is similar to the approach I've taken in build splendid [2].
The only difference is the build definition is living in somewhere not
Makefile.in's.

We don't have details on how exactly the migration will be carried
about. But, it should be seamless. So, unless you touch the build
system, you should be able to continue living in blissful ignorance.

If you have any concerns over this transition, please voice them.

File Format
===========

I hinted at bikeshedding in the tl;dr. We want feedback on the file
format to use for the new build manifest files. The requirements are as
follows (feel free to push back on these):

1. Easy for humans to grok and edit. An existing and well-known format
is preferred. We don't want a steep learning curve here.
2. Simple for computers to parse. We will use Python to load the build
manifest files. Python can do just about anything, so I'm not too
worried here.
3. Efficient for computers to load. As these files need to be consulted
to perform builds, we want to minimize the overhead for reading them
into (Python) data structures.
4. Native support for list and maps. Make files only support strings.
The hacks this results in are barely tolerable.
5. Ability to handle conditionals. We need to be able to conditionally
define things based on the presence or value of certain "variables."
e.g. "if the current OS is Linux, append this value to this list." I
quote "variables" because there may not be a full-blown variable system
here, just magic values that come from elsewhere and are addressed by
some convention.
6. Ability to perform ancillary functionality, such as basic string
transforms. I'm not sure exactly what would be needed here. Looking at
make's built-in functions might be a good place to start. We may be able
to work around this by shifting functionality to side-effects from
specially named variables, function calls, etc. I really don't know.
7. Evaluation must be free from unknown side-effects. If there are
unknown side-effects from evaluation, this could introduce race
conditions, order dependency, etc. We don't want that. Evaluation must
either be sandboxed to ensure nothing can happen or must be able to be
statically analyzed by computers to ensure it doesn't do anything it
isn't supposed to.
8. Doesn't introduce crazy build dependencies. We /might/ be able to get
away with something well-known. But, new build dependencies are new
build dependencies.

Ideally, the data format is static and doesn't require an interpreter
(something like YAML or JSON). Unfortunately, the need for conditionals
makes that, well, impossible (I think).

We could go the route of GYP and shoehorn conditionals into a static
document (JSON) [3]. Actually, using GYP itself is an option! Although,
some really don't like the data format because of this shoehorning (I
tend to agree).

On the other end of the spectrum, we could have the build manifest files
be Python "scripts." This solves a lot of problems around needing
functionality in the manifest files. But, it would be a potential foot
gun. See requirement #7.

Or, there is something in the middle. Does anyone know of anything that
can satisfy these requirements? I think Lua is perfect for this (it was
invented to be a configuration language after all). But, I'm not sure it
satisfies #1 nor #8.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=774049
[2]
http://gregoryszorc.com/blog/2012/08/15/build-firefox-faster-with-build-splendid/
[3] https://code.google.com/p/gyp/wiki/GypLanguageSpecification

[xunxun]

于 2012/8/22 7:36, Gregory Szorc 写道:
> tl;dr We're proposing moving away from Makefile's as the sole source
> of the build system definition. This will lead to faster build times.
> Bikeshedding^wFeedback on the file format is requested.
>
> The existing build system is defined by Makefile.in's scattered around
> the source tree (typically one Makefile.in per directory). At
> configure time, these Makefile.in's get preprocessed into Makefile's
> using simple variable substitution. Then make/pymake is let loose on
> the result. It is a very traditional model.
>
> We are attempting to move to a model where the build definition is
> generic and data-driven. By treating the build definition as data
> (rather than a glorified shell script that is Makefiles), this will
> allow us to take that data and convert it into formats understood by
> other, better/faster build backends, such as non-recursive make files,
> Tup, Ninja, or even Visual Studio.

Don't like VS project, and we will find that VS project is a so big work
in future development.

>
> Up until now, the focus has been on making Makefile.in's themselves
> generic and data-driven [1]. We would use pymake's API to parse, load,
> and extract data from Makefile.in's to construct the build definition.
> In the long run, we'd realize that using make files for data
> definition was silly (and a large foot gun) and thus we would switch
> to something else.

At present, we also can use pymake, I don't know and I want to know when
your work is done, we will abandon pymake?

And will the work become a new development branch?

> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=774049
> [2]
> http://gregoryszorc.com/blog/2012/08/15/build-firefox-faster-with-build-splendid/
> [3] https://code.google.com/p/gyp/wiki/GypLanguageSpecification

> _______________________________________________
> dev-builds mailing list
> dev-b...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-builds


--
Best Regards,
xunxun

[Gregory Szorc]

On 8/21/2012 6:12 PM, xunxun wrote:
> 于 2012/8/22 7:36, Gregory Szorc 写道:
>> Up until now, the focus has been on making Makefile.in's themselves
>> generic and data-driven [1]. We would use pymake's API to parse,
>> load, and extract data from Makefile.in's to construct the build
>> definition. In the long run, we'd realize that using make files for
>> data definition was silly (and a large foot gun) and thus we would
>> switch to something else.
> At present, we also can use pymake, I don't know and I want to know
> when your work is done, we will abandon pymake?

As long as we are supporting building with make on Windows, we will
support pymake.

> And will the work become a new development branch?

It's too early to tell how this work will play out. I imagine the goal
will be to merge work into mozilla-central as soon as it is ready to
minimize the potential for bit rot. We don't want two sources of truth
for build system data which can get out of sync.

[Blair McBride]

On 22/08/2012 11:36 a.m., Gregory Szorc wrote:
> I think Lua is perfect for this (it was invented to be a configuration
> language after all). But, I'm not sure it satisfies #1 nor #8.

+1 for Lua - it seems perfect for this. For #1, I find it far easier to
read (and write) than Gyp, when it comes to things like conditionals.
For #8, we could just ship the entire runtime in the tree for Tier 1
platforms (its small enough!), then its only an additional dependency
for the 0.01%.

- Blair

[Mike Hommey]

But then, you have a chicken and egg kind of problem, where your build
system depends on building something...

My preference would go do simple, preprocessed .ini-like files. Or
somewhere between a .ini and a jar.mn.

Mike

[Neil]

Gregory Szorc wrote:

> Up until now, the focus has been on making Makefile.in's themselves

> generic and data-driven. We would use pymake's API to parse, load, and

> extract data from Makefile.in's to construct the build definition. In
> the long run, we'd realize that using make files for data definition
> was silly (and a large foot gun) and thus we would switch to something
> else.

Switch to something else for all Makefile.in's or just the generic ones?

--
Warning: May contain traces of nuts.

[Blair McBride]

On 22/08/2012 5:58 p.m., Mike Hommey wrote:
>> +1 for Lua - it seems perfect for this. For #1, I find it far easier
>> >to read (and write) than Gyp, when it comes to things like
>> >conditionals. For #8, we could just ship the entire runtime in the
>> >tree for Tier 1 platforms (its small enough!), then its only an
>> >additional dependency for the 0.01%.
> But then, you have a chicken and egg kind of problem, where your build
> system depends on building something...

When I say "runtime" I mean the actual pre-built runtime, not the source.

- Blair

[Robert Kaiser]

Gregory Szorc schrieb:

> We could go the route of GYP and shoehorn conditionals into a static
> document (JSON) [3].

JSON is a good format for data for the most part, but IMHO we *really*
want comments in those files, and unfortunately JSON doesn't have those
and therefore probably must be thrown out of the equation. :(

Actually, I know a format that allows everything we need: Makefile! :p

Seriously, this is hard, and needing to parse even more files to build
doesn't sound faster and cleaner, but rather slower and more complex,
esp. given that basic file I/O is often costly (from watching my CPU
usage, a lot of the build time is spent in I/O wait when using spinning
disks - SSDs improve that hugely).

Robert Kaiser

[Benjamin Smedberg]

On 8/21/2012 7:36 PM, Gregory Szorc wrote:
>
>
>
> On the other end of the spectrum, we could have the build manifest
> files be Python "scripts." This solves a lot of problems around
> needing functionality in the manifest files. But, it would be a
> potential foot gun. See requirement #7.

I don't think this would be a big deal. We could just enforce no side
effects at review, or with a small bit of python we could enforce some
basic restraints in code:

* allow imports from only a small whitelist of known-safe modules:
perhaps just 're', or disallow imports altogether (by modifying the
globals in the execution environment) and pre-import the safe modules
* "fix" output variables to ensure that they are the expected
string/list-of-strings/whatever types

I really think that python manifests are the best choice here, since
python is already optimized to parse them efficiently, the control
structures are fairly straightforward, and most people (both regular
engineers and build-sytem hackers) are going to know enough to get
started without confusion.

One thing that wasn't clear to me from the original post is whether we
are planning on automatically transforming some parts of the existing
makefiles (e.g. DIRS, EXPORTS, preprocessor stuff) into the new manifest
format, or whether the plan is to just migrate by hand. Or is that still
TBD?

--BDS

[Mike Hommey]

Which you then need to ship as a windows binary, an osx binary, and a
linux binary. And that other platforms need to build or install
separately.

Mike

[Jeff Hammel]

I don't have a magic bullet here. My inclination is to not introduce
new technology/languages/syntax to keep the learning curve lower/comparable.

If we're going to go for a whole language, I am +1 on python. We
already use it...well, all over the place. It is well known, solid, and
not "yet another thing to learn or syntax switch" to. Unless a very
compelling argument argument could be made for the functionality of
other languages vs python for this problem space, python seems a natural
solution and can build on our existing python ecosystem (which likewise
can grow driven by needs here).

My actual inclination is to use a pure data format. I'm not sure if
.ini is sufficient: guessing not. My inclination here is JSON. That
said, it won't have conditionals or logic, so as :gps puts it, these
would have to be shoehorned in. To expand on that a bit, the different
platforms/configurations/states will have to be recorded and the runner
would have to have the logic to decide what to use. I haven't looked at
this problem in depth, so I don't know how good/bad/horrible this would
be in practice.

If we do go with python, it would be nice to keep the configuration
files as much configuration as possible. The reason I question having
any "full" language as a configuration language is that in practice I
see a lot of logic creep in to "configuration" files and then they
become very difficult for a consumer to parse/use as configuration.
While a bit of an unfair example, our buildbot-configs fall into this
category.

TL;DR - python if we want/need a full language, JSON if we can get away
with POD

Jeff

[Ehsan Akhgari]

On 12-08-22 12:43 PM, Jeff Hammel wrote:
> TL;DR - python if we want/need a full language, JSON if we can get away
> with POD

I think JSON is the wrong choice here. It will not satisfy the need for
conditionals, which causes us to invent terrible hacks inside the JSON
like we had to do with the xpcshell manifest format, and it's not really
easy to be consumed or edited by humans.

Python is the clear winner here, IMO.

Ehsan

[Gregory Szorc]

The plan would be to have no Makefile.in's, as the presence of a
Makefile implies usage of make, which may not always be the case. We
want to support building with alternate build backends and that is
difficult if large parts involve one-off usages of make. We may use make
for some specific tasks (because it is a decent dependency-action DSL).
But, for building the core of mozilla-central, use of Makefile's as the
*definition* of the build config will likely be marginalized.

On 8/22/12 6:10 AM, Robert Kaiser wrote:> Gregory Szorc schrieb:

>> We could go the route of GYP and shoehorn conditionals into a static
>> document (JSON) [3].
>

> JSON is a good format for data for the most part, but IMHO we *really*
> want comments in those files, and unfortunately JSON doesn't have those
> and therefore probably must be thrown out of the equation. :(
>
> Actually, I know a format that allows everything we need: Makefile! :p

I actually agree with you! This was our initial plan: load Makefiles
into pymake and evaluate the values of specific variables, like CPPSRCS.
Unfortunately, there are some deal-breakers. The evaluation model of
make is... complicated. Lazy evaluation, rules that must be executed
before variables can be read, side-effects from evaluating variables,
etc. There are a *lot* of pitfalls.

Switching to something else also has another advantage: the opportunity
for a clean slate. I'm hoping we'll use the opportunity to scrape away
10+ years of cruft.

> Seriously, this is hard, and needing to parse even more files to build
> doesn't sound faster and cleaner, but rather slower and more complex,
> esp. given that basic file I/O is often costly (from watching my CPU
> usage, a lot of the build time is spent in I/O wait when using spinning
> disks - SSDs improve that hugely).

On my browser build, we place 19000+ files in the objdir. That's not
counting all the files in the srcdir that need to be stat() or read
during builds. I think the overhead of a few hundred build manifest
files won't be too bad. It will also likely be mitigated through caching.

Yes, I/O wait can significantly reduce build times. I recommended at [1]
that aside from a modern CPU, investing in an SSD is the best thing you
can do for build times. If a build machine doesn't have an SSD, it's
effectively throwing away CPU cycles in I/O wait. It's true that the
build system today wastes a lot of CPU cycles due to inefficient use of
all available cores (lack of wide parallelism due to recursive make).
But, the efficiency should rise drastically with our build system
improvements. That will make the impact of an SSD even more pronounced.
On my MBP (with 8GB), building with an SSD reduced overall build time by
a few minutes, with libxul linking going from ~55s to ~5s. That's
without any build system changes! There is no doubt in my mind that SSDs
are worth the investment.

[1] http://gregoryszorc.com/blog/2012/07/29/mozilla-central-build-times/

[Justin Wood (Callek)]

Jeff Hammel wrote:
> While a bit of an unfair example, our buildbot-configs fall into this
> category.

IMO not unfair at all.

(p.s. to stay on topic, +1 to all else you said)

~Justin Wood (Callek)

[Ben Hearsum]

On 08/22/12 12:43 PM, Jeff Hammel wrote:
> If we do go with python, it would be nice to keep the configuration
> files as much configuration as possible. The reason I question having
> any "full" language as a configuration language is that in practice I
> see a lot of logic creep in to "configuration" files and then they
> become very difficult for a consumer to parse/use as configuration.
> While a bit of an unfair example, our buildbot-configs fall into this
> category.

A million times this. Putting our configs in Python has let to terrible,
terrible hacks - to the point where we don't even know how to do things
like "turn off leopard tests for some branches" anymore
(https://bugzilla.mozilla.org/show_bug.cgi?id=773120). I strongly
discourage using Python when you don't need a fully featured language.
You will do bad things. You will regret it.

[Ben Hearsum]

On 08/22/12 12:43 PM, Jeff Hammel wrote:

> If we do go with python, it would be nice to keep the configuration
> files as much configuration as possible. The reason I question having
> any "full" language as a configuration language is that in practice I
> see a lot of logic creep in to "configuration" files and then they
> become very difficult for a consumer to parse/use as configuration.
> While a bit of an unfair example, our buildbot-configs fall into this
> category.

[Jeff Hammel]

On 08/22/2012 12:29 PM, Gregory Szorc wrote:
>
> Switching to something else also has another advantage: the
> opportunity for a clean slate. I'm hoping we'll use the opportunity to
> scrape away 10+ years of cruft.

I support the sentiment, though this reasoning always makes me cautious,
as it often translates to (unintentionally) eliminating the old system
(cruft + accrued knowledge), replacing with a new system that initially
feels clean but once all the features have added and it needs to be
extended is just different cruft, and cruft that is no longer standard.

Don't get me wrong: I don't think you would do this and I am very
grateful to see you tackle making the build system better. I just
wanted to point out to the list that this is a "WARNING! Big change!
Proceed with caution!" scenario.

[Gregory Szorc]

There are a lot of bullets in "batteries included" languages. The Python
foot gun worries me greatly. We even have this problem in make, and that
is a much simpler language!

Let's think of what can be done to secure/limit Python. Disabling import
has already been mentioned. That's a start.

What about the ast module [1]? I /think/ it could be used to ensure
parsed code conforms to whatever we allow. e.g. no import statements, no
function definitions, no classes, etc. Actually, it may be better to
think of it in terms of what would be allowed. Only declaring variables
that are strings, lists, or dicts. Only calling pre-defined functions
"exported" by the build system, etc. We start with the minimum and open
it up as we need to.

Is there anyone out there with experience with this module? What's
performance like for walking the AST? Is this something we could do at
manifest read/eval time without incurring too much of a penalty? (If
not, there's always automated auditing, which could even be part of the
test suite so trees would go red if someone checked in something in
violation.) Are there any caveats to the low-level nature of this
module? i.e. is there a burden to supporting ast functionality on
multiple versions of Python (we'll eventually need to handle the 2.7 ->
3.x transition).

For those who don't know Lua, the main reason I said it was perfect in
the original post is it does all of this automatically! It's designed to
be embeddable. A newly-created Lua context/interpreter can do very
little. You need to explicitly load the standard library from the C API!
A new context doesn't even have a string module nor the ability to load
modules: These must be selectively enabled through the C API by the
process creating the context instance! It's still a full programming
language. But, it's so constricted without any external modules or
module loading support that you almost don't have to worry. It is a more
natural fit for the task at hand. But, given Mozilla's requirements
(extra build dependency, maintaining the Lua bridge code - I know the
Lua C API, does anyone else?), I think shoehorning Python would be
preferred over Lua.

[1] http://docs.python.org/library/ast.html

[Ben Hearsum]

On 08/22/12 04:38 PM, Gregory Szorc wrote:
> Let's think of what can be done to secure/limit Python. Disabling import
> has already been mentioned. That's a start.

I think it's worth noting that even if you *do* limit what you can do
through some technical means, you still have the option to change that
later, disable it some places, etc. It's really easy to get into that
game when you're fixing blockers or working on chemspills, too.

[Ben Hearsum]

On 08/22/12 04:38 PM, Gregory Szorc wrote:

> Let's think of what can be done to secure/limit Python. Disabling import
> has already been mentioned. That's a start.

[Ian Bicking]

On Wed, Aug 22, 2012 at 3:38 PM, Gregory Szorc <g...@mozilla.com> wrote:

> Let's think of what can be done to secure/limit Python. Disabling import
> has already been mentioned. That's a start.
>

> What about the ast module [1]? I /think/ it could be used to ensure parsed
> code conforms to whatever we allow. e.g. no import statements, no function
> definitions, no classes, etc. Actually, it may be better to think of it in
> terms of what would be allowed. Only declaring variables that are strings,
> lists, or dicts. Only calling pre-defined functions "exported" by the build
> system, etc. We start with the minimum and open it up as we need to.
>

Instead of disallowing imports, you could replace the importer (during
build file evaluation) with something that allows only whitelisted modules,
and potentially only allows subsets of those modules. For instance, you
might want os.environ, but not os.open. You might also want to do
something like replace os.environ with a wrapper that tracks access, so you
can detect the environmental dependencies.

I don't think functions and classes can make things too much more
complicated, so long as the build files can't share those definitions
outside the file (which could lead to an ad hoc and hidden library).
Something like open() seems perhaps more dangerous in terms of creating
hard-to-understand dependencies.

Doing strict type checks on the resulting data would be a good check.
E.g., you don't want a file to return a clever str subclass that overrides
equality.

[Robert O'Callahan]

If someone is that desperate, what would you have them do instead of hack
the configuration file? Aren't they likely to respond by doing some even
worse hack that gets the job done?

I think it makes a ton of sense to use automation to stop developers
accidentally doing something they shouldn't. But if someone's desperate
enough to disable the automation, and can get a review for it, then I don't
think it makes sense to try to stop them.

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]

[Nicholas Nethercote]

So...

Data-only formats such as JSON are too weak -- no comments, no conditionals.

Full languages (e.g. Python, Lua) are too powerful, which means it's
too tempting to use the full language features.

Should you invent something new? If it's just lists, maps, comments
and conditionals, that should be easy. And looking at Gregory's
initial list of eight requirements, it sounds like they could be
satisfied nicely.

Just a thought :)

Nick

[Mike Hommey]

On Wed, Aug 22, 2012 at 01:38:22PM -0700, Gregory Szorc wrote:
> On 8/22/12 1:00 PM, Ben Hearsum wrote:

> There are a lot of bullets in "batteries included" languages. The
> Python foot gun worries me greatly. We even have this problem in
> make, and that is a much simpler language!

As I said on irc, i think the "language" for build manifests should be
something between .ini and jar.mn. Because they're simple,
straightforward, and are well known. And simple.

I understand why people are eager to use python or lua, or other
scripting languages. But the fact is you really only need scripting in
very rare cases. The vast majority of our build is "take this file, put
it there", "take this file, preprocess it, put it there", "compile this"
or "link that". And occasionally, we need to add flags for specific files.
The build manifests should make defining those extremely simple.

There's also the case of conditionals. But i think the vast majority of
the conditionals we use are of the type "do this only on windows", or
"do this only on arm"

The few remaining things can have python scripts, but let's worry about
them later.

Mike

[Jeff Hammel]

On 08/22/2012 11:19 PM, Mike Hommey wrote:
> On Wed, Aug 22, 2012 at 01:38:22PM -0700, Gregory Szorc wrote:
>> On 8/22/12 1:00 PM, Ben Hearsum wrote:

>> There are a lot of bullets in "batteries included" languages. The
>> Python foot gun worries me greatly. We even have this problem in
>> make, and that is a much simpler language!
> As I said on irc, i think the "language" for build manifests should be
> something between .ini and jar.mn. Because they're simple,
> straightforward, and are well known. And simple.
>
> I understand why people are eager to use python or lua, or other
> scripting languages. But the fact is you really only need scripting in
> very rare cases. The vast majority of our build is "take this file, put
> it there", "take this file, preprocess it, put it there", "compile this"
> or "link that". And occasionally, we need to add flags for specific files.
> The build manifests should make defining those extremely simple.
>
> There's also the case of conditionals. But i think the vast majority of
> the conditionals we use are of the type "do this only on windows", or
> "do this only on arm"
>
> The few remaining things can have python scripts, but let's worry about
> them later.
>
> Mike

Maybe the thing to do is to stub this out for a few existing cases in
various formats and see how they look. It seems like the consensus is
that the format should be
- as simple as possible
- self-evident: easy to write, easy to edit, preferably something
Mozillians are familiar with
- able to do the simple things simply
- able to handle complex specialization for the cases that require it

But its hard (for me, anyway) to visualize what this would look like
without seeing more concrete proposals.

If .ini suffices, I am all for it.

Jeff

P.S. In the past, I was quite fond of buildit:
http://agendaless.com/Members/chrism/software/buildit . AFAIK, it is not
actively maintained, but it could be a source of inspiration.

[Dirkjan Ochtman]

On Wed, Aug 22, 2012 at 1:36 AM, Gregory Szorc <g...@mozilla.com> wrote:
> Up until now, the focus has been on making Makefile.in's themselves generic

> and data-driven [1]. We would use pymake's API to parse, load, and extract

> data from Makefile.in's to construct the build definition. In the long run,
> we'd realize that using make files for data definition was silly (and a
> large foot gun) and thus we would switch to something else.

Can you expand on that? From the discussion so far, JSON is not
expressive enough and Python is too expressive. There are some very
understandable reservations about inventing a new language, as well as
the desire for a "clean slate" (admittedly attractive). Why doesn't it
make sense to use a very-restricted dialect of Makefiles? pymake
already has a parser, the dumbing down of which to disallow arbitrary
shell expressions would supposedly be fairly straightforward. No one
would have to learn a new language, and you can start with the current
files, duplicate the parser inside pymake, then start to dumb it down
as you weed the complexity out of the Makefile.ins.

Cheers,

Dirkjan

[Ben Hearsum]

On 08/23/12 12:05 AM, Robert O'Callahan wrote:
> On Thu, Aug 23, 2012 at 8:40 AM, Ben Hearsum <bhea...@mozilla.com> wrote:
>
>> On 08/22/12 04:38 PM, Gregory Szorc wrote:
>>> Let's think of what can be done to secure/limit Python. Disabling import
>>> has already been mentioned. That's a start.
>>
>> I think it's worth noting that even if you *do* limit what you can do
>> through some technical means, you still have the option to change that
>> later, disable it some places, etc. It's really easy to get into that
>> game when you're fixing blockers or working on chemspills, too.
>>
>
> If someone is that desperate, what would you have them do instead of hack
> the configuration file? Aren't they likely to respond by doing some even
> worse hack that gets the job done?

It's hard to answer this in the abstract. In my experience, sometimes it
takes only a little more effort to do things the "right" way (that is,
the way you would do them if you weren't under time pressure). These
cases are where limiting peoples' ability to things in a bad or
difficult to maintain way really pays off.

For cases where the "right" fix can only happen through rearchitecting
or an otherwise large change I'm not sure how it would play out.

> I think it makes a ton of sense to use automation to stop developers
> accidentally doing something they shouldn't. But if someone's desperate
> enough to disable the automation, and can get a review for it, then I don't
> think it makes sense to try to stop them.

Yeah, I think I agree. My experience in RelEng has biased me strongly
towards not allowing even temporary hacks like that, because it's rare
that we ever remove them, and end up with a pile of very difficult to
maintain hacks (Tinderbox client, buildbot-configs). This may not be the
case for the build system in general, though.

[Ben Hearsum]

On 08/23/12 12:05 AM, Robert O'Callahan wrote:

> On Thu, Aug 23, 2012 at 8:40 AM, Ben Hearsum <bhea...@mozilla.com> wrote:
>
>> On 08/22/12 04:38 PM, Gregory Szorc wrote:
>>> Let's think of what can be done to secure/limit Python. Disabling import
>>> has already been mentioned. That's a start.
>>
>> I think it's worth noting that even if you *do* limit what you can do
>> through some technical means, you still have the option to change that
>> later, disable it some places, etc. It's really easy to get into that
>> game when you're fixing blockers or working on chemspills, too.
>>
>
> If someone is that desperate, what would you have them do instead of hack
> the configuration file? Aren't they likely to respond by doing some even
> worse hack that gets the job done?

It's hard to answer this in the abstract. In my experience, sometimes it
takes only a little more effort to do things the "right" way (that is,
the way you would do them if you weren't under time pressure). These
cases are where limiting peoples' ability to things in a bad or
difficult to maintain way really pays off.

For cases where the "right" fix can only happen through rearchitecting
or an otherwise large change I'm not sure how it would play out.

> I think it makes a ton of sense to use automation to stop developers
> accidentally doing something they shouldn't. But if someone's desperate
> enough to disable the automation, and can get a review for it, then I don't
> think it makes sense to try to stop them.

[Mike Hommey]

On Thu, Aug 23, 2012 at 08:43:31AM -0400, Ben Hearsum wrote:
> Yeah, I think I agree. My experience in RelEng has biased me strongly
> towards not allowing even temporary hacks like that, because it's rare
> that we ever remove them, and end up with a pile of very difficult to
> maintain hacks (Tinderbox client, buildbot-configs). This may not be the
> case for the build system in general, though.

The current build system *is* a pile of 10+ years old hacks. That alone
tells why we'd like to change it.

Mike

[David Rajchenbach-Teller]

While I do not advocate the use of ocamlbuild, we could draw some
inspiration from the design.

Essentially, an ocamlbuild-based build system is based around a library
of rules (implemented in OCaml, we could do the same in Python, of
course), and any number of trivial description files. OCamlBuild itself
contains a solver that finds one possible solution to build the target
by applying the rules to the description files and source files.

For instance, to extract documentation from some source code:
- create a file foo.odocl, containing the list of all files that should
be passed to the documentation generator;
- ocamlbuild's standard rule library knows how to produce %.html from
%.odocl;
- it is pretty easy to add a rule to produce %.pdf from %.odocl.

Generally, this design is rather good at separating tasks that need a
full language (implemented as rules) from tasks that only need a trivial
description (implemented as description files).

Cheers,
David

On 8/23/12 8:19 AM, Mike Hommey wrote:
> As I said on irc, i think the "language" for build manifests should be
> something between .ini and jar.mn. Because they're simple,
> straightforward, and are well known. And simple.
>
> I understand why people are eager to use python or lua, or other
> scripting languages. But the fact is you really only need scripting in
> very rare cases. The vast majority of our build is "take this file, put
> it there", "take this file, preprocess it, put it there", "compile this"
> or "link that". And occasionally, we need to add flags for specific files.
> The build manifests should make defining those extremely simple.
>
> There's also the case of conditionals. But i think the vast majority of
> the conditionals we use are of the type "do this only on windows", or
> "do this only on arm"
>
> The few remaining things can have python scripts, but let's worry about
> them later.
>
> Mike

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>


--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla

[Ted Mielczarek]

On Thu, Aug 23, 2012 at 9:39 AM, David Rajchenbach-Teller
<dte...@mozilla.com> wrote:
> Generally, this design is rather good at separating tasks that need a
> full language (implemented as rules) from tasks that only need a trivial
> description (implemented as description files).

This discussion is focused solely on the "description" portion. The
rules portion will be the "backend" here, which will be GNU make for
the time being, and possibly something else in the future.

The reason this is complicated is that a "trivial description" turns
out to not be trivial within our build system. Most of what we do is
just assigning lists of values, certainly, but there's a lot of
platform conditionals. I suspect we could get by with some sort of
language that allowed assignment, conditionals, and append/update for
data structures.

I think the appeal of using Python or Lua is that it's something that
exists, and people are familiar with the syntax. I don't want people
writing full-fledged programs in replacement of Makefiles, but writing
something with real data structures and a sane syntax would be nice.

-Ted

[Ehsan Akhgari]

Unfortunately inventing something new is only appealing on paper. Let
me give you a concrete example of what is wrong with one of the new
formats we invented (xpcshell tests manifest). [1] I recently had to
disable a test which was failing very often in order to make the tree
green. So, thinking that the manifest files are regular ini files, I
just commented it out [2] and... boom! The build system blew up when
trying to use the manifest file to figure out what tests are run. Then,
puzzled by why comments do not do what they would normally do in ini
files, I removed the commented out line [3] (surely if the test name is
not mentioned, it won't run, right?). Wrong. We check to make sure
that all xpcshell js files *are* mentioned in the manifest file.
Frustrated to find this out on the Tinderbox machines, I cried for help
on IRC. And then people pointed me to this blatantly unfamiliar syntax
which needs to be used if you want to disable a test. [4]

I think Gregory's suggestion of using the python's ast module to limit
the sorts of syntax constructs that we accept in the build manifest
files is a great one. That would let us make python powerful enough for
our needs, but not more powerful. And we can easily advertize the build
scripts as "python files that only allow variable assignments and
conditionals, etc." The advantage of not inventing a new language is
too high for us to give up, IMO.

Cheers,
Ehsan


[1] Don't be tricked by its looks. xpcshell.ini's are _not_ normal ini
files. They have semantics built on top of ini, because ini was not
powerful enough for our needs there.
[2] http://hg.mozilla.org/mozilla-central/rev/1ff862485759
[3] http://hg.mozilla.org/mozilla-central/rev/39b384528de3
[4] http://hg.mozilla.org/mozilla-central/rev/d88590f8a245

[Hanno Schlichting]

On 23.08.2012, at 17:07 , Ehsan Akhgari <ehsan....@gmail.com> wrote:
> I think Gregory's suggestion of using the python's ast module to limit the sorts of syntax constructs that we accept in the build manifest files is a great one. That would let us make python powerful enough for our needs, but not more powerful. And we can easily advertize the build scripts as "python files that only allow variable assignments and conditionals, etc." The advantage of not inventing a new language is too high for us to give up, IMO.

Instead of using Python's ast module, you can also do a simple trick with the exec statement and limit the global scope and only allow certain whitelisted names.

An example implementation is at https://gist.github.com/3437909. Download it as restricted.py, put it into a directory next to an empty Python file called evil.py and run "python restricted.py evil.py

If your evil.py contains:

CONST = True

l = [1, 2, 3]
if 1 in l:
l.append(4)

You get the pretty printed output:

Globals
{'__builtins__': {'False': False, 'None': None, 'True': True},
'uname': ('Darwin',
'hanno-air.local',
'12.0.0',
'Darwin Kernel Version 12.0.0: Sun Jun 24 23:00:16 PDT 2012; root:xnu-2050.7.9~1/RELEASE_X86_64',
'x86_64',
'i386')}
Locals
{'CONST': True, 'l': [1, 2, 3, 4]}

I exposed platform.uname as uname to showcase how you could expose information about the current system. In the same way putting os.environ.copy() in there works as well.

By default there's no import statement and no way to open files, or even just any of the exception names. You can define functions, but not classes. One could add a special import statement or some other function like "include" to allow re-use across files.

You can achieve the same with the ast module, but I find that more difficult to read and understand than this approach.

Hanno

[Steve Fink]

On Thu 23 Aug 2012 09:11:46 AM PDT, Hanno Schlichting wrote:
> By default there's no import statement and no way to open files, or even just any of the exception names. You can define functions, but not classes. One could add a special import statement or some other function like "include" to allow re-use across files.
>
> You can achieve the same with the ast module, but I find that more difficult to read and understand than this approach.

That's really slick and simple.

+1

[Neil]

Gregory Szorc wrote:

> We want feedback on the file format to use for the new build manifest
> files.

Is there a good reason to shoehorn everything into a single file format?

--
Warning: May contain traces of nuts.

[Ted Mielczarek]

On Fri, Aug 24, 2012 at 9:17 AM, qheaden <qhe...@phaseshiftsoftware.com> wrote:
> Is there any special reason why an existing build system such as SCcons couldn't be used as a new build system for Mozilla? I know the Mozilla source has a lot of special build instructions, but SCons does allow you to create your own special builders in Python code.

Build systems like SCons are just a different coat of paint over make.
They wouldn't really solve any of our problems, it'd just be
busy-work. In addition, SCons (among other build systems) tries to
solve more problems than we need, by providing the features of
autoconf as well as make. Finally, for SCons in particular, I have
doubts about its ability to scale to a project of Mozilla's size. KDE
tried to switch to SCons and failed, and wound up using CMake.

In short, most build systems suck at large scale. Almost any will
suffice for a small project, but for a project of Mozilla's size
there's no perfect solution.

-Ted

[qheaden]

Yes, I guess you are right about that. A custom, simplified build system would be the best choice in this situation.

[qheaden]

On Friday, August 24, 2012 10:32:46 AM UTC-4, Ted Mielczarek wrote:

[Brian Smith]

Gregory Szorc wrote:
> 4. Native support for list and maps. Make files only support strings.
> The hacks this results in are barely tolerable.
>
> 5. Ability to handle conditionals. We need to be able to
> conditionally define things based on the presence or value of certain
> "variables."
> e.g. "if the current OS is Linux, append this value to this list." I
> quote "variables" because there may not be a full-blown variable
> system here, just magic values that come from elsewhere and are
> addressed by some convention.
>
> 6. Ability to perform ancillary functionality, such as basic string
> transforms. I'm not sure exactly what would be needed here. Looking
> at make's built-in functions might be a good place to start. We may
> be able to work around this by shifting functionality to side-effects
> from specially named variables, function calls, etc. I really don't
> know.
>
> 7. Evaluation must be free from unknown side-effects. If there are
> unknown side-effects from evaluation, this could introduce race
> conditions, order dependency, etc. We don't want that. Evaluation
> must either be sandboxed to ensure nothing can happen or must be able
> to be statically analyzed by computers to ensure it doesn't do anything
> it isn't supposed to.

...

> On the other end of the spectrum, we could have the build manifest
> files be Python "scripts." This solves a lot of problems around
> needing functionality in the manifest files. But, it would be a
> potential foot gun. See requirement #7.

I do not think it is reasonable to require support for alternate build systems for all of Gecko/Firefox.

But, let's say were were to divide the build into three phases:
1. Generate any generated C/C++ source files.
2. Build all the C/C++ code into libraries and executables
3. Do everything else (build omnijar, etc.)

(I imagine phase 3 could probably run 100% concurrently with the first two phases).

It would be very nice if phase #2 ONLY could support msbuild (building with Visual Studio project files, basically), because this would allow smart editors'/IDEs' code completion and code navigation features to work very well, at least for the C/C++ source code. I think this would also greatly simplify the deployment of any static analysis tools that we would develop.

In addition, potentially it would allow Visual Studio's "Edit and Continue" feature to work. ("Edit and Continue" is a feature that allows you to make changes to the C++ source code and relink those changes into a running executable while execution is paused at a breakpoint, without restarting the executable.)

I think that if you look at the limitations of gyp, some (all?) of them are at least partially driven by the desire to provide such support. I am sure the advanced features that you list in (4), (5), (6), (7) are helpful, but they may make it difficult to support these secondary use cases.

That said, getting the build system to build as fast as it can is much more important.

Cheers,
Brian

[Gregory Szorc]

On 8/24/12 12:42 PM, Brian Smith wrote:
> I do not think it is reasonable to require support for alternate build systems for all of Gecko/Firefox.
>
> But, let's say were were to divide the build into three phases:
> 1. Generate any generated C/C++ source files.
> 2. Build all the C/C++ code into libraries and executables
> 3. Do everything else (build omnijar, etc.)
>
> (I imagine phase 3 could probably run 100% concurrently with the first two phases).
>
> It would be very nice if phase #2 ONLY could support msbuild (building with Visual Studio project files, basically), because this would allow smart editors'/IDEs' code completion and code navigation features to work very well, at least for the C/C++ source code. I think this would also greatly simplify the deployment of any static analysis tools that we would develop.
>
> In addition, potentially it would allow Visual Studio's "Edit and Continue" feature to work. ("Edit and Continue" is a feature that allows you to make changes to the C++ source code and relink those changes into a running executable while execution is paused at a breakpoint, without restarting the executable.)
>
> I think that if you look at the limitations of gyp, some (all?) of them are at least partially driven by the desire to provide such support. I am sure the advanced features that you list in (4), (5), (6), (7) are helpful, but they may make it difficult to support these secondary use cases.
>
> That said, getting the build system to build as fast as it can is much more important.

Agreed.

Changing how we define the build config would enable us to do everything
you mentioned and more. With the current "architecture" of our
Makefile's, we effectively have different build phases called tiers. See
[1] for more. As much as I would love to split things up into more
distinct phases/tiers, the overhead for recursive make traversal would
be prohibitive. As far as prioritizing work to enable basic Visual
Studio project generation, I'm all for that: my build-splendid branch
[2] had its roots in VS generation after all (if you go back far enough
in the history you can still see this)!

Once we treat the build system as a giant data structure, we are free to
transform that any way we want. We feed that data structure into a
generator and spit something out the other side. This is very similar to
GYP's model. Essentially what we are proposing is reinventing GYP, but
with a different frontend. It's entirely possible we will implement
things using GYP's APIs so we can reuse GYP's existing generators! Time
will tell.

[1] http://gregoryszorc.com/blog/2012/07/29/mozilla-build-system-overview/
[2] https://github.com/indygreg/mozilla-central/tree/build-splendid

[Gregory Szorc]

Yes, we could use a subset of make to define things. On the surface, the
functionality is just what you want: simple conditionals, built-in
functions, simple appends, well-understood (more or less). Using make
*is* very tempting. And, we could probably use pymake as-is: we could
validate the parser's "statement list" output for conformance with our
limited make dialect. We should seriously consider using make files,
albeit in a restricted form to make parsing easier.

That being said, there are a few cons:

* Everything is a string. There is no null, false, true, or arrays. "if"
is "ifneq (,$(foo))" or even "ifneq (,$(strip $(foo)))" in case some
extra whitespace snuck in there. Arrays are strings delimited by
whitespace. This results in lots of ugly and hard-to-read code. Maps
don't exist at all. So, you have to normalize everything down to
key-value pairs. This results in weird foo like the library/module or
EXPORTS boilerplate.
* Syntax for complex behavior is hard to read. You inevitably need to
call functions for more advanced behavior. This results in code like
https://hg.mozilla.org/mozilla-central/file/ad7963c93bd8/config/rules.mk#l1623.
That's just as bad as poorly-written Perl. And, unlike Perl, it *has* to
be that way. Fortunately, this complexity *should* be hidden to everyone
but build system people. But, we need to maintain it and that's no fun.
* Performance issues. = in make is deferred assignment. There's a lot of
overhead in resolving values (although bsmedberg proposed a solution for
pymake that may combat this).
* Might require stapling some new features onto pymake. Not a deal breaker.

Individually, they aren't too bad. A lot are superficial. But, when you
combine them, it gets ugly.

[Gregory Szorc]

On 8/24/12 8:05 AM, John Hopkins wrote:
> Suggestion: identify the ugliest sections of Makefile usage and use
> those as a benchmark for evaluating different solutions. ie. how could
> it be implemented in SCons, pymake, etc. or even, how could it be
> reimplemented in Make in a clean fashion.

I've started [1] to compare things. I just plucked a random Makefile.in
out of the source tree that seemed to have a nice mix of things. We may
want to throw some more complexity in there just for completeness.

If someone familiar with some of the empty sections has time to fill
those out, it would be appreciated (I'm too busy!).

[1] https://wiki.mozilla.org/User:Gszorc/Build_frontend_shootout

[Gregory Szorc]

On 8/23/2012 9:11 AM, Hanno Schlichting wrote:
> Instead of using Python's ast module, you can also do a simple trick
> with the exec statement and limit the global scope and only allow
> certain whitelisted names. An example implementation is at
> https://gist.github.com/3437909. Download it as restricted.py, put it
> into a directory next to an empty Python file called evil.py and run
> "python restricted.py evil.py

This is so awesome and simple I just had to try it out!

I've built what *could* become the underpinnings of our new build system
using this technique.

Results are checked in at [1]. Code for doing the file loading is at
[2]. It's not perfect, but the proof-of-concept is solid enough. It's
also fast. My machine is reading 159 files in ~30ms. That's with a
single thread too. Granted, it isn't doing anything too complicated yet.

A typical frontend file is at [3]. An uglier one is at [4]. I only
ported the DIRS and TIERS variables, so it isn't too indicative of the
final product.

Obviously there is no shortage of things to bike shed about. But, it's a
start.

Mike Hommey just updated the comparison wiki page [5] with an INI format
that looks pretty clean. I may give it the same porting treatment so
people can compare on a larger scale...

[1] https://github.com/indygreg/mozilla-central/tree/python-build-files
[2]
https://github.com/indygreg/mozilla-central/blob/python-build-files/python/mozbuild/mozbuild/frontend/reader.py
[3]
https://github.com/indygreg/mozilla-central/blob/python-build-files/browser/components/build.mozbuild
[4]
https://github.com/indygreg/mozilla-central/blob/python-build-files/build.mozbuild
[5] https://wiki.mozilla.org/User:Gszorc/Build_frontend_shootout

[Anthony Jones]

On 29/08/12 19:45, Gregory Szorc wrote:
> On 8/23/2012 9:11 AM, Hanno Schlichting wrote:
>> Instead of using Python's ast module, you can also do a simple trick
>> with the exec statement and limit the global scope and only allow
>> certain whitelisted names. An example implementation is at
>> https://gist.github.com/3437909. Download it as restricted.py, put it
>> into a directory next to an empty Python file called evil.py and run
>> "python restricted.py evil.py
>
> This is so awesome and simple I just had to try it out!

It makes the most sense to use existing language and one we are already
using. Think very carefully before inventing a new language or syntax,
then think again. You throw everything away when you create a new language.

We can begin by simply using convention to limit the use of language
features. Until we have a working build system the enforcement is just
feature creep.

Why would we not use Python?

Anthony

[Robert Kaiser]

Gregory Szorc schrieb:
> * "if"

> is "ifneq (,$(foo))" or even "ifneq (,$(strip $(foo)))" in case some
> extra whitespace snuck in there.

One thing that was hoped we could achieve with pymake was to possibly
switch to it everywhere at some time and then be able to replace those
ugly conditionals with just python ones.
We definitely should go for nicer conditionals with all this. :)

Robert Kaiser

[Karl Tomlinson]

> On 8/22/12 6:10 AM, Robert Kaiser wrote:

>> [...]
>> esp. given that basic file I/O is often costly (from watching my CPU
>> usage, a lot of the build time is spent in I/O wait when using spinning
>> disks - SSDs improve that hugely).

On Wed, 22 Aug 2012 12:29:24 -0700, Gregory Szorc wrote:

> Yes, I/O wait can significantly reduce build times. I recommended
> at [1] that aside from a modern CPU, investing in an SSD is the
> best thing you can do for build times. If a build machine doesn't
> have an SSD, it's effectively throwing away CPU cycles in I/O
> wait. It's true that the build system today wastes a lot of CPU
> cycles due to inefficient use of all available cores (lack of wide
> parallelism due to recursive make). But, the efficiency should
> rise drastically with our build system improvements. That will
> make the impact of an SSD even more pronounced. On my MBP (with
> 8GB), building with an SSD reduced overall build time by a few
> minutes, with libxul linking going from ~55s to ~5s. That's
> without any build system changes! There is no doubt in my mind
> that SSDs are worth the investment.
>
> [1] http://gregoryszorc.com/blog/2012/07/29/mozilla-central-build-times/

This sounded good to me, so before searching for an SSD, I did
some experimentation.

Although suggested in Gregory's blog, I want to say more clearly
that if you have a filesystem designed for performance and enough
page cache then an SSD will not make any significant difference to
warm build times. The filesystem will perform the I/O
asynchronously and so I/O is not the limiting factor in build
times.

On this test system with a 1.6GHZ 4-core i7 and 16GB RAM, a gcc
4.5.3 64-bit debug build from scratch takes 23-24 minutes, whether
on a tmpfs ramdisk or magnetic disk with ext4 data=ordered.
libxul link times are 14s on each disk.

I/O does have an effect on some filesystems. The home partition
here is mounted with data=journal so that sqlite does not need
fsync for consistency. data=journal also turns off delalloc which
on its own affects performance.

Switching from data=ordered to data=journal increased
binutils-2.22 gold libxul link times from 14s to 76s. The effect
was a little less dramatic with bfd links, which increased from
28s to 54s.

The system has the default journal size of only 128M, which I
imagine could be relevant for data=journal when writing 0.5GB of
libxul.so. Linux 3.3.8.

[Gregory Szorc]

For warm build times, this is expected. In the measurements I performed
in January, I concluded that you needed ~9GB of memory on Linux to avoid
page cache eviction and thus incur no device I/O during builds [1].

If you have 9+GB of memory dedicated to building mozilla-central and you
are doing warm builds all the time, great, you probably don't need an
SSD. But, those conditions are specialized. You have to get things into
the page cache. That's a few GB of source files, the object files,
ccache files, etc over thousands of files. That's a lot of I/O and a lot
of potential for I/O wait. You also have everything else on your
computer contending for that page cache's memory.

Essentially if you have a server that does nothing but build
mozilla-central all day, you are probably fine without an SSD. (Note
that Mozilla's build infra doesn't qualify because AFAIK the machines
are rebooted after every build, purging the page cache in the process.)
If you are building on your everyday computer which is also running a
window manager, Firefox, etc, you'll likely have more page cache
eviction and slower build times assuming I/O wait isn't short.

FWIW, my build-splendid branch [2] now incorporates the psutil Python
package to measure system resource usage during builds. Every 0.5s it
records the I/O, memory, and CPU counters. At the end of the build, it
correlates build events (configure, js, nspr, export, libs, app, etc)
with resource usage and prints a summary. It paints an extremely
detailed picture of how system resources are being used during builds.
This data has already taught me a lot about where the inefficiencies in
the existing build system are. I'm hoping to get it checked in so others
can also produce data. And, once enough data is being produced, we
should be able to start drawing statistically sound correlations between
things like build times and I/O wait.

[1]
https://groups.google.com/forum/#!topic/mozilla.dev.builds/FJclsTA_OBQ/discussion
[2] https://github.com/indygreg/mozilla-central/tree/build-splendid/
[3]
https://github.com/indygreg/mozilla-central/blob/build-splendid/python/mozbuild-bs/mozbuild/util.py#L82

[Aryeh Gregor]

On Thu, Aug 30, 2012 at 1:25 AM, Gregory Szorc <g...@mozilla.com> wrote:
> If you have 9+GB of memory dedicated to building mozilla-central and you are
> doing warm builds all the time, great, you probably don't need an SSD. But,
> those conditions are specialized. You have to get things into the page
> cache. That's a few GB of source files, the object files, ccache files, etc
> over thousands of files. That's a lot of I/O and a lot of potential for I/O
> wait. You also have everything else on your computer contending for that
> page cache's memory.
>
> Essentially if you have a server that does nothing but build mozilla-central
> all day, you are probably fine without an SSD. (Note that Mozilla's build
> infra doesn't qualify because AFAIK the machines are rebooted after every
> build, purging the page cache in the process.) If you are building on your
> everyday computer which is also running a window manager, Firefox, etc,
> you'll likely have more page cache eviction and slower build times assuming
> I/O wait isn't short.

In practice, I've noticed minimal difference using SSDs on my desktop,
which has 16G of RAM. I have lots of other stuff open, but all that
only uses a couple of gigs of RAM, so there should certainly be 9G for
page cache. (free -m reports about 7G used for buffers/cache.)

If you use a desktop and work on compiled code even part-time, at any
reasonable pay rate, I can't see any reason why you wouldn't get 16G
of RAM. Although it seems most people don't -- maybe people mostly
use laptops? 2x8G of desktop RAM is about $75 right now on Newegg,
and any decent motherboard should take that much these days.

[Ehsan Akhgari]

On 12-08-31 8:28 AM, Aryeh Gregor wrote:
> If you use a desktop and work on compiled code even part-time, at any
> reasonable pay rate, I can't see any reason why you wouldn't get 16G
> of RAM. Although it seems most people don't -- maybe people mostly
> use laptops? 2x8G of desktop RAM is about $75 right now on Newegg,
> and any decent motherboard should take that much these days.

A major reason why many people don't have that is that they use a
MacBook Pro, which until recently supported a maximum of 8GB of RAM.

[Boris Zbarsky]

On 8/31/12 8:28 AM, Aryeh Gregor wrote:
> In practice, I've noticed minimal difference using SSDs on my desktop,
> which has 16G of RAM. I have lots of other stuff open, but all that
> only uses a couple of gigs of RAM, so there should certainly be 9G for
> page cache. (free -m reports about 7G used for buffers/cache.)

That's been my experience on a Linux desktop as well. Minimal win from
an SSD compared to a 10k rpm spinny disk and lots of RAM.

On a Mac laptop, the situation is pretty different, for three reasons:

1) As Ehsan points out these were limited to 8GB until recently.
2) Laptop spinny disks are slower.
3) MacOS may not cache disk in RAM as much as Linux does; certainly
Windows didn't use to.

-Boris

[Gregory Szorc]

A decision has been made: we will be using Python files executing in a
sandboxed environment (using the technique that Hanno posted).
Supporting this decision are Ted (build system owner) and Benjamin
Smedberg and myself (build system peers).

The first step is moving all the directory traversal definitions from
existing make files into Python files. This is being tracked in bug
784841 [1]. If all goes according to plan, this should be a transparent
transition: this won't change how you build the tree.

Initially, this transition will seem like a lot of busy work with no
real benefit. The real wins come after we've moved more exciting pieces
such as C/C++ compilation and IDL generation to the new Python frontend
files. When those are in place, we should be able to do things such as
generate non-recursive make files, which should make builds faster.

More information about the new world order will be communicated once
things are closer to landing. If you wish to influence it, please follow
bug 784841.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=784841

On 8/21/2012 4:36 PM, Gregory Szorc wrote:
> tl;dr We're proposing moving away from Makefile's as the sole source
> of the build system definition. This will lead to faster build times.
> Bikeshedding^wFeedback on the file format is requested.
>
> The existing build system is defined by Makefile.in's scattered around
> the source tree (typically one Makefile.in per directory). At
> configure time, these Makefile.in's get preprocessed into Makefile's
> using simple variable substitution. Then make/pymake is let loose on
> the result. It is a very traditional model.
>
> We are attempting to move to a model where the build definition is
> generic and data-driven. By treating the build definition as data
> (rather than a glorified shell script that is Makefiles), this will
> allow us to take that data and convert it into formats understood by
> other, better/faster build backends, such as non-recursive make files,
> Tup, Ninja, or even Visual Studio.

>
> Up until now, the focus has been on making Makefile.in's themselves
> generic and data-driven [1]. We would use pymake's API to parse, load,
> and extract data from Makefile.in's to construct the build definition.
> In the long run, we'd realize that using make files for data
> definition was silly (and a large foot gun) and thus we would switch
> to something else.
>

> After a long IRC conversation, Mike Hommey and I concluded that we
> want to begin the transition away from Makefile.in's ASAP.
>
> Essentially, the proposal is to move (not duplicate) some data from
> Makefile.in's into new files. Initially, this would include things
> like subdirectories to descend into and files to copy/preprocess.
> Simple stuff to start with. Eventually, scope would likely increase to
> cover the entirety of the build system definition (like compiling),
> rendering Makefile.in's obsolete. But, it will take a *long* time
> before we get there.
>
> In the new world, the source of truth for the build system is jointly
> defined by existing Makefile.in's and whatever these new files are
> that we create. I'll call these not-yet-existing files "build manifest
> files." Somewhere in the build process we read in the build manifest
> files and generate output for the build backend of choice.
>
> Our existing non-recursive make backend should integrate with this
> seamlessly. Instead of a dumb variable substitution phase for
> configuring the build backend, we'll have some additional logic to
> write out new make files derived from the contents of the build
> manifest files. This is similar to the approach I've taken in build
> splendid [2]. The only difference is the build definition is living in
> somewhere not Makefile.in's.
>
> We don't have details on how exactly the migration will be carried
> about. But, it should be seamless. So, unless you touch the build
> system, you should be able to continue living in blissful ignorance.
>
> If you have any concerns over this transition, please voice them.
>
> File Format
> ===========
>
> I hinted at bikeshedding in the tl;dr. We want feedback on the file
> format to use for the new build manifest files. The requirements are
> as follows (feel free to push back on these):
>
> 1. Easy for humans to grok and edit. An existing and well-known format
> is preferred. We don't want a steep learning curve here.
> 2. Simple for computers to parse. We will use Python to load the build
> manifest files. Python can do just about anything, so I'm not too
> worried here.
> 3. Efficient for computers to load. As these files need to be
> consulted to perform builds, we want to minimize the overhead for
> reading them into (Python) data structures.

> 4. Native support for list and maps. Make files only support strings.
> The hacks this results in are barely tolerable.
> 5. Ability to handle conditionals. We need to be able to conditionally
> define things based on the presence or value of certain "variables."
> e.g. "if the current OS is Linux, append this value to this list." I
> quote "variables" because there may not be a full-blown variable
> system here, just magic values that come from elsewhere and are
> addressed by some convention.
> 6. Ability to perform ancillary functionality, such as basic string
> transforms. I'm not sure exactly what would be needed here. Looking at
> make's built-in functions might be a good place to start. We may be
> able to work around this by shifting functionality to side-effects
> from specially named variables, function calls, etc. I really don't know.
> 7. Evaluation must be free from unknown side-effects. If there are
> unknown side-effects from evaluation, this could introduce race
> conditions, order dependency, etc. We don't want that. Evaluation must
> either be sandboxed to ensure nothing can happen or must be able to be
> statically analyzed by computers to ensure it doesn't do anything it
> isn't supposed to.

> 8. Doesn't introduce crazy build dependencies. We /might/ be able to
> get away with something well-known. But, new build dependencies are
> new build dependencies.
>
> Ideally, the data format is static and doesn't require an interpreter
> (something like YAML or JSON). Unfortunately, the need for
> conditionals makes that, well, impossible (I think).
>
> We could go the route of GYP and shoehorn conditionals into a static
> document (JSON) [3]. Actually, using GYP itself is an option!
> Although, some really don't like the data format because of this
> shoehorning (I tend to agree).

>
> On the other end of the spectrum, we could have the build manifest
> files be Python "scripts." This solves a lot of problems around
> needing functionality in the manifest files. But, it would be a
> potential foot gun. See requirement #7.
>

> Or, there is something in the middle. Does anyone know of anything
> that can satisfy these requirements? I think Lua is perfect for this
> (it was invented to be a configuration language after all). But, I'm
> not sure it satisfies #1 nor #8.
>
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=774049
> [2]
> http://gregoryszorc.com/blog/2012/08/15/build-firefox-faster-with-build-splendid/
> [3] https://code.google.com/p/gyp/wiki/GypLanguageSpecification
> _______________________________________________
> dev-builds mailing list
> dev-b...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-builds

[Gregory Szorc]

On 8/31/2012 5:28 AM, Aryeh Gregor wrote:
> On Thu, Aug 30, 2012 at 1:25 AM, Gregory Szorc <g...@mozilla.com> wrote:
>> If you have 9+GB of memory dedicated to building mozilla-central and you are
>> doing warm builds all the time, great, you probably don't need an SSD. But,
>> those conditions are specialized. You have to get things into the page
>> cache. That's a few GB of source files, the object files, ccache files, etc
>> over thousands of files. That's a lot of I/O and a lot of potential for I/O
>> wait. You also have everything else on your computer contending for that
>> page cache's memory.
>>
>> Essentially if you have a server that does nothing but build mozilla-central
>> all day, you are probably fine without an SSD. (Note that Mozilla's build
>> infra doesn't qualify because AFAIK the machines are rebooted after every
>> build, purging the page cache in the process.) If you are building on your
>> everyday computer which is also running a window manager, Firefox, etc,
>> you'll likely have more page cache eviction and slower build times assuming
>> I/O wait isn't short.

> In practice, I've noticed minimal difference using SSDs on my desktop,
> which has 16G of RAM. I have lots of other stuff open, but all that
> only uses a couple of gigs of RAM, so there should certainly be 9G for
> page cache. (free -m reports about 7G used for buffers/cache.)
>

> If you use a desktop and work on compiled code even part-time, at any
> reasonable pay rate, I can't see any reason why you wouldn't get 16G
> of RAM. Although it seems most people don't -- maybe people mostly
> use laptops? 2x8G of desktop RAM is about $75 right now on Newegg,
> and any decent motherboard should take that much these days.

I re-ran some builds on my desktop with my instrumented build system
that records system metrics and an SSD clobber build was only ~50s
faster than the same build on a 7200RPM drive. This comes out to ~5%,
which isn't much. Both results were on a freshly booted virtual machine
(read: empty page cache) that had access to 12GB of memory. This was
also with ccache disabled (all my previous measurements on list posts
and my personal blog were with ccache enabled, which definitely has an
impact on I/O).

I would say this result validates your statement of "minimal difference
using SSDs." Honestly, this surprised me, as it is counter to what I've
measured before, especially on my MBP, where the difference between SSD
and non-SSD is a few minutes.

As I said, these measurement were without ccache. What impact does
ccache have? Well, on that same 7200RPM drive, enabling ccache and
building with a purged ccache (the worst case) increased build times
from 1077s to 1245s. That's 15.6% slower! Granted, this is the worst
case for ccache (empty at start of build). And, I wasn't using any
CCACHE_HARDLINK or CCACHE_COMPRESS magic, so every object added to
ccache effectively incurred 2x write I/O. If we look at the raw numbers,
I see that total device write bytes with ccache nearly doubled to
~6.5GB! ccache also increased I/O wait times by ~2x. Given how I/O wait
is measured on Linux, it's very difficult to convert I/O wait to wall
clock time. This is why I refrain from specifying raw numbers. But,
there does seem to be a correlation. I stopped short of measuring a
ccache build on my SSD, which should isolate I/O wait and prove or
disprove I/O wait as the main contributor to the longer build times. So,
the "ccache builds considered harmful?" thread will have to wait. If I
had infinite time...

[qheaden]

On Sunday, September 2, 2012 5:15:40 PM UTC-4, Gregory Szorc wrote:
> A decision has been made: we will be using Python files executing in a
>
> sandboxed environment (using the technique that Hanno posted).
>
> Supporting this decision are Ted (build system owner) and Benjamin
>
> Smedberg and myself (build system peers).

This question might be impossible to answer right now, but about how long do you think it will take to convert the entire build system to the new format?

[qheaden]

On Sunday, September 2, 2012 5:15:40 PM UTC-4, Gregory Szorc wrote:

> A decision has been made: we will be using Python files executing in a
>
> sandboxed environment (using the technique that Hanno posted).
>
> Supporting this decision are Ted (build system owner) and Benjamin
>
> Smedberg and myself (build system peers).

[Aryeh Gregor]

On Tue, Sep 4, 2012 at 3:19 AM, Gregory Szorc <g...@mozilla.com> wrote:
> I re-ran some builds on my desktop with my instrumented build system that
> records system metrics and an SSD clobber build was only ~50s faster than
> the same build on a 7200RPM drive. This comes out to ~5%, which isn't much.
> Both results were on a freshly booted virtual machine (read: empty page
> cache) that had access to 12GB of memory. This was also with ccache disabled
> (all my previous measurements on list posts and my personal blog were with
> ccache enabled, which definitely has an impact on I/O).

Having no page cache and no ccache is great for getting stable
numbers, but not very realistic. In practice, most compiles will have
the page cache populated, so it probably won't even be 5%.

(On the other hand, in principle ccache should increase the amount of
on-disk data being accessed, and therefore *increase* the value of an
SSD. But I haven't noticed this in practice.)

[Mike Hommey]

On the other hand, the ccache miss rate is pretty high, at least it is
for me, so in the end, ccache might not be a win at all.

Mike

[Nicholas Nethercote]

On Tue, Sep 4, 2012 at 9:39 PM, Mike Hommey <m...@glandium.org> wrote:
>
> On the other hand, the ccache miss rate is pretty high, at least it is
> for me, so in the end, ccache might not be a win at all.

That's what I've found, for my compiling patterns.

Nick

[Gregory Szorc]

I finally got my resource measuring patches to work on Try! The impact
of ccache on the OS X builders isn't pretty. An extra ~7.5GB write I/O
and an additional ~4 minutes to build times.

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=789317 (with the
raw numbers) to propose disabling ccache on the builders for a few days
to see what kind of results we see on a larger scale.