Persistently storing build system state

[Gregory Szorc]

As I wrote at [1] there are some scenarios where the build system and
related tools would like to store persistent state. Some uses for this
include:

* Automatically recording previous build logs, compiler warnings, and
test results.
* Holding build environments (downloaded Mercurial extensions,
downloaded or built toolchains, etc).
* Global mach/mozconfig config files
* Caching of state such as last bootstrap time, last Mercurial extension
update check, etc.

Some of these we can't do today because state in the object directory
frequently gets lost (from clobbers) or is not shared among source
directories. As a result, I'd argue that build system UX and
productivity suffers because we can't tap this potential. I'd like to
tap this potential.

Since we can't store all state in the object directory or in the source
directory, I'm proposing that at some time in the future the build
system will automatically use ~/.mozbuild for persistent state storage.
The location will be configurable via an environment variable or
something, of course. And, when the directory is initially created, we
can add a prompt or a long delay with a prominent notification message
or something. Of course, things would be implemented such that multiple
source directories and multiple object directories continue to work.

Initially, I intend to utilize the state directory for holding a global
mach config file and bootstrap state (so mach and/or the build system
can prompt/notify you to rebootstrap every N days or something).

Are there any objections or concerns?

[1] http://gregoryszorc.com/blog/2013/05/13/mozilla-build-system-brain-dump/

[Nick Alexander]

I also think we should tap this potential. But:

Some of the things you mention are truly global: mach configuration, hg
extensions. But some things are more source-tree specific: build logs,
compiler warnings, test results. I would not want to share too much
state between my mozilla-inbound, mozilla-central, and services-central
trees.

I think making this distinction explicit will help: ~/.mozbuild and
$SRCDIR/.mozbuild.

Best,
Nick

[Nick Alexander]

On 13-05-16 12:53 PM, Gregory Szorc wrote:
> As I wrote at [1] there are some scenarios where the build system and
> related tools would like to store persistent state. Some uses for this
> include:
>
> * Automatically recording previous build logs, compiler warnings, and
> test results.
> * Holding build environments (downloaded Mercurial extensions,
> downloaded or built toolchains, etc).
> * Global mach/mozconfig config files
> * Caching of state such as last bootstrap time, last Mercurial extension
> update check, etc.

Let me suggest another: keeping track of successfully built object
directories, so that remote testing commands can guess MOZ_HOST_BIN
candidates.

Nick

[Gregory Szorc]

No issues were raised publicly or privately so I have landed this in bug
873798.

The next time you run mach you will receive a one-time notice of the
creation of this directory. Expect features built on top of this to land
soon! If you don't run mach, there is no impact (i.e. nothing for RelEng
to worry about).

On 5/16/13 12:53 PM, Gregory Szorc wrote:
> As I wrote at [1] there are some scenarios where the build system and
> related tools would like to store persistent state. Some uses for this
> include:
>
> * Automatically recording previous build logs, compiler warnings, and
> test results.
> * Holding build environments (downloaded Mercurial extensions,
> downloaded or built toolchains, etc).
> * Global mach/mozconfig config files
> * Caching of state such as last bootstrap time, last Mercurial extension
> update check, etc.
>

> Some of these we can't do today because state in the object directory
> frequently gets lost (from clobbers) or is not shared among source
> directories. As a result, I'd argue that build system UX and
> productivity suffers because we can't tap this potential. I'd like to
> tap this potential.
>
> Since we can't store all state in the object directory or in the source
> directory, I'm proposing that at some time in the future the build
> system will automatically use ~/.mozbuild for persistent state storage.
> The location will be configurable via an environment variable or
> something, of course. And, when the directory is initially created, we
> can add a prompt or a long delay with a prominent notification message
> or something. Of course, things would be implemented such that multiple
> source directories and multiple object directories continue to work.
>
> Initially, I intend to utilize the state directory for holding a global
> mach config file and bootstrap state (so mach and/or the build system
> can prompt/notify you to rebootstrap every N days or something).
>
> Are there any objections or concerns?
>

> [1]
> http://gregoryszorc.com/blog/2013/05/13/mozilla-build-system-brain-dump/
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform