Overhauling MozillaBuild

[Ryan VanderMeulen]

As some of you on this list are already aware of, I have recently been toying with the idea of overhauling MozillaBuild. I'm sending this message out to flesh out the plan more and collect some early feedback before moving forward formally with this.

As I see it, the current MozillaBuild has a few problems:
* High barrier of entry for new contributors due to requiring a specialized build setup (or a custom VM image) to package
* Difficulty updating core components due to being pinned to an extremely old revision of MSYS (version 1.0.11)
- This leaves us unable to fix bugs like bug 703986 without jumping through hoops to do so
* Inability to ship small incremental updates (even an update to just Hg requires a full 80+MB release)
* Difficulty including git due to side-by-side MSYS versions

I think that the above issues are largely fixable, but difficult without a more extensive rebuild of MozillaBuild. Therefore, I would like to take this on as a Q2 project. The rewrite would be based around MSYS2 (http://sourceforge.net/projects/msys2/), a more recent cygwin fork.

MSYS2 offers many nice improvements to the current MSYS:
* A built-in package manager (ported from Arch Linux - https://wiki.archlinux.org/index.php/pacman), enabling smaller updates of core components.
* Native builds of many core components included with MozillaBuild (or on the wish list), like Hg, git, python, etc. No more side-by-side installations, py2win, or other ugliness.
* Includes the mintty terminal, a very capable improvement over the native Windows command line.

In the end, I think the rebuilt MozillaBuild will end up being simpler than the current setup. Easier to package, easier to update, and easier to customize.

Let me know what you think :)

-Ryan

[Gregory Szorc]

This all sounds fantastic. Thanks for taking this up!

I feel improving MozillaBuild is long overdue. I feel many developers
(myself included) avoid developing Firefox on Windows because the
development experience is horrible. The MozillaBuild environment is
partly to blame (build speed being the other, but mozmake / GNU make 4
has changed the game). We really should have more developers developing
Firefox on Windows because that's where our users are. I feel an
overhauled MozillaBuild would be a terrific carrot to lure people over.

It isn't a shipping requirement, but I'd like to eventually roll Windows
support into the in-tree bootstrapper.
https://github.com/indygreg/mozilla-central/compare/master...windows-bootstrap
contains some experimental work using a VBScript to bootstrap a Python
environment from which we can do real bootstrap work. We should be able
to easily plug msys2 package management into the mix.

The only major question I have is over the impact to release automation.
I'm not sure to what degree they rely on MozillaBuild, if any. We want
the build environments to remain as similar as possible to prevent
surprises. It would be great if you could get a commitment from
automation to transition to msys2 at or around the same time as
MozillaBuild. IIRC they are talking about moving their Windows infra out
of Mozilla data centers and into the cloud. So now would seem like the
best time to approach them with this looming requirement.

I can't wait for this to happen.

[Ben Hearsum]

On 03/11/14 01:51 PM, Gregory Szorc wrote:
> The only major question I have is over the impact to release automation.
> I'm not sure to what degree they rely on MozillaBuild, if any. We want
> the build environments to remain as similar as possible to prevent
> surprises. It would be great if you could get a commitment from
> automation to transition to msys2 at or around the same time as
> MozillaBuild. IIRC they are talking about moving their Windows infra out
> of Mozilla data centers and into the cloud. So now would seem like the
> best time to approach them with this looming requirement.

Our automation uses a heavily patched older version of MozillaBuild
(based on 1.2 or 1.3 IIRC). With recent changes to how we deploy things
on Windows, it might be easier.

One complication is how we support older branches. If this is something
that requires in tree changes that ride the trains, supporting older
branches may get very complicated, because Buildbot runs inside of
MozillaBuild, so delegating to a different set of tools from inside of
Buildbot may not be possible.

[Chris Cooper]

That said, we would make every reasonable effort to support this change.

Package installation and maintenance on Windows has gotten better with
our recent move to GPO, but as Ben says, we're still layering newer
versions of required tools over-top of an ancient MozillaBuild install.
Having an easier way to update and experiment would be great.

Is part of the plan to keep a manifest of required tool versions in the
tree, similar to what we do now with tooltool?

cheers,
--
coop

[rya...@gmail.com]

My thought is that pacman would enable us to set up our own internal update servers, so you could control exactly what packages it sees and when. Then it would just be a matter of telling pacman when to look for updates.

[Wesley Hardman]

Can there be an effort to make this work better on Windows (not sure about other OSes)? I have to create a junction at C:\mozilla-build\ pointing to C:\Program Files (x86)\mozilla-build\ because it currently doesn't work properly with spaces. If it is too much of an issue, I can continue using the junction.

[Mike Hommey]

Here's a thought: how about just making things work with plain msys2 and
not require MozillaBuild at all anymore? We don't have a MozillaBuild
for mac, we just rely on homebrew or whatever else. Why not just aim
for the same?
What, in a msys2-based MozillaBuild would not come from msys2? The
environment scripts to find msvc? I'd argue that could/should be handled
by the build system.

Mike

[Ehsan Akhgari]

Does msys2 come with a decent package manager by default? Having to end
up with installing prerequisites manually would be a shame. Unless
there is something better that we can do through the mach bootstrap script?

If that can be taken care of, I think this is a great idea!

Cheers,
Ehsan

[Mike Hommey]

On Wed, Mar 12, 2014 at 10:51:40PM -0400, Ehsan Akhgari wrote:
> >Here's a thought: how about just making things work with plain msys2 and
> >not require MozillaBuild at all anymore? We don't have a MozillaBuild
> >for mac, we just rely on homebrew or whatever else. Why not just aim
> >for the same?
> >What, in a msys2-based MozillaBuild would not come from msys2? The
> >environment scripts to find msvc? I'd argue that could/should be handled
> >by the build system.
>
> Does msys2 come with a decent package manager by default? Having to
> end up with installing prerequisites manually would be a shame.
> Unless there is something better that we can do through the mach
> bootstrap script?
>
> If that can be taken care of, I think this is a great idea!

As Ryan said, msys2 comes with a port of pacman, which is Arch Linux's
package manager.

Probably not a complete list, but something like:
pacman -S make python2 git mercurial perl autoconf2.13
ought to be enough, and that can be documented on the wiki like the
homebrew and apt-get/yum invocations for OSX/Linux. And can be automated
with mach bootstrap too. I can't tell more because there are many issues
preventing the build from working with msys2 currently. I gave it a shot
for a couple minutes, and here are the most immediate problems:
- The base installation comes as a tar.xz, which is not the most
convenient format to install from on windows.
- The python 2.7 package doesn't install a "python" executable. Only a
"python2.7" executable, and both mach and client.mk default to using
"python".
- Our config.guess is old and doesn't output the right triplet.
- Using the 32 bits mode of the 64 bits msys2 still gives a triplet for
64 bits, so the 32 bits msys2 needs to be used (or a different
--target given)
- virtualenv installs the python executable in _virtualenv/bin/python
and we're looking in _virtualenv/Scripts/python.exe.
- either virtualenv or python doesn't work properly: sys.path doesn't
have the virtualenv paths when running the virtualenv python with a
windows path (c:/foo), although it works when running with a msys path
(/c/foo). (And the build system really prefers windows paths)

None of those except the first are problems that need a MozillaBuild
package. And the first should be dealt with at the msys2 level anyways.
The rest are issues with the build system.

That being said, I see msys python uses msys paths, and msys APIs, and
I'm sure many things will break as a consequence. We might still want to
stick with a native win32 python, and *that* would need a MozillaBuild
package.

Mike

[Benjamin Smedberg]

> Here's a thought: how about just making things work with plain msys2 and
> not require MozillaBuild at all anymore? We don't have a MozillaBuild
> for mac, we just rely on homebrew or whatever else. Why not just aim
> for the same?

Let me provide a bit of history/context for how Mozillabuild came to be
in the first place!

Back when I started at Mozilla, the process of building on Windows was
far worse. We used cygwin, and the process of getting everything working
was something like this:

* Install visual studio
* Install cygwin
* In the cygwin GUI package manager, select a list of packages to
install based on a list.
* The cygwin maintainers changed the package list frequently enough that
our documented list of package names was always wrong.
* You had to configure the Windows -> cygwin path mappings by hand to
deal with emulated unix permission handling and case sensitivity because
the defaults weren't correct.
* You had to create a batch file by hand to load the visual studio
environment variables and then load the cygwin shell.
* You had to configure an editor (for commit messages and whatnot) by hand.
* Setup put all sorts of configuration in the registry which wasn't
undone if you tried to uninstall, so people frequently ended up in
situations where they had a broken system and couldn't figure out how to
fix it.

I don't oppose using more modern system to do this, but I do want to
express the following items as being really important:

* the number of steps from "download from an MDN link" to "working
shell" should be as small as possible. Contributors who are used to
working in Windows are already at a disadvantage having to learn how to
use a unix-style shell, and we shouldn't make them responsible for
installing the right list of packages.
* It should include sensible defaults so that mercurial, git, and
commit-message editors all work out of the box
* Something early in the process should be able to check whether you
have a working setup including a working compiler and whatever D3D/DX
headers we need (maybe mach bootstrap can serve this role).
* Uninstall should work correctly and not leave residue.

--BDS