Exploring requiring Node.js to build Firefox

[Gregory Szorc]

Speaking as a maintainer of the Firefox build system, we try to be conservative about the set of dependencies required to build Firefox from source. We recognize that every required dependency is a potential source of failure and complexity. Dependencies can create complications for packagers. From a build system perspective, it's in our best interest to minimize the dependencies required to build Firefox.

Our desire to keep things simple can be at odds with the wishes of Firefox developers who wish to leverage new and exciting tools and technologies. In short, the policy of minimizing build dependencies can externalize costs onto overall Firefox development by hindering people from using better tools. This adversely affects the development velocity of Firefox and can cause product quality to suffer.

For a few years now, various pockets of Firefox development have wanted to use Node.js as part of the development workflow. I wouldn't say they directly want to use Node.js: they want to use the large ecosystem of tools built around Node.js. But that's splitting hairs. Our build system policy has been that leveraging Node.js and its ecosystem for supplemental workflows is fine, but shipping a build dependency on Node.js is not. Many Firefox developers are now using Node.js in their day-to-day workflow for things like running ESLint. We're using Node.js in CI. But we're not forcing people to have Node.js installed to build Firefox.

Various groups have routed around the limitation that Node.js can't be required to build Firefox. There are now Firefox features that do require Node.js to build. However, the output from the Node.js tools is checked into the Firefox source repository. So from the perspective of the Firefox build system, Node.js doesn't exist and therefore isn't a build dependency.

The status quo is not ideal. The more people I speak with, the more apparent it is that our current policy of not allowing Node.js tooling in the build system is causing more problems than it is preventing. Speaking as the build system module owner and someone who cares about developer workflows, tooling, and developer productivity, I don't think the current policy is good for Mozilla.

I'd like to start a discussion about requiring Node.js to build Firefox.

What do I mean by "require Node.js?" Let's assume I mean having a usable Node.js executable on the host system to be used during a Firefox build.

What about npm or a package manager? I would strongly prefer to limit the required dependency to Node.js itself. While the Firefox build system would depend on 3rd party packages and tools (such as Babel), I'm pretty insistent (as a build system maintainer) that these dependencies be vendored into the Firefox source repository so as to not incur a run-time dependency on a packaging service. I've seen the chaos that "left-pad" caused. I don't fully trust the security model of JavaScript package distribution. I don't think we can risk the ability to build Firefox or the integrity of the Firefox product by the availability and integrity of a 3rd party packaging service. That may sound like a harsh thing to say. But it's the posture we've applied elsewhere (such as to Python packages and PyPI). So, this means that all JavaScript executed by Node.js as part of the build would either be provided by Node.js itself or the Firefox source repository. If we needed to use a package manager as part of the build, that package manager could be vendored in the Firefox repository along with other JavaScript libraries (not unlike how we currently vendor Python's pip package manager).

A few people at Mozilla have poked at this problem already. We have a general sense of where some pain points for us will be. We know that getting modern versions of Node.js installed on various distributions requires using 3rd party package repositories. We know that Windows support could be painful. We know that installing common packages can result of dozens if not hundreds of dependencies being added. We know this could lead to us having to install thousands of files as part of the Firefox build - an overhead I'm not keen on seeing. We know all of this can add up to a significant amount of overhead to support. (Yet it still feels like a lesser problem than having people work around not being able to use Node.js directly.)

What we don't generally know is the impact requiring Node.js would have on downstream packagers. Our adoption of Rust last year was a long and sometimes painful process. I have a feeling that requiring Node.js would be a similar experience. But like Rust, I feel that adopting Node.js is in the best long-term interest for Firefox development velocity and product quality. I'm reluctant to cause more hardship by introducing a new build dependency. But it's very difficult to keep saying we can't use Node.js in the Firefox build system. I wish I could say "we'll build SpiderMonkey and use that instead." Unfortunately, many Node.js tools don't work with SpiderMonkey, so that's not an option. Plus there are difficulties with cross-compilation. As sad as it makes me to say it, SpiderMonkey is not an option: Node.js is the only viable option.

If we require Node.js to build Firefox, what are the requirements, desires, and hardships of downstream packagers and consumers of the Firefox build system? Keep in mind that mozilla-central right now is Firefox 60. That will become ESR 60 in May.

Gregory Szorc

g...@mozilla.com

Firefox build system module owner

[Gregory Szorc]

Quick follow-up. First to add mozilla-linux-taskforce. Second to note that we almost certainly wouldn't make a change before Firefox 61. That would give everyone only caring about ESR an extra ~1 year to deal with fallout.

[ISHIKAWA,chiaki]

The following issue related to npm package affects only a subset of
mozilla developers: only the patch contributors who develop on their PCs
that run Debian GNU/Linux.

|mach bootstrap| to set up the development tools and niceties fails now
under Debian GNU/Linux.
This is because "npm" package is no longer in the standard package
archive :-(

This is happening since mid-December, I think.
I was trying to fix a problem caused by "setupterm" error: see
https://groups.google.com/forum/#!topic/mozilla.dev.apps.thunderbird/mzEu2KI9foA

I thought of upgrading the packages from the linux distribution and
then tried to see if refreshing tools by "mach bootstrap" would help
when I noticed the problem seeing |mach| execution failed.

Moral of the story is the fewer dependencies, the better.

However, as I noted, this particular issue affects only the minority
users of Debian GNU/Linux on their individual machines, I suppose.
(I hate to think mozilla forces its developers to use a different
distribution. I don't believe it does.)

Also, NPM developers have prepared a web page to cope with this:
https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions
(Oops, according to the web page, Ubunto may be also affected by this.
If so, the future ubuntu users need to be warned. I am afraid that there
are many more ubuntu users than Debian users among patch developers.)

I think I will file both issues bugzilla: the latter for making it easy
for patch developers who use Debian GNU/Linux to become aware of the
solution. The former issue needs careful analysis ONCE I find the
solution to this particular issue on one of my PCs.


That said, "mach bootstrap" failing due to the missing npm package *IS A
NUISANCE*. I think a casual patch developer will quit doing so then and
there.

I am not a casual patch developer (doing this for the last several
years) and have used Debian distribution for more than 15 years.
AND I have been doing the software development on many hardware/software
including OS development itself close to 40 years now, and YET, I cannot
solve the original problem of setupterm failing right now :-(
(Admittedly, I am trying to solve the issue in my spare time which is
dwindling and so can't say exactly how hard the issue is. But usually,
this type of the problem can be fixed in a few days...)

So here it goes again:
Moral of the story is the fewer dependencies, the better.

If you expect non-paid free volunteer types to contribute to mozilla
software work, make the hurdle for development as low as possible.

TIA

[Dustin Mitchell]

http://nvm.sh/ seems to be a great way to avoid distro-specific issues
with node/npm installs, and to run exactly the expected version. Mach
could likely use that or do something similar, installing node and npm
in the objdir and using it from there.

Dustin

> _______________________________________________
> dev-builds mailing list
> dev-b...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-builds

[Sylvestre Ledru]

On 24/01/2018 08:08, ISHIKAWA,chiaki wrote:

The following issue related to npm package affects only a subset of mozilla developers: only the patch contributors who develop on their PCs that run Debian GNU/Linux.

|mach bootstrap| to set up the development tools and niceties fails now under Debian GNU/Linux.
This is because "npm" package is no longer in the standard package archive :-(

Could you please report a bug?

Mark added npm as a dependency on Debian [1], as I run Debian unstable, I didn't see it wasn't available for stretch or testing.

Sorry about that.

I also reported a bug on Debian side [2] so that it is clear that npm needs love in Debian.

Cheers,
Sylvestre

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1424921

[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888272

[ISHIKAWA,chiaki]

I will report a bugzilla.

Yes, I think npm nees supporters on Debian repository management now
that it seems to be used in many applications so much so that I find
them quite unexpected very often as in the case of mozilla build (!).

After an upgrade of mozilla source tree in December (the first after
late summer) and OS upgrade, I encountered many issues.
- blessing bug or an issue:
I have reported it now.
https://bugzilla.mozilla.org/show_bug.cgi?id=1432867
- then strange missing files issue when I enable official-branding
ac_add_options --enable-official-branding
The following closed bug is for M-C portion of the tree.
https://bugzilla.mozilla.org/show_bug.cgi?id=1416968
But I found that there is an issue on C-C tree side, too.
We are missing under the directory
comm-central/other-licenses/branding/thunderbird
two files
mailicon48.png
mailicon64.png
jar.mn modified in December refers to these files.
I said "strange missing files" because I thought these files are
supposed to have been
added to the tree,
https://bugzilla.mozilla.org/show_bug.cgi?id=1417334
But maybe the testing with official-branding enabled was not done.

- There could have been a few other issues before I could finally run
|mach build|, but now I can finally.
Then I am seeing tons of compiler error messages that basically says
ISO C++
does not allow certain things in constant expression (I am afraid
clang is too lenient on this) from GCC, and then I got segfault from
clang-4 when I switched to clang to see how it goes on clang side.
clang-5 still spews out many error message. I have no idea what is going
on with C-C tree right now. I will check with dev-apps-thunderbird ML
after I report the few unreported issues including this one in this mail
thread.

TIA

[Nicholas Alexander]

Greg, thanks for starting this discussion.

On Tue, Jan 23, 2018 at 4:34 PM, Gregory Szorc <g...@mozilla.com> wrote:

Speaking as a maintainer of the Firefox build system, we try to be conservative about the set of dependencies required to build Firefox from source. We recognize that every required dependency is a potential source of failure and complexity. Dependencies can create complications for packagers. From a build system perspective, it's in our best interest to minimize the dependencies required to build Firefox.

Our desire to keep things simple can be at odds with the wishes of Firefox developers who wish to leverage new and exciting tools and technologies. In short, the policy of minimizing build dependencies can externalize costs onto overall Firefox development by hindering people from using better tools. This adversely affects the development velocity of Firefox and can cause product quality to suffer.

For a few years now, various pockets of Firefox development have wanted to use Node.js as part of the development workflow. I wouldn't say they directly want to use Node.js: they want to use the large ecosystem of tools built around Node.js. But that's splitting hairs. Our build system policy has been that leveraging Node.js and its ecosystem for supplemental workflows is fine, but shipping a build dependency on Node.js is not. Many Firefox developers are now using Node.js in their day-to-day workflow for things like running ESLint. We're using Node.js in CI. But we're not forcing people to have Node.js installed to build Firefox.

I got interested in "paving Node.js/NPM cowpaths" in existing projects that are in some way in mozilla-central a little while ago.  I started interviewing people and teams in such projects, and have produced an "ecosystem matrix" of what such projects are doing at

https://docs.google.com/document/d/1beGX-xA3qtZQme-JKa2gOgyXo-GPdEXbapeYO3dY858/

(That document is supposed to be open for comment to anyone with the link; if it's not, please let me know!)  I hope this matrix can help inform this discussion by illuminating what's actually in use right now.  That document also includes notes from my interviews (with the permission of the interview subjects).

 

Various groups have routed around the limitation that Node.js can't be required to build Firefox. There are now Firefox features that do require Node.js to build. However, the output from the Node.js tools is checked into the Firefox source repository. So from the perspective of the Firefox build system, Node.js doesn't exist and therefore isn't a build dependency.

In the dependency matrix above, the projects that are doing this are the projects using WebPack.

 

The status quo is not ideal. The more people I speak with, the more apparent it is that our current policy of not allowing Node.js tooling in the build system is causing more problems than it is preventing. Speaking as the build system module owner and someone who cares about developer workflows, tooling, and developer productivity, I don't think the current policy is good for Mozilla.

I'd like to start a discussion about requiring Node.js to build Firefox.

What do I mean by "require Node.js?" Let's assume I mean having a usable Node.js executable on the host system to be used during a Firefox build.

What about npm or a package manager? I would strongly prefer to limit the required dependency to Node.js itself. While the Firefox build system would depend on 3rd party packages and tools (such as Babel), I'm pretty insistent (as a build system maintainer) that these dependencies be vendored into the Firefox source repository so as to not incur a run-time dependency on a packaging service. I've seen the chaos that "left-pad" caused. I don't fully trust the security model of JavaScript package distribution. I don't think we can risk the ability to build Firefox or the integrity of the Firefox product by the availability and integrity of a 3rd party packaging service. That may sound like a harsh thing to say. But it's the posture we've applied elsewhere (such as to Python packages and PyPI). So, this means that all JavaScript executed by Node.js as part of the build would either be provided by Node.js itself or the Firefox source repository. If we needed to use a package manager as part of the build, that package manager could be vendored in the Firefox repository along with other JavaScript libraries (not unlike how we currently vendor Python's pip package manager).

I whole-heartedly agree that vendoring required code -- that is, things that must be NPM `dependencies` -- into mozilla-central is required for our release management and product integrity efforts.

It's my belief that vendoring the NPM-based *tooling* -- that is, things that could be NPM `devDependencies` -- that Firefox developers are actively using right now would be impractical.

Just the node_modules from the testing configuration of our ES lint plugin (eslint-plugin-{mozilla,spidermonkey}) is roughly 150 NPM packages totaling 5000 files.  We might have, say, 500-1000 NPM packages to vendor into mozilla-central to support the existing toolchains of the 6 NPM-using projects in the tree.  Some of those packages probably require multiple versions.  Updating this vendored sprawl of code that *doesn't ship in Firefox directly* seems like a waste of time and effort.  Yes, we could add tooling to make this less painful, but I don't think we should eat that cost; and any layer of tooling to handle this vendoring will make all subsequent efforts more complex, and lead us to not update the vendored code.  That in turn will lead us to not "follow the ecosystem" and make it less likely that we'll use the best tool for the job at hand.

If we really want to avoid a build time dependency on NPM, then I think we should think through a small set of build system primitives that we want to support and not really think about NPM-based tooling at all.  That is, can we support Babel, WebPack, ES lint, and perhaps Flow, for arbitrary NPM packages checked into the tree?  (And vendor those hard dependencies into the tree.)  If we can do that, we might even be able to stretch to running those _limited_ dependencies in SpiderMonkey with the help of WebPack or browserify or some other Node.js-in-the-browser conversion, plus special sauce for SpiderMonkey and our build environment.

But that leaves us not using the NPM unit testing ecosystem (Mocha, Chai, Sinon, Jasmine, Jest, etc), and not using the code coverage ecosystem, and not using the editor integration ecosystem (natively), etc, etc.

On an entirely different front, Mozilla has some experience with requiring a package service at build time.  Firefox for Android requires third-party Maven repositories (BinTray's jcenter and Google's Android repository) at build-time.  We have a two-stage build process in automation that collects and caches dependencies, to isolate us from third-party outages.  Obviously Firefox for Android is a small project compared to Gecko and Firefox for Desktop, but early experience has been very positive.  And using Android Studio and the features of the Android-Gradle build plugin has been a phenomenal productivity win for the Firefox for Android teams; has enabled proper unit testing and static analysis of the Firefox for Android code base; and has allowed GeckoView to architect itself as a separable component of Firefox for Android, which is critical for the long-term success of GeckoView.

A few people at Mozilla have poked at this problem already. We have a general sense of where some pain points for us will be. We know that getting modern versions of Node.js installed on various distributions requires using 3rd party package repositories. We know that Windows support could be painful. We know that installing common packages can result of dozens if not hundreds of dependencies being added. We know this could lead to us having to install thousands of files as part of the Firefox build - an overhead I'm not keen on seeing. We know all of this can add up to a significant amount of overhead to support. (Yet it still feels like a lesser problem than having people work around not being able to use Node.js directly.)

What we don't generally know is the impact requiring Node.js would have on downstream packagers. Our adoption of Rust last year was a long and sometimes painful process. I have a feeling that requiring Node.js would be a similar experience. But like Rust, I feel that adopting Node.js is in the best long-term interest for Firefox development velocity and product quality. I'm reluctant to cause more hardship by introducing a new build dependency. But it's very difficult to keep saying we can't use Node.js in the Firefox build system. I wish I could say "we'll build SpiderMonkey and use that instead." Unfortunately, many Node.js tools don't work with SpiderMonkey, so that's not an option. Plus there are difficulties with cross-compilation. As sad as it makes me to say it, SpiderMonkey is not an option: Node.js is the only viable option.

Even though I floated an alternative above (browserify and labour), I concur that Node.js is the only viable option.

 

If we require Node.js to build Firefox, what are the requirements, desires, and hardships of downstream packagers and consumers of the Firefox build system? Keep in mind that mozilla-central right now is Firefox 60. That will become ESR 60 in May.

I wish I could say more about downstream packagers but I have nothing to add here.

Thanks again!  Long live the glorious web development ecosystem!

Nick

[ISHIKAWA,chiaki]

On 2018/01/24 22:42, Sylvestre Ledru wrote:
> On 24/01/2018 08:08, ISHIKAWA,chiaki wrote:
>>
>> The following issue related to npm package affects only a subset of
>> mozilla developers: only the patch contributors who develop on their
>> PCs that run Debian GNU/Linux.
>>
>> |mach bootstrap| to set up the development tools and niceties fails
>> now under Debian GNU/Linux.
>> This is because "npm" package is no longer in the standard package
>> archive :-(
>
> Could you please report a bug?
>

I filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1432892

TIA

[m...@mykzilla.org]

On Wednesday, January 24, 2018 at 11:10:06 AM UTC-8, Nicholas Alexander wrote:

> If we really want to avoid a build time dependency on NPM, then I think we should think through a small set of build system primitives that we want to support and not really think about NPM-based tooling at all. That is, can we support Babel, WebPack, ES lint, and perhaps Flow, for arbitrary NPM packages checked into the tree? (And vendor those hard dependencies into the tree.) If we can do that, we might even be able to stretch to running those _limited_ dependencies in SpiderMonkey with the help of WebPack or browserify or some other Node.js-in-the-browser conversion, plus special sauce for SpiderMonkey and our build environment.

This would be a reasonable strategy for a situation in which we want to integrate a third-party project whose build process relies on Node but don't otherwise want to use Node ourselves. Then we might do the minimum necessary to support that build process without enabling more general use of Node in the build system.

An (imperfect, to be sure) example of that approach is the way we added minimal support for GYP in order to integrate some third-party libraries that depended on it. (We've since discovered other uses for GYP, and improved support for it, so the approach has changed over time.)

However, I don't think that's the situation we're in currently. Rather, the breadth of Firefox projects using Node, and the breadth of Node features (+ NPM packages) they're using, suggests that our strategy should accommodate more general use of Node (and evolution of its use).

> As sad as it makes me to say it, SpiderMonkey is not an option: Node.js is the only viable option.

As a SpiderNode contributor (and fanboy), I reluctantly concur. Even if we could kludge together a solution for the subset of Node (+ NPM packages) in use today (which itself seems like a stretch), it wouldn't have the same developer ergonomics and would be inflexible in the face of evolving usage patterns.

Maybe it'll be possible to run Node (or Node packages) on SpiderMonkey one day, but Node itself is the only viable option for the foreseeable future.

-myk

[Gregory Szorc]

On Fri, Jan 26, 2018 at 1:10 AM, Wolfgang Rosenauer <wolf...@rosenauer.org> wrote:

Hi,

(wondering if this goes through. Somehow I miss another answer sent to
this list earlier.)

Am 24.01.2018 um 10:26 schrieb Landry Breuil:

>> Quick follow-up. First to add mozilla-linux-taskforce. Second to note that
>> we almost certainly wouldn't make a change before Firefox 61. That would
>> give everyone only caring about ESR an extra ~1 year to deal with fallout.
>

> The adoption of rust has been painful, i guess the requirement of npm
> will be even more painful, and if i'd be given the choice, i'd strongly
> raise my voice against it. Just recently it's been made harder for
> packagers to work on betas by not providing source tarballs anymore but
> only a link to hg that we get to hammer (#1432591) and now this ?
>
> But since the decision has already been made, i guess we can't do
> anything about it, but adapt or die ?
> As long as the required version isnt too recent (we have 6.12.3 in
> OpenBSD right now, and i have no idea what this means in terms of nodejs
> versions) and *you stick to vendoring the nodejs modules HELL* (build
> systems are forbidden to download stuff themselves here), i guess we'll
> swallow the bitter pill.
> Or give up and go do something else, in the greener pastures outside of
> this crazy javascript bloat ecosystem.

I can only mainly agree to what Landry wrote.
With (open)SUSE we have basically the same situation.

Our buildsystem is not allowed to download any resources. Everything
must be available in the source tree or in the distribution we build for.
As node.js and modules is quite a moving target there is a high risk
that we end up being unable to build Firefox+1 leaving users at risk.
While there is ESR this helps indeed because this is what we nowadays
(are forced to) ship because mainly of rust since we simply cannot
update rust in a released distribution every around 6 weeks.
But "dealing with the fallout" by having a year time does not help. This
is an ongoing burden which cannot be solved but would need to be
addressed always when the node.js requirement is increased basically.

For our rolling variant Tumbleweed this is typically not much of an
issue because also node.js is rolling there and can be updated via the
normal process if required. Tumbleweed is also the only distribution we
nowadays can ship regular release on (see above).

That context explained I think our downstream requirements are summarized:
- all node.js modules need to be shipped via the mozilla source tree to
  be on the safe side (no download possible)
- problem is with node.js itself and the module compatibility. I have no
  experience with that stuff
- try to be conservative and also keep node.js requirements in ESR
  cycles unchanged

Just to give an impression which node.js versions we currently have in
supported releases:

Stable distributions:
node.js 4.8.7 and 6.11.1 (choice)

Rolling distribution:
node.js 6.12.3 and 8.9.4

Obviously those are changing over time.

While we are at it something a bit offtopic:
Currently we are scared about python 2.7 requirements since the next
stable version is in the works which will last for at least 3-5 years
and our goal is to get rid of python 2.7 in that release because it will
(officially) run out of maintenance before EOL.
Are there any plans about that?

We are just now requiring Python 3.5+ to build.

While some aspects of the build system will adopt Python 3, we have no timeline for moving everything away from Python 2.7. From our perspective, Python 2.7 "just works" and we'd rather spend time making meaningful improvements rather than porting little-changed code to Python 3. That being said, I concede that I'm not a fan of running on a language platform that will be unsupported after 2020. But that platform is stable and until there is more pressure or a compelling need to mass port everything to Python 3, I don't see that happening any time soon.