Fwd: Intent to implement: Use NodeJS-based tools in Chromium’s toolchain

[Demetrios Papadopoulos]

Sent from the correct account this time. Apologies for those who received this twice.

-------------------------------------------

Hello Chromium developers and Node users,

Summary

We need to use NodeJS-based tools to make Chrome Material Design Settings fast enough to ship.  More details in the document (also linked below).

Tracking bug

crbug.com/673825

Details

See https://docs.google.com/document/d/1G3JLn-BM-fItuJVUiILIz7X2qAv4eta5DljSSdQ94Fg/edit

Contacts

dpa...@chromium.org, db...@chromium.org

Thoughts?

-- Demetrios & Dan

[Dan Beam]

On Thu, Dec 15, 2016 at 4:40 PM, Demetrios Papadopoulos <dpa...@google.com> wrote:

Hello Chromium developers and Node users,

Summary

We need to use NodeJS-based tools to make Chrome Material Design Settings fast enough to ship.  More details in the document (also linked below).

Tracking bug

crbug.com/673825

Details

See https://docs.google.com/document/d/1G3JLn-BM-fItuJVUiILIz7X2qAv4eta5DljSSdQ94Fg/edit

Contacts

dpa...@chromium.org, db...@chromium.org

Thoughts?

-- Demetrios & Dan

You have my sword.

-- Dan

[Elliott Sprehn]

And my AXE!

This is very exciting. Having node.js will let us reuse the existing vulcanize tooling instead of checking in the build output, allow us to remove the python version of vulcanize, and let us reuse lots of tools and infrastructure used by regular web developers.

I know there's many folks around chrome who've been asking for node.js for a while outside Web UI as well. :)

[Demetrios Papadopoulos]

Fixed some typos in the recipients addresses.

On Fri, Dec 16, 2016 at 5:44 PM, Elliott Sprehn <esp...@chromium.org> wrote:

And my AXE!

Thanks!

[Anthony Berent]

Great news; we are also looking at using node.js to allow us to run the JS version of the closure compiler at build time to minify Chrome's Javascript (see http://crbug/393874) 

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Richard Maher]

What? No jQuery bollocks optimizer? No Bootstrap "absolutely essential performance optimizer"?

Did I really just hear grown men say "You have my sword and my axe"? Are there no adults here?

When will this Lord of the Flies twilight zone give way to background geolocation and broadcast notifications :-(

[Colin Blundell]

Please reread https://www.chromium.org/conduct.

[Nico Weber]

I remain skeptical that having two dependencies on v8 (on because we depend on v8, and one through node) is the best way forward here. We already have a JS engine in chromium's repo. What parts of node does Polymer need? Is it hard to stub those out ourselves?

Why did we add a strong dependency on Polymer when the integration story there is apparently still "Node"?

(I realize that this is a complicated space and that this might be the best tradeoff, but the linked doc didn't say anything about these points.)

[Dirk Pranke]

These are good questions.

The JS community has very strongly shifted to using tools based on top of Node to do things like lint, minify, combine modules, and run tests.

Attempting to stub those out or duplicate the functionality in a different language like Python at this point would IMO both be a bunch of work

that is unnecessary and very much swimming upstream. A few years ago when we first discussed adding Node I also thought it might be

tractable to avoid it and re-implement the work in Python and avoid the 100's of KLOCs of code that doing anything in Node appears to require.

I no longer really think that's the case. Or, at least, I'm not going to be the one putting in the work, and no one else seems to want to , either.

Also, by pulling in a binary dependency on Node we don't actually have to build it, nor do we have to worry about it breaking on tip-of-tree 

and thus crippling our build process. 

-- Dirk

[Nico Weber]

> To be clear, no part of Polymer depends on Node.

Well, if you don't need to use Node but "We need to use NodeJS-based tools to make Chrome Material Design Settings fast enough to ship", then you kinda need to use Node ;-)

On Mon, Dec 19, 2016 at 1:46 PM, Dirk Pranke <dpr...@chromium.org> wrote:

These are good questions.

The JS community has very strongly shifted to using tools based on top of Node to do things like lint, minify, combine modules, and run tests.

Attempting to stub those out or duplicate the functionality in a different language like Python at this point would IMO both be a bunch of work

that is unnecessary and very much swimming upstream. A few years ago when we first discussed adding Node I also thought it might be

tractable to avoid it and re-implement the work in Python and avoid the 100's of KLOCs of code that doing anything in Node appears to require.

I no longer really think that's the case. Or, at least, I'm not going to be the one putting in the work, and no one else seems to want to , either.

I mean we already have a JS engine, it only isn't able to write files and things, which is what Node provides bindings for. I meant to suggest to stub out those pieces -- most of Polymer is probably/hopefully/maybe "just" JS which we already can run.

[Dirk Pranke]

On Mon, Dec 19, 2016 at 10:56 AM, Nico Weber <tha...@chromium.org> wrote:

> To be clear, no part of Polymer depends on Node.

Well, if you don't need to use Node but "We need to use NodeJS-based tools to make Chrome Material Design Settings fast enough to ship", then you kinda need to use Node ;-)

On Mon, Dec 19, 2016 at 1:46 PM, Dirk Pranke <dpr...@chromium.org> wrote:

These are good questions.

The JS community has very strongly shifted to using tools based on top of Node to do things like lint, minify, combine modules, and run tests.

Attempting to stub those out or duplicate the functionality in a different language like Python at this point would IMO both be a bunch of work

that is unnecessary and very much swimming upstream. A few years ago when we first discussed adding Node I also thought it might be

tractable to avoid it and re-implement the work in Python and avoid the 100's of KLOCs of code that doing anything in Node appears to require.

I no longer really think that's the case. Or, at least, I'm not going to be the one putting in the work, and no one else seems to want to , either.

I mean we already have a JS engine, it only isn't able to write files and things, which is what Node provides bindings for. I meant to suggest to stub out those pieces -- most of Polymer is probably/hopefully/maybe "just" JS which we already can run.

It's not Polymer I'm worried about. I'm worried about whatever dependencies things like the closure_compiler and karma might have.

-- Dirk

[Mark Dittmer]

Dirk,

Are you concerned of the build-time costs of depending on (deps-of-closure_compiler/karma)?

[Dirk Pranke]

( Trimming the cc' list because there are a couple of incorrect addresses and because we're all on chromium-dev anyway ... )

I expect the node modules and other stuff that closure_compiler and karma pull in will also be pulled in as binary dependencies.

So, they'll impose a one-time cost in download time and disk space on the initial pull (or again when they're updated), but will not

impose costs on each build.

I believe the costs are not excessive (a few MB to maybe 10s of MB), but maybe I'm wrong. If so, that would be good to know.

This is still probably significantly better than, say, the NaCl toolchains :).

I'm guessing it's not really tractable to make using Node for WebUI a sync-time/compile-time option, since that would probably

force WebUI devs to maintain two different code paths, and that seems somewhere between intractable and annoying, but

if we could do it so that there wasn't much overhead to maintaining it, *and* if the overall download time was really large,

that might be worth it.

-- Dirk

[Nico Weber]

On Mon, Dec 19, 2016 at 3:22 PM, Dirk Pranke <dpr...@chromium.org> wrote:

( Trimming the cc' list because there are a couple of incorrect addresses and because we're all on chromium-dev anyway ... )

I expect the node modules and other stuff that closure_compiler and karma pull in will also be pulled in as binary dependencies.

So, they'll impose a one-time cost in download time and disk space on the initial pull (or again when they're updated), but will not

impose costs on each build.

I believe the costs are not excessive (a few MB to maybe 10s of MB), but maybe I'm wrong. If so, that would be good to know.

This is still probably significantly better than, say, the NaCl toolchains :).

That seems like a pretty bad point of reference, almost everything looks small compared to those :-P

Initially, the idea was that WebUI could depend on a modern browser, so that it could be "just JS" and we wouldn't need jquery or whatever the big web thing was back then. Now that transpiling JS is popular on the web, it sounds like we're close committing to depending on:

* Node

* Closure-compiler-in-js-compiled-through-gwt(?)-from-Java

* Etc

To me, that looks like one of the biggest dependencies (or rather, dependency chains) we've added so far. Hence, I'd personally love to see a larger "other evaluated approaches" and their drawbacks section in that doc.

[Chris Greene]

Estimated size of dependencies based on my machine.

node: 24M-33M
npm: 20M
crisper vulcanize polymer-css-build uglifyjs: 60 M

Total estimate: ~100M

> You received this message because you are subscribed to the Google Groups
> "Chromium-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to chromium-dev...@chromium.org.

[Mark Dittmer]

[[cc-trim again]]

Is there any dead code hiding in the 60M dependencies? If the external-repo-providing-dependencies could ship code that provides the interface WebUI needs after having done some serious "tree shaking" against npm dependencies, that could shrink the largest of the three dependency chains listed.

Is such a thing possible? It's not clear to me whether WebUI is hoping to pull into Chromium a large API surface to use as it needs (in which case, "no"), or specific interfaces that it knows (today) it needs (in which case, "yes").

//Mark

> email to chromium-dev+unsubscribe@chromium.org.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
    http://groups.google.com/a/chromium.org/group/chromium-dev
---

You received this message because you are subscribed to a topic in the Google Groups "Chromium-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chromium-dev/H2IqgqwdUqs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chromium-dev+unsubscribe@chromium.org.

[Demetrios Papadopoulos]

Regarding on how much size this adds to a Chromium checkout, here are the numbers:

• Node binaries: Linux 11MB (gzipped), Mac 13.5MB (gzipped), Win: 17MB
  
• NPM dependencies of  vulcanize and remaining tools: 25MB, which will be pulled down via a googlesource git repo (as explained in the doc).
  
• The NPM binary itself is not needed and will not be used by Chromium's toolchain. We could probably remove it from the Linux/Mac gzip file and reduce size even more.

Remaining answers inlined.

On Mon, Dec 19, 2016 at 9:50 AM, Nico Weber <tha...@chromium.org> wrote:

I remain skeptical that having two dependencies on v8 (on because we depend on v8, and one through node) is the best way forward here. We already have a JS engine in chromium's repo.

Node is not just a JS engine. It includes a large set of cross-platform APIs

(https://nodejs.org/api/), that make it much more useful than V8's shell (d8).

 

What parts of node does Polymer need? Is it hard to stub those out ourselves?

Finding exactly which Node APIs all Polymer tools we use (and their transitive
dependencies use), and stubbing those out sounds like an immense amount of
Work. It basically means that we implement and maintain (on all of Mac/Linux/Win) our own
subset of Node just so that we don't use proper Node. Even if that was possible,
what would be the benefit? Does saving a few MBs from a Chromium checkout for a
binary that will be updated infrequently (whenever a new LTS version of
Node exists), justify the seemingly huge cost of the alternative?

 

Why did we add a strong dependency on Polymer when the integration story there is apparently still "Node"?

Custom elements has been a multi-year, cross-browser effort, and represents the type of advancement the Web community has been eagerly looking forward to. Polymer provides Chrome:

• The best way we know on how to consume Custom Elements
  
• Data binding between JS and HTML (greatly reduces code complexity)
  
• A library of Material Design UI components
  
• Great tooling, product, and performance support (as a result of being part of Chromium)

We adopted Polymer for these reasons.

Even if we had not used Polymer in the first place, some other framework could have been

used (there is plenty of them out there), and the necessity of using Node would most likely

come up either way, because modern JS development requires Node. Almost modern tools for

JS are:

• written in JS
  
• run with Node
  
• distributed via NPM/Bower (this part does not really matter for Chromium though)

The development bottlenecks caused by not using Node (mentioned in the doc), have been worked

around for quite a while, but as the nature of all workarounds is, they are not scalable. Now is the time

where those workarounds need to be replaced with a proper solution.

Thanks,

Demetrios

[Adam Klein]

On Mon, Dec 19, 2016 at 10:56 AM, Nico Weber <tha...@chromium.org> wrote:

> To be clear, no part of Polymer depends on Node.

Well, if you don't need to use Node but "We need to use NodeJS-based tools to make Chrome Material Design Settings fast enough to ship", then you kinda need to use Node ;-)

On Mon, Dec 19, 2016 at 1:46 PM, Dirk Pranke <dpr...@chromium.org> wrote:

These are good questions.

The JS community has very strongly shifted to using tools based on top of Node to do things like lint, minify, combine modules, and run tests.

Attempting to stub those out or duplicate the functionality in a different language like Python at this point would IMO both be a bunch of work

that is unnecessary and very much swimming upstream. A few years ago when we first discussed adding Node I also thought it might be

tractable to avoid it and re-implement the work in Python and avoid the 100's of KLOCs of code that doing anything in Node appears to require.

I no longer really think that's the case. Or, at least, I'm not going to be the one putting in the work, and no one else seems to want to , either.

I mean we already have a JS engine, it only isn't able to write files and things, which is what Node provides bindings for. I meant to suggest to stub out those pieces -- most of Polymer is probably/hopefully/maybe "just" JS which we already can run.

As a v8 developer who's recently made some changes v8's shell, d8, to do more with file loading, I would say that this is grossly underestimating the work that would be required to make v8 "stub out" the parts of Node that it's missing. The fact that Node uses v8 internally is an implementation detail; it does not imply that it's easy to re-build a Node-compatible environment on top of v8 (and in fact that seems like it would be more likely to lead to breakages even if it were a tractable problem).

- Adam

[PhistucK]

We can always go with Chakra-Node. ;)

☆PhistucK

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Dan Beam]

On Mon, Dec 19, 2016 at 9:50 AM, Nico Weber <tha...@chromium.org> wrote:

I remain skeptical that having two dependencies on v8 (on because we depend on v8, and one through node) is the best way forward here. We already have a JS engine in chromium's repo. What parts of node does Polymer need? Is it hard to stub those out ourselves?

From what I've heard: node's use of v8 is involved.  They used to have a fork (or something?) and now track versions of v8 that aren't HEAD.  I agree with Adam that v8 is an implementation detail; web developers would still use node without v8 (and so should we).

Why did we add a strong dependency on Polymer when the integration story there is apparently still "Node"?

We use Polymer because it's the best way we know to harness Custom Elements / Shadow DOM and push the web platform forward (a long-standing goal of Chrome ;)).

There's been need for node in Chromium for a long time.  Folks have just found crafty ways around it (see: dev tools, catapult, webui's vulcanize and polymer rolling flows).

However, at this point, the cost of these node-prohibited solutions higher than the benefit (at least for WebUI).  As mentioned on the document, we need to integrate with grit at compile-time, which is why we need node in the toolchain itself.  The [super lame] alternatives are to check in like 1000 x 1MB generated code bundles or move all of webui to github or something.

-- Dan

[Dan Beam]

On Mon, Dec 19, 2016 at 12:45 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Dec 19, 2016 at 3:22 PM, Dirk Pranke <dpr...@chromium.org> wrote:

( Trimming the cc' list because there are a couple of incorrect addresses and because we're all on chromium-dev anyway ... )

I expect the node modules and other stuff that closure_compiler and karma pull in will also be pulled in as binary dependencies.

So, they'll impose a one-time cost in download time and disk space on the initial pull (or again when they're updated), but will not

impose costs on each build.

I believe the costs are not excessive (a few MB to maybe 10s of MB), but maybe I'm wrong. If so, that would be good to know.

This is still probably significantly better than, say, the NaCl toolchains :).

That seems like a pretty bad point of reference, almost everything looks small compared to those :-P

Initially, the idea was that WebUI could depend on a modern browser, so that it could be "just JS" and we wouldn't need jquery or whatever the big web thing was back then.

Most of WebUI runs on HEAD Chrome + blink + v8.  But parts run on Chrome for iOS, which differs in its support.

 

Now that transpiling JS is popular on the web, it sounds like we're close committing to depending on:

* Node

* Closure-compiler-in-js-compiled-through-gwt(?)-from-Java

* Etc

We don't yet transpile.  We could if we'd like to make sure we don't break iOS users.  Right now we basically check manually (and I added a presubmit about 1 specific feature (=>) recently).

Also, we don't yet need Closure-thinger-from-GWT; we currently use uglifyjs (an alternative approach).  This saved like 300KB from Chrome's installer.  But uglifyjs' support of ES6 features is still behind a branch, and we'd prefer to use Closure compiler because it has great ES6 support and is Google-maintained (and may produce more optimized code).

To me, that looks like one of the biggest dependencies (or rather, dependency chains) we've added so far. Hence, I'd personally love to see a larger "other evaluated approaches" and their drawbacks section in that doc.

While our browser environment may be modern, our web development tooling stack is not.

The pre-node web dev tooling story was fragmented.  Random python/shell/Java scripts that required additional language knowledge to build your website.  That's why the ability to write tools in node is a game changer for the web development community: JavaScript is already strictly required to make useful web apps, so why add N+1 languages?  That's the fundamental shift.

I don't see this conversation as "we need node for this single UI this one time".  I see this conversation as "every new web development tool is likely to use node, and if we'd like to use any of them we need this".  Perhaps with a hint of "we're already using node in sneaky one-off ways, which could be improved".

-- Dan

[Dan Beam]

On Mon, Dec 19, 2016 at 4:18 PM, Dan Beam <db...@chromium.org> wrote:

On Mon, Dec 19, 2016 at 12:45 PM, Nico Weber <tha...@chromium.org> wrote:

On Mon, Dec 19, 2016 at 3:22 PM, Dirk Pranke <dpr...@chromium.org> wrote:

( Trimming the cc' list because there are a couple of incorrect addresses and because we're all on chromium-dev anyway ... )

I expect the node modules and other stuff that closure_compiler and karma pull in will also be pulled in as binary dependencies.

So, they'll impose a one-time cost in download time and disk space on the initial pull (or again when they're updated), but will not

impose costs on each build.

I believe the costs are not excessive (a few MB to maybe 10s of MB), but maybe I'm wrong. If so, that would be good to know.

This is still probably significantly better than, say, the NaCl toolchains :).

That seems like a pretty bad point of reference, almost everything looks small compared to those :-P

Initially, the idea was that WebUI could depend on a modern browser, so that it could be "just JS" and we wouldn't need jquery or whatever the big web thing was back then.

Most of WebUI runs on HEAD Chrome + blink + v8.  But parts run on Chrome for iOS, which differs in its support.

 

Now that transpiling JS is popular on the web, it sounds like we're close committing to depending on:

* Node

* Closure-compiler-in-js-compiled-through-gwt(?)-from-Java

* Etc

We don't yet transpile.  We could if we'd like to make sure we don't break iOS users.  Right now we basically check manually (and I added a presubmit about 1 specific feature (=>) recently).

Also, we don't yet need Closure-thinger-from-GWT; we currently use uglifyjs (an alternative approach).  This saved like 300KB from Chrome's installer.  But uglifyjs' support of ES6 features is still behind a branch, and we'd prefer to use Closure compiler because it has great ES6 support and is Google-maintained (and may produce more optimized code).

To me, that looks like one of the biggest dependencies (or rather, dependency chains) we've added so far. Hence, I'd personally love to see a larger "other evaluated approaches" and their drawbacks section in that doc.

I added an "Alternatives Considered" section at the bottom of the document.

-- Dan

[John Abd-El-Malek]

FWIW this proposal sounds reasonable given all the constraints listed.

One thing I didn't see is: what's the effect on build times?

[Demetrios Papadopoulos]

On Mon, Dec 19, 2016 at 5:37 PM, John Abd-El-Malek <j...@chromium.org> wrote:

FWIW this proposal sounds reasonable given all the constraints listed.

One thing I didn't see is: what's the effect on build times?

The additional build time for creating JS/HTML bundles takes:

• 5.5 seconds (averaged) for Downloads
  
• 10-12 seconds (averaged) for Settings

If "use_vulcanize = false" is used, then there is no impact at all in build times. Needless to say that things only get rebuild if a dependency of the output HTML/JS changes, so it would have a minimal impact on developers who don't modify given WebUI pages. I can send more instructions if you want to try it locally (it involves patching 4 CLs).

[John Abd-El-Malek]

Can you provide numbers for all the webui pages on mac/win/linux? How many webui pages would go through this at the end (I presume all of them?). If so, what's the upper range for this new build step?

If it's going to be a few minutes, then perhaps this should only be done on optimized or official builds by default?

On Mon, Dec 19, 2016 at 6:37 PM, Demetrios Papadopoulos <dpa...@chromium.org> wrote:

On Mon, Dec 19, 2016 at 5:37 PM, John Abd-El-Malek <j...@chromium.org> wrote:

FWIW this proposal sounds reasonable given all the constraints listed.

One thing I didn't see is: what's the effect on build times?

The additional build time for creating JS/HTML bundles takes:

• 5.5 seconds (averaged) for Downloads
  
• 10-12 seconds (averaged) for Settings

If "use_vulcanize = false" is used, then there is no impact at all in build times. Needless to say that things only get rebuild if a dependency of the output HTML/JS changes, so it would have a minimal impact on developers who don't modify given WebUI pages. I can send more instructions if you want to try it locally (it involves patching 4 CLs).

Note: I sync every day or so and that's effectively a clean build each time because of all the changes that happen since then.

[Dan Beam]

On Mon, Dec 19, 2016 at 6:44 PM, John Abd-El-Malek <j...@chromium.org> wrote:

Can you provide numbers for all the webui pages on mac/win/linux? How many webui pages would go through this at the end (I presume all of them?). If so, what's the upper range for this new build step?

We haven't gotten GN integration working on Windows yet.

Here's some timings of all the node-based optimization tools we currently run:

https://gist.github.com/danbeam/f52c980973520623d4262bce972d5df9

To optimize all current pages (downloads, history), in total it takes:

• 9.5s on my 2013 Macbook pro
  
• 12.5s on my z620 (Ubuntu)
  
• 17s on my t3500 (Windows 7)

Settings is a big page, so I'd guess about ~10s on Mac/Linux and about ~20s on Windows.

As Demetrios said, we have a use_vulcanize GN flag that we could disable by default for debug or chromium builds if we'd like.

-- Dan

[Daniel Cheng]

I am mostly neutral about NodeJS. However, I've found it necessary to modify WebUI sometimes in the past, and historically, it's not been so easy for me to figure what JS file to modify. Will throwing the NodeJS toolchain in the mix make this easier or harder? =)

Daniel

[Dan Beam]

On Tue, Dec 20, 2016 at 11:47 AM, Daniel Cheng <dch...@chromium.org> wrote:

I am mostly neutral about NodeJS. However, I've found it necessary to modify WebUI sometimes in the past, and historically, it's not been so easy for me to figure what JS file to modify. Will throwing the NodeJS toolchain in the mix make this easier or harder? =)

You're probably talking about following this:

https://chromium.googlesource.com/chromium/src/+/master/docs/vulcanize.md

If we bundle the NPM modules and sync down node, everything will be way easier.  You will not need to install anything locally and everybody will be on the same versions of any related tool.

-- Dan

[John Abd-El-Malek]

Thanks. So how many pages in total will this run on once everything is using MD?

[Dan Beam]

On Tue, Dec 20, 2016 at 12:20 PM, John Abd-El-Malek <j...@chromium.org> wrote:

Thanks. So how many pages in total will this run on once everything is using MD?

Sorry, forgot that part of your question.

It really depends.  I think at most: extensions, history, downloads, settings/about, bookmarks, print preview (6 pages, 7 if you count settings/about as doubly complex).  We probably only need to vulcanize for user-facing pages with lots of HTML imports and that aren't fast enough already.

If I were to give you a total guesstimate, it'd be that we need to make:

• 1 bundle for downloads (known)
• 2 bundles for history (known)
  
• 1 bundle for extensions (?)
• 2 bundles for settings (semi-known)
• 1 bundle for bookmarks (?)
• 1.5 bundles for print preview (?, either 1 or 2)
  

So, if we multiply that by the average time to create a bundle we get the total extra amount of time:

• 8.5 bundles x ~3s / bundle = 25.5s on Mac (MBP 2013)
  
• 8.5 bundles x ~4s / bundle = 34s on Linux (Z620)
  
• 8.5 bundles x ~6s / bundle = 51s on Windows (T3500)
  

Note: I don't see a reason why this bundling couldn't be fully parallelized.

We're also likely to apply GRD's compress="gzip" to these bundles, which greatly reduces their on-disk size (and hence, the installer size).  I'm landing some metrics to track the first meaningful paint on downloads so we can measure the effect of the streaming decompression when pulled out of a resource pak.  MD History is probably ready to try this now.  We've been doing this on internals pages, especially on Android, with great success.

-- Dan

[John Abd-El-Malek]

Thanks for the data. Hopefully the numbers you mentioned for Windows will be closer to the Linux ones if using a Z840. This should happen in parallel as you mention so shouldn't impact compile time that much. If there's a flag to switch it on and off, then we can wait till all these pages are switches and run experiments to see if it adds non-negligible cost to debug compiles, and if so, turn them off by default.

PS I highly suggest getting a Z840 for Windows development!

[Demetrios Papadopoulos]

Hello Chromium devs,

Summarizing the discussion so far (per my understanding), as well as giving some updates.

To Node or not to Node?

There is enough consensus for allowing usage of NodeJS for WebUI related pre-processing tasks (a mix between people positive about Node in general, and people just acknowledging that is an unfortunate necessity of JS tooling).

How about size?

• Size of NodeJS downloaded binaries is 11-17MB (depending on platform) which is reasonable.
  
• NPM dependencies are currently 25MB. WebUI team is working on reducing those as follows:
1. Contacting authors of vulcanize/crisper to remove unnecessary deps, or replace them with better alternatives. For example command line parsing could be done with nopt (192kb) insetad of command-line-args (7.5MB).
2. Ensuring that obviously unnecessary files will not be pulled down via gclient sync (package.json, *.md, */test/ etc), saves at least another 4MB.

How about security?

• It was made clear that none of the NPM deps is shipped with Chrome, only the generated JS/HTML bundled output is shipped (which is no different than existing situation, History/Downloads already shipped such bundles).
• It was also made clear that NPM itself is not invoked from Chrome's toolchain, (instead deps will be downloaded from a googlesource repo maintained by the WebUI team).
• Adding new NPM deps in the future will require to go through the existing review process (just like any other third_party/ dependency).

How about build times?

• Additional build time by the new compilation steps is a concern (especially when more WebUI pages will be vulcanized). We are considering configuring use_vulcanize GN flag to be false by default, and true only on official builds.

Unless I missed some major concern that has not been addressed yet, we plan to move ahead with the security and open-source licensing review in the coming days.

Thank you,

Demetrios

[PhistucK]

On Thu, Dec 22, 2016 at 8:56 PM, Demetrios Papadopoulos <dpa...@chromium.org> wrote:

Additional build time by the new compilation steps is a concern (especially when more WebUI pages will be vulcanized).

One lovely consequence of using Node is that as V8 gets faster, the build gets faster (unless most of the time is spent in system calls, obviously). :)​

☆PhistucK

[Nico Weber]

Thanks for the update, that sounds right to me.

Do you know if this tooling produces deterministic outputs? As in, if I run the build step, delete the outputs, and run it again, does it produce exactly the same files again?

[Dan Beam]

On Fri, Dec 23, 2016 at 6:41 AM, Nico Weber <tha...@chromium.org> wrote:

Thanks for the update, that sounds right to me.

Do you know if this tooling produces deterministic outputs? As in, if I run the build step, delete the outputs, and run it again, does it produce exactly the same files again?

It already says in this in the doc, but just re-iterating (because it's come up many times): the nodejs binaries and npm deps would be stored in Chromium-controlled git repos.  These tools would be synced locally and run explicitly; never the system-wide versions on $PATH.

We would never run `npm install` on a bot or developer machine.  Npm modules and nodejs binaries would only change by somebody explicitly pushing.

Once synced: yes, I would expect exactly the same output on each run.  I can't prove all the code involved is deterministic because I'd need to look through all of nodejs' source code and the npm deps with a fine toothed comb (and would still likely be wrong).  

As a data point: I have never seen a difference when re-running vulcanize.py (our optimization script); I've done that hundreds of times.

Do we audit every change to clang or gcc or <insert build tool here> for non-determinism?  Addressing issues as they come up sounds like a lot less effort.

-- Dan

[Dirk Pranke]

On Wed, Dec 28, 2016 at 10:49 AM, Dan Beam <db...@chromium.org> wrote:

Do we audit every change to clang or gcc or <insert build tool here> for non-determinism?  Addressing issues as they come up sounds like a lot less effort.

Yes, we have builders on the FYI waterfall that check to make sure the builds are (and stay) deterministic. We're looking at adding them to the CQ as well,

though we haven't decided for sure about that yet.

-- Dirk

[Dan Beam]

Sweet!  We'd be happy to integrate with these builders.

-- Dan

 

-- Dirk

[Demetrios Papadopoulos]

Hi Chromium devs,

Thanks for all the discussion. Following up on the summary sent on Dec 22nd, I am

moving forward with:

• open-source-third-party-reviews review.
  
• secu...@chromium.org review
  
• Setting up a chromium.googlesource.com repository to hold the NPM dependencies.

Thank you,

Demetrios