Should we minify Javascript resources on desktop platforms?

[Anthony Berent]

I recently implemented minification of Javascript resources on Android, removing all comments and unnecessary whitespace, but not otherwise modifying the Javascript. This has reduced the size of the Chrome resources.pak file (and hence the Android APK) by 600KB. See https://codereview.chromium.org/2179033002/.

On desktop versions of Chrome I believe this would have a far greater effect (simply because there are many Javascript resources that are only included in desktop versions of Chrome), however the tool I am using to do the minification is the Closure compiler, which is a Java executable, so would require Java support on all build and development machines. Is it acceptable to add this requirement?

[Alexei Svitkine]

I believe we've seen evidence that smaller download size directly affects number of successful downloads. So I think that order of KB savings would be worthwhile.

However, I believe in the past, there's been push backs on adding a Java requirement to the build system.

My personal opinion is that if you could make it be transparent to devs (i.e. depot tools would install and update Java - just like e.g. it does clang or other build dependencies), it shouldn't be much of a burden on developers. If you can manage that, then I think it would definitely be worth it.

(But I don't speak for the project as a whole and others may have different opinions.)

On Tue, Aug 9, 2016 at 8:16 AM, Anthony Berent <abe...@chromium.org> wrote:

I recently implemented minification of Javascript resources on Android, removing all comments and unnecessary whitespace, but not otherwise modifying the Javascript. This has reduced the size of the Chrome resources.pak file (and hence the Android APK) by 600KB. See https://codereview.chromium.org/2179033002/.

On desktop versions of Chrome I believe this would have a far greater effect (simply because there are many Javascript resources that are only included in desktop versions of Chrome), however the tool I am using to do the minification is the Closure compiler, which is a Java executable, so would require Java support on all build and development machines. Is it acceptable to add this requirement?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Nico Weber]

I don't think we want to add a dependency on Java.

I believe dbeam has been looking at this some (there's an FYI bot or some such?)

[Ken Rockot]

Seems worth doing, but I think it should be done behind an optional (default-off) build setting so developers generally don't have to care about Java.

[Lei Zhang]

If all we are doing is stripping out comments and whitespace, can we
use a Python implementation on the desktop to avoid the Java
dependency?

[Anthony Berent]

Yes, we could make it optional; it is already controlled internally by a flag, so it would be quite easy to add a GN argument for this.

I did wonder about using a python implementation, but I couldn't find an existing one, and these things always end up more complicated than one expects, mainly because of false matches within quoted strings (and quotes within comments etc.).

[Brett Wilson]

On Tue, Aug 9, 2016 at 8:25 AM, Alexei Svitkine <asvi...@chromium.org> wrote:

I believe we've seen evidence that smaller download size directly affects number of successful downloads. So I think that order of KB savings would be worthwhile.

For this case we care about the compressed size. I would expect the compression algorithm to do much of the same job of the minifier. We should get final numbers on the minified vs. non-minified compressed installer before making a decision to do this.

Brett

[Scott Violet]

What are the effects of this change on the various perf bots we have?

-Scott

On Tue, Aug 9, 2016 at 8:16 AM, Anthony Berent <abe...@chromium.org> wrote:

[Elliott Sprehn]

Note that this also changes the memory mapped for the pak file, and also the amount of memory we have to transport from the browser to the renderer to load the resource, and the amount code that v8 then needs to process, and the amount of memory blink needs to keep around for it.

I don't think we should use a python minifier. We should either use something that depends on node.js (which is what the community uses) or Closure (Java) which is what Google uses internally.

I don't think the push back on Java here is reasonable. Anyone who's on any sherrifing rotation needs Java, anyone who builds Clank (which should be *all* Google Chrome engineers) needs Java. :)

[Mike Frysinger]

we have java in the CrOS SDK specifically for closure, so we'd be fine with turning it on.  it would also mean shrinking our rootfs usage.

-mike

[Christian Biesinger]

Why do sheriffs need Java? I didn't need it on my last shift I don't think?

-Christian

[Lei Zhang]

Curious, why not a Python minifier? I have no idea how minifiers
compare in terms of speed / features / correctness. I suggested a
Python minifier simply because the existing process involves GRIT,
which is Python, and we already use Python for other tools, so it is
readily available.

[Mike Frysinger]

i think it's more "can you find an existing one written in python that works well and is maintained".  the existing ones (noted earlier in the thread) with large user-bases aren't written in python.

-mike

[Matthew Menke]

I think it's worth noting that minified Javascript may actually increase the size of updates.  Javascript normally doesn't change that much, however, if you change a few bytes at the top of a minified Javascript file, variable names for a significant chunk of the rest of the file may change.  Since our update system (On desktop, at least) relies on deltas, this could theoretically result in bigger updates.

[Anthony Berent]

On Wed, 10 Aug 2016 at 18:01 Matthew Menke <mme...@google.com> wrote:

I think it's worth noting that minified Javascript may actually increase the size of updates.  Javascript normally doesn't change that much, however, if you change a few bytes at the top of a minified Javascript file, variable names for a significant chunk of the rest of the file may change.  Since our update system (On desktop, at least) relies on deltas, this could theoretically result in bigger updates.

The minification we are currently doing on Android is only stripping whitespace and comments, not changing variable names or otherwise changing the code. As such this should not be an issue.

Of course, the first update after minification probably will generate a large delta.

It might be possible to do more than this, but doing so would require considerably more work, and it is unclear whether it is worth the effort. 

[Elliott Sprehn]

On Aug 10, 2016 5:02 AM, "Christian Biesinger" <cbies...@chromium.org> wrote:

Why do sheriffs need Java? I didn't need it on my last shift I don't think?

Because the presubmit if you touch a Java file (revert, disable a test) requires it. I guess you could skip all the checks but that seems pretty silly to avoid having Java when all sheriffs are Chrome engineers, and all Chrome engineers should be setup to build clank.

 

-Christian

- E 

[Dan Beam]

Y'all probably know this BUT here's more context on web resource† compression:

Closure compiler

• Language: Java
  
• Used on: all platforms (typechecking) via fyi / try bots, local install + script (typechecking)
• Used on: Android (minfication)

Closure compiler is a Google-maintained JavaScript compiler written in Java.  For a few years we've used it to typecheck JS files in Chrome (yes, those @jsdoc tags do something!).  This is solely static analysis: we don't create any build artifacts / directly change the code with this tool.

Recently, aberent@ added lossy simple minification on Android (already requires Java).  This saves a bit on disk (limited on Android) and probably less in the installer (apks are presumably compressed).

Node.js-based tools

• Language: JavaScript
• Used on: desktop platforms via local install + script + check in results

We use node.js and a few npm modules to update Polymer (npm modules: bower, crisper) and make the downloads page faster (npm modules: vulcanize) and soon history.  These are the standard Polymer tools, also used by web developers.

We'd like to start lossy compression via uglify real soon.

gzip/brotli

• Languages: C/C++/Python
  
• Used on: Android (for now) through build process
  

smaier@ and agrieve@ have used lossless brotli/gzip compression when packing debug-only chrome:// page resources on Android.  I think this is mainly for on-disk size.  I hope to improve the guts soon.

tl;dr - From what I gather, Chrome does not yet harness

• lossy compression (i.e. minfication via Closure or uglify) because:
  
• the savings are uncertain and vary contextually (installer, on-disk, resource loading, etc.)
• initial and continual breakage will probably occur
• it'd require archiving/loading source maps (similar to C++ symbols) for debugging
• there's constant pushback against adding Java or node to the build toolchain

• lossless compression because:
• uncompressing resources dynamically has not been tested in performance critical / user-facing situations yet

• both types of compression fully because:
• the effect on incremental update size is unknown

-- Dan

† And we haven't even started on CSS or HTML yet!

--

[PhistucK]

Plus, the Developer Tools team seems to use a Python based whitespace-only minifier. Maybe just leverage that one (in hopes that one day all of you switch to Closure Compiler with ADVANCED_OPTIMIZATIONS)?

☆PhistucK

---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Mike Frysinger]

or just switch to the closure compiler now and be done.  installing only a JRE (not the full JDK) on systems nowadays is pretty trivial (especially for Ubuntu systems), and a drop in the bucket compared to all the other tools & source trees that people have to install now in order to build Chromium.  this discussion seems more of a general/political aversion to Java than it is having any actual technical problems -- we aren't talking about anyone having to look at, understand, write, or even think about Java code.

-mike

[Anthony Berent]

Nico - you are the only person, so far, who has explicitly objected to depending on Java; can you expand on your objection?

Scott - what is your concern about the perf bots? I don't see why this change should have any effect on them, other than slightly reducing memory usage. 

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Scott Violet]

My concern is I'm wondering what the performance impact of this change
is. It would be good to get concrete numbers. It may not effect
performance at all, but it would be nice to know that.

-Scott

[Ken Rockot]

(from the right address)

We already have build configs which require Java, obviously. Minification doesn't seem like something which needs to be on in default build configs for developers (isn't Closure kind of slow?), so the state of the world wouldn't really change with respect to Java dependency.

Why not move forward with enabling minification behind a GN flag and then we can explore specific concerns over size and performance?

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Scott Graham]

On Thu, Aug 11, 2016 at 1:25 AM, Anthony Berent <abe...@chromium.org> wrote:

Nico - you are the only person, so far, who has explicitly objected to depending on Java; can you expand on your objection?

I also object to installing Java. Especially on Windows, it's a horrible user experience.

In general, if it's going to be another thing we all have to install to complete some flavour of build, update periodically, and generally care about, there's a pretty high bar to demonstrate clear end-user benefit before imposing that on all devs.

 

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Alexei Svitkine]

I'm hoping it can be done in a way where you don't need to have a system install of it (since that would popup for updates and I agree would be a horrible user experience).

I'm pretty sure it's possible to just have an on-disk JRE install that could be fetched and updated by depot tools (just as python exe and clang are) and doesn't impact the system except when it needs to be run by the specific build step that needs it.

(Not sure if that's the plan, but if we consider doing it I strongly think it should be done the above way rather than requiring devs to do an install themselves and maintain it.)

[Dirk Pranke]

I believe that installing the OpenJDK still requires you to accept a license agreement, and so it can't be done as seamlessly as python and clang.

-- Dirk

[Hans Wennborg]

Just out of curiosity, what are our largest JS resources? 600 KB of
whitespace and comments sounds like a lot.

[Brendan Kenny]

FWIW there is now a version of Closure Compiler that is pure javascript (compiled to JS via GWT):

https://github.com/google/closure-compiler-js

Slower than the Java version and not all configuration is exposed yet, but it may be worth it to avoid the Java dependency (it also currently runs in node, but with a little work it could probably run in d8 and avoid the node dependency if desired).

[Rachel Blum]

Maybe we should talk about node.js again, then... (We do need it to vulcanize as well). 

[Peter Kasting]

On Thu, Aug 11, 2016 at 8:55 AM, Scott Graham <sco...@chromium.org> wrote:

On Thu, Aug 11, 2016 at 1:25 AM, Anthony Berent <abe...@chromium.org> wrote:

Nico - you are the only person, so far, who has explicitly objected to depending on Java; can you expand on your objection?

I also object to installing Java. Especially on Windows, it's a horrible user experience.

I too object to adding a Java dependency.

I have no objection to adding this under an off-by-default flag, in such a way that developers who don't turn it on don't see any changes regarding Java requirements.  I think the idea of minification before we ship to the public, but not in default developer build configurations, has promise.

Another possibility, depending on what resources we have and how often they're touched, would be manual minification like we have with .pngs.

PK

[Ojan Vafai]

Just an additional data point I haven't heard mentioned is the closure compiler does considerably more than minification, e.g. dead code elimination,  function inlining, etc. I don't know what the code we are compiling looks like, but back when I worked on a web product at Google, the closure compiler got considerably more savings than minifiers possibly could, on the order of 10-30% iirc. 

To Scott's question about performance, minification should be a strict performance improvement with some rare exceptions. Obviously function inlining and whatnot that the closure compiler doest should be a performance improvement as well. I'd expect to see modest perf gains in js parsing and execution time.

Imo, we should have really strong reasons for not using closure compiler. If someone can work out how to make deps/gclient handle the Java transparently, then I think it's definitely worth it. 

--

[Kevin Marshall]

Does the Closure Compiler still work well when the JS isn't written with Closure functions (e.g. goog.provide/goog.require/goog.export*) and comment @annotations?

[Ojan Vafai]

What I know of the closure compiler is definitely dated. It used to. Of course, with type annotations it worked better though.

[PhistucK]

It does. I never used goog.foos, I just annotated the code and used my own ways.

☆PhistucK

[PhistucK]

Note that it only does those great things in ADVANCED_OPTIMIZATIONS mode, which is not on the table for now, unfortunately. I agree that it is worth the effort. Especially for the great mass of code that the Developer Tools feature has.

And since the JavaScript version of it is now available (that is a pretty recent development), you might not need Java at al. It depends on the flags you need.

☆PhistucK

[Anthony Berent]

I don't work Fridays, so am a little bit behind with this discussion.

My current proposal is to implement it for all platforms, but to make it optional through a GN flag. This should certainly be enabled on official release builds, and on at least some of the waterfall bots for each platform. For now it probably makes sense to disable this by default on, at least, Windows development machines.

I still need to discuss with infra installing Java on the (non-Android) build machines.

sky@ - This has landed on Android, and there is no evidence of any change in performance.  In theory there might be a marginal performance improvement since disk reads and memory usage will go down, but I would be surprised if this is measurable.

rokot@ - Yes, Closure is slow, however, at least for Android, rather to my surprise, ninja parallelization of the build hid this, so adding this made no difference to the total build time. I did some investigation of why Closure is slow (around 1 second per compilation), and most of the time in Closure is spent starting Java VMs. We actually have a Closure wrapper that allows multiple compilations without restarting the Java VM, so if build performance does become a problem then we could try using this, however doing so would complicate the interface between grit and Closure.

schottmg@ and others - Since I work mainly on Clank, I don't have a Windows machine, and have not investigated Java installation on Windows. For Android we have a similar problem with licences with the Android SDK. The install is run from a script, but the first time it is run developers have to accept the license. Oddly I don't remember ever having to accept any Java license on my Linux desktop, but I may have simply forgotten.

hans@ - Even on Android we actually had more than 600kB of comments and whitespace (see https://docs.google.com/document/d/1XUte4m_wkPwQ9I-MfVZfBhksoL0E2_7Eb7N-PkXQKvw/edit?usp=sharing), but some of this is in CSS and HTML files so is not currently being removed. On desktop I would expect the total saving to be far more than 600KB, the Android resource file is about 20% of the size of, for example, the Linux resource file.

After a bit of investigation (and playing with Linux scripting) it seems;we have, in total, just under 1000 unique javascript resource files (as reported by grit buildinfo), for a total of 9.5MB of javascript resources. The largest single javascript resource file is 137KB. This is, however, across all target platforms, and no one platform will include all the resources.

bckenny@ - How mature is the JS version of the Closure compiler? I can see this as a possible route forward if we can't find a good way managing Java installation on Windows.

groby@ - From what I can find on Node.js minification, it always seems to call out to some other external tool. Is there a Node.js minifier that is pure Javascript?

pkasting@ - We have just under 1000 relevant javascript files, and just over 900 commits to them since the beginning of the year, so manual minification seems impractical. 

ojan@ - Yes, the closure compiler can do considerably more than this; it has three main optimization modes:

WHITESPACE_ONLY - This is the option we are using now, it simply removes unnecessary whitespace and comments, and as such is low risk. Using the developer tools' pretty-printing it is easy, for debugging, to get back to something that looks very like the original code (although, of course, without comments).

SIMPLE_OPTIMIZATION - I did some very brief experiments with this, and it didn't seems to gain much over WHITESPACE_ONLY. It also made debugging of the Javascript much more difficult, since it changes local variable names etc.. I didn't investigate it far enough to know if it would otherwise work correctly.

ADVANCED_OPTIMIZATION - This would, probably, produce major further size reductions. To get this working correctly we would, however, need to add a great many annotations. I am not sure how we would test that we had annotated the Javascript correctly, and since its output looks nothing like its input, it would be almost impossible to debug our Javascript using Chrome developer tools.

[Mike Frysinger]

wrt debugging, doesn't source maps address that ?  and i thought closure could produce source maps automatically.

-mike

[Anthony Berent]

On Mon, 15 Aug 2016 at 15:49 Mike Frysinger <vap...@chromium.org> wrote:

wrt debugging, doesn't source maps address that ?  and i thought closure could produce source maps automatically.

Sadly no, at least not at the moment. The problem is that, before compiling the files we have to process the <include> and <if...> preprocessing statements that we use in these files, so what we are actually compiling are temporary files that may already be significantly different from the original sources, hence meaning that the source maps don't map to anything useful. I don't know if it would be possible to add a pass to the closure compiler to handle these while correctly maintaining the source maps (rather than preprocessing the files); however if it was this would probably be the best solution.

[PhistucK]

I am not sure I understand the complication here - source maps can be created with the sources (that are used for compilation) embedded within them (I am not sure Closure Compiler currently supports generating this, but I will check if it simplifies the approach). You could save the source map on a Google Cloud Storage bucket and point the minified files to those source maps.

☆PhistucK

[Jeremy Roman]

On Mon, Aug 15, 2016 at 11:15 AM, Anthony Berent <abe...@chromium.org> wrote:

On Mon, 15 Aug 2016 at 15:49 Mike Frysinger <vap...@chromium.org> wrote:

wrt debugging, doesn't source maps address that ?  and i thought closure could produce source maps automatically.

Sadly no, at least not at the moment. The problem is that, before compiling the files we have to process the <include> and <if...> preprocessing statements that we use in these files, so what we are actually compiling are temporary files that may already be significantly different from the original sources, hence meaning that the source maps don't map to anything useful. I don't know if it would be possible to add a pass to the closure compiler to handle these while correctly maintaining the source maps (rather than preprocessing the files); however if it was this would probably be the best solution.

Alternatively, maybe grit could be modified to emit a source map, which could then be composed with the one emitted by closure-compiler.

[Dimitri Glazkov]

node.js-based minification sounds pretty appealing to me, at least in principle.

:DG<

[Anthony Berent]

On Mon, 15 Aug 2016 at 16:36 PhistucK <phis...@gmail.com> wrote:

I am not sure I understand the complication here - source maps can be created with the sources (that are used for compilation) embedded within them (I am not sure Closure Compiler currently supports generating this, but I will check if it simplifies the approach). You could save the source map on a Google Cloud Storage bucket and point the minified files to those source maps.

That sounds like a good option if it works.  

[Torne (Richard Coles)]

Yeah, just invoking uglifyjs with node (like a lot of the web ecosystem do) seems pretty reasonable - but people are still going to have to install node I guess? We can probably manage to depend on a relatively old version and have the one in the ubuntu repos work, but mac/win i guess have to install from elsewhere..

Separately: OpenJDK doesn't require a clickthrough license, but there don't appear to be ready-made binaries for it on windows/mac; it's under a sufficiently open license that Linux distros can build their own and package it, but the openjdk site mostly just directs people toward installing Oracle's JDK instead.

[Elliott Sprehn]

On Mon, Aug 15, 2016 at 11:53 AM, Torne (Richard Coles) <to...@chromium.org> wrote:

Yeah, just invoking uglifyjs with node (like a lot of the web ecosystem do) seems pretty reasonable - but people are still going to have to install node I guess? We can probably manage to depend on a relatively old version and have the one in the ubuntu repos work, but mac/win i guess have to install from elsewhere..

In general we need node for a bunch of reasons, vulcanize, uglify, karma, etc. I don't think we want to depend on the system one, we should have it checked in to an infra repo and then we DEPS it in. We already do that for node_modules/ for various things.

[Tyler Breisacher]

From our (Closure Compiler team) point of view, we would like to see Chrome use Closure Compiler for minification (either running in Java or in node.js). We've worked with dbeam@ a lot already to use Closure Compiler as a typechecker, so it would be great to see the same tool used for minification/dead code elimination as well. Using Uglify may be easier to get set up in the short term but I would hope that Chrome will eventually switch to using Closure, and we'd be happy to help get source maps working smoothly and handle other issues that may come up. 

[Dan Beam]

On Mon, Aug 15, 2016 at 8:55 AM, Elliott Sprehn <esp...@chromium.org> wrote:

On Mon, Aug 15, 2016 at 11:53 AM, Torne (Richard Coles) <to...@chromium.org> wrote:

Yeah, just invoking uglifyjs with node (like a lot of the web ecosystem do) seems pretty reasonable - but people are still going to have to install node I guess? We can probably manage to depend on a relatively old version and have the one in the ubuntu repos work, but mac/win i guess have to install from elsewhere..

In general we need node for a bunch of reasons, vulcanize, uglify, karma, etc. I don't think we want to depend on the system one, we should have it checked in to an infra repo and then we DEPS it in. We already do that for node_modules/ for various things.

 

Separately: OpenJDK doesn't require a clickthrough license, but there don't appear to be ready-made binaries for it on windows/mac; it's under a sufficiently open license that Linux distros can build their own and package it, but the openjdk site mostly just directs people toward installing Oracle's JDK instead.

On Mon, 15 Aug 2016 at 16:47 Dimitri Glazkov <dgla...@chromium.org> wrote:

node.js-based minification sounds pretty appealing to me, at least in principle.

Adding to what dglazkov@ and esprehn@ have said: using node.js directly in the build toolchain would be immensely helpful for a number of reasons.

1) The web is embracing node; one less language to learn if your build scripts or server are in the same language.  All of Polymer's tools are in node, for instance (get polymer = bower, make it faster = vulcanize, CSP compliance = crisper, build = polybuild).

This is relevant to Chrome because we show a number of important UIs in web technologies.  Those UIs are often less modernly written/tooled than they could be because of pushback against node.

2) Node often finishes before the JVM begins :P, especially for smaller tasks.

3) Adding node to our build toolchain would reduce the size of our source checkout.  We currently check in a bunch of locally-node-processed artifacts because we can't use node to build (we bower+crisp polymer and check it into third_party/polymer, we vulcanize downloads and soon history and check them in to chrome/browser/resources).  We can't make the new settings page fast yet because each grit <if> would need it's own precompiled bundle, and we really don't want check in 600KB x each variable in an <if> (i.e. mac, linux, chromeos, _google_chrome, etc.).

4) If we decide to use a node.js-based minifier, it'd save us 200KB in on-disk space quickly (probably with source map support) by running it on the History and Downloads vulcanize bundles.

The WebUI team is working on a larger list of reasons why node.js would be useful, and a list of responses to those that have objected in the past (lots has changed over the last couple years).  I hope to follow up with that soon (next couple days), we're just full investigating some options to things like vulcanize (server push, for example) to hone what we'd need node for.

-- Dan

[Dan Beam]

On Tue, Aug 16, 2016 at 12:12 PM, Dan Beam <db...@chromium.org> wrote:

On Mon, Aug 15, 2016 at 8:55 AM, Elliott Sprehn <esp...@chromium.org> wrote:

On Mon, Aug 15, 2016 at 11:53 AM, Torne (Richard Coles) <to...@chromium.org> wrote:

Yeah, just invoking uglifyjs with node (like a lot of the web ecosystem do) seems pretty reasonable - but people are still going to have to install node I guess? We can probably manage to depend on a relatively old version and have the one in the ubuntu repos work, but mac/win i guess have to install from elsewhere..

In general we need node for a bunch of reasons, vulcanize, uglify, karma, etc. I don't think we want to depend on the system one, we should have it checked in to an infra repo and then we DEPS it in. We already do that for node_modules/ for various things.

 

Separately: OpenJDK doesn't require a clickthrough license, but there don't appear to be ready-made binaries for it on windows/mac; it's under a sufficiently open license that Linux distros can build their own and package it, but the openjdk site mostly just directs people toward installing Oracle's JDK instead.

On Mon, 15 Aug 2016 at 16:47 Dimitri Glazkov <dgla...@chromium.org> wrote:

node.js-based minification sounds pretty appealing to me, at least in principle.

Adding to what dglazkov@ and esprehn@ have said: using node.js directly in the build toolchain would be immensely helpful for a number of reasons.

1) The web is embracing node; one less language to learn if your build scripts or server are in the same language.  All of Polymer's tools are in node, for instance (get polymer = bower, make it faster = vulcanize, CSP compliance = crisper, build = polybuild).

This is relevant to Chrome because we show a number of important UIs in web technologies.  Those UIs are often less modernly written/tooled than they could be because of pushback against node.

2) Node often finishes before the JVM begins :P, especially for smaller tasks.

3) Adding node to our build toolchain would reduce the size of our source checkout.  We currently check in a bunch of locally-node-processed artifacts because we can't use node to build (we bower+crisp polymer and check it into third_party/polymer, we vulcanize downloads and soon history and check them in to chrome/browser/resources).  We can't make the new settings page fast yet because each grit <if> would need it's own precompiled bundle, and we really don't want check in 600KB x each variable in an <if> (i.e. mac, linux, chromeos, _google_chrome, etc.).

4) If we decide to use a node.js-based minifier, it'd save us 200KB in on-disk space quickly (probably with source map support) by running it on the History and Downloads vulcanize bundles.

Oh yeah, last but not least:

5) Currently all of node-based things are manual scripts.  It's harder to automate them and it's easy to forget to re-run your script.[1]

-- Dan

[1] https://codereview.chromium.org/2252633002/#msg12

[Dirk Pranke]

On Tue, Aug 16, 2016 at 12:12 PM, Dan Beam <db...@chromium.org> wrote:

On Mon, Aug 15, 2016 at 8:55 AM, Elliott Sprehn <esp...@chromium.org> wrote:

On Mon, Aug 15, 2016 at 11:53 AM, Torne (Richard Coles) <to...@chromium.org> wrote:

Yeah, just invoking uglifyjs with node (like a lot of the web ecosystem do) seems pretty reasonable - but people are still going to have to install node I guess? We can probably manage to depend on a relatively old version and have the one in the ubuntu repos work, but mac/win i guess have to install from elsewhere..

In general we need node for a bunch of reasons, vulcanize, uglify, karma, etc. I don't think we want to depend on the system one, we should have it checked in to an infra repo and then we DEPS it in. We already do that for node_modules/ for various things.

 

Separately: OpenJDK doesn't require a clickthrough license, but there don't appear to be ready-made binaries for it on windows/mac; it's under a sufficiently open license that Linux distros can build their own and package it, but the openjdk site mostly just directs people toward installing Oracle's JDK instead.

On Mon, 15 Aug 2016 at 16:47 Dimitri Glazkov <dgla...@chromium.org> wrote:

node.js-based minification sounds pretty appealing to me, at least in principle.

Adding to what dglazkov@ and esprehn@ have said: using node.js directly in the build toolchain would be immensely helpful for a number of reasons.

1) The web is embracing node; one less language to learn if your build scripts or server are in the same language.  All of Polymer's tools are in node, for instance (get polymer = bower, make it faster = vulcanize, CSP compliance = crisper, build = polybuild).

This is relevant to Chrome because we show a number of important UIs in web technologies.  Those UIs are often less modernly written/tooled than they could be because of pushback against node.

2) Node often finishes before the JVM begins :P, especially for smaller tasks.

3) Adding node to our build toolchain would reduce the size of our source checkout.  We currently check in a bunch of locally-node-processed artifacts because we can't use node to build (we bower+crisp polymer and check it into third_party/polymer, we vulcanize downloads and soon history and check them in to chrome/browser/resources).  We can't make the new settings page fast yet because each grit <if> would need it's own precompiled bundle, and we really don't want check in 600KB x each variable in an <if> (i.e. mac, linux, chromeos, _google_chrome, etc.).

Given that we'd have to bundle Node as part of our checkout dependencies, would we really save enough space to offset the size increase for node itself? 

4) If we decide to use a node.js-based minifier, it'd save us 200KB in on-disk space quickly (probably with source map support) by running it on the History and Downloads vulcanize bundles.

The WebUI team is working on a larger list of reasons why node.js would be useful, and a list of responses to those that have objected in the past (lots has changed over the last couple years).  I hope to follow up with that soon (next couple days), we're just full investigating some options to things like vulcanize (server push, for example) to hone what we'd need node for.

I have no particular love for Node, but it does seem like the way thing are headed and my Node-enamored colleagues seem disinclined to use (the far preferable ;) Python instead, so I'm okay with us adding Node as long as we do it properly. And, I do think it would be better and more useful to require Node than it would be to require Java.

I think we use Closure in the iOS build; can we use the Node version for that as well, so we can drop the Java requirement for iOS?

Of course, we'll still need Java on Android regardless.

-- Dirk

[Dan Beam]

On Tue, Aug 16, 2016 at 12:22 PM, Dirk Pranke <dpr...@chromium.org> wrote:

On Tue, Aug 16, 2016 at 12:12 PM, Dan Beam <db...@chromium.org> wrote:

On Mon, Aug 15, 2016 at 8:55 AM, Elliott Sprehn <esp...@chromium.org> wrote:

On Mon, Aug 15, 2016 at 11:53 AM, Torne (Richard Coles) <to...@chromium.org> wrote:

Yeah, just invoking uglifyjs with node (like a lot of the web ecosystem do) seems pretty reasonable - but people are still going to have to install node I guess? We can probably manage to depend on a relatively old version and have the one in the ubuntu repos work, but mac/win i guess have to install from elsewhere..

In general we need node for a bunch of reasons, vulcanize, uglify, karma, etc. I don't think we want to depend on the system one, we should have it checked in to an infra repo and then we DEPS it in. We already do that for node_modules/ for various things.

 

Separately: OpenJDK doesn't require a clickthrough license, but there don't appear to be ready-made binaries for it on windows/mac; it's under a sufficiently open license that Linux distros can build their own and package it, but the openjdk site mostly just directs people toward installing Oracle's JDK instead.

On Mon, 15 Aug 2016 at 16:47 Dimitri Glazkov <dgla...@chromium.org> wrote:

node.js-based minification sounds pretty appealing to me, at least in principle.

Adding to what dglazkov@ and esprehn@ have said: using node.js directly in the build toolchain would be immensely helpful for a number of reasons.

1) The web is embracing node; one less language to learn if your build scripts or server are in the same language.  All of Polymer's tools are in node, for instance (get polymer = bower, make it faster = vulcanize, CSP compliance = crisper, build = polybuild).

This is relevant to Chrome because we show a number of important UIs in web technologies.  Those UIs are often less modernly written/tooled than they could be because of pushback against node.

2) Node often finishes before the JVM begins :P, especially for smaller tasks.

3) Adding node to our build toolchain would reduce the size of our source checkout.  We currently check in a bunch of locally-node-processed artifacts because we can't use node to build (we bower+crisp polymer and check it into third_party/polymer, we vulcanize downloads and soon history and check them in to chrome/browser/resources).  We can't make the new settings page fast yet because each grit <if> would need it's own precompiled bundle, and we really don't want check in 600KB x each variable in an <if> (i.e. mac, linux, chromeos, _google_chrome, etc.).

Given that we'd have to bundle Node as part of our checkout dependencies, would we really save enough space to offset the size increase for node itself? 

It's unclear exactly how much space would be used.  It depends on how many vulcanize bundles we'd require and how we share common code between them (i.e. if a <custom-element> is shared on 2 pages, we make a shared bundle for them).  Ultimately, because Settings bundles are like 1MB each with lots of variables, it's possible all the bundles combined are larger than Node (which is 25MB on Linux x64 as a data point).

BUT a node binary isn't similar to source code like these bundles are.  A node binary wouldn't be:

• indexed by code search
• show up / yield duplicates in grep
• blow up our presubmit tools (i.e. style checks, copyright scanning)
• accidentally get edited

etc. like the current bundles do.  ^ These are all things we could make better, but I was always hoping we'd just ... add node to the toolchain instead of legitimizing checking in these artifacts.

4) If we decide to use a node.js-based minifier, it'd save us 200KB in on-disk space quickly (probably with source map support) by running it on the History and Downloads vulcanize bundles.

The WebUI team is working on a larger list of reasons why node.js would be useful, and a list of responses to those that have objected in the past (lots has changed over the last couple years).  I hope to follow up with that soon (next couple days), we're just full investigating some options to things like vulcanize (server push, for example) to hone what we'd need node for.

I have no particular love for Node, but it does seem like the way thing are headed and my Node-enamored colleagues seem disinclined to use (the far preferable ;) Python instead, so I'm okay with us adding Node as long as we do it properly. And, I do think it would be better and more useful to require Node than it would be to require Java.

I think we use Closure in the iOS build; can we use the Node version for that as well, so we can drop the Java requirement for iOS?

It sounds like it's worth a try to me.

-- Dan

[Anthony Berent]

I let this discussion drop some time ago, since it was always a background task for me, and I had higher priorities. I would, however, now like to restart it.

(First question, has anybody done anything on this since we last discussed it?)

This is my understanding of where we left it:

• Nobody is objecting in principle to stripping Javascript whitespace and comments from, at least, official release builds on all platforms.
• The Android builds are currently using the Closure compiler to do the stripping, but this requires Java on the build machines, which we only currently require for Android build.
• There are some alternatives, in particular uglifyJS, that don't use Java but these are untested and would presumably require third party approval before we could use them.

Have I summarised this correctly?

What exactly are the objections to requiring Java on all build machines?

For uglifyJS, would we need to install anything new on the build machines, or could it all be in the Chromium repositories?

How do we move this forward?

[PhistucK]

Though Closure Compiler can be JavaScript based, so that is a better alternative, since it is tested.

☆PhistucK

[Anthony Berent]

Doesn't any Javascript solution require Nodejs (or its equivalent) on the build machines? Doesn't this have the same problems as requiring a Java runtime?

[Nico Weber]

Yes.

[Elliott Sprehn]

node.js is not quite the same as java, we can build it from source if needed and don't need to go through any installer to have it available on windows, we could just unpack it into a directory. We're going to need node.js available very soon for devtools, web ui, and testing infra. The python vulcanize and checking in compiled binaries is not a long term solution, we're going to need to start including node.js in the build requirements.

[Anthony Berent]

Thanks, that makes sense. Do we have, or are we getting, third party approval for Nodejs, and for either uglifyJS or the JS version of the Closure Compiler?

[Dirk Pranke]

Good question(s). I don't know and will check.

We actually have a hook in place already to optionally download node binaries so we can run some tests for devtools.

-- Dirk

[Mike Frysinger]

you can build java (OpenJDK) from source as well.  but for node.js, i'm not sure you're looking deep enough -- node.js requires a v8 vm to run the js.  i suspect you plan on bundling the vm binary and not actually building that from scratch.

i'm not disagreeing that, if we need node.js for other tooling, trying to use that everywhere makes sense.

-mike

[Dirk Pranke]

Correct, we're pulling down node binaries, not building it from scratch as part of a chromium compile. But, there's no license agreement to accept.

The java installs we use for Android and iOS are actually the oracle-distributed binaries (hence the license agreement). 

Do we know that we could build a java from source that would work (and supported by Android for use with their SDK)? That would certainly be an

interesting data point ...

-- Dirk

[Torne (Richard Coles)]

Current versions of OpenJDK work fine with the Android SDK. (since 1.7)

I don't think this is officially recommened/supported, but it's widely relied on anyway, and the actual Android AOSP build *requires* OpenJDK and rejects the oracle JDK explicitly.

[Mike Frysinger]

seems like that mode would be feasible with OpenJDK too which is what Android is using with N+ instead of Oracle's JDK.  can we re-use Android's work in this space ?

-mike

[Rachel Blum]

+dpapad who's currently figuring out how to integrate Node.js into the webui toolchain

[Dirk Pranke]

Ah. Based on your response and Torne's, that definitely sounds worth exploring.

vapier@, mind finding out if there's already a bug open to investigate switching and, if not, file one? One less prompt during setup and a switch to a fully open toolchain would be nice things.

Though we've also wandered afield from the thread, so let's end this digression on that note :).

-- Dirk

[Mike Frysinger]

there is crbug.com/662467 about converting the bots to Java 8 / OpenJDK.  wrt the Chromium build itself, i hesitate filing bugs as i know little about the clank side of things and what actually needs to be changed.  any parts of Chromium i work on are far far away from Java :).

-mike

[Dirk Pranke]

That bug looks like a good fit, thanks! 

-- Dirk

[Demetrios Papadopoulos]

FYI, I am in the process requesting approval for Nodejs. I have prepared a document for chrome-eng-review with enough context, explaining how it will work (node binaries + npm deps), and I am planning to share this doc publicly very soon (most likely tomorrow). The CL is here. Stay tuned.

--

[dpapad]

Reviving this old-ish thread. Node has been added to the toolchain for quite a while now, and is being used to optimize (bundle) Polymer WebUI pages.

I recently tried minifying Print Preview code with Uglify (which is not using Polymer). The result was a decrease of browser_resources.pak file from 2.8mb to 2.5mb, which seems non negligible to me, even though I suspect these are pre-gzipping gains. If you think these are significant gains, let me know and I'll look into possibly cleaning my experiment CL and sending it for review.

Having said that, I realized that the usage of the <include> tag in Chromium's code to inline files at build time makes the use of other JS tools more complex (definitely a topic for a whole other thread). Eventually I would like to eliminate all usage of the custom <include> and use normal mechanisms like <script src="foo.js">, since <include> was relevant at a time where files could not be bundled together via other means, which is no longer the case. Either way, I'll probably start a new discussion on that when I have a more concrete plan on how to eliminate <include>.

Yes.

<br class="m_-5117706417764818761m_45009514111447

[Dan Beam]

On Fri, Jul 21, 2017 at 4:40 PM, dpapad <dpa...@chromium.org> wrote:

Reviving this old-ish thread. Node has been added to the toolchain for quite a while now, and is being used to optimize (bundle) Polymer WebUI pages.

I recently tried minifying Print Preview code with Uglify (which is not using Polymer). The result was a decrease of browser_resources.pak file from 2.8mb to 2.5mb, which seems non negligible to me, even though I suspect these are pre-gzipping gains.

Note (that I assume most people are aware of but mayyybe not): these are desktop-specific (likely Linux) numbers.  Also, I'd be weary of size measurements without doing a clobber build (i.e. gn clean <out_dir>) between measuring (not sure if you did or not).

If you think these are significant gains, let me know and I'll look into possibly cleaning my experiment CL and sending it for review.

Now that we have node.js in Chromium's build toolchain, maybe we could just minify every .js file before placing it into resources.pak with Uglify (kind of like what Android does w/compiler.jar)?

For what it's worth: I'm all for optimization tools; the details are just super subtle and important.  For example: I preferred lossless gzip before lossy minification because source maps are kind of a pain (as painful as symbols).  Also, we uglify on official builds only or also release, maybe?

Having said that, I realized that the usage of the <include> tag in Chromium's code to inline files at build time makes the use of other JS tools more complex (definitely a topic for a whole other thread). Eventually I would like to eliminate all usage of the custom <include> and use normal mechanisms like <script src="foo.js">, since <include> was relevant at a time where files could not be bundled together via other means, which is no longer the case. Either way, I'll probably start a new discussion on that when I have a more concrete plan on how to eliminate <include>.

+1

Yeah, I think <include> has surpassed its usefulness.  I think it can truly die when ES6 modules land (if they haven't yet).

For what it's worth: <include> satisfies that the code exists 1+ times by pasting it every time, but that's not really want we want most of the time.  There's a bunch duplication with <include>, as folks will speculatively do it just to make their dependencies are satisfied.  There's also situations where we read source files from src/ directly in browser tests that don't honor <include> (because it bypasses Grit) and tools like clang-format don't understand <include> in code (it tries to parse as JS, i.e. "less than" <ident> "greater than" and blows up).

We probably should have bit the bullet and implemented some goog.require() long ago, but there were strong opinions about closure library that prevented this.

-- Dan

[Andrew Grieve]

Not sure about uglify, but I know that jscompiler has always optimized for gzipped size, so I'm sure there are gains to be had over plain gzip. 

Whatever module system you come up with, I'd ask that you design it such that unused modules are detected and removed on a per-platform basis. E.g., Grit currently has a mode where it generates C++ files that registers URL handlers for all files in a .grd. The result of this is that our unused resources logic considers them all as used when that file is compiled, even if they aren't actually used.

[Michael Giuffrida]

On Fri, Jul 21, 2017 at 5:57 PM Dan Beam <db...@chromium.org> wrote:

On Fri, Jul 21, 2017 at 4:40 PM, dpapad <dpa...@chromium.org> wrote:

Reviving this old-ish thread. Node has been added to the toolchain for quite a while now, and is being used to optimize (bundle) Polymer WebUI pages.

I recently tried minifying Print Preview code with Uglify (which is not using Polymer). The result was a decrease of browser_resources.pak file from 2.8mb to 2.5mb, which seems non negligible to me, even though I suspect these are pre-gzipping gains.

Note (that I assume most people are aware of but mayyybe not): these are desktop-specific (likely Linux) numbers.  Also, I'd be weary of size measurements without doing a clobber build (i.e. gn clean <out_dir>) between measuring (not sure if you did or not).

Probably also best to measure the .deb installer size itself (by building "//chrome/installer/linux/debian:stable") or whatever the Windows equivalent is. And doing the internal build along with "is_chrome_branded = true" and "is_official_build = true".

 

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CANNW_Qqfdq5v_w2%3DLp4qq5H21VNesTNJY%2B3NgzkpVJuPGU1E6A%40mail.gmail.com.

[Dan Beam]

On Fri, Jul 21, 2017 at 6:21 PM, Andrew Grieve <agr...@chromium.org> wrote:

Not sure about uglify, but I know that jscompiler has always optimized for gzipped size, so I'm sure there are gains to be had over plain gzip. 

In my experience (though it's corpus-dependent), minify+gzip is better than just gzip, but not by that much.  Just gzip is pretty much always better than just minify, with less costs (no code munging).  That's why I encourage (and worked on) gzip first.

 

Whatever module system you come up with, I'd ask that you design it such that unused modules are detected and removed on a per-platform basis. E.g., Grit currently has a mode where it generates C++ files that registers URL handlers for all files in a .grd. The result of this is that our unused resources logic considers them all as used when that file is compiled, even if they aren't actually used.

It's very easy to wrap the .grdp entries that create URLs in a per-platform <if> tag.  It's harder to determine which platforms use these URLs, but I'm not sure that's a grit-specific issue...

-- Dan

[Anthony Berent]

One problem with using gzip is that one has to unzip the JS files before use. This increases CPU and memory usage at run time. Which it has more effect on depends on how much caching one does, but both are undesirable. See, also, this document which describes an (admittedly Android focused) investigation of this that was done some time ago. 

An additional consideration is that the effectiveness of gziping is depends on what exactly you are trying to reduce. If you are simply trying to reduce the file size on disk then gzip is good. If however you are trying to reduce the data usage when Chrome is installed (or upgraded) it is less good. The issue is that the install image is probably compressed for download (it certainly is on Android), and recompressiong gzip tends to, if anything, expand it.