What is the recommended workflow for using closure library + compiler?

[Andre Tannus]

Hey all,

I'm starting a new project and I want to use closure library + compiler. I've been using a bash+plovr-based setup for ages, but for this project I'd like to figure out how to setup a more up to date npm-tools-based workflow.

I've spend a few hours looking at how to put this together, but I found it hard to navigate the numerous plugin options, plus it seems like a lot of the packages and recipes are pretty outdated. There seems to be grunt, brewer, gulp, webpack and npm options, but I could not get them to play nice...

The closest I got to a working case was using gulp, with which I managed to compile a standalone "hello world" script using the following:

./src/js/index.js

console.log('hello world');

./gulpfile.js

var gulp = require('gulp');

const closureCompiler = require('google-closure-compiler').gulp();

gulp.task('default', function () {

  return gulp.src([

  './src/js/**/*.js'

], {base: './'})

      .pipe(closureCompiler({

          compilation_level: 'SIMPLE',

          warning_level: 'VERBOSE',

          language_in: 'ECMASCRIPT6_STRICT',

          language_out: 'ECMASCRIPT5_STRICT',

          output_wrapper: '(function(){\n%output%\n}).call(this)',

          js_output_file: 'output.min.js'

        }, {

          platform: ['native', 'java', 'javascript']

        }))

      .pipe(gulp.dest('./dist/js'));

});

./dist/js/output.min.js

(function(){

'use strict';console.log("hello world");

}).call(this)

That is fine, but as soon as I try to require something (say goog.crypt.Sha1'), I get in trouble.

Simply passing the entire closure library (via 'node_modules/google-closure-library/closure/goog/**/*.js') as a gulp.src gives me a funny error (in fact 5 of them).

gulp-google-closure-compiler: node_modules/google-closure-library/closure/goog/json/json_perf.js:31: ERR
OR - Variable table declared more than once. First occurrence: node_modules/google-closure-library/closu
re/goog/crypt/bytestring_perf.js 
var table = new goog.testing.PerformanceTable(goog.dom.getElement('perfTable')); 
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I was able to make it work by passing the dependencies manually into gulp.src (goog.crypt.Sha1, goog.crypt.Hash ans base.js), and that worked but specifying every dependency like this is not feasible.

I then tried to get gulp-google-closure-deps to work, but following the documentation here generates and empty deps.js.

Does anyone have a recipe or tutorial I could use? I'm pretty lost and overwhelmed.

Thanks!

[Stephen Hicks]

Unfortunately I don't have much to help with here.  Externally I've had a lot of good experience with the Closure Compiler gulp plugin, but have never tried to get Closure Library to work from gulp.  Internally we use Bazel for all of our builds, and that works great, but it depends on an intermediate step that automatically prunes a flag file to only include sources that are actually (transitively) pulled in, and this is far from trivial at the scale of even a small-to-medium-sized project.

Potentially what needs to happen is that you need to add a google-closure-deps step before passing the sources to gulp to extract this depgraph and then only pass the sources specified by the depgraph.  This is not about generating a deps.js file, but an (ordered?) list of filenames, which I believe should be possible.  I'll bring this to John Plaisted's attention, since he's the most recent person to touch the deps stuff.

[John Plaisted]

I think the main issue here is our npm packages has some tests it shouldn't :/ Those "perf" files are tests. If we didn't have those you could glob the entire library to the compiler. Granted, the compiler should also be pruning by default, and so those files should get pruned and then the type checker won't complain. idk why the gulp compiler doesn't prune.

What exactly did you try with google-closure-deps and how did it not work? Evidently something is unclear, and we at least need to update documentation.

So I think you have several options to get this to work.

- A pass in the files in dependency order, meaning the compiler doesn't need to do this.

- Ask the compiler to do it for you.

Option 1:

fwiw I can do the following to get just a list of needed files with google-closure-deps.

// Assuming in root of closure's git repo:

const {parser, depGraph} = require('google-closure-deps');

// NEW FEATURE: Parse the existing deps.js file, no need to scan all files ourselves.

const deps = parser.parseFile('closure/goog/deps.js').dependencies;

// Note: when we can a deps.js file these don't know what the path on disc is

// of these deps, they only know their closure relative path. So if we tell them

// explicitly what the path to closure is, they can resolve the path on disc.

deps.forEach(d => d.setClosurePath(process.cwd() + '/closure/goog'));

// TODO add more of your own files to "deps" here

// The example on github scans a fake base.js directly with parseText. But we

// want a real one.

// Now that it is set up we can make a dep graph:

const graph = new depGraph.Graph(deps);

// Now we can try getting ever file needed from an entry point (meaning

// we need an entry point for our program). Let's try goog.log:

const glog = deps.find(d => d.closureRelativePath === 'log/log.js');

console.log(graph.order(glog).map(d => d.path).join('\n'));

Option 2:

The compiler can prune the files for you. Which is basically what the above is doing too. It should default to doing this... idk why the gulp version doesn't. Try setting this option explicitly maybe?

https://github.com/google/closure-compiler/blob/971e95aa97353427c83b67d4c3722ec027fbc7af/src/com/google/javascript/jscomp/CommandLineRunner.java#L796

===============

I don't think you need to make a deps.js file (unless you want to debug outside the compiler). But the most basic deps.js file is generated like so:

$ closure-make-deps --closure-path closure/goog --file closure/goog/deps.js --file myfile.js

goog.addDependency('../../myfile.js', ['baz.bar'], ['goog.log'], {'lang': 'es6', 'module': 'goog'});

Note that I pass in --file closure/goog/deps.js for the same reasons above - by default it will try to validate your requires are valid, so it needs to know about goog.log either by reading in that file directly OR seeing it in a deps.js file. I could also just do this to disable validation:

$ closure-make-deps --closure-path closure/goog --validate false --file myfile.js

goog.addDependency('../../myfile.js', ['baz.bar'], ['goog.log'], {'lang': 'es6', 'module': 'goog'});

Try closure-make-deps --help for more info? Let me know where the confusion was so we can make it clearer for the next person :)

[Ádám Rocska]

Hi Andre!

My crew & I went all in on Bazel about a year ago, and it was one of the best (professional) bets of our career.

I'd be more than happy to help you with whatever learning outcomes we had.

Let me know if you have any questions about our journey.

Cheers,

Adam

[Erik Neumann]

I'm curious, what are the advantages of "more up to date npm-tools-based workflow"?  I'm using just "make" and bash scripts.  I like having as few dependencies as possible, and understanding my tools as much as possible.

Actually the main problems I encounter are with Dossier (documentation tool for closure compiler) because it requires Bazel which keeps needing care and feeding to keep it working.

--ErikN

--

---
You received this message because you are subscribed to the Google Groups "Closure Library Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to closure-library-d...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Laertes]

As others have commented, the way the config is written might not be letting the compiler do the pruning.

A contrived example based on your configuration suggests that all that's needed for the pruning to be effective is telling the compiler which namespace to start with. Assuming you are goog.providing or using goog.module for the entry point, you can add the entry_point compiler arg:

.pipe(closureCompiler({

          entry_point: 'goog:my.app.main',

          compilation_level: 'SIMPLE',

          warning_level: 'VERBOSE',

          // ...

         })

However, AFAIU this delegates the initial globbing to gulp, which doesn't remove unneeded files while walking the tree, leading to high RAM usage. If this is ever a problem, you can do the globbing via the compiler while staying in the gulp ecosystem[1]—with the downside that it restricts such usage to the Java compiler.

[1] https://github.com/google/closure-compiler-npm/blob/master/packages/google-closure-compiler/docs/gulp.md#use-without-gulpsrc-java-version-only

[Andre Tannus]

Thank you all for you input, very helpful.

@Stephen, @Ádám, thanks, I'll be sure to take a look at Bazel.

@John, I was pretty burned out when I wrote that message; I later got it to work, but I do not recall what I was doing wrong, just tired I guess. I'll look into your suggestions, my current solution has the drawback of dumping the entire CL + third party + application code into the compiler at once. The app is quite small, so the harm is limited, but having a more streamlined solution that takes advantage of the dependency tree is definitely in my road map.

@Erik, like I said, I've been (still am on a rather large project) using Plovr with an assortment of bash scripts. What I love about it is that it's fast and handles RAW mode, which serves files independently, allowing for very comfortable debugging without the pitfalls of source maps. It's been fine, even though my setup feels a bit Jerry rigged and Plovr support might get spotty sometimes. I'm mostly looking for alternative setups which might be easier to put together and give me more security and confidence to throw the power of Closure Tools onto smaller projects without the need for a confusing build mechanism. I'm poking around for alternatives, I guess.

@Alex, right on. I was missing the entry_point flag.

To wrap up, here's a sample gulp file I turned up, which uses the following Closure Tools: Library, Compiler, Stylesheets and Templates.

I hope this is useful to someone, I sure would have loved itas a starting point.

https://gist.github.com/atannus/4dc9c4ec5036499d0b49681283a24174

Thank you all.

André

--

---
You received this message because you are subscribed to the Google Groups "Closure Library Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to closure-library-d...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

André Tannús | Epungo | +55 11 2389-4360

We are a layer