Cross Module Code Motion

[Jay Young]

I'm writing a script that uses the compiler's Java API to compile a
project into modules. I can swallow the idea of creating a config
file to represent my module tree and which of my own js input files
should go into each module. My question is whether or not I can
simply include all of the goog.* dependency files in the top-most
module and trust CrossModuleCodeMotion and CrossModuleMethodMotion to
put them in the lowest appropriate part of the module tree.

core:
....module a
........module b
............module c
........module d
............module e
....module f

To put it another way, if one of my app files in "module c" and
another in "module e" both require goog.fx.AwesomeFireworksAnimation,
will these optimizations move that class (and its dependencies) down
to "module a"? I can extend my script to do dependency analysis to do
this, but I'd rather not write it if the capability is built into the
compiler already.

Thanks.

[Jay Young]

[Nick Santos]

In the general case, no.

those passes will only move code if they can prove, without a doubt,
that it is safe to do so. There are many "reasonable" ways to write
code that will prevent code motion. For example, if your file
initializes some complex data structure in the global scope, this will
often prevent code motion.

We've been experimenting with different compilation models that do
"automatic dependency management", so that the compiler would
automatically figure out which module each file should go in, and you
would just list the entry points in each module.
jscomp.deps.SortedDependencies may help you do this sort of deps
analysis.

[Jay Young]

Thanks, Nick.

[Andrew Mattie]

It'd be great if you could post your experiences with this to the list after you've played around with it a bit. I've been musing about this myself but haven't yet reached a point where I've needed to make a decision about it.

On Thu, Dec 16, 2010 at 3:30 PM, Jay Young <jaydevf...@gmail.com> wrote:

Thanks, Nick.

[bolinfest]

Jay, have you tried using plovr to help you with modules? I just put
together a demo page that shows plovr serving modules in action:

http://www.plovr.com/demo/modules.html

It even generates a module diagram using SVG:

http://plovr.com:9810/modules?id=module-example

This doesn't cover cross-module code motion, but it does show how your
input files are bucketed into modules.

Unlike the complicated build script that is normally required to use
modules:

http://www.plovr.com/demo/compile-modules.sh

plovr uses a more compact config file where you only have to specify
the module dependencies and the required inputs for each module:

http://plovr.googlecode.com/hg/testdata/modules/plovr-config.js

For each transitive dependency, plovr puts it in the least common
ancestor module of the modules that require it.

As you can see from the unit test (check out the ASCII art in the
Javadoc! http://plovr.googlecode.com/hg/test/org/plovr/ModuleConfigTest.java),
plovr can also handle module graphs that are DAGs, not just trees.

HTH,
Michael

On Dec 17, 12:14 am, Andrew Mattie <amat...@gmail.com> wrote:
> It'd be great if you could post your experiences with this to the list after
> you've played around with it a bit. I've been musing about this myself but
> haven't yet reached a point where I've needed to make a decision about it.
>

> On Thu, Dec 16, 2010 at 3:30 PM, Jay Young <jaydevfollo...@gmail.com> wrote:
> > Thanks, Nick.

[Jay Young]

Hey Michael.

I haven't used Plovr yet, but I have been following your work on it
for a while. I like the idea and the features, but I only casually
code, so I haven't spent much time in picking it up and trying it out.

Since I posted this, I've been working on a script that does pretty
much what you just described: given a config.json file (I even stole
your idea of allowing comments; hope you don't mind) that outlines the
entry points for a module, it reads in a given set of deps files
(generated by depswriter.py), analyzes the module and namespace
dependencies, and throws things into the lowest part of the tree that
works. It also handles the fancy module dependency graphs Plovr does.

Except I wrote mine in Javascript (run on top of Rhino-based RingoJS)
because I'm not very familiar with Java. My goal was to figure out
how this could best be done and then integrated back into the Compiler
so that an external tool is not needed. I would have tried this in
Java, but my Java skills are very lacking and I was able to pump out a
working prototype a lot faster.

The project is up on Google Code at: http://code.google.com/p/closure-module-builder/

I started putting up a few wiki pages regarding the data structures
and algorithms I use, but they're not done yet. I'm not a code ninja,
so there are definitely areas for algorithm improvement, not to
mention a few corner cases I'm sure I've missed. I'm going to spend
some time building up a small sample app to try to exercise the cross
module namespace motion to find and patch the leaks.

By the way, thanks for the awesome book. I wouldn't have gotten this
far into Closure without it.

[Fredrik Blomqvist]

Hi,
I had similar needs a while ago and created a closurebuilder extension
in Python - coincidentally it is also called modulebuilder :)
https://github.com/blq/mochikit/blob/master/scripts/Closure/bin/build/modulebuilder.py
https://github.com/blq/mochikit/blob/master/scripts/Closure/bin/build/modulebuilder_help.txt
(See the "--module" flag).

Essentially it uses the same cmd-line as closurebuilder except there's
no individual file references, i.e
the "--js" flag is removed in favor of "--module", module only input.

Here's an example that compiles my Closure adapted MochiKit fork using
modulebuilder.py:
https://github.com/blq/mochikit/blob/master/scripts/build_closure.py

Haven't studied your take, nor Plovr, in detail yet, but seems like
you for example use explicit module-deps whereas I leave that
to be implicit. Hopefully ideas can be built upon!

Regards & Merry Christmas to all on the list!
// Fredrik Blomqvist

ref. initial thread where I discussed the setup:
http://groups.google.com/group/closure-compiler-discuss/browse_thread/thread/479a54ecfd10a2f0/2a918fd905b6c8b9?#2a918fd905b6c8b9