AMD and Fanstatic

[Martijn Faassen]

Hi there,

I've recently been learning more about Asynchronous Module Definition
(AMD) for JavaScript libraries, its require.js implementation, and
been thinking about how this might fit into Fanstatic. Here are some
of my first thoughts sketched out in the hope we can prompt some
discussion.

Fanstatic solves a number of problems; the problem I want to address
now is one of the first problems it tackled in its hurry.resource
implementation: the javascript module problem.

JavaScript, at least in the browser, does not have a module
implementation. Instead the developer has to manage this manually; if
a.js depends on b.js, the developer has to include <script> tags on
the page in the right order.

Fanstatic solves this on the server-side by letting people declare JS
resources and their dependencies in a simple Python object structure.
Fanstatic then makes sure the script tags are included in the right
order.

require.js has taken an other route; it includes script tags
dynamically in the web page on the client side. It loads JS files
dynamically and asynchronously from the server. This has some
advantages (no special server support is needed) and some
disadvantages (css dependencies won't work so nicely).

How does require.js know what JS file depends on what other JS? It
requires the developer to add this information to the JS files
directly, similar to how Python modules have import statements.
Because modules are loaded asynchronously, all this is wrapped in a
define function.

For instance, here's a module that depends on jquery:

define(['jquery'], function($) {
return {
myapi: function() { }
}
});

the 'jquery' module is looked up and loaded and passed into the
function as the $ parameter. In the function the module specific code
can be written. Finally the module returns an object with the module
API. This, in brief, is the AMD definition.

How does require.js know where to look for 'jquery'? The basic rule is
to look for jquery.js under the base URL (typically a directory with
all the scripts in it). In addition the developer (or tools) can
create a structure of paths in a configuration, for instance one that
maps 'jquery' to jquery.js in some other directory.

Another relevant bit is that require.js has a optimization tool that
can bundle all JS files (including dependencies) into a single
minified .js file so it can be loaded quickly into the browser.

Now how could this tie into Fanstatic?

One approach to integration is that Fanstatic maintains a paths
structure for require.js according to what Fanstatic js.* packages are
installed, and leaves the loading up to require.js. It'd be
integration, but I'm not sure this gains us anything above having
plain Fanstatic.

The more interesting possibility in my mind is that Fanstatic could
grow a way to parse AMD-based .js files and create Resource objects
automatically for them. That way for AMD-based code you don't have to
maintain any Python code at all anymore. We could investigate how
require.js does its parsing (there is some syntactic sugar it supports
that I didn't address) and implement that for Fanstatic as a first
step. A tool that can create Python packages for arbitrary JS
libraries that use AMD would be a nice first step.

On top of that we could consider building a Fanstatic that can deal
with non-Python JS packages, i.e. get packages from some a "JS place",
instead of PyPI.

For this I will look at the bower project, at least for inspiration,
though perhaps code can be used as well. bower is a package
installation tool that simply pulls JS code off tags on git (often
github), and installs them into a directory in your project. It has a
mechanism for resolving inter-package dependencies. Fanstatic could
grow code that can do the same, perhaps using bower directly. After
this it can use AMD module parsing to figure out a module-map. Once we
have created a bunch of resources, we can .need them from our python
code.

There are a whole host of issues to work out. If there's no Python
package anymore installed, how do you get a resource then to need() it
in your code? A new API to get such resources might be useful. How
does the old and the new work together: can we make Python-declared
Resource depend on an automatically created Resource?

Bits and pieces we might need:

* AMD parser

* parsed AMD info to Resources converter

* write out Resources to Python file (we have this already)

* take JS directory (or tarball, or git URL) and convert it to
Fanstatic js.* package

* take JS directory and create virtual js.* module that we can import
resources from (this might require too much work during import time,
though, so other API to expose resources might be better)

Regards,

Martijn

[Wolfgang Schnerring]

Hello,

> The more interesting possibility in my mind is that Fanstatic could
> grow a way to parse AMD-based .js files and create Resource objects
> automatically for them. That way for AMD-based code you don't have to
> maintain any Python code at all anymore.

Yes! I don't really have anything constructive to add right now, but I just
wanted to say that I like this idea very much! :-)

> There are a whole host of issues to work out. If there's no Python
> package anymore installed, how do you get a resource then to need() it
> in your code? A new API to get such resources might be useful. How
> does the old and the new work together: can we make Python-declared
> Resource depend on an automatically created Resource?

Hmm, above you talk about synthesizing Resource objects. While I'm a little
wary of synthetic Python modules, we /could/ have Fanstatic create something on
the fly, e.g. from foo/bar.js we'd make it so that clients could call
js.foo.bar.need() or something -- we'd need to think about naming conventions
and namespaces and such, though.

The other option would be a new string-based API, like
AMDResource('foo.bar').need(). (This, too, raises the
naming/namespace question, of course.)
As for combining new and old ways, I think one might spell it like this:
Resource(lib, 'local.js', depends=[AMDResource('other.js')])
where AMDResource is a stand-in that resolves to the automatically created
thing at the right time *waves hands*.

Oh. I just saw you already talked about that:

> take JS directory and create virtual js.* module that we can import
> resources from (this might require too much work during import time,
> though, so other API to expose resources might be better)

Well. So we agree about the possibilities here. :)

Wolfgang

[Martijn Faassen]

Hi there,

On Fri, May 3, 2013 at 8:18 AM, Wolfgang Schnerring <w...@gocept.com> wrote:
> The other option would be a new string-based API, like
> AMDResource('foo.bar').need(). (This, too, raises the
> naming/namespace question, of course.)

I think a string based lookup API would be the right way to go, as AMD
uses strings (with potential paths in them, i.e. foo/bar) as resource
identifiers.

Whether we want to expose something like AMDResource explicitly in the
API is another question; perhaps a 'get_js_module' function API would
be better. In the end it might all resolve to plain Resource objects
in the end anyway.

I'd like to explore this before we go into virtual module generation,
which is a bit too magic for my tastes. Though we might end up there
as we both speculated, we'll see. :)

Regards,

Martijn