Cross-domain loading of package metadata

[Kris Zyp]

We have recently been having discussions on Dojo and RequireJS about
being CommonJS package aware from browser-based loaders. The package
specification and transport specification are all appropriate in the
browser, loaders can certainly parse package.json and handle package
mappings just fine, and the transport format makes it easy to load
modules even cross-domain, but the one missing piece is that
browser-based loaders can't load a package.json file from a cross-domain
source. I think this could be resolved pretty easily though. I'd propose
that if a package is made available for direct loading from the browser
(that is, the modules are accessible as individual URL/resources wrapped
with the transport format), then the package should also include a
package.js (in the root, sibling to package.json) that can be requested
as a script to register any the package information. We can then define
a require.registerPackage(packageJsonData) that can be called by
package.js to register the package metadata. Of course package.js is
just a script that is recommended to be requested first (by client side
loaders that are package aware), and thus we shouldn't preclude
package.js from also making require.define calls (per the transport
spec) as well, This would allow "built" packages to provide likely
modules to the browser ahead of time without necessitating additional
requests.

So basically there are two simple proposals:
* Client-side loaders that wish to be made aware of package metadata
from a cross-domain package (that would otherwise be available in
package.json) should try to request package.js.
* require.registerPackage(packageData) should be defined as the
API/callback for registering package metadata.

Does that seem reasonable?

--
Thanks,
Kris

[Kris Kowal]

These ideas seem grand to me.

* would it make sense to have a dynamic JSONP analog such that
packages are not self-naming?
* would it make sense to separate the specifications for packaging and
package transporting in the same fashion that we separate modules from
module transport? I suspect we could greatly simplify the burden on
package authors and the mappings specification if we separate the
concern of browser transport, such that simple archives can be
trivially "complied" to browser transport packages.

Kris Kowal

[Kris Zyp]

On 8/18/2010 1:05 PM, Kris Kowal wrote:
> These ideas seem grand to me.
>
> * would it make sense to have a dynamic JSONP analog such that
> packages are not self-naming?

The problem with JSONP is that it require dynamic creation of the
response (to include the client provided callback name). This doesn't
work well with CDN-hosted static content.

Does a static callback (require.registerPackage) preclude client
assigned aliases and anonymous packages? JSONP or self-named packages
solves the problem of associating the callback to the request
(particularly in the situation of parallel requests), but can't that
also be solved with onload/onreadystatechange handlers?

AFAICT, we are not introducing any problems with package aliasing here.

> * would it make sense to separate the specifications for packaging and
> package transporting in the same fashion that we separate modules from
> module transport? I suspect we could greatly simplify the burden on
> package authors and the mappings specification if we separate the
> concern of browser transport, such that simple archives can be
> trivially "complied" to browser transport packages.

Yes, that makes sense.

I think another key thing we should clarify/specify is the namespace
used for the module ids in the transport format for hosted packages. I
believe this should just be based on the root of the package. For
example, if we mapped "foo" to "http://my-cdn.com/packages/foo/" and did:
require.ensure(["foo/bar"], function(require){...});

And generated a request for http://my-cdn.com/packages/foo/bar.js, it
should return:

require.define({
"bar": function(...){...}
}, []);

And a package aware client side loader would be responsible for
resolving the "bar" module within the "foo" aliased package as
satisfying the "foo/bar" module id.

Also, I was wondering if we should specify within the package mapping
specification that when a "location" is specified (a base URL for
directly GETting modules) that the modules SHOULD be "transported"
(using transport/D presumably). The "location" property has been
somewhat contentious since the package management folks don't see the
point, but on the browser side, the "location" is essential and
"archive" is almost equally useless, the browser has to request modules
individually. But if the browser side is one of the main use cases for
"location" than it seems like it would be more useful if it could be
expected to browser consumable with script tags. The other use case for
"location" is with runtime package aware loaders like Nodules, but
Nodules is fine with plain CommonJS modules or transport-wrapped modules.

--
Thanks,
Kris

[Kris Zyp]

Here is the proposal I wrote for asynchronous definition of packages:
http://wiki.commonjs.org/wiki/Packages/AsynchronousDefinition

Thanks,
Kris