Module Loaders

[Kris Kowal]

Having discussed the topic with Ash Berlin on #narwhal IRC, I've added
a section for and included a draft specification for a supplement to
the module system that would make it possible to interoperably
construct and use the module loader objects used by the module system
on some implementations.

http://wiki.commonjs.org/wiki/Modules/Loaders

Kris Kowal

[ihab...@gmail.com]

On Sat, Nov 21, 2009 at 8:38 PM, Kris Kowal <cowber...@gmail.com> wrote:

... a supplement to the module system that would make it possible to interoperably
construct and use the module loader objects used by the module system ...
http://wiki.commonjs.org/wiki/Modules/Loaders

From http://wiki.commonjs.org/wiki/Modules/Loaders/B:

1. A package may provide a module loader.

How can a package do that? A package is just code. How does it acquire authority to provide a loader?

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

[Kris Kowal]

On Sat, Nov 21, 2009 at 9:16 PM, <ihab...@gmail.com> wrote:
> A package may provide a module loader.
>
> How can a package do that? A package is just code. How does it acquire
> authority to provide a loader?

System package, the permissive package management case. This is for
adding loaders for .so, .dylib, Objective-J, &c. and would not be
available in the packages of packages system you proposed.

In related news though, Christoph Dorn has an implementation of
two-arg require(id, pkg) with a package.json ~ {"using": {"name":
{"location": "url"}}} notation, in the spirit of your remote package
proposal. Trying it out now. It's coming along; we probably will
have some ideas for the next draft soon.

Kris Kowal

[ihab...@gmail.com]

On Sat, Nov 21, 2009 at 9:23 PM, Kris Kowal <cowber...@gmail.com> wrote:

In related news though, Christoph Dorn has an implementation of

two-arg require(id, pkg) with a package.json ~ {"using": {"name":
{"location": "url"}}} notation, in the spirit of your remote package
proposal.  Trying it out now.  It's coming along; we probably will
have some ideas for the next draft soon.

Cool!

Ihab

[kkasravi]

A module's require() no doubt leads to the requirements and
implementation of the loader.
However burdening the caller of require() to determine whether it
should be
async or sync is in fact wrong. Also exposing a promise API to the
caller of require is also wrong. Why?

1. No lexical mandate
We're introducing runtime metadata to require()'s return value
arbitrarily when in
fact there is nothing within require()'s lexical environment that
would suggest a 'special' return type.
We are asking the caller of require() to help in determining when
the return type
will be accessed. This is the parser's job and answered by
evaluating the next token.
In reality this should be part of the language runtime.
Examples:
a. var bar = require('foo').bar;
Next token '.'
Requirements: needs to be sync
b. { logger: require('log/logger'), name: undefined }
Next token ','
Requirements: can be async, return type implementation could be
a lazy function definition (http://peter.michaux.ca/articles/lazy-
function-definition-pattern)
c. return streamType === 'File' ? require('io/File') : require('io/
Socket');
Next tokens ':', ';'
Requirements: can be async (same as b.)

2. Distorts correct dependency resolution
The various suggestions for when to call require(), or embed meta-
data associated with require()
tend to suffer from trying to resolve all the dependencies within a
module prior to loading the module.
Some suggestions say to list all the requires at the top similar to
java import statements other
suggestions place all the dependencies in a module metadata record
and suggest that
they should all be loaded as a pre-requirement. Both suggestions
result in an incorrect runtime
dependency list, and have the loader preload potentially never
called modules.
Not only does this impact performance but also impacts runtime
memory requirements

Resolution:
The correct implementation of loader requires partial exportation of
its lexical environment.
This information should be hidden and transparent to a caller of
require().
Obviously this suggests that require() be a native function, but how
could it be implemented
non-natively? A possible implementation is shown below, where a parser
has annotated a function
with a context dictionary as a result of parsing the function.
function require(moduleId) {
return arguments.callee.caller.context[moduleId] === 'Sync' ? /*
loader.sync */ : /* loader.async */
}
Arguments.callee.caller.context[moduleId] is metadata inserted by the
parser
Examples
1. function survey() {
var user = new require('survey/user').User();
require('survey/user/home');
user.provideHomeAddress();
}
The parser determines the first require() needs to load the 'bar'
module synchronously.
The parser determines the second require() can load the 'survey/
user/home' module asynchronously.
The parser does the following:
survey.context = survey.context || {};
survey.context['survey/user'] = 'Sync';
survey.context['survey/user/home'] = 'Async';
At runtime the loader will know the dependencies of the caller and
their sync/async requirements.

Advantages:
1. Correct dependency management that is naturally cached.
Because the lexical context is encoded within the functions
properties that is static to the function,
this information is available to require and its loader even when
the loaders are different.
2. No explicit dependency management policy.
Presumable the loader retains private information as to whether it
has loaded the given module.
In cases where race conditions occur, its up to the loader to
manage module requests in flight
that is not exposed to the caller.
3. Choice of metadata insertion
This type of information could be inserted at 'compile' time or
runtime
4. Heuristically extensible and tunable
A loader's policy could be tuned to load aggressively or lazily and
could be based on profiling
5. Shareable
Different loaders could share the information inserted by the first
loader
6. Migratable to native implementations

-kam