`require()` system and prototypal programming

[elliottcable]

I have to add this little prequel: it’s 6:54 AM here, I am just
wrapping up the last paragraph of this post, and I am *exhausted*.
Please, please, please excuse any typos or grammar-o’s or brain-o’s or
Cheetos… mmm… now I’m hungry… and really, really, really sleepy…
anyway, this is extremely long, ~1,500 words, because I had a *lot* to
say on the subject. I realize it may take a chunk out of your time,
but it’s a subject *very* near and dear to my heart as a person who’s
written dozens, if not hundreds, of libraries over the years… so I’d
really, really appreciate it if you’d take the time to read it. (-:


Well, I’ve not really been following the CommonJS project (I’m a
Node.js guy, not really all that interested in the other
implementations of ‘SSJS’).

Unfortunately, it’s come to affect me: Ryah just implemented a rather
large change in the way `require()` works, that very much affects many
things I am working on. I talked with him a bit about it, and he
suggested I come here and hash it out with y’all. I understand his
point of view; he wants Node.js to be library-compatible with other
‘SSJS’ implementations.

So, right off the bat, up until Node.js 0.1.16, I had been writing my
own implementation of an alternative to `require()` that solved the
(at that time) minor problems I was having with `require()`. The code
is unfinished (as 0.1.16 threw a huge wrench in my plans, with the
CommonJS `require()` that was no longer possible to make backwards-
compatible from my implementation), but you can check it out here:
http://github.com/elliottcable/poopy.js/blob/master/lib/acquire.js

So, onwards to the problems I see with the CommonJS implementation of
`require()`.


First off, it makes true prototypal programming impossible. When I
first came to JavaScript, the moment I grokked the beautiful
inheritance system, I was absolutely appalled at how all of the major
JS libraries seemed to do their absolute best to push it away, hide
it, ignore it. Most of them seem to try to implement some half-baked
‘classes’ system on top of it, with the intention of replacing it
entirely.

I’ve been working on a ‘framework’ (it’s really much less code, and
much more ideology) to foster purist prototypal development in
JavaScript. I use it exclusively, myself. Unfortunately, you have to
create objects *yourself* (for obvious reasons) to have control over
their prototypal inheritance; the CommonJS `require()` implemented in
Node.js creates this `exports` object for you, leaving you with no way
to control it; you have to use what you’re handed, making prototypal
programming (or really, anything very powerful, even non-prototypally)
improbably difficult or even impossible.


Next, and second on my list of complaints, the `require()` function
returns the `exports` module object. This problem is entirely linked
into the one above, but I expect it to return what the `require()`’d
file `return`s. Again, same problem; I don’t want the system telling
me what my file returns, *I* want to tell the *system* what *I* want
to return, with `return`.


Third, `require()` is not inline-equivalent to the source code of the
file you’re requiring. A huge, huge requirement for many JavaScript
libraries to come in the near future, with JavaScript-as-a-system
taking off, is the ability to take each `acquire()` (the name of my
alternative to `require()`, possibly a bit *too* close, I guess) in a
block of code, remove the string, and replace it wholesale with the
*content* read from the file that the `acquire()`, when evaluated,
would run.

This means that, if I write a module in `subModule.js` as follows:

(function(){
var subModule = parentModule.beget();
subModule['something'] = "Yay!";
return subModule;
})();

… and then require it into another file, `myLibrary.js`:

(function(){
var myLibrary = {};
myLibrary['subModule'] = acquire('subModule.js');
return myLibrary;
})();

… I can then run a ‘client compiler’ that collapses the entire library
into one file, and then minifies/gzips it for the client:

(function(){
var myLibrary = {};
myLibrary['subModule'] = (function(){
var subModule = parentModule.beget();
subModule['something'] = "Yay!";
return subModule;
})();
return myLibrary;
})();

This functionality is absolutely essential for any web-application
libraries which use to be useful in both your client-side and server-
side code; I myself was intending to eventually write a web
development framework that unified the client- and server-side
development environments into one beautiful whole. The inline
semanticity of `acquire()` is what would make that possible.


Fourth, the asynchronicity of `require()` disappeared with 0.1.16.
This is *extremely* out of sorts with Node.js; for those of you not
familiar with Node, even freaking *printing to stdout* is asyncronous.
Requiring external code from the filesystem should *certainly* be
asynchronous.

I am suspicious that this change was also for compatibility with
CommonJS; after all, less purist-async systems than Node.js most
likely directly return a value from `require()` instead of returning a
`Promise`, as I expect Node to… I don’t have a solution to this,
because Node-specific code would really be a lot more consistent if
`require()` were async, and you had to `require().wait()` as with any
other `Promise` to throw it in a coroutine and pass on a continuation…
but then that code wouldn’t work with libraries that had lines like
`var myThing = require('myThing.js');`, and as I gather, the whole
point of CommonJS is to allow libraries like that to work in Node,
Rhino, and so on, at the same time…

I won’t really be that unhappy if `require()` is left synchronous
(even though that’s out of keeping with the *entire* Node.js API), for
compatibility with CommonJS, if an async alternative is provided,
though. Maybe that’s the least of two evils? )-:


Finally, not related to the semantics, but rather to the
implementation… and not so much a requirement, as something that
annoys me, personally, and I am sure would annoy others…

I come from a Ruby history, and having source code files to my library
in the root folder of my library *really* bothers me. I much prefer to
keep it all under `PROJ/lib/`, with the ‘root file’ (what the user
gets if they `require 'project'`) as `PROJ/lib/my_project.rb`.

The file lookup path I was building for `acquire()` worked something
like this, with full backwards compatibility to the *old* Node.js
`require()` function (not, unfortunately, the new one)`:

For `require('foo.js')` in `myProject/lib/myProject.js`:
- `…/myProject/lib/foo.js`

For `require('/foo.js')` in `myProject/lib/myProject.js`:
- `/foo.js` (yes, this comes first; Node.js puts the root directory
*last* in the lookup path for some reason, I remove it from the array
and universally look there first. I’m of the opinion that if I *say* I
want `/foo/bar/gaz.js`, the very first place looked should be, uh, `/
foo/bar/gaz.js`. duh.)
- `/Users/elliottcable/.node_libraries/foo.js`
- `/usr/local/lib/node/libraries/foo.js`

For `require('foo')` in `myProject/lib/myProject.js`:
- `…/myProject/lib/foo` (as a file, not a directory)
- `…/myProject/lib/foo.js`
- `…/myProject/lib/foo.node` (Node.js’s compiled file extension)
- `…/myProject/lib/foo/foo` (as a file, not a directory)
- `…/myProject/lib/foo/foo.js`
- `…/myProject/lib/foo/foo.node`

For `require('/foo')` in `myProject/lib/myProject.js`:
- `/foo`
- `/foo.js`
- `/foo.node`
- `/Users/elliottcable/.node_libraries/foo`
- `/Users/elliottcable/.node_libraries/foo.js`
- `/Users/elliottcable/.node_libraries/foo.node`
- `/Users/elliottcable/.node_libraries/*/lib/foo`
- `/Users/elliottcable/.node_libraries/*/lib/foo.js`
- `/Users/elliottcable/.node_libraries/*/lib/foo.node`
- `/Users/elliottcable/.node_libraries/*/lib/foo/foo`
- `/Users/elliottcable/.node_libraries/*/lib/foo/foo.js`
- `/Users/elliottcable/.node_libraries/*/lib/foo/foo.node`
- `/usr/local/lib/node/libraries/foo`
- `/usr/local/lib/node/libraries/foo.js`
- `/usr/local/lib/node/libraries/foo.node`
- `/usr/local/lib/node/libraries/*/lib/foo`
- `/usr/local/lib/node/libraries/*/lib/foo.js`
- `/usr/local/lib/node/libraries/*/lib/foo.node`
- `/usr/local/lib/node/libraries/*/lib/foo/foo`
- `/usr/local/lib/node/libraries/*/lib/foo/foo.js`
- `/usr/local/lib/node/libraries/*/lib/foo/foo.node`

The last one looks in quite a few places; note that `/Users/
elliottcable/.node_libraries` and `/usr/local/lib/node/libraries` are
simply the *default* paths in `node.libraryPaths` (I think the name of
that variable changed in 0.1.16, though); that search pattern is
repeated for each of them.

More important things to note here:
- For the absolute-path notation, the *actual* absolute-path is always
looked at first, because users might not know better, and developers
might just want to throw an absolute path in there when debugging
something (to be explicit)
- We always look at the filename given, even if it has no extension;
if somebody distributes a useful library inside their `./binary` with
a shebang, it will likely not have an extension, and is still
conceivably acquire-worthy
- Projects can simply be distributed however the author wants, to
eventually be unzipped as a folder into `/usr/local/lib/node/
libraries` or another libraries path; no complex ‘gem’ generation is
necessary, as with Ruby and Python and such
- All of a projects’ code resides in a sub-directory of the project’s
distribution directory (in the example above, that is `…/lib/`, but I
am open to alternatives, as long as it is standardized)
- Distributions of code may have multiple separate projects or modules
within them, you could easily have `PROJ/lib/foo` and `PROJ/lib/bar`
in the same git repository, distributed in the same `.tar.gz`

The final benefit, which needs a little explanation, is that each
module or sub-module can have an initialization file of its own name,
inside a folder of its own name: this means groupings of code can be
stored in folders, with a file of the same name in the folder that
requires and otherwise manipulates the code in the other files in that
folder.

For instance, `require('/myLibrary')` from an arbitrary location would
locate `<libraryPaths>/<projectFolder>/lib/myLibrary.js`, but it would
also locate `<libraryPaths>/<projectFolder>/lib/myLibrary/
myLibrary.js`, allowing you to wrap *all* `myLibrary`-related-code
into a single `myLibrary` folder, instead of having one file (the
initialization file, `myLibrary.js` itself) in the folder above in the
hierarchy.

In the same way, inside your project’s `myLibrary.js`, you can `require
('subModule')`, and get `…lib/myLibrary/subModule.js` (for a
traditional, Ruby/Perl/Python-esque layout), or `…lib/myLibrary/
subModule/subModule.js` (for a layout intent on keeping everything
related to a subject in a single folder).

I will likely continue to develop my `acquire()` function, and I will
likely use it instead of Node.js’s new `require()`, for all of the
reasons listed above; but I thought it was worth my while to, at the
very least, explain my reasoning, before parting from the crowd. I
write code primarily for myself, but I’d be sad to see my code become
useless to anyone who simply chooses a different SSJS system than
Node.js, for no other reason than the implementation of the `require()
` or `acquire()` function.

[Ash Berlin]

Comments inline, original message snipped

On 8 Nov 2009, at 16:07, elliottcable wrote:

So, onwards to the problems I see with the CommonJS implementation of
`require()`.


First off, it makes true prototypal programming impossible. When I
first came to JavaScript, the moment I grokked the beautiful
inheritance system, I was absolutely appalled at how all of the major
JS libraries seemed to do their absolute best to push it away, hide
it, ignore it. Most of them seem to try to implement some half-baked
‘classes’ system on top of it, with the intention of replacing it
entirely.

No it doesn't. It just means you have to do:

exports.MyClass = funciton MyClass() { }.

and then in the consuming module:

var MyClass = require('my-class').MyClass;

var MySubClass = exports.MySubClass = function() {}

MySubClass.prototype = new MyClass(); // Or more likely something with Object.create

Perhaps a little bit verbose, and I believe that HelmaNG has a include() function that means you can do `include('my-class')` instead.

I’ve been working on a ‘framework’ (it’s really much less code, and
much more ideology) to foster purist prototypal development in
JavaScript. I use it exclusively, myself. Unfortunately, you have to
create objects *yourself* (for obvious reasons) to have control over
their prototypal inheritance; the CommonJS `require()` implemented in
Node.js creates this `exports` object for you, leaving you with no way
to control it; you have to use what you’re handed, making prototypal
programming (or really, anything very powerful, even non-prototypally)
improbably difficult or even impossible.

Modules export things. A module is not a class. for instance a port of perl's Template Toolkit i wrote exports two things: 1) a Template class, and 2) a render function to streamline the most common case.

Next, and second on my list of complaints, the `require()` function
returns the `exports` module object. This problem is entirely linked
into the one above, but I expect it to return what the `require()`’d
file `return`s. Again, same problem; I don’t want the system telling
me what my file returns, *I* want to tell the *system* what *I* want
to return, with `return`.

Yes these two points are the same. And they are by design. If we want to be able to make module securable (sandboxes etc) then they should have no way of altering their calling environment, which they would do if they could randomly create variables in the global context.

Also modules are not specified to be functions. They probably could have been, but there was a *big* discussion about this (module behaviour in general) at the start of the year -- See the mailing list archives if you are curious.

Third, `require()` is not inline-equivalent to the source code of the
file you’re requiring. A huge, huge requirement for many JavaScript
libraries to come in the near future, with JavaScript-as-a-system
taking off, is the ability to take each `acquire()` (the name of my
alternative to `require()`, possibly a bit *too* close, I guess) in a
block of code, remove the string, and replace it wholesale with the
*content* read from the file that the `acquire()`, when evaluated,
would run.

Yes, this is due to the reason above.

This means that, if I write a module in `subModule.js` as follows:

[example about including submodule snipped]

If you are compiling the module anyway, then you just have to make the compiler slightly smarter. It is still doable.

Fourth, the asynchronicity of `require()` disappeared with 0.1.16.
This is *extremely* out of sorts with Node.js; for those of you not
familiar with Node, even freaking *printing to stdout* is asyncronous.
Requiring external code from the filesystem should *certainly* be
asynchronous.

To be honest, this is one of the things that I don't like about node.js: its in your face required async model. Sure its powerful, but a lot of the time it would do the job to just have `require()` block the current 'worker'. i.e. behave something like the following example:

function require(id) {

  var promise = require.async(id);

  promise.start();

  while (!promise.complete) { run_event_loop() };

  return promise.exports;

}

(Okay you might want to fire off 5 async requires in one go, etc. This is when the promise type api is worth it)

(The other reason I don't get on with node is it uses v8 which doesn't have any of the cool JS extensions that mozilla and to a lesser degree, ES5, have which make programming in JS a much more pleasant experience for me. Just personal preference)

[snip]

I won’t really be that unhappy if `require()` is left synchronous
(even though that’s out of keeping with the *entire* Node.js API), for
compatibility with CommonJS, if an async alternative is provided,
though. Maybe that’s the least of two evils? )-:

So I think narwhal has a require.async(id) behaviour which returns a promise

I come from a Ruby history, and having source code files to my library
in the root folder of my library *really* bothers me. I much prefer to
keep it all under `PROJ/lib/`, with the ‘root file’ (what the user
gets if they `require 'project'`) as `PROJ/lib/my_project.rb`.

Ditto - all my projects have code under lib/*: http://github.com/ashb/juice/tree/master/lib/

For `require('foo.js')` in `myProject/lib/myProject.js`:
- `…/myProject/lib/foo.js`

require('foo.js') should first look for 'foo.js.js' in the include paths, then it is allowable, but not required, to then look for 'foo.js' in the include paths. It should not look for 'myProject/lib/foo.js' - if you want that then require('./foo') is the syntax required. That or to add 'myProject/lib' to the include paths, but that is probably the wrong way to do it.

For `require('/foo.js')` in `myProject/lib/myProject.js`:
- `/foo.js` (yes, this comes first; Node.js puts the root directory
*last* in the lookup path for some reason, I remove it from the array
and universally look there first. I’m of the opinion that if I *say* I
want `/foo/bar/gaz.js`, the very first place looked should be, uh, `/
foo/bar/gaz.js`. duh.)
- `/Users/elliottcable/.node_libraries/foo.js`
- `/usr/local/lib/node/libraries/foo.js`

Behaviour when passing an absolute path to require is currently outside of the spec (and specd to do whatever the hell you want), but looking inside the include paths is slightly counter-intuitive to me given that 'foo' is a top-level-id, and './foo' being a relative one. Your behaviour for an absolute id is how the commonjs spec (or at least most/all implementations) treat a top level id.

For `require('foo')` in `myProject/lib/myProject.js`:
- `…/myProject/lib/foo` (as a file, not a directory)
- `…/myProject/lib/foo.js`

- `…/myProject/lib/foo.node` (Node.js’s compiled file extension)

1) this is the behaviour that you should get from require('./foo')

2) 'foo.js' really *should* be found in preference to 'foo'.

3) shouldn't .node be searched for first? or is .node a binary (dll,dylib,so) file?

Remember, require doesn't take filenames, it takes a module identifier.

- All of a projects’ code resides in a sub-directory of the project’s
distribution directory (in the example above, that is `…/lib/`, but I
am open to alternatives, as long as it is standardized)

Standardizing this might be a long time in coming, as there are a lot of strong feelings about doing this in quite different ways. But the way the spec is written node.js looking inside */lib/foo.js is allowed. I think the most likely approach for this is a require function that comes from a package manager module or similar:

const P = require('my-package-manager');

var x P.require('my-project');

For instance, `require('/myLibrary')` from an arbitrary location would
locate `<libraryPaths>/<projectFolder>/lib/myLibrary.js`, but it would
also locate `<libraryPaths>/<projectFolder>/lib/myLibrary/
myLibrary.js`, allowing you to wrap *all* `myLibrary`-related-code
into a single `myLibrary` folder, instead of having one file (the
initialization file, `myLibrary.js` itself) in the folder above in the
hierarchy.

No. having myLibrary/myLibrary.js be loaded on a require('myLibrary') seems wrong. If you want something like this, make it a special filename, not just the name of the directory the file is in (something like python's __init__.py)

A lot of your confusion/complaints seems to be with 1) the reasoning behind why the CommonJS module system works the way it does and 2) the types of IDs you pass to require.

I'm not saying its not going to be a shock or some work to port it over, but the differences aren't insurmountable. 

-ash

[Kris Kowal]

On Sun, Nov 8, 2009 at 8:07 AM, elliottcable
<googl...@elliottcable.com> wrote:
> First off, it makes true prototypal programming impossible. When I
> first came to JavaScript, the moment I grokked the beautiful
> inheritance system, I was absolutely appalled at how all of the major
> JS libraries seemed to do their absolute best to push it away, hide
> it, ignore it. Most of them seem to try to implement some half-baked
> ‘classes’ system on top of it, with the intention of replacing it
> entirely.

As Ash mentioned, modules are not designed to be classes but carriers
for classes. The question of whether to conflate modules and a type
system has come up, and we will probably revisit the possibility of
supporting constructors and prototype inheritance. But, blessing a
type system and a module system would have been multiplicatively more
difficult to ratify than either one alone, and the conversation about
module systems began with the title "Module Systems: Difficult to
Agree". If it were "Module and Type System", it would have been
"Impossible to Agree".

> Next, and second on my list of complaints, the `require()` function
> returns the `exports` module object. This problem is entirely linked
> into the one above, but I expect it to return what the `require()`’d
> file `return`s. Again, same problem; I don’t want the system telling
> me what my file returns, *I* want to tell the *system* what *I* want
> to return, with `return`.

We opted to use the exports strategy so that modules could be mutually
dependent, which is a pretty common occurrence. I recommend looking
through the archives for the details of this argument.

> Third, `require()` is not inline-equivalent to the source code of the
> file you’re requiring. A huge, huge requirement for many JavaScript
> libraries to come in the near future, with JavaScript-as-a-system
> taking off, is the ability to take each `acquire()` (the name of my
> alternative to `require()`, possibly a bit *too* close, I guess) in a
> block of code, remove the string, and replace it wholesale with the
> *content* read from the file that the `acquire()`, when evaluated,
> would run.

On one side you have naïve C-like name spaces, and all the
machinations of require_once, manual cyclic dependency resolution, and
centralized global namespace management; and on our side, the author
of a module has lexical analyzability, sovereignty (the right to
manage the names in their scope), and securability (the guarantee that
your dependencies will not be subverted by a third-module).

> … I can then run a ‘client compiler’ that collapses the entire library
> into one file, and then minifies/gzips it for the client:
>
>    (function(){
>      var myLibrary = {};
>      myLibrary['subModule'] = (function(){
>        var subModule = parentModule.beget();
>        subModule['something'] = "Yay!";
>        return subModule;
>      })();
>      return myLibrary;
>    })();
>
> This functionality is absolutely essential for any web-application
> libraries which use to be useful in both your client-side and server-
> side code; I myself was intending to eventually write a web
> development framework that unified the client- and server-side
> development environments into one beautiful whole. The inline
> semanticity of `acquire()` is what would make that possible.

"Bundling" and "Combination" is possible to implement with CommonJS
modules by using a separate "module transport format" that involves
registering modules as factory functions and instantiating them on
demand at the point-of-require. We have two proposals for this
format. There are also two ways to facilitate the use of "module
transport format", either via cooperation of a server-side component
that bundles-on-demand, or a tool compile a set of modules into a
single overlay in that format.

For "F5 or Command-R to Debug" deployments, you can use sync-XHR to
load CommonJS modules on the client.

There are implementations for most of these approaches.

> Fourth, the asynchronicity of `require()` disappeared with 0.1.16.
> This is *extremely* out of sorts with Node.js; for those of you not
> familiar with Node, even freaking *printing to stdout* is asyncronous.
> Requiring external code from the filesystem should *certainly* be
> asynchronous.

We have proposals for a require.async(id):Promise system. We haven't
yet ratified promises, but there are two implementations of this
strategy so far, that I know of.

> I am suspicious that this change was also for compatibility with
> CommonJS; after all, less purist-async systems than Node.js most
> likely directly return a value from `require()` instead of returning a
> `Promise`, as I expect Node to… I don’t have a solution to this,
> because Node-specific code would really be a lot more consistent if
> `require()` were async, and you had to `require().wait()` as with any
> other `Promise` to throw it in a coroutine and pass on a continuation…
> but then that code wouldn’t work with libraries that had lines like
> `var myThing = require('myThing.js');`, and as I gather, the whole
> point of CommonJS is to allow libraries like that to work in Node,
> Rhino, and so on, at the same time…
>
> I won’t really be that unhappy if `require()` is left synchronous
> (even though that’s out of keeping with the *entire* Node.js API), for
> compatibility with CommonJS, if an async alternative is provided,
> though. Maybe that’s the least of two evils? )-:

I spoke to Ryan this weekend. It seems to me that there must be
separate and complete API's for both async and sync styles of just
about everything.

> Finally, not related to the semantics, but rather to the
> implementation… and not so much a requirement, as something that
> annoys me, personally, and I am sure would annoy others…

I've written async servers before, too, and agree that the
unavailability of universal async support makes those kinds of
projects very difficult. But, it's something you have to be willing
to pay for. I agree that there would be much better support for async
API's if they were the only API's that were possible to write, but the
fact of the matter is that they're harder to code for, and infectious;
that's why languages like E exist where any function call can be made
asynchronously and thus remotely. But, it's not possible to force
everyone to pay that price. We have to pay as we go, pay for the
things we want.

So, the issue of making universal asynchronous API's available is
orthogonal to the concern of making synchronous API's available. We
simply need to provide both and offer our users both options.
Exhaustively. *Not* providing synchronous API's to provide leverage
for universal asynchronous API's isn't really an option.

> I come from a Ruby history, and having source code files to my library
> in the root folder of my library *really* bothers me. I much prefer to
> keep it all under `PROJ/lib/`, with the ‘root file’ (what the user
> gets if they `require 'project'`) as `PROJ/lib/my_project.rb`.

You mention here many things about require.paths that you claim are
impossible. All of them are supported by some CommonJS
implementations; the specification does not preclude intelligent
manipulation of require.paths or well-defined layout of packages.
Narwhal does exactly what you describe with package lib/ paths. The
specification may be worded poorly, but the intention is that the
top-level identifier notation is a strict subset of both UNIX paths,
URI's, and URL's. The verbiage does restrict identifiers for use in
the CommonJS standard library and interoperable modules, but does not
intentionally constrain implementations from using
beyond-specification identifiers.

> I will likely continue to develop my `acquire()` function, and I will
> likely use it instead of Node.js’s new `require()`, for all of the
> reasons listed above; but I thought it was worth my while to, at the
> very least, explain my reasoning, before parting from the crowd. I
> write code primarily for myself, but I’d be sad to see my code become
> useless to anyone who simply chooses a different SSJS system than
> Node.js, for no other reason than the implementation of the `require()
> ` or `acquire()` function.

Thanks. It would be a loss for the cause of interoperability if I
haven't convinced you that our goals are compatible.

Kris Kowal

[Erik Garrison]

I really like the idea of commonjs. But I do not like the modules
system. It seems incredibly complex. If not rethought, I imagine it
will be a thorn in the side of the project. But I worry that it is
too late to rethink, even though there are constant small pushes to do
so by people who became aware of this discussion only after it had
been going on for some time. It seems that people who come to
commonjs who haven't been part of the discussions on the module system
(many months ago) are either confused or appalled at the results. We
are asked 'hey where were you?' as if there was something wrong with
us for showing up late.

As a developer, I would like to be able to use a module system only by
adding 'require' clauses to existing libraries to denote dependencies
between modules. I would not like to be forced to write libraries
that only work in the presence of commonjs, or port everything I care
about into this arrangement when simpler code loading schemes exist.

Like Elliott, I also wonder how the ratified module system will
promote code reuse. Pre-existing js libraries will not work on both
the commonjs-compliant server and the in-the-wild UA client without
code level modifications. (Assuming first that we have a shim that
lets us 'require' things in the UA). The result will be that people
use simpler import systems. Do not believe that the technical clarity
of a spec means it no longer presents a source of friction to
contribution to your project. I have seen many instances in which
'tiny' amounts of this kind of friction were enough to inhibit
contribution. A lower rate of contribution means your beneficial
network of code will grow more slowly.

If commonjsen are to be successful unifiers of an enormous and
immensely fragmented community of software developers, engineers,
hackers, and users, we are going to have to seek a lower-energy
position on which to base the foundation of this work. Otherwise
people are just going to ignore it and keep doing things in a
worse-but-easier way.

I think that the existing spec is quite correct and complete. But
correctness seems irrelevant in this situation. It doesn't matter how
correct the existing spec is. If it doesn't work well within its
target social context, it is not going to be successful.

This whole rant is just polite suggestion to step back and ask
yourselves how the module system is going to be adopted into an
environment which has a much simpler concept of 'import code.' My
suggestion is that the group should try to add to the existing
conventions instead of producing new ones.

To be specific, if everyone who writes reusable javascript libraries
is doing so via closures and single-variable namespacing, maybe it
would be preferable to make a system which plays nice with them. What
such a convention is lacking is not a securable, perfect module
loading scheme, but critical things like path resolution and module
caching which make it possible to build larger code systems without
pain.

> --
>
> You received this message because you are subscribed to the Google Groups "CommonJS" group.
> To post to this group, send email to comm...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/commonjs?hl=.
>
>
>

[Nathan Stott]

On Mon, Nov 16, 2009 at 10:54 AM, Erik Garrison <erik.g...@gmail.com> wrote:

To be specific, if everyone who writes reusable javascript libraries
is doing so via closures and single-variable namespacing, maybe it
would be preferable to make a system which plays nice with them.  What
such a convention is lacking is not a securable, perfect module
loading scheme, but critical things like path resolution and module
caching which make it possible to build larger code systems without
pain.

That actually sounds really insightful. 

[Kris Kowal]

On Mon, Nov 16, 2009 at 8:54 AM, Erik Garrison <erik.g...@gmail.com> wrote:
> To be specific, if everyone who writes reusable javascript libraries
> is doing so via closures and single-variable namespacing, maybe it
> would be preferable to make a system which plays nice with them.  What
> such a convention is lacking is not a securable, perfect module
> loading scheme, but critical things like path resolution and module
> caching which make it possible to build larger code systems without
> pain.

I'm not really following this argument. Modules that use the global
object to communicate their API's to each other are very expensive to
secure; take a look at Valija and Cajita. I read your argument as
"such systems do not exist", but perhaps you mean "there is no need
for such systems", or "such systems can do their own thing because I'm
not willing to pay for them." Also, CommonJS modules do specify
support for module caching and path resolution.

I do, however, understand what you're saying about migration from
current module systems. Dojo, Doctype, YUI before v3 and Closure do
name-spacing with provide and require. YUI3 uses something more like
CommonJS module transport format, where you register a module factory
function with its dependency list. jQuery is probably the closest
example of what you describe as "single variable name spacing".

But, jQuery is *easy* to port to CommonJS in such a way that it still
works with the old' <script src> approach.

I think it's very natural for people joining our community to
initially assume that CommonJS modules are complicated. They
certainly require a bit more scaffolding than previous approaches,
with the exception of YUI which is similar under the hood. It's worth
nothing, though, that simple implementations don't take long to write;
there are a lot of them already. But, it's my hope that these folks
who gently urge us to do something more comfortable and familiar, and
there have been a few, have been convinced rather than ostracized.
Sure, some members of this community are more polite and understanding
than others, but as a whole, we embrace argument.

To that end, perhaps you could post a counter-proposal or some
idea-code so we can have a technical argument. It's my best guess
that something like the "module-transport-format" we've been
discussing in a couple other threads would partially address your
concerns. Your concerns may already be addressed by the fact that
modules that just use require and not exports will probably work in
browsers anyway, even though they aren't strongly compliant with
CommonJS and thus not candidates for our standard library and won't
work in secure environments when such things exist. The beauty of the
system, I think, is that we can gradually migrate to something closer
to perfect, rather than camping out in a particular kind of imperfect
and just deal with the shortcomings.

Kris Kowal

[Wes Garland]

Erik;

On Mon, Nov 16, 2009 at 11:54 AM, Erik Garrison <erik.g...@gmail.com> wrote:

I really like the idea of commonjs.  But I do not like the modules
system.  It seems incredibly complex.  

math.js:
exports.add = function add(a,b) {
  return a + b;
}

program.js:
var math = require("math");
print(math.add(1, 2));

...really? "incredibly complex"?

Like Elliott, I also wonder how the ratified module system will
promote code reuse.  Pre-existing js libraries will not work on both
the commonjs-compliant server and the in-the-wild UA client without
code level modifications.

Can you show me what the standard UA JavaScript module loader looks like?
And, also, how to use that from the server to access the filesystem?

people are just going to ignore it and keep doing things in a
worse-but-easier way.

like?
 

I think that the existing spec is quite correct and complete.  But
correctness seems irrelevant in this situation.  It doesn't matter how
correct the existing spec is.  If it doesn't work well within its
target social context, it is not going to be successful.

My team and I use it in daily, real-life work.  We don't find typing "require('X')" to be particularly burdensome, and enjoy knowing that loading a module is not going to have any side-effects on the global environment, entrain garbage, or cause dependency problems.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Kris Zyp]

>
> Like Elliott, I also wonder how the ratified module system will
> promote code reuse. Pre-existing js libraries will not work on both
> the commonjs-compliant server and the in-the-wild UA client without

> code level modifications. (Assuming first that we have a shim that
> lets us 'require' things in the UA).

I presume that what is meant by browser/UA style code is something like
this (borrowing from Wes's example), which AFAIKT works on all the
currently available CommonJS platforms:
mymath.js:
mymath={
add: function add(a,b) {

return a + b;
}
};

program.js:

require("mymath");
print(mymath.add(1, 2));

The caveat is that this is not a true portable commonjs module since it
won't work on securable object capability that prohibit global variable
assignment (because obviously it assigns a global). But, I believe we
were careful to allow platforms to allow this behavior if desired, and
so platforms that were more concerned with code compatibility than
object capability security could run such modules even if they weren't
true commonjs modules. Isn't this style of modules that we have already
accommodated what you are wanting?
Kris

[Wes Garland]

Kris:

For what you've documented there is also possible a slightly more general way, provided we tweak the spec to insist that all properties of exports are enumerable; something like this could be made to work in pretty much any JS environment:

/* Completely untested */
function loadModule(moduleId, global) {
  var module = require('moduleId');

  if (!module)
    throw(module + ": no such module");

  for (var exp in module)
  {
    if (global[exp])
      throw('namespace collision for ' + exp);

    global[exp] = module[exp];
  }
}

loadModule("math", this);

[Daniel Friesen]

^_^ Requiring enumerability isn't necessary.
You can only make things non-enumerable in a ES5 context, and in an ES5
context we have Object.getOwnPropertyNames(o);

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

[Erik Garrison]

On Mon, Nov 16, 2009 at 2:31 PM, Wes Garland <w...@page.ca> wrote:
> Erik;
>
> On Mon, Nov 16, 2009 at 11:54 AM, Erik Garrison <erik.g...@gmail.com>
> wrote:
>>
>> I really like the idea of commonjs.  But I do not like the modules
>> system.  It seems incredibly complex.
>
> math.js:
> exports.add = function add(a,b) {
>   return a + b;
> }
>
> program.js:
> var math = require("math");
> print(math.add(1, 2));
>
> ...really? "incredibly complex"?
>

Wes, Sorry to not be technical or clear about this point. I was
discussing problems with *writing* modules, not using them.

>> people are just going to ignore it and keep doing things in a
>> worse-but-easier way.
>
> like?
>

Such as encapsulating their code in closures and loading it with a
loader that doesn't do anything more than read the file.

>>
>> I think that the existing spec is quite correct and complete.  But
>> correctness seems irrelevant in this situation.  It doesn't matter how
>> correct the existing spec is.  If it doesn't work well within its
>> target social context, it is not going to be successful.
>
> My team and I use it in daily, real-life work.  We don't find typing
> "require('X')" to be particularly burdensome, and enjoy knowing that loading
> a module is not going to have any side-effects on the global environment,
> entrain garbage, or cause dependency problems.
>

So perhaps I have misunderstand the target social context for commonjs.

Erik

[Erik Garrison]

Sorry I didn't give examples. Yes, it is much more along the lines of
what I was describing.

Perhaps, rather than producing a competing way of doing things, a
module loader could accept that globals happen, and sandbox them into
the 'module' object returned from require. So that:

require('mymath')

Would yield an object with a tree like such:

mymath.mymath.add

That way you could be assured that there wouldn't be problems even if
you used code that wasn't worked into compliance before-hand.

This just seems much friendlier to both promiscuous (using code from
everywhere) and paranoid (fearful of globals) developers.

But I'm probably missing something.

Erik

[Erik Garrison]

This doesn't solve code reuse issues.... All environments would have
to have the capability to create new global contexts....

[Wes Garland]

> a module loader could accept that globals happen, and sandbox them into
> the 'module' object returned from require.  So that:
>
> require('mymath')
>
> Would yield an object with a tree like such:
> mymath.mymath.add

This is equivalent to

var mymath = {mymath: require('mymath') };

If that's too burdensome, you can always write a loader similar to the one I posted a few messages back.

A promiscuous module loader can always be built on top of a securable one, but not the other way around.

[tim]

Here are some snippets from my own notes on these subjects, having
experimented with current platforms looking for feasible common
ground. These are of course my opinions, but because of limited time,
I’ve expressed them as matters-of-fact and probably out of order.
Thanks for bearing with me if you do.

Synchronicity
-----------------
async-to-sync is apples-to-oranges. This might one day be expressed
and well-maintained as two separate copies of similar functionalities
(at twice the cost). Not today, not in retrospect, and not by
accident.

Security
----------
Minimally-Practical vs. Secure is also apples-to-oranges. Python’s
“restricted mode” abandoned its noble efforts likely because of too
many loopholes hidden by applying security in retrospect. In all
practical ways, CommonJS is in retrospect here. (FWIW, I had a need
to explore sharing global state between modules/contexts, and am
currently using an inconspicuous security loophole across several
recent proto-platforms to accomplish this).

Thought Vehicles
---------------------
I see lots of bike-shaped stenciling on minivans, monster trucks, and
unicycles.
‘JavaScript’ is 40% ‘Java’ in terms of surface stenciling, but…

Separation of Concerns
----------------------------
Very little overlap happens by random chance. How are these
different needs to be managed now and in the future, both in
discussion and in code? Are current specifications catering to just
one or two of these?

* The CommonJS-as-a-language
* The Module Librarians
* The End-User Developers
* The Platform Builders
* The Module Authors

Going with the Flow
-------------------------
Current proposals, when viewed across the above concerns and across
existing codebases, remain suspicious and burdensome. What other
approaches might better serve to maintain inter-project rapport while
revealing palatable solutions? How about an incremental approach?

(1) vanilla ES3 module concept
No prescribed module identifiers or resolution mandates or paths, just
an embryonic, abstract mechanism for an End-User Developer to express
a need, and for a Module Author to similarly identify dependencies.
This is doable in useful ways through code comments.

(2) vanilla ES3+Core concept
With the addition of bare-minimum, majority concurred CommonJS APIs.
Both Synchronous and Asynchronous interfaces provided (fine if
emulated). Nothing dependent on any module or packaging system. The
APIs expressed as superglobals, not module imports, with no choices
whatsoever for the End-User Developer to make here (beyond all or
nothing ES3+Core).

(3) An ES3+Core+Stdlib reference implementation
Basically ES3+Core+MoreES3s, with any MoreES3s’ synchronicity support
TBD. Again superglobals used to reference the larger API, and just
one decision for a user to make.

(4) ??

Tangential Needs
---------------------
* DISCUSSION FORUM: ES3+xAPI *
To vet some good interfaces to already-common libraries like mysql/
memcache/etc. This seems completely overlooked by CommonJS movement.
It needs to be addressed quickly to re-unify current divides and to
avoid new exponential divides across platforms.

* DISCUSSION FORUM: ES3+Synchronicity *
To coordinate async vs. sync standards any possible interop; to track
relatable efforts w/o confusing them with module or security
discussions.
------------------------

Again, just some thoughts from my notes offered with a grain of salt.

Best, -Tim

[Erik Garrison]

On Mon, Nov 16, 2009 at 4:41 PM, Wes Garland <w...@page.ca> wrote:
>> a module loader could accept that globals happen, and sandbox them into
>> the 'module' object returned from require.  So that:
>>
>> require('mymath')
>>
>> Would yield an object with a tree like such:
>> mymath.mymath.add
>
> This is equivalent to
>
> var mymath = {mymath: require('mymath') };
>
> If that's too burdensome, you can always write a loader similar to the one I
> posted a few messages back.
>
> A promiscuous module loader can always be built on top of a securable one,
> but not the other way around.
>

Right. I think I'm going to see what I can do there.