CommonJS Modules/2.0

[Wes Garland]

Hi, All!

Discussions on this list over the last few months - mostly since late September - have shown that the current module format has some rough edges, in particular where it comes to dependency resolution and browser operation.

I have put together a draft proposal for consideration by the community, available at http://www.page.ca/~wes/CommonJS/modules-2.0-7/, and started a task in the BetterMeans CommonJS Modules workstream.

This proposal attempts to address several issues
  - Modules/1.1.1 compatibility guarantees
  - Minor Modules/1.1.1 spec bug fixes
  - Formalizing the CommonJS environment
  - Defining a wrapped module format
  - Defining a formal way to specify explicit dependencies
  - Defining a way to lazy-load modules (similar to require.async) in the base-level specification
  - Defining cross-platform hooks which can be used by future work (including transports, packages, mappings)
  - Recommendations for platform-dependent bootstrapping

Beyond the proposal, I have implemented draft 5 in both GPSEE on the server side, and BravoJS on the browser (current is draft 7 -- I expect to update BravoJS to draft 7 this week).

BravoJS, available at http://code.google.com/p/bravojs/, includes a wrapped version of the old Modules/1.0 test suite, as well as several plug-in module loaders.  Of one these loaders shows how to load Modules/1.1.1 into a Modules/2.0 environment, using jQuery to load the modules.  Another shows how to implement a transport which talks to a server-side process to aggregate multiple modules into a single HTTP request.  A live, browsable version of BravoJS is online at http://www.page.ca/~wes/BravoJS/.  Don't be afraid to "view source"!

A blog post on the GPSEE blog at http://gpsee.blogspot.com/2010/12/wrapped-modules-with-gpsee-02-gsr-10.html shows how to upgrade the current GPSEE release candidate to load modules written this way.  This upgrade allows us to freely mix and match Modules/1.1.1 and Modules/2.0 formatted modules on the server side without measurable penalty.

I am still looking for co-editors and any other feedback you can offer.  In particular, if there is an organizational wizard out there with a word processor who can refactor my work into a technically superior document, it would be very helpful. If anybody would like to contribute sample implementations, test suites, etc., that would be excellent, too.

Please let me know what you think -- and please try to avoid replying directly to this message; change the subject to start a new thread instead.  Otherwise the ensuing discussion may be difficult to follow, if Q3 2010 was any indicator!

Cheers
Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Florian Traverse]

Hi,
Great job of compiling all of this Wes :)

I really have a problem with both current AMD, Wrapping and this Modules 2.0 proposal. I know this may seems minor to most of you, but each time I see any code using one of these, I can't really stand the way the "imports" are defined, using an array. The only advantage I see for the Array is it's conciseness, but being concise is not a great goal in a language if it's a trade off for expressiveness.

So, I would like it to be more expressive, natural to read, and self-explanatory:

instead of (Wrapping like, taken from you spec)

module.declare(["increment"], function(require, exports, module) {

  var inc = require('increment').increment;

  var a = 1;

  inc(a); // 2

  // module.id === "program"

})

or (AMD like, taken from RequireJS website)

//my/shirt.js now has some dependencies, a cart and inventory
//module in the same directory as shirt.js
define(["./cart", "./inventory"], function(cart, inventory) {
        //return an object to define the "my/shirt" module.
        return {
            color: "blue",
            size: "large"
            addToCart: function() {
                inventory.decrement(this);
                cart.add(this);
            }
        }
    }
);

I would prefer something involving the word "import" rather than just the array, and maybe give an "as" option as well for shortcut or easy refactoring purpose.

Some example could be :
(Wrappings style)

module

.import("increments")

.declare(function(require, exports, module) {

  var inc = require('increment').increment;

  var a = 1;

  inc(a); // 2

  // module.id === "program"

})

(Wrappings style, with "as" alias)

module

.import("increments").as("inc")

.declare(function(require, exports, module) {

  var inc = require('inc').increment;

  var a = 1;

  inc(a); // 2

  // module.id === "program"

})

(AMD Style)

module
.import("./cart")


.import("./inventory")


.define(function(cart, inventory) {
        //return an object to define the "my/shirt" module.
        return {
            color: "blue",
            size: "large"
            addToCart: function() {
                inventory.decrement(this);
                cart.add(this);
            }
        }
    }
);

Ok it's a little harder for writing the module loader this way, it's however not that hard either, switching from an array to function chaining, and it feels way more natural to me, and I think(hope) I'm not alone.

BTW, Wes, I can try to help you at least on the layout/structure but do you mind it's primarly designed in OOo/ODP format rather than MSWord (and exported to Word format if you care about this) ?

Regards,
Florian Traverse

2011/1/5 Wes Garland <w...@page.ca>

--
You received this message because you are subscribed to the Google Groups "CommonJS" group.
To post to this group, send email to comm...@googlegroups.com.
To unsubscribe from this group, send email to commonjs+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/commonjs?hl=en.

[khs4473]

Sorry, Wes, I'm replying in your post : )

Florian, for the record, the current Wrappings proposal avoids these
issues (and they are tough ones) by not having a dependency array at
all.

Kevin

On Jan 6, 5:37 am, Florian Traverse <florian.trave...@gmail.com>
wrote:

> > available athttp://www.page.ca/~wes/CommonJS/modules-2.0-7/<http://www.page.ca/%7Ewes/CommonJS/modules-2.0-7/>,

> > and started a task in the BetterMeans CommonJS Modules workstream.
>
> > This proposal attempts to address several issues
> >   - Modules/1.1.1 compatibility guarantees
> >   - Minor Modules/1.1.1 spec bug fixes
> >   - Formalizing the CommonJS environment
> >   - Defining a wrapped module format
> >   - Defining a formal way to specify explicit dependencies
> >   - Defining a way to lazy-load modules (similar to require.async) in the
> > base-level specification
> >   - Defining cross-platform hooks which can be used by future work
> > (including transports, packages, mappings)
> >   - Recommendations for platform-dependent bootstrapping
>
> > Beyond the proposal, I have implemented draft 5 in both GPSEE on the server
> > side, and BravoJS on the browser (current is draft 7 -- I expect to update
> > BravoJS to draft 7 this week).
>

> > BravoJS, available athttp://code.google.com/p/bravojs/, includes a

> > wrapped version of the old Modules/1.0 test suite, as well as several
> > plug-in module loaders.  Of one these loaders shows how to load
> > Modules/1.1.1 into a Modules/2.0 environment, using jQuery to load the
> > modules.  Another shows how to implement a transport which talks to a
> > server-side process to aggregate multiple modules into a single HTTP
> > request.  A live, browsable version of BravoJS is online at

> >http://www.page.ca/~wes/BravoJS/<http://www.page.ca/%7Ewes/BravoJS/>.

> > Don't be afraid to "view source"!
>
> > A blog post on the GPSEE blog at

> >http://gpsee.blogspot.com/2010/12/wrapped-modules-with-gpsee-02-gsr-1...how to upgrade the current GPSEE release candidate to load modules

> > written this way.  This upgrade allows us to freely mix and match
> > Modules/1.1.1 and Modules/2.0 formatted modules on the server side without
> > measurable penalty.
>
> > I am still looking for co-editors and any other feedback you can offer.  In
> > particular, if there is an organizational wizard out there with a word
> > processor who can refactor my work into a technically superior document, it
> > would be very helpful. If anybody would like to contribute sample
> > implementations, test suites, etc., that would be excellent, too.
>
> > Please let me know what you think -- and please try to avoid replying
> > directly to this message; change the subject to start a new thread instead.
> > Otherwise the ensuing discussion may be difficult to follow, if Q3 2010 was
> > any indicator!
>
> > Cheers
> > Wes
>
> > --
> > Wesley W. Garland
> > Director, Product Development
> > PageMail, Inc.
> > +1 613 542 2787 x 102
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "CommonJS" group.
> > To post to this group, send email to comm...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > commonjs+u...@googlegroups.com<commonjs%2Bunsubscribe@googlegroups.c om>
> > .

[Florian Traverse]

Sorry Kevin, didn't notice both flyscript/wrappings update !

BTW, I'm definitively not a big fan of the "fn.toString" approach, and the current Module 2.0 draft 7 proposal includes the previous Wrappings spec ;P

If you want to use toString in a standard spec, at least inject "require" in the function scope 'either through a global or a closure if you can) so the wrapping function does not provide a way to change function(require){var foo=require("foo")} for something else like function(load){var foo=load("foo"}, but only function(){var foo=require("foo")} IMHO it's at least less confusing on what you can or can't do as a programmer (and you could even want to do a function(){var r=require, foo=r("foo")}" and it should work, but toString will have some problems handling such a use case ...

More importantly developer wouldn't know why it doesn't work, you would at least need to provide an explicit error message when using an aliased require that doesn't work ( like a "module not found in the current module pool, please verify you didn't aliased the 'require' function but directly used it as is" message through console.warn if available, if Exception is not an option) and it has to be part of the spec, so it wouldn't differ depending on the implementation.

Apart from that, I have added a vote for choosing the best candidate between AMD and Wrappings on Bettermeans for Module 2.0, I consider this as a high priority for us to build further on CommonJS, I know more and more people watching at this everyday and waiting for a "winner" to go on with CommonJS for the browser. I'm quite neutral here, I do not find either one or the other perfect, but I prefer one that is not perfect for me choice rather than a battle between to pretty good candidates.

I've got a big project yesterday that has started with a RequireJS/jQuery/jQuery-UI stack (so AMD, because it has more alternative implementation and tooling right now), they weren't easy to convince to use CommonJS because this project is browser based and no real choice has been made between Wrappings and AMD. So now I have some eggs on the AMD side, but I'll be pretty happy to make the project switch to Wrappings if not "too late" (= finished), and will try to provide a tool to do the switch if needed, if I can count on a massive Module 2.0 adoption by all those interested in CommonJS.

IMHO, and because I want to push a number of other project to use CommonJS, I think we have to choose a true winner, even if this choice may be a bad one in the long term, because we'll can always do a Module 3.0 and some tools to move to it if needed. Bettermeans (or any pretty close alternative) is by far a better way for us to vote for this than just plain mail, so please go and vote for what you like, vote against one if you can't stand it, be honest, so we can decide. Besides, if everyone (or at least the vast majority) is ok to use Wrappings for module 2.0 here as stated currently in the draft, call me sucker, don't take in account what I've just written, close the vote on Bettermeans, I will be more than happy to pass as an idiot, but I would really want the current state to change and having something I can count on, not in years or month, but in the next weeks. I will definitively try to help as much as possible to achieve such a goal.

Regards,
Florian

2011/1/6 khs4473 <khs...@gmail.com>

To unsubscribe from this group, send email to commonjs+u...@googlegroups.com.

[khs4473]

> If you want to use toString in a standard spec, at least inject "require" in
> the function scope 'either through a global or a closure if you can) so the
> wrapping function does not provide a way to change function(require){var
> foo=require("foo")} for something else like function(load){var
> foo=load("foo"}, but only function(){var foo=require("foo")} IMHO it's at
> least less confusing on what you can or can't do as a programmer (and you
> could even want to do a function(){var r=require, foo=r("foo")}" and it
> should work, but toString will have some problems handling such a use case

Hmmm... I understand the issue, but it's not really solvable in a
satisfactory way. I suppose an implementation could have a global
require function that threw a helpful error. That might not be too
bad of an idea, actually.

I believe that in practice it's a minor issue though. These are
judgement calls, of course, but I believe the convenience and
scalability of implicit dependency resolution (scanning) outweighs the
unambiguousness of explicit (the dependency array). Think about big
libraries here. What would it be like to explicitly specify the
dependencies for this:

https://github.com/khs4473/Codera/blob/master/lib/text-panel.js#L3-29

That would be an array of 19 elements or 19 calls to import/as or
something similar. Having to remember the boilerplate rules seems
like a small price to pay not to have to maintain such a monster.

> Apart from that, I have added a vote for choosing the best candidate between
> AMD and Wrappings on Bettermeans for Module 2.0, I consider this as a high
> priority for us to build further on CommonJS, I know more and more people
> watching at this everyday and waiting for a "winner" to go on with CommonJS
> for the browser. I'm quite neutral here, I do not find either one or the
> other perfect, but I prefer one that is not perfect for me choice rather
> than a battle between to pretty good candidates.

I totally understand your impatience! I think we need to have one
more knock-down-drag-out debate before we're ready though. We need to
put all of the technical aspects on the table so that everyone can see
what it really comes down to. I need to get my transport spec out
there first though...


On Jan 6, 10:35 am, Florian Traverse <florian.trave...@gmail.com>
wrote:

> 2011/1/6 khs4473 <khs4...@gmail.com>

> >http://gpsee.blogspot.com/2010/12/wrapped-modules-with-gpsee-02-gsr-1...upgrade the current GPSEE release candidate to load modules

> > > > written this way.  This upgrade allows us to freely mix and match
> > > > Modules/1.1.1 and Modules/2.0 formatted modules on the server side
> > without
> > > > measurable penalty.
>
> > > > I am still looking for co-editors and any other feedback you can offer.
> >  In
> > > > particular, if there is an organizational wizard out there with a word
> > > > processor who can refactor my work into a technically superior
> > document, it
> > > > would be very helpful. If anybody would like to contribute sample
> > > > implementations, test suites, etc., that would be excellent, too.
>
> > > > Please let me know what you think -- and please try to avoid replying
> > > > directly to this message; change the subject to start a new thread
> > instead.
> > > > Otherwise the ensuing discussion may be difficult to follow, if Q3 2010
> > was
> > > > any indicator!
>
> > > > Cheers
> > > > Wes
>

> ...
>
> read more »

[Florian Traverse]

2011/1/6 khs4473 <khs...@gmail.com>

> If you want to use toString in a standard spec, at least inject "require" in
> the function scope 'either through a global or a closure if you can) so the
> wrapping function does not provide a way to change function(require){var
> foo=require("foo")} for something else like function(load){var
> foo=load("foo"}, but only function(){var foo=require("foo")} IMHO it's at
> least less confusing on what you can or can't do as a programmer (and you
> could even want to do a function(){var r=require, foo=r("foo")}" and it
> should work, but toString will have some problems handling such a use case

Hmmm...  I understand the issue, but it's not really solvable in a
satisfactory way.  I suppose an implementation could have a global
require function that threw a helpful error.  That might not be too
bad of an idea, actually.

Seems a good way, yes : should be at least a recommendation to implementors in the spec ?

I believe that in practice it's a minor issue though.  These are
judgement calls, of course, but I believe the convenience and
scalability of implicit dependency resolution (scanning) outweighs the
unambiguousness of explicit (the dependency array).  Think about big
libraries here.  What would it be like to explicitly specify the
dependencies for this:

https://github.com/khs4473/Codera/blob/master/lib/text-panel.js#L3-29

That would be an array of 19 elements or 19 calls to import/as or
something similar.  Having to remember the boilerplate rules seems
like a small price to pay not to have to maintain such a monster.

I believe scalability is a non-issue :

1. if you have tons of imports, chances are that you are writting a very big module, which seems to me as an anti-pattern in a modularized application, so chances are that you should split your code in submodules, which will have probably have less dependencies, and imports those submodules.

2. it's really easy to make convenient smart modules for regrouping dependencies, e.g. :

//brushes.js
module
.import("./text-brush")


.import("./gutter-brush")
.declare(function(require) {


    return  {
        Text : require("./text-brush"),
        Gutter: require("./gutter-brush")


    }
});

So you can do (I alias this as I don't like telling the path explicitly multiple time for refactoring concern)

module


.import("./brushes").as("brushes")
.declare(function(require) {


 ...
 var Brushes = require("brushes);
 ...
} 

It sounds quite neat to me, you can do this multiple time for this module, have at least half of the dependencies grouped in a logical way, have less requires in you code, and finally have a more concise module by removing some require boilerplate by trading things like "GutterBrush" to "Brush.Gutter" , no ?

> Apart from that, I have added a vote for choosing the best candidate between
> AMD and Wrappings on Bettermeans for Module 2.0, I consider this as a high
> priority for us to build further on CommonJS, I know more and more people
> watching at this everyday and waiting for a "winner" to go on with CommonJS
> for the browser. I'm quite neutral here, I do not find either one or the
> other perfect, but I prefer one that is not perfect for me choice rather
> than a battle between to pretty good candidates.

I totally understand your impatience!  I think we need to have one
more knock-down-drag-out debate before we're ready though.  We need to
put all of the technical aspects on the table so that everyone can see
what it really comes down to.  I need to get my transport spec out
there first though...

Ok for me and thank for having finished your Transport spec today !! :)

--

[James Burke]

On Wed, Jan 5, 2011 at 1:48 PM, Wes Garland <w...@page.ca> wrote:
> Please let me know what you think -- and please try to avoid replying
> directly to this message; change the subject to start a new thread instead.
> Otherwise the ensuing discussion may be difficult to follow, if Q3 2010 was
> any indicator!

How much do you want to consider recent developments? For me, there
are two that should impact something like this:
a) Simple Module modules strawman is gaining more prominence.
b) AMD has some real world adoption.

So for me, I would rather see AMD as a base because it has some
adoption and real world usage, the most of any other wrapped format or
transport alternative. It has been proven in the market.

The arguments against using AMD are bikesheds except for two:
1) Specifying a transport and a wrapped format in one API
2) Reliance on Function toString().

#1: This is necessary to bootstrap module usage on the web. Most
single file web libraries need to specify their ID today because they
can still be used in projects that may not know how to add module IDs
when combining scripts. In order to make sure those libraries work
also when used in other environments, it is best to specify a base
transport format that all implementations understand. Implementations
can provide alternatives but having a commonly understood is
important.

#2: Not using toString creates more typing for the developer, and in
practice toString works well enough. It seems reasonable to expect
better parsing APIs will come to JS environments over time, and asking
the ES committee to formalize the expectations of toString seem
possible, particular given how environments work today.

To me, forcing an ES5 baseline is worse than toString usage since ES5
cuts out older IE browsers which are still in use today. At a minimum
the ES5 baseline should not be in the proposal.

Usage of "module" should be avoided given the possible confusion or
conflict with Simple Modules. For the "module" free variable that may
be used inside a factory function, maybe something more like "self" is
more appropriate. Using AMD as a base avoids the other "module" uses.

I would prefer to see constraints on the use of require to only be for
static dependencies that should be expected to run at the top of the
factory function, and any use of computed require uses a callback
style require. This will also match Simple Modules behavior better.

Those require rules, using AMD as a base, and picking up some of the
Modules 1.1 wording is my preferred approach.

James