Another idea on unifying Transport/C and /D
Kris Zyp
would be possible bridge the two main divergent specs (I know we haven't
discussed this in a while, so all the discussions aren't fresh). Could
we use the transport/C argument list, but allow the argument list to be
repeating sets of id/dependencies/factory:
require.def(id, deps, factory, id, deps, factory, id, deps, factory, ...);
And then say the dependencies/injections arguments are optional
(defaults to ["require", "exports", "module"]), and one additional set
of dependencies can be suffixed as the last parameter. So it can also be
in the form:
require.def(id, factory, id, factory, deps);
This would at least satisfy my desire to have a concise way to define
multiple modules (avoiding multiple calls and pause and resume). It
would also eliminate the concern Tobie had about IE bugs with property
enumeration. And of course normal transport/C usage would be still
supported.
Another example:
require.def("foo", ["bar"], function(bar){
bar.test();
},
"bar", function(require, exports){
require("another");
exports.test = function(){}
}, ["another"]);
Also, one more thought/question, is it possible to make the very first
id optional? Can that be implied from module/script that was requested,
associating the script with the require.def call by the script element's
onload/onreadystate event that require.def precedes (or do browsers
sequence the script executions so you can determine the id by the order
of requested by scripts)?
--
Thanks,
Kris
James Burke
> This would at least satisfy my desire to have a concise way to define
> multiple modules (avoiding multiple calls and pause and resume). It
> would also eliminate the concern Tobie had about IE bugs with property
> enumeration. And of course normal transport/C usage would be still
> supported.
pause and resume are in there to allow legacy scripts to be included
which may define global variables via var. Wrapping them in a function
wrapper would break those kinds of scripts, but I expect that is not
of interest in the context of CommonJS. As far as RequireJS is
concerned, I still want to support legacy scripts for now, so I will
likely still support pause/resume.
That said, it does not mean I could not support the transport proposal
in this thread. Although I do have a question (after next quote):
> Another example:
> require.def("foo", ["bar"], function(bar){
> bar.test();
> },
> "bar", function(require, exports){
> require("another");
> exports.test = function(){}
> }, ["another"]);
What if "bar" depended on "baz"? How would that work for the factory arguments?
require.def("bar", ["baz"], function(baz?, require, exports) {});
As I recall, there were objections from Kris Kowal about treating
require, exports, module as dependencies that could be listed in the
dependency array as strings, then have them listed in same order as
factory function arguments, and I do not like forcing the only
arguments to the factory function to be just require, exports, module.
So I am not sure how to resolve that for RequireJS: I do not mind
supporting a CommonJS transport format, but I would not want to give
up the matching order for dependency string array to factory function
argument names for things that are just coded in RequireJS module
format.
I am not sure you are implying that, I think you are indicating both
would work, but I seem to be missing it.
> Also, one more thought/question, is it possible to make the very first
> id optional? Can that be implied from module/script that was requested,
> associating the script with the require.def call by the script element's
> onload/onreadystate event that require.def precedes (or do browsers
> sequence the script executions so you can determine the id by the order
> of requested by scripts)?
The order is not guaranteed. In particular, IE does not fire the
readystate change directly after executing the script. There is a test
in RequireJS at this location if you want to confirm:
http://github.com/jrburke/requirejs/tree/master/tests/browsertests/scriptload/
Other browsers seem to match them up. Perhaps IE 9 will work better
too. I am having trouble testing IE 9 Platform Preview 4 at the
moment. Looking at the HTML5 spec at this URL:
http://dev.w3.org/html5/spec/Overview.html#executing-a-script-block
Step 5 in the "If the load was successful" section seems to indicate
that onload should fire immediately after the script is executed, but
if an inline script, then the onload is queued in what I believe is
the normal event queue, which to me indicates it may not fire exactly
after the execution of the inline script.
It might be good to get clarification from the HTML5 folks to see if
we could match better scripts with their elements. Hmm, IIRC someone
made a proposal to the HTML5 group that allowed a script to get at its
related element? If so, that would work too. But all future stuff.
Depends on how important the now is to you. I still want to work in
the now, so for RequireJS, the module name always needs to be
specified.
James
Kris Zyp
On 9/1/2010 10:59 PM, James Burke wrote:
> On Wed, Sep 1, 2010 at 6:03 AM, Kris Zyp <kri...@gmail.com> wrote:
>> This would at least satisfy my desire to have a concise way to define
>> multiple modules (avoiding multiple calls and pause and resume). It
>> would also eliminate the concern Tobie had about IE bugs with property
>> enumeration. And of course normal transport/C usage would be still
>> supported.
> pause and resume are in there to allow legacy scripts to be included
> which may define global variables via var. Wrapping them in a function
> wrapper would break those kinds of scripts, but I expect that is not
> of interest in the context of CommonJS. As far as RequireJS is
> concerned, I still want to support legacy scripts for now, so I will
> likely still support pause/resume.
That makes sense. Although even in the RequireJS world, it is reasonable
that in situations where pure RequireJS format is used (everything
enclosed in require.def calls, could be signaled with a build flag or
detected from code), that the build could combine modules with
sequential arguments instead of sequential calls between pause() and
resume() calls, saving some bytes in the built files, right?
> That said, it does not mean I could not support the transport proposal
> in this thread. Although I do have a question (after next quote):
>
>> Another example:
>> require.def("foo", ["bar"], function(bar){
>> bar.test();
>> },
>> "bar", function(require, exports){
>> require("another");
>> exports.test = function(){}
>> }, ["another"]);
> What if "bar" depended on "baz"? How would that work for the factory arguments?
>
> require.def("bar", ["baz"], function(baz?, require, exports) {});
In the example, bar already depends on another module, I just spelled it "another" instead of "baz". But if you want that dependency to be declared directly on that module, like your example, that would be:
require.def("bar", ["baz", "require", "exports"], function(baz, require, exports) {});
(just like Transport/C has always been)
> As I recall, there were objections from Kris Kowal about treating
> require, exports, module as dependencies that could be listed in the
> dependency array as strings, then have them listed in same order as
> factory function arguments, and I do not like forcing the only
> arguments to the factory function to be just require, exports, module.
> So I am not sure how to resolve that for RequireJS: I do not mind
> supporting a CommonJS transport format, but I would not want to give
> up the matching order for dependency string array to factory function
> argument names for things that are just coded in RequireJS module
> format.
I agree, I am definitely suggesting maintaining RequireJS/transport/C
style of mixing injection variables with dependencies for the arguments.
I understand Kowal's concern with this. I just don't agree that it is
really a practical problem. The cost of reserving a few module ids is
negligible (one already reserves module ids to deal with existing
code/modules). If we really wanted to preserve the namespace of module
ids, we could spell require, exports with some reserved characeter like
"!require", "!exports".
> I am not sure you are implying that, I think you are indicating both
> would work, but I seem to be missing it.
>
>> Also, one more thought/question, is it possible to make the very first
>> id optional? Can that be implied from module/script that was requested,
>> associating the script with the require.def call by the script element's
>> onload/onreadystate event that require.def precedes (or do browsers
>> sequence the script executions so you can determine the id by the order
>> of requested by scripts)?
> The order is not guaranteed. In particular, IE does not fire the
> readystate change directly after executing the script. There is a test
> in RequireJS at this location if you want to confirm:
> http://github.com/jrburke/requirejs/tree/master/tests/browsertests/scriptload/
Awesome, great tests. And yes, I can definitely reproduce the scripts
executing out of order, and the onreadystatechange definite does not
fire directly after the execution, but... Everytime I run the test, the
order that the scripts are executed exactly matches the order of the
firing of the onreadystatechange events. If script five executes before
script four, then the onreadystatechange for five will fire before the
onreadystatechange for four. While it is not as convenient as having the
event fire directly after the script executes, it does seem to provide
the information necessary to associate requests with script executions
(and thus anonymous require.def calls with module ids). For example,
annotating from your tests:
one.js script
two.js script
-> four.js script # four gets executed before three
-> three.js script
five.js script
six.js script
seven.js script
eight.js script
one.js loaded
nine.js script
two.js loaded
-> four.js loaded # the onreadystatechange/loaded event preserves the
order (four before three)
-> three.js loaded
five.js loaded
six.js loaded
seven.js loaded
eight.js loaded
nine.js loaded
Am I missing something?
--
Thanks,
Kris
jbrantly
> This would at least satisfy my desire to have a concise way to define
> multiple modules (avoiding multiple calls and pause and resume).
multiple modules with Transport/C in the same file. It does use
multiple calls to require.def of course, but no pause/resume
necessary: http://github.com/jbrantly/yabble/blob/master/test/transportC/tests/multipleDefines/program.js
I'm not sure I see a huge benefit in reducing the calls to one
(perhaps you could explain that better?). On the other hand, I don't
see a problem with it either, so why not? :) I think the main thing is
being able to stick them all into one file/request which is already
possible.
I like how you've solved the explicit injections difference. This
would be a requirement for me to be on board. I prefer D's approach of
a sane default with overrides if desired. I think that your function
signatures might should make a distinction between injects and deps
though. For example:
require.def(id, injects, factory); // normal transport C
require.def(id, factory, deps); // the additional set of dependencies
which are *not* injected, with a default injects of ['require',
'exports', 'module']
Both of the above are possible with your format but injects/deps don't
work quite the same way.
Also, I think you're right about associating script execution with
script onload by order, and thus it should be possible to name the
modules appropriately. Just push the first module into a holding queue
and pop when onload fires (or something like that).
Kris Zyp
On 9/2/2010 2:07 PM, jbrantly wrote:
> On Sep 1, 9:03 am, Kris Zyp <kris...@gmail.com> wrote:
>> This would at least satisfy my desire to have a concise way to define
>> multiple modules (avoiding multiple calls and pause and resume).
> Not sure if you meant it this way, but it's already possible to have
> multiple modules with Transport/C in the same file. It does use
> multiple calls to require.def of course, but no pause/resume
> necessary: http://github.com/jbrantly/yabble/blob/master/test/transportC/tests/multipleDefines/program.js
Say I do a require.ensure(["A"],...), which should trigger a request for
A.js. Let's say the response includes both modules A and B, and they
have a circular dependency:
require.def("A", ["B"], function(B){
});
require.def("B", ["A"], function(A){
});
The problem is that when the first require.def executes, it satisfies
the request for A, but indicates that module B is still needed. A loader
might then request module B, because there is no way for the loader to
know that another require.def call is going to be executed that will
provide module B. Perhaps this would occur in Yabble, although there are
ways around this, maybe you use setTimeout to wait until the current
execution is finished before deciding if unsatisfied dependencies still
need to be requested However, setTimeout(func) solutions are still not
optimally performant because of the minimum delay resolution for
setTimeout in the browser is something like 15ms or something, which can
add up quickly with lots of modules. Being able to explicitly define a
set of modules that should be defined (with a clear finish) before
requesting unsatisfied dependencies is the fastest, most reliable solution.
The other motivation for adding multiple modules in a single call is for
optimized compression of combined files. Doing multiple definitions in a
single call takes less bytes than doing multiple calls. Most things we
define in CommonJS don't need to be very terse. But, in this situation,
we are dealing with code that must be used to wrap every module sent to
browser. Brevity is extremely important for a call that is going be used
for the output of build processes that combine multiple files and minify
JavaScript to squeeze out the best performance. This is arena where
battles occur with just a few byte differences between google closure
compiler, yui compressor, shrinksafe, packer, and uglify fighting to
give the best user experience onband-width constrained mobile devices
and less-than optimal connections.
> I'm not sure I see a huge benefit in reducing the calls to one
> (perhaps you could explain that better?). On the other hand, I don't
> see a problem with it either, so why not? :) I think the main thing is
> being able to stick them all into one file/request which is already
> possible.
>
> I like how you've solved the explicit injections difference. This
> would be a requirement for me to be on board. I prefer D's approach of
> a sane default with overrides if desired. I think that your function
> signatures might should make a distinction between injects and deps
> though. For example:
>
> require.def(id, injects, factory); // normal transport C
> require.def(id, factory, deps); // the additional set of dependencies
> which are *not* injected, with a default injects of ['require',
> 'exports', 'module']
>
> Both of the above are possible with your format but injects/deps don't
> work quite the same way.
That make sense, having injection variables in the trailing dependency
list would be incoherent anyway. And that is a good point, we should
make sure the trailing dependency list could always be used for a clean,
unencumbered namespace for module ids (in case you really need to have a
module named "require" or "exports").
--
Thanks,
Kris
jbrantly
> Perhaps this would occur in Yabble, although there are
> ways around this, maybe you use setTimeout to wait until the current
> execution is finished before deciding if unsatisfied dependencies still
> need to be requested However, setTimeout(func) solutions are still not
> optimally performant because of the minimum delay resolution for
> setTimeout in the browser is something like 15ms or something, which can
> add up quickly with lots of modules. Being able to explicitly define a
> set of modules that should be defined (with a clear finish) before
> requesting unsatisfied dependencies is the fastest, most reliable solution.
the minimum delay but (I think) it only happens once per file (so if
you have your whole application in one file, there's only one delay).
However, your statement about this the fastest, most reliable solution
makes sense.
> The other motivation for adding multiple modules in a single call is for
> optimized compression of combined files. Doing multiple definitions in a
> single call takes less bytes than doing multiple calls. Most things we
> define in CommonJS don't need to be very terse. But, in this situation,
> we are dealing with code that must be used to wrap every module sent to
> browser. Brevity is extremely important for a call that is going be used
> for the output of build processes that combine multiple files and minify
> JavaScript to squeeze out the best performance. This is arena where
> battles occur with just a few byte differences between google closure
> compiler, yui compressor, shrinksafe, packer, and uglify fighting to
> give the best user experience onband-width constrained mobile devices
> and less-than optimal connections.
part (which is the redundant part) would probably get compressed
fairly well using gzip. Everyone *is* using gzip, right? :)
In any case you've answered my question and like I said before I don't
see anything bad about it, so two thumbs up from me.
Kris Zyp
> On Sep 2, 4:44 pm, Kris Zyp <kris...@gmail.com> wrote:
>> Perhaps this would occur in Yabble, although there are
>> ways around this, maybe you use setTimeout to wait until the current
>> execution is finished before deciding if unsatisfied dependencies still
>> need to be requested However, setTimeout(func) solutions are still not
>> optimally performant because of the minimum delay resolution for
>> setTimeout in the browser is something like 15ms or something, which can
>> add up quickly with lots of modules. Being able to explicitly define a
>> set of modules that should be defined (with a clear finish) before
>> requesting unsatisfied dependencies is the fastest, most reliable solution.
> You got it. I use a setTimeout to handle this situation. I'm aware of
> the minimum delay but (I think) it only happens once per file (so if
> you have your whole application in one file, there's only one delay).
> However, your statement about this the fastest, most reliable solution
> makes sense.
But on the otherhand, you could use the onload/onreadystatechange event
to delineate the module definitions. I don't believe there are any extra
delays with this event, and in this case of anonymous modules, you would
have to listen/wait for this event anyway. One could combine multiple
calls with this approach with changing transport/C (in this regard).
.
>> The other motivation for adding multiple modules in a single call is for
>> optimized compression of combined files. Doing multiple definitions in a
>> single call takes less bytes than doing multiple calls. Most things we
>> define in CommonJS don't need to be very terse. But, in this situation,
>> we are dealing with code that must be used to wrap every module sent to
>> browser. Brevity is extremely important for a call that is going be used
>> for the output of build processes that combine multiple files and minify
>> JavaScript to squeeze out the best performance. This is arena where
>> battles occur with just a few byte differences between google closure
>> compiler, yui compressor, shrinksafe, packer, and uglify fighting to
>> give the best user experience onband-width constrained mobile devices
>> and less-than optimal connections.
> Understood, but if we want to get really technical, the "require.def"
> part (which is the redundant part) would probably get compressed
> fairly well using gzip. Everyone *is* using gzip, right? :)
>
> In any case you've answered my question and like I said before I don't
> see anything bad about it, so two thumbs up from me.
Good point.
So to summarize I think there are four parts to what I have suggested as
changes to transport/C:
1. Make the injection+dep array optional (defaulting to ["require",
"exports", "module"]) to make it easy to wrap commonjs modules succinctly.
2. Make the module id optional, determining this from the requested
module and the order of the onload/onreadystatechange event. James
Burke's tests convinced me that this is feasible, and I think it would
be great to support anonymous modules and make it much easier to
hand-code modules without hard-coding them to a specific location.
3. Allow for repeating sets of id/injections/factory in the arguments.
This is in intended to improve the performance of combined modules.
After thinking about the ability of loaders to use the onload event and
gzip's elimination of redundancy, perhaps this doesn't buy us that much.
4. Allow for a trailing dependency array in the arguments. This array
does not mix with injection variables, so it is an encumbered namespace,
and it is also is important for succinct wrapping of commonjs modules so
the dependencies can be declared without having to write out "require",
"exports", "modules".
Thanks,
Kris
4. Allow for a trailing dependency list
--
Thanks,
Kris
James Burke
> That makes sense. Although even in the RequireJS world, it is reasonable
> that in situations where pure RequireJS format is used (everything
> enclosed in require.def calls, could be signaled with a build flag or
> detected from code), that the build could combine modules with
> sequential arguments instead of sequential calls between pause() and
> resume() calls, saving some bytes in the built files, right?
Yes it could. In the context of CommonJS modules only you can get away
with the format you suggest. Just mentioning why pause/resume is
supported in the more general case, but for the purposes of a CommonJS
transport, what you suggest is fine. Although your point about using
the script onload to know when to trace dependencies is a nice way to
to avoid pause/resume. Hmm, although for other environments outside
the browser it means coding in special handling if it were to support
files that had more than one module in the file. That may be OK
though.
> Awesome, great tests. And yes, I can definitely reproduce the scripts
> executing out of order, and the onreadystatechange definite does not
> fire directly after the execution, but... Everytime I run the test, the
> order that the scripts are executed exactly matches the order of the
> firing of the onreadystatechange events. If script five executes before
> script four, then the onreadystatechange for five will fire before the
> onreadystatechange for four.
Ah, great observation! There does seem to be a way to tie them
together. It does make implementation a bit more complicated, but
certainly doable. I am a bit wary of depending on the behavior -- it
would be good to confirm with browser vendors that this behavior
always matches up, but it is promising. I am happy to confirm with
browser vendors if we get general agreement.
However, I want to push it a little further:
To me the criticisms I heard for using a require.def type of syntax as
the CommonJS source file module syntax were:
1) specifying the name of the module was seen as bad, makes moving
files to different directories harder.
2) Explicitly reserving "require", "exports", and "module" as
dependency names in the array to map to the CommonJS module
definitions was seen as bad.
3) Perceived to be more typing.
We could get rid of #1, and only use names when combining more than
one module together in a file.
#2: given that a function callback is used, and dependencies are
listed in an array/function args, "require" is not normally needed
inside the factory function (may want to for some circular
dependency/generic module referencing). The factory function can
return the module exports, so "exports" would not normally be needed
(maybe only for some circular dependency cases). And I believe
"module" is not needed that often either (but there are valid cases
for needing it). So in practice for most modules, they would not need
to specify any of those special dependency names. Yes, they would
still need to be reserved, but as far as practical impact on
developers or code weight, it seems negligible.
#3: is really just bikeshedding. I believe the code weight is about
the same, since return {} is used instead of typing exports more than
once, "require" normally only needs to be typed once, and since return
{} is used for setting exports, constructor functions can be used as
the module export, meaning that you do not have to see extra typing
like "new require('foo').Foo()". So in the end, the typing ends up to
be about the same.
So the differences between "source module" and "transport" end up
being the addition of a name for the module as the first arg. I like
that, and it means getting an easy to debug, fast loading source
module format for the browser. How does that sound? :)
If that goes over well, then a jbrantly mentioned, having extra
require.def calls in the transported file should be negligible with
gzip. It loses some optimization in that a common dependency will be
listed many times in the array of dependency/injections, but I think
gzip also helps a bit there too.
If the inertia for considering a change to the CommonJS source module
format is too great, then what you propose is fine, and I will focus
on obsoleting the old CommonJS source module format via RequireJS
adoption, possibly removing the need for the name in require.def
syntax in RequireJS. I am curious though, who else is interested in
the transport format? You, jbrantly and me, I wonder who else actively
tries to implement one of the transport proposals. I am sure there are
others, it has just been a while since transports were brought up.
James
Kris Zyp
Sounds great! Maybe we can nail down exactly what should be changed in
the transport/C spec (optional first module id, optional injection list,
trailing dependency list) and you can do any additional verification of
script onload association .
> If that goes over well, then a jbrantly mentioned, having extra
> require.def calls in the transported file should be negligible with
> gzip. It loses some optimization in that a common dependency will be
> listed many times in the array of dependency/injections, but I think
> gzip also helps a bit there too.
>
> If the inertia for considering a change to the CommonJS source module
> format is too great, then what you propose is fine, and I will focus
> on obsoleting the old CommonJS source module format via RequireJS
> adoption, possibly removing the need for the name in require.def
> syntax in RequireJS. I am curious though, who else is interested in
> the transport format? You, jbrantly and me, I wonder who else actively
> tries to implement one of the transport proposals. I am sure there are
> others, it has just been a while since transports were brought up.
Well, potentially Dojo, I think that's an important one :).
--
Thanks,
Kris
James Burke
> So to summarize I think there are four parts to what I have suggested as
> changes to transport/C:
> 1. Make the injection+dep array optional (defaulting to ["require",
> "exports", "module"]) to make it easy to wrap commonjs modules succinctly.
> 2. Make the module id optional, determining this from the requested
> module and the order of the onload/onreadystatechange event. James
> Burke's tests convinced me that this is feasible, and I think it would
> be great to support anonymous modules and make it much easier to
> hand-code modules without hard-coding them to a specific location.
> 3. Allow for repeating sets of id/injections/factory in the arguments.
> This is in intended to improve the performance of combined modules.
> After thinking about the ability of loaders to use the onload event and
> gzip's elimination of redundancy, perhaps this doesn't buy us that much.
> 4. Allow for a trailing dependency array in the arguments. This array
> does not mix with injection variables, so it is an encumbered namespace,
> and it is also is important for succinct wrapping of commonjs modules so
> the dependencies can be declared without having to write out "require",
> "exports", "modules".
I think gzipping makes the need for #3 and #4 less necessary. If #4
really made your side of processing noticeably better, I could
probably live with it.
I do like #2, want to prototype it some more.
#1 is a bit trickier. If no explicit dependencies are specified in
RequireJS, then I do not bother with creating an exports object, since
I allow return in the factory function to define the exports,
similarly, I do not bother with manufacturing a "module" object. So I
am concerned about doing extra work when it is not needed. At first
glance, I prefer just to be explicit with the dependencies, but may be
able to be convinced otherwise. Here again I think gzip helps to
collapse the size of those things if it is a common pattern.
I think the bigger thing that is indicated by this approach is that
"require", "module" and "exports" do become reserved strings that map
to the free variables expected in traditional CommonJS modules. I want
to call that out since it was contentious before.
James
Kris Zyp
I had hoped the trailing dependency argument would mitigate this
problem. However, I think stating it as two optional arguments is too
confusing. Let me restate my proposal. The require.def could take two forms:
require.def(id?, injections, factory); // existing transport/C (with
optional first arg, hopefully)
require.def(id?, factory, dependencies);
This second form is like the first, except that arguments to the factory
are always "require", "module" and "exports", and the third argument has
no reserved module names ("require" would request the module with that
name). Basically, this form would be a super easy way to wrap existing
CommonJS modules with minimal alteration to require.def and without
reserved strings in the dependency list.
--
Thanks,
Kris
Kris Zyp
* require.def should not depend |this|. You should be able to use
require.def in both forms:
require.def(...);
var def = require.def;
def(...);
This is already true for RequireJS and Yabble, but seems like it should
be explicitly stated in the specification.
* If we are going to support anonymous modules, we should certainly also
allow relative module ids. This is an important element to keeping
modules portable as well. Relative ids should be supported in the
dependency list and I believe also in the module id argument. The
dependency list module ids should be resolved relative to the module
that is being defined.
Allowing relative ids for the module id (the first param) helps to
address a broader issue of how packages could be accessed from a client
side loader. How do we create "built" packages for browsers? If I
request a module "a" from a package at http://somesite.com/foo, and I
have built "a" to also include it's dependencies "b" what should
http://somesite.com/foo/lib/a.js return? One possibility is it could
assume that it is being mapped to the "foo" namespace:
require.def("foo/a",["./b"], function(b){...});
require.def("foo/b",[], function(){...});
but then the modules are hard-coded to a particular expected location. A
much more portable solution is anonymous modules plus relative modules:
require.def(["./b"], function(b){...});
require.def("./b",[], function(){...});
I believe we would need to specify that relative module ids are resolved
relative to the last require.def call.
* Is a "transport" even still the right name for require.def anymore? As
we add the ability to do anonymous modules, maybe relative ids, and with
the existing ability to conveniently map dependencies to arguments, it
seems like it is not strictly a transport (I think Kris Kowal has
pointed this out before), it is perhaps an async module format, module
registration API or module definition API.
Thanks,
Kris
James Burke
> but then the modules are hard-coded to a particular expected location. A
> much more portable solution is anonymous modules plus relative modules:
>
> require.def(["./b"], function(b){...});
> require.def("./b",[], function(){...});
>
> I believe we would need to specify that relative module ids are resolved
> relative to the last require.def call.
I do not understand, I would have thought not specifying the module
name at all was good enough. I suppose you may want to explain more by
what a "built" package means.
James
Kris Zyp
By built package, I mean a package that may have modules that have
multiple dependencies in a single file like we do with builds in Dojo.
Taking an example from Dojo, the DataGrid.js we distribute to CDNs
(http://o.aolcdn.com/dojo/1.5/dojox/grid/DataGrid.xd.js) includes
multiples modules. Since all these modules are a single file
(DataGrid.js), only the DataGrid module can be anonymous, but the other
ones could at least be defined relatively so that aren't harded coded to
a certain path:
require.def(["./_Grid","./DataSelection",...], function(){ /*DataGrid
module */});
require.def("./_Grid",[...], function(){ /*_Grid module*/});
require.def("./DataSelection",[...], function(){ /*_Grid module*/});
...
Does that make sense?
--
Thanks,
Kris
James Burke
> By built package, I mean a package that may have modules that have
> multiple dependencies in a single file like we do with builds in Dojo.
> Taking an example from Dojo, the DataGrid.js we distribute to CDNs
> (http://o.aolcdn.com/dojo/1.5/dojox/grid/DataGrid.xd.js) includes
> multiples modules. Since all these modules are a single file
> (DataGrid.js), only the DataGrid module can be anonymous, but the other
> ones could at least be defined relatively so that aren't harded coded to
> a certain path:
That would only work if modules from just one package were included in
the built file. I am not sure how common that is, or if it is worth
supporting. Once there modules from two packages in the file, a full
ID is needed.
For me, if we are talking about optimized, built code, it means the
module IDs have been set/burned in, so I do not see a great need to
have relative module IDs in that case.
James
Kris Zyp
Yes, that makes sense. But relative ids in the dependency list for
anonymous modules would certainly still be reasonable (since it is
represents pre-built source code that may be built to any location),
right? And I would think it would be pretty easy to support as well.
--
Thanks,
Kris
James Burke
> Yes, that makes sense. But relative ids in the dependency list for
> anonymous modules would certainly still be reasonable (since it is
> represents pre-built source code that may be built to any location),
> right? And I would think it would be pretty easy to support as well.
Agreed.
James
Kris Zyp
relative ids (unless someone else wants to). And I still think that
Transport/C deserves a better name. Any objections to renaming it to
"Module Registration" or "Asynchronous Module Definition" specification?
Also, would relative ids in dependency lists *only* be supported for
anonymous modules or should it be for all modules? Is it more work to
disable this for non-anonymous modules if relative id is supported?
--
Thanks,
Kris
James Burke
> OK, I'll update the proposal with the optional first argument and
> relative ids (unless someone else wants to). And I still think that
> Transport/C deserves a better name. Any objections to renaming it to
> "Module Registration" or "Asynchronous Module Definition" specification?
For me, it is a module definition, but if it helps for the purposes of
this group to call it a Transport proposal, that is fine. I would be
OK if it was revved to a Transport/E or whatever the next letter is,
just in case support for the anonymous modules falls through, but
reusing /C also works if you already did changes.
> Also, would relative ids in dependency lists *only* be supported for
> anonymous modules or should it be for all modules? Is it more work to
> disable this for non-anonymous modules if relative id is supported?
Seems fine to allow relative ids in the dependency list when name is
specified for the module. That works today in RequireJS, and I do not
think it forces any new burdens on the loader, since relative ids need
to be supported, they are always resolved in relation to the module id
for the require.def call. In the anonymous require.def case, the
module id gets applied later by the system before dependencies are
dealt with anyway.
James
Kris Zyp
ideas from Transport/C and recent changes discussed:
http://wiki.commonjs.org/wiki/Modules/AsynchronousDefinition
Feedback welcome, of course.
Thanks,
Kris
Tom Robinson
If you perceived the issues with loading CommonJS modules in a browser to be a big deal then it should have been dealt with a year and a half ago when we were defining CommonJS modules. I thought we had concluded it wasn't a major problem.
I don't have a problem with CommonJS modules in the browser because I've been using a similar system (Cappuccino's load system) for several years without major issues.
</rant>
-tom
> --
> You received this message because you are subscribed to the Google Groups "CommonJS" group.
> To post to this group, send email to comm...@googlegroups.com.
> To unsubscribe from this group, send email to commonjs+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/commonjs?hl=en.
>
James Burke
> So we now have two different ways of defining CommonJS modules? There was a reason the transport specs were named "transport". This significantly complicates the CommonJS module story.
>
> If you perceived the issues with loading CommonJS modules in a browser to be a big deal then it should have been dealt with a year and a half ago when we were defining CommonJS modules. I thought we had concluded it wasn't a major problem.
I have just as much frustration on the other side. A ServerJS group
switching names to CommonJS, but then treating the browser as second
class has not sat well with me. I did try to engage, about a year ago,
but was informed that the browser was not the first target for this
group. It would be bad to assume that this group has had enough of a
cross section of JS developers to know if they got the format right,
particularly given the ServerJS origins.
I do not think I got it perfect with Transport/C-RequireJS, and I am
happy that Kris Zyp has noticed a pattern I missed before that would
allow not specifying the name in a require.def call if there is just
one module in a script.
I believe it brings the format closer to something that could be used
in the browser directly and meets the goals for CommonJS. As I
mentioned in the other thread[1], the issues of typing is a bikeshed,
and the reservation of "require", "module" and "exports" do not seem
that bad, particularly given that most modules will not need them as
much when using a format that has a function wrapper, and it reduces
the overall typing in the format.
> I don't have a problem with CommonJS modules in the browser because I've been using a similar system (Cappuccino's load system) for several years without major issues.
Cappuccino and Objective-J are not what I would consider mainstream
front end development though. Not coding in the language that the
browser already knows is not natural for many front end developers,
including me. Which is fine, the browsers are capable enough to allow
variations on that spectrum. You find it works for you. We in Dojo
have found xhr+eval to be workable for many years, but it does not
work as well as something that uses script tags. We know this from
experience. It makes adoption of the toolkit harder. There are
complications with xdomain loading, debugging and speed. They are
workable, but there are real costs. Using a function wrapper avoids
those costs. It may be tempting to think the tradeoff is the amount of
typing, but as mentioned[1], that can be a hard argument to make
definitively. YUI seems to operate well with a function wrapper too.
I do not want to start a flame war on this, we are likely not to get
anywhere on it. I will try not to respond more on this thread about
it. I just wanted to point out that there is a nontrivial number of
developers that feel differently, and not all want to try to engage
with this group because it has not directly impacted them yet, and
there has not been much in it for them to participate, since it has
been mentioned a couple times that CommonJS is mainly concerned with
non-browser environments. If you want to keep it that way, fine by me,
but know that other solutions may gain more ground, regardless of the
transport or module format labels.
[1] http://groups.google.com/group/requirejs/msg/6922316ab3b66bbb
James
Tom Robinson
On Sep 11, 2010, at 2:24 AM, James Burke wrote:
> On Fri, Sep 10, 2010 at 10:45 PM, Tom Robinson <tlrob...@gmail.com> wrote:
>> So we now have two different ways of defining CommonJS modules? There was a reason the transport specs were named "transport". This significantly complicates the CommonJS module story.
>>
>> If you perceived the issues with loading CommonJS modules in a browser to be a big deal then it should have been dealt with a year and a half ago when we were defining CommonJS modules. I thought we had concluded it wasn't a major problem.
>
> I have just as much frustration on the other side. A ServerJS group
> switching names to CommonJS, but then treating the browser as second
> class has not sat well with me. I did try to engage, about a year ago,
> but was informed that the browser was not the first target for this
> group. It would be bad to assume that this group has had enough of a
> cross section of JS developers to know if they got the format right,
> particularly given the ServerJS origins.
Personally I always intended to use ServerJS for more than servers and never liked the ServerJS name. But I also never thought there would be significant opposition to using the proposed module system in the browser.
> I do not think I got it perfect with Transport/C-RequireJS, and I am
> happy that Kris Zyp has noticed a pattern I missed before that would
> allow not specifying the name in a require.def call if there is just
> one module in a script.
>
> I believe it brings the format closer to something that could be used
> in the browser directly and meets the goals for CommonJS. As I
> mentioned in the other thread[1], the issues of typing is a bikeshed,
> and the reservation of "require", "module" and "exports" do not seem
> that bad, particularly given that most modules will not need them as
> much when using a format that has a function wrapper, and it reduces
> the overall typing in the format.
>
>> I don't have a problem with CommonJS modules in the browser because I've been using a similar system (Cappuccino's load system) for several years without major issues.
>
> Cappuccino and Objective-J are not what I would consider mainstream
> front end development though. Not coding in the language that the
> browser already knows is not natural for many front end developers,
> including me. Which is fine, the browsers are capable enough to allow
> variations on that spectrum. You find it works for you.
The language is irrelevant to this discussion so I'm going to ignore your superficial criticism on that. The Objective-J loader can load standard JavaScript, and we've even modified it to support CommonJS modules (it was a couple dozen line change). It works with Firebug and WebKit debuggers (and presumably anything else that supports @sourceURL). It's also got something similar to the transport format for loading cross domain, but that format is automatically generated by a tool upon deployment, of course.
Does CommonJS even need to appease "mainstream front end" developers anyway?
> We in Dojo
> have found xhr+eval to be workable for many years, but it does not
> work as well as something that uses script tags. We know this from
> experience. It makes adoption of the toolkit harder.
I suspect confusing people with multiple module formats is going to hurt adoption more.
> There are
> complications with xdomain loading, debugging and speed. They are
> workable, but there are real costs. Using a function wrapper avoids
> those costs. It may be tempting to think the tradeoff is the amount of
> typing, but as mentioned[1], that can be a hard argument to make
> definitively. YUI seems to operate well with a function wrapper too.
xdomain and speed are solved by the *transport* format. Debugger support is solved by @sourceURL, or the transport format.
What other value does using a script tag provide? Certainly not familiarity to "mainstream" JavaScript developers, since using any kind of dependency management system is vastly different than manually including script tags.
> I do not want to start a flame war on this, we are likely not to get
> anywhere on it. I will try not to respond more on this thread about
> it. I just wanted to point out that there is a nontrivial number of
> developers that feel differently, and not all want to try to engage
> with this group because it has not directly impacted them yet, and
> there has not been much in it for them to participate, since it has
> been mentioned a couple times that CommonJS is mainly concerned with
> non-browser environments. If you want to keep it that way, fine by me,
> but know that other solutions may gain more ground, regardless of the
> transport or module format labels.
It certainly seems we're not going to convince each other. If the majority of CommonJS contributors agree with you then I'll back off, but so far it seems like it's mostly you and a couple others.
I'll just say my preference if you're going to invent a new module format is to not call it CommonJS modules. We absolutely should not have two incompatible ways of defining modules.
If we do end up having two formats then we need to specify that implementations should support both, otherwise what's the point?
Alternatively, you could make your boilerplate something backwards compatible with CommonJS modules, something like:
(require.def || function(f) { f(exports, require, module); })(function(exports, require, module) {
// ...
}, "foo", ["bar", "baz"]);
Obviously it's a bit verbose and error prone. Not a problem if it's auto-generated though.
-tom
Kris Zyp
On 9/11/2010 5:42 AM, Tom Robinson wrote:
> On Sep 11, 2010, at 2:24 AM, James Burke wrote:
>
>> On Fri, Sep 10, 2010 at 10:45 PM, Tom Robinson <tlrob...@gmail.com> wrote:
>>> So we now have two different ways of defining CommonJS modules? There was a reason the transport specs were named "transport". This significantly complicates the CommonJS module story.
But we never did have consensus on the transport API. We have always had
multiple APIs of CommonJS module transport with out a clear winning
proposal. The basic underlying difference of approach between
hand-coding an asynchronous module vs auto-generated wrappings of
CommonJS has constantly divided such discussions. The proposal doesn't
introduce anything new (well it altered the transport/C API slightly,
making the first param optional and removed require.pause and
require.resume), it simply calls Transport/C what it really is. If I got
the name wrong, suggest something different (I asked for suggestions
before creating the wiki page), but it ain't a "transport".
> [snip]
> xdomain and speed are solved by the *transport* format.
Absolutely, but the transport methodology is to wrap a CommonJS module.
Doing this by hand is a lot of extra work (AMD is vastly less effort),
and doing it with a tool creates an extra tool dependency. Within Dojo
(which wants to use CommonJS), requiring the use of a tool (for
wrapping) is a non-starter.
--
Thanks,
Kris
Wes Garland
> I'll just say my preference if you're going to invent a new module format is to not call it CommonJS
> modules. We absolutely should not have two incompatible ways of defining modules.
I think in order to resolve this issue, though, we need to look hard at the REAL issues with using CommonJS on the browser. They all boil down to: how can we load modules asynchronously?
Async loading was never a design consideration for CommonJS modules. This is not all that surprising, given it's origins -- async is not really a requirement on the server, and those who are that server-side developers should be forced into an async-only paradigm are wrong.
When I first looked at using CommonJS on the browser, it was my assumption that modules would simply be loaded ahead of time, without consideration for lazy-loading. It is still my belief that this is a perfectly via technique, and that browser authors would easily learn to load the modules they need in an effecient way: after all, we all learned about image preloading and CSS sprites as those techniques became necessary in our work.
With respect to transports, I have always seen a Transport + a Web-Client Framework to go hand-in-hand in presenting CommonJS modules to the user via the require() statement. The mechanics of doing so strike me mostly as a tradeoffs/tuning exercise, and that modifying modules depending on which transport system you happen to favour is quite smelly.
Kris Zyp:
What's to stop Dojo from wrapping modules on the client side, when there is no server-side support?
Wes
--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
James Burke
> The language is irrelevant to this discussion so I'm going to ignore your superficial criticism on that. The Objective-J loader can load standard JavaScript, and we've even modified it to support CommonJS modules (it was a couple dozen line change). It works with Firebug and WebKit debuggers (and presumably anything else that supports @sourceURL). It's also got something similar to the transport format for loading cross domain, but that format is automatically generated by a tool upon deployment, of course.
I pointed out the language to illustrate that since you got something
that works with your choice of language, it does not translate to a
more general statement about the module format being ideal for front
end development. I mentioned YUI to illustrate someone else's
perspective of getting something to work just fine for them, but it
happens to use a function wrapper format. I mentioned Dojo because it
does exactly all the things you just described you do for standard
JavaScript in the Objective-J loader. Those tradeoffs are real.
@sourceURL is not helpful if there is a syntax error in the file, does
not work for IE. It makes development slower, speed is not just about
deployment speed. An extra tool step to get xdomain support adds more
steps, more things to know to deploy code.
The possibility to avoid those problems with a function wrapper format
seem worth it particularly since the amount of typing for it is not
that much different from the existing CommonJS module format, and it
allows setting the exported value in a natural way. While setting
exports may be contentious, the fact that it is implemented in more
than one implementation should indicate it is not a feature wanted by
a vocal minority that has no implementation to back up the talk.
> Does CommonJS even need to appease "mainstream front end" developers anyway?
You call it appeasement, I call it getting them involved in the
discussion, see what works best for them and if there is enough
overlap in goals to agree on something that is generally useful. I
think there is.
> I suspect confusing people with multiple module formats is going to hurt adoption more.
Right now there are three things someone needs to understand in
CommonJS to effectively use code in the browser:
- A module format
- A transport proposal (but which one?)
- An async require syntax proposal (but which one?)
It is confusing now. Compare that with one format proposal that would
define a module syntax, then say, "for optimizing transport/including
multiple modules in one file, you MAY place the ID of the module as
the first argument to require.def. If you do not want to define a
module, but just use some dependencies, use the same argument syntax
as require.def anonymous modules, but drop the .def property access".
That said, I am completely fine just calling the proposal from Kris
Zyp Transport/E. Just expect it to be pushed as a source format for
modules for some systems, and we can let implementation and adoption
sort it out.
James
Tom Robinson
On Sep 11, 2010, at 5:17 AM, Kris Zyp wrote:
>
>
> On 9/11/2010 5:42 AM, Tom Robinson wrote:
>> On Sep 11, 2010, at 2:24 AM, James Burke wrote:
>>
>>> On Fri, Sep 10, 2010 at 10:45 PM, Tom Robinson <tlrob...@gmail.com> wrote:
>>>> So we now have two different ways of defining CommonJS modules? There was a reason the transport specs were named "transport". This significantly complicates the CommonJS module story.
> But we never did have consensus on the transport API. We have always had
> multiple APIs of CommonJS module transport with out a clear winning
> proposal. The basic underlying difference of approach between
> hand-coding an asynchronous module vs auto-generated wrappings of
> CommonJS has constantly divided such discussions. The proposal doesn't
> introduce anything new (well it altered the transport/C API slightly,
> making the first param optional and removed require.pause and
> require.resume), it simply calls Transport/C what it really is. If I got
> the name wrong, suggest something different (I asked for suggestions
> before creating the wiki page), but it ain't a "transport".
Sorry, I hadn't been following the details of the transport proposals closely. They were still being called "transport", so I assumed they were still intended to be used only as a transport mechanism (though RequireJS's abuse of the transport proposals bugged me from the beginning). It has come to my attention that that is not the case, and I'm not happy about the change in direction.
>
>> [snip]
>> xdomain and speed are solved by the *transport* format.
> Absolutely, but the transport methodology is to wrap a CommonJS module.
> Doing this by hand is a lot of extra work (AMD is vastly less effort),
> and doing it with a tool creates an extra tool dependency. Within Dojo
> (which wants to use CommonJS), requiring the use of a tool (for
> wrapping) is a non-starter.
And introducing a new first-class module format intended to be written by hand and distributed as source is not acceptable to me. It's crazy to have two incompatible module formats, and if popular projects like Dojo start using the transport format as their module format it's going to confuse the hell out of people coming to CommonJS who see a totally different format elsewhere (as if we don't already have enough non-standard/incompatible features between implementations!)
How is this significantly different than Dojo's current system, or Cappuccino's, or SproutCore's, or YUI's, or basically any other large-ish JavaScript framework. Which frameworks actually use function wrapper boilerplate around every file (in their source)?
To be clear, I have no problem with the idea of a standard module transport format, only with the idea that the transport format should be a first-class module format intended to be written by hand, distributed as source.
-tom
Kris Zyp
On 9/11/2010 7:11 AM, Tom Robinson wrote:
> On Sep 11, 2010, at 5:17 AM, Kris Zyp wrote:
>
>>
>> On 9/11/2010 5:42 AM, Tom Robinson wrote:
>>> On Sep 11, 2010, at 2:24 AM, James Burke wrote:
>>>
>>>> On Fri, Sep 10, 2010 at 10:45 PM, Tom Robinson <tlrob...@gmail.com> wrote:
>>>>> So we now have two different ways of defining CommonJS modules? There was a reason the transport specs were named "transport". This significantly complicates the CommonJS module story.
>> But we never did have consensus on the transport API. We have always had
>> multiple APIs of CommonJS module transport with out a clear winning
>> proposal. The basic underlying difference of approach between
>> hand-coding an asynchronous module vs auto-generated wrappings of
>> CommonJS has constantly divided such discussions. The proposal doesn't
>> introduce anything new (well it altered the transport/C API slightly,
>> making the first param optional and removed require.pause and
>> require.resume), it simply calls Transport/C what it really is. If I got
>> the name wrong, suggest something different (I asked for suggestions
>> before creating the wiki page), but it ain't a "transport".
> Sorry, I hadn't been following the details of the transport proposals closely. They were still being called "transport", so I assumed they were still intended to be used only as a transport mechanism (though RequireJS's abuse of the transport proposals bugged me from the beginning). It has come to my attention that that is not the case, and I'm not happy about the change in direction.
>
>>> [snip]
>>> xdomain and speed are solved by the *transport* format.
>> Absolutely, but the transport methodology is to wrap a CommonJS module.
>> Doing this by hand is a lot of extra work (AMD is vastly less effort),
>> and doing it with a tool creates an extra tool dependency. Within Dojo
>> (which wants to use CommonJS), requiring the use of a tool (for
>> wrapping) is a non-starter.
> And introducing a new first-class module format intended to be written by hand and distributed as source is not acceptable to me. It's crazy to have two incompatible module formats, and if popular projects like Dojo start using the transport format as their module format it's going to confuse the hell out of people coming to CommonJS who see a totally different format elsewhere (as if we don't already have enough non-standard/incompatible features between implementations!)
I don't see these as directly competing, they serve different roles.
There is one and only one CommonJS module format, which defines a set of
guaranteed free variables that will be available for synchronously
requiring modules and exporting functionality within a CommonJS
controlled evaluation context. Asynchronous module definition API, on
the other hand, defines of way of registering a module from outside a
CommonJS controlled evaluation context. The expected context is no
different than the transport API, but it differs in the purpose of being
optimized for hand-coding.
> How is this significantly different than Dojo's current system, or Cappuccino's, or SproutCore's, or YUI's, or basically any other large-ish JavaScript framework. Which frameworks actually use function wrapper boilerplate around every file (in their source)?
Dojo has used synchronous dependency loading for years, and most of the
committers (who have used this for years) are pretty much in agreement
that it is wrong and don't want it anymore. Asynchronous dependency
loading is necessary, regardless of whether is called a transport API or
async module definition API.
IIUC, YUI added dependency loading in version 3. And it appears that
they do indeed use a function wrapper around each module. If you look at
the source, you'll see a YUI.add(moduleId, factory, version,
options-with-dependency-list) around each module.
> To be clear, I have no problem with the idea of a standard module transport format, only with the idea that the transport format should be a first-class module format intended to be written by hand, distributed as source.
So your suggestion is that a client side library use synchronous loading
+ eval-based module loading? Or use CommonJS raw format plus a tool for
transport wrapping? I know in the context of Dojo, the former is what
years of experience have lead us away from. I suggested the latter on
the Dojo ML, and it was thoroughly shot down, Dojo won't limit its
availability to those who are willing to use a transport tool.
-- Thanks, Kris
Kris Zyp
On 9/11/2010 9:56 AM, James Burke wrote:
> [snip]
> That said, I am completely fine just calling the proposal from Kris
> Zyp Transport/E. Just expect it to be pushed as a source format for
> modules for some systems, and we can let implementation and adoption
> sort it out.
I thought the letters were for creating alternate proposals (someone
correct me if I am wrong). This is clearly just an upgrade to
Transport/C (optional first arg, and removal of pause and resume due to
our realization that they weren't necessary) and not meant to compete
with Transport/C, and thus should be draft 2 if it keeps the name. It is
the name (and not the changes) that seem to be contentious. We could
revert back to calling it a "transport", but really? That is just a
lousy name for it (it is appropriate for Transport/D, but not
AMD/Transport/C). I don't see how we are helping the process but giving
it an inappropriate name, just so it has the same name as another API
within CommonJS. I'd love to hear other name suggestions (hopefully
better than "transport").
But perhaps the issue isn't so much with the name change, but just the
fact that AMD/transport/c exists, regardless of the name or minor updates...
--
Thanks,
Kris
Tom Robinson
> On 9/11/2010 7:11 AM, Tom Robinson wrote:
>>> [snip]
>>>
>> And introducing a new first-class module format intended to be written by hand and distributed as source is not acceptable to me. It's crazy to have two incompatible module formats, and if popular projects like Dojo start using the transport format as their module format it's going to confuse the hell out of people coming to CommonJS who see a totally different format elsewhere (as if we don't already have enough non-standard/incompatible features between implementations!)
> I don't see these as directly competing, they serve different roles.
> There is one and only one CommonJS module format, which defines a set of
> guaranteed free variables that will be available for synchronously
> requiring modules and exporting functionality within a CommonJS
> controlled evaluation context. Asynchronous module definition API, on
> the other hand, defines of way of registering a module from outside a
> CommonJS controlled evaluation context. The expected context is no
> different than the transport API, but it differs in the purpose of being
> optimized for hand-coding.
Ok, but if I can't share modules between the client and server then using CommonJS on the client loses some of it's appeal.
>> How is this significantly different than Dojo's current system, or Cappuccino's, or SproutCore's, or YUI's, or basically any other large-ish JavaScript framework. Which frameworks actually use function wrapper boilerplate around every file (in their source)?
> Dojo has used synchronous dependency loading for years, and most of the
> committers (who have used this for years) are pretty much in agreement
> that it is wrong and don't want it anymore. Asynchronous dependency
> loading is necessary, regardless of whether is called a transport API or
> async module definition API.
We can (and already do) support async loading of regular CommonJS modules without wrappers using async XHR. See my second paragraph below.
> IIUC, YUI added dependency loading in version 3. And it appears that
> they do indeed use a function wrapper around each module. If you look at
> the source, you'll see a YUI.add(moduleId, factory, version,
> options-with-dependency-list) around each module.
>
>
>> To be clear, I have no problem with the idea of a standard module transport format, only with the idea that the transport format should be a first-class module format intended to be written by hand, distributed as source.
>
> So your suggestion is that a client side library use synchronous loading
> + eval-based module loading? Or use CommonJS raw format plus a tool for
> transport wrapping? I know in the context of Dojo, the former is what
> years of experience have lead us away from. I suggested the latter on
> the Dojo ML, and it was thoroughly shot down, Dojo won't limit its
> availability to those who are willing to use a transport tool.
I would advocate both, except use async loading instead of sync. Use the eval-based loader during development for ease of use, then optimize by bundling the modules in the transport format using a tool during deployment (when you'll most likely be running other build processes like minification anyway). This has always been our approach in Cappuccino. If you want to load libraries from a CDN, just make sure they're deployed in the transport format. IMO the loader should support mixing of both module formats side-by-side, so I can point my jQuery (or whatever) package to a CDN and my own modules locally during development.
Regarding async vs. sync loading, I thought it was well understood that you can do asynchronous module loading with regular CommonJS modules and no wrappers or tool. You asynchronously *download* all the modules and their transitive dependencies up front, then synchronously *execute* them. It adds a small amount of complexity to the loader, but not much. This is the reason we mandate strings passed to require() are literals, so that they're statically analyzable.
The transport formats get you a few distinct things:
1) The benefits from loading in a script tag:
a) cross-domain loading
b) better debugging in debuggers that don't support @sourceURL
c) maybe faster parsing/execution (?)
2) Combining multiple modules into a single file.
3) Dependency information that you don't have to parse out yourself.
(note that I don't include async loading because that can also be done with XHR+eval)
So 1a, 1c, 2, and 3 are probably only important during deployment. 1b is the only one that might matter during development, if you're using a debugger that doesn't support @sourceURL, but that's a tradeoff I'm willing to make since the most common debuggers (by far, Firebug and WebKit) support it.
There are already multiple implementations of CommonJS modules for browsers using this async loading / sync executing pattern. I think Yabble, Tiki, and my modified Objective-J loader all do this, off the top of my head.
-tom
Kris Zyp
On 9/11/2010 12:52 PM, Tom Robinson wrote:
> On Sep 11, 2010, at 10:54 AM, Kris Zyp wrote:
>
>> On 9/11/2010 7:11 AM, Tom Robinson wrote:
>>>> [snip]
>>>>
>>> And introducing a new first-class module format intended to be written by hand and distributed as source is not acceptable to me. It's crazy to have two incompatible module formats, and if popular projects like Dojo start using the transport format as their module format it's going to confuse the hell out of people coming to CommonJS who see a totally different format elsewhere (as if we don't already have enough non-standard/incompatible features between implementations!)
>> I don't see these as directly competing, they serve different roles.
>> There is one and only one CommonJS module format, which defines a set of
>> guaranteed free variables that will be available for synchronously
>> requiring modules and exporting functionality within a CommonJS
>> controlled evaluation context. Asynchronous module definition API, on
>> the other hand, defines of way of registering a module from outside a
>> CommonJS controlled evaluation context. The expected context is no
>> different than the transport API, but it differs in the purpose of being
>> optimized for hand-coding.
> Ok, but if I can't share modules between the client and server then using CommonJS on the client loses some of it's appeal.
Using the standard CommonJS module format with the transport API is
totally the way to go if you are doing SSJS, I completely agree (that's
why I wrote transporter). For something like Dojo, probably less than
10% (maybe less than 1%) are using SSJS. Forcing the 90% who aren't
using JS anywhere but in the browser to use a format that is can't be
directly loading in script tags without additional processing is
untenable in Dojo.
>>> How is this significantly different than Dojo's current system, or Cappuccino's, or SproutCore's, or YUI's, or basically any other large-ish JavaScript framework. Which frameworks actually use function wrapper boilerplate around every file (in their source)?
>> Dojo has used synchronous dependency loading for years, and most of the
>> committers (who have used this for years) are pretty much in agreement
>> that it is wrong and don't want it anymore. Asynchronous dependency
>> loading is necessary, regardless of whether is called a transport API or
>> async module definition API.
> We can (and already do) support async loading of regular CommonJS modules without wrappers using async XHR. See my second paragraph below.
Right, Nodule does this (static analysis for the purpose of async), but
you have to control the entire loading process and Dojo does not. Once a
sync require is made (from any script, inline or in a file), all the
transient dependencies have to sync loaded as well.
>> IIUC, YUI added dependency loading in version 3. And it appears that
>> they do indeed use a function wrapper around each module. If you look at
>> the source, you'll see a YUI.add(moduleId, factory, version,
>> options-with-dependency-list) around each module.
>>
>>
>>> To be clear, I have no problem with the idea of a standard module transport format, only with the idea that the transport format should be a first-class module format intended to be written by hand, distributed as source.
>> So your suggestion is that a client side library use synchronous loading
>> + eval-based module loading? Or use CommonJS raw format plus a tool for
>> transport wrapping? I know in the context of Dojo, the former is what
>> years of experience have lead us away from. I suggested the latter on
>> the Dojo ML, and it was thoroughly shot down, Dojo won't limit its
>> availability to those who are willing to use a transport tool.
> I would advocate both, except use async loading instead of sync. Use the eval-based loader during development for ease of use, then optimize by bundling the modules in the transport format using a tool during deployment (when you'll most likely be running other build processes like minification anyway). This has always been our approach in Cappuccino. If you want to load libraries from a CDN, just make sure they're deployed in the transport format. IMO the loader should support mixing of both module formats side-by-side, so I can point my jQuery (or whatever) package to a CDN and my own modules locally during development.
That is the way it currently works in Dojo.
> Regarding async vs. sync loading, I thought it was well understood that you can do asynchronous module loading with regular CommonJS modules and no wrappers or tool. You asynchronously *download* all the modules and their transitive dependencies up front, then synchronously *execute* them. It adds a small amount of complexity to the loader, but not much. This is the reason we mandate strings passed to require() are literals, so that they're statically analyzable.
>
> The transport formats get you a few distinct things:
>
> 1) The benefits from loading in a script tag:
> a) cross-domain loading
> b) better debugging in debuggers that don't support @sourceURL
> c) maybe faster parsing/execution (?)
> 2) Combining multiple modules into a single file.
> 3) Dependency information that you don't have to parse out yourself.
>
> (note that I don't include async loading because that can also be done with XHR+eval)
>
> So 1a, 1c, 2, and 3 are probably only important during deployment. 1b is the only one that might matter during development, if you're using a debugger that doesn't support @sourceURL, but that's a tradeoff I'm willing to make since the most common debuggers (by far, Firebug and WebKit) support it.
From what I've seen, and from what I remember of the test results, the
performance difference is dramatic. And this is enormously important to
me for development. Faster loading equals faster development. Also, even
with @sourceURL, stack traces don't work in any browser, AFAICT.
> There are already multiple implementations of CommonJS modules for browsers using this async loading / sync executing pattern. I think Yabble, Tiki, and my modified Objective-J loader all do this, off the top of my head.
Yep, and that's exactly what Dojo does as well (just not with the
CommonJS API, but all the same patterns). And after years of supporting
multiple core loaders, and suffering through slow eval based loading, we
are ready to be done with it. Anyway, I think we will definitely have an
auxillary loader for CommonJS plain modules in Dojo, but having Dojo
modules be written in and designed for CommonJS has been discussed and
totally rejected.
Also, in regards to implementations, I know Yabble, RequireJS, and
Nodules (and soon Dojo) all implement Transport/C, so it is a reasonably
well implement spec, so I don't think we could just make it go away even
if we wanted to. We can give it a less confusing name though (having two
transport specs, differentiated by a letter is horrible).
--
Thanks,
Kris
Eugene Lazutkin
build tool (Dojo has an async loader too). Unfortunately there are
problems with this approach.
An example close to my heart: right now I am working on an
application, which doesn't work with the sync loader as intended ---
it shows "script takes too much time to run, do you want to cancel?"
twice breaking internal timers. So I am forced to do a build.
The build works great --- but now I have a major problem with
debugging: everything is in one (two, three) humongous file(s)
(obviously I do not use a minification or any other source code
processing for debugging). If there is a problem in line 12,345 I
don't know where it is in my original source code. It is a real
problem to find the actual file and the correct offset in it. It takes
time and makes me less productive --- I hope everybody knows that
programmers frequently fall in "groove", and it takes small kinks in
the overall development process to break the "groove". Probably I
forgot to mention that the build of that application takes about a
minute (Dojo's builder does more than JS concatenation --- it does the
same for CSS, and inlines internal resources).
That's about real-life working with transformation/build tools.
An automated loader, which does all these things on the server on
demand has the same problems and it brings more problems to the table.
The first and foremost problem is the much higher barrier of entry ---
you have to set up/run a web application to serve CommonJS modules,
which is more complex than running from a file system, or serving
static resources with a web server. The web app for that should be
written, or some stock one should be used. It always leads to "Why do
you use Flask? You should be using Ruby on Rails like me!", "IIS?
Windows? Start using PHP on Linux like all cool guys!", and so on.
To sum it up: a specialized loader is a non-starter. It is a hassle to
set it up especially for people who are not sysadmins, but need it for
development. At the moment and in a near future it is not available on
majority of platforms. Legacy platforms are especially bad at that.
The more burden we put on programmers, the more barriers of entry we
create.
Another thing I sense (it was not said openly but assumed) is that
many people afraid of ... asynchronous module loading in general. I
can understand arguments against doing it asynchronously in non-
browser environments. I just don't see anything in Kris' proposal,
which forces to load such modules asynchronously. You can do it
synchronously and "stop the world" while loading dependencies without
any problems. His proposal is about one thing only: *allow* to load
such modules asynchronously in a browser or anywhere else you want it.
Nobody forces implementors to do it one way or another.
If we don't start sharing modules between servers and clients, we will
rob themselves of this unique ability afforded by JS "all the way".
This is a huge advantage over Java, Python, Ruby, PHP, or any other
server-side languages. So far we are shooting ourselves in the foot by
incessant bike-shedding.
<rant>
We pooh-poohed it a year ago. We can pooh-poohed it now. The reason is
simple --- our majority either doesn't understand the idea of sharing
modules yet because the whole technique is in infancy, or lives in
closed gardens, which they fiercely guard from "outsiders". Either way
we prevent the technique to go mainstream ('Does CommonJS even need to
appease "mainstream front end" developers anyway?'), we reduce number
of programmers, who can benefit from it => the less the pool of
participant is => the less sustainable community is, the less new
ideas we get, the less real-world experience we have as a community.
But mark my words: in 2-3 years even former Visual Basic programmers
would want sharing modules, and the problem would be solved anyway,
and the very same people in this community, who oppose now, would be
agitated about it, and they will say "that was the whole idea of
CommonJS, you know!".
Why the opposition against an inevitable? Is Kris' proposal that
difficult to implement? I don't think so --- all machinery is already
in place in existing loaders. Does it make everything we wrote
instantly obsolete? Last time I checked it is fully backward
compatible. So what is the problem? Wrong name?
</rant>
Cheers,
Eugene
PS: Obviously I agree that require.def() or whatever it is called
should be universally supported --- otherwise the whole idea of
platform-neutral migrating modules doesn't work.
Eugene Lazutkin
compel client-side developers to explore and use SSJS. Right now
browsers and SSJS are completely separate environments with different
libraries and everything. No much difference to do the server with
Python or Ruby. With truly shared code we gain instantly the benefit
of familiarity. Just think about it.
Cheers,
Eugene
