Client-side modules
Ben Newman
the primary goal of this group, but I was just wondering if anyone is
interested in discussing (or has already discussed, somewhere
unbeknownst to me) mechanisms for transmitting JS modules to the
client, using existing/unmodified web technologies.
If this is a priority at all, I have a few concerns to voice,
primarily in the interest of perceived client-side performance / page-
load time.
Cheers,
Ben
Daniel Friesen
There have been discussions on list on async and browser friendly
require() (Though iirc mostly rejected as a part of require itself)
search through the archives for stuff related to require() with keywords
like async and promises.
Personally I prefer more of the classic inclusion of scripts into pages.
My Wrench.js which provides the stdlib of array/string manipulation
methods to MonkeyScript also works browser-side.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]
Nickolay Platonov
http://groups.google.com/group/jsan/browse_thread/thread/8035ed10b2e6750c# ,
I'll be very glad to discuss it.
Regards, Nickolay
Nickolay Platonov
If you'll have some patience to examine the information on this link:
http://groups.google.com/group/jsan/browse_thread/thread/8035ed10b2e6750c# ,
http://extjs.com/forum/showthread.php?t=69161
Nickolay
Kevin Dangoor
I think it is a priority, because one of the benefits of using server side JavaScript is the ability to also use the modules client side. Feel free to voice said concerns...
I have an experimental implementation of a module loader here:
http://hg.mozilla.org/labs/bespin/file/2a108f9a97fc/frontend/js/bespin/plugins/loader.js
This particular approach requires the modules to either have a wrapper around them in the file or to have a little bit of wrapping applied by the server. That wrapping wouldn't be necessary if XHR was used, but I wanted to use <script> tags to ensure good debuggability.
Anyhow, it's not a great bit of code, but it shows that you can do securablemodule loading without too much work.
Kevin
--
Kevin Dangoor
work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com
Joshaven Potter
I am currently looking into using JSON and or XMPP (http://xmpp.org/) to do what I think your talking about.
Also I will be doing something to allow transferring of js objects between server & client sides.
These may interest you (in regards to using XMPP):
http://metajack.im/2009/03/12/awesome-demo-of-real-time-xmpp-powered-app/
http://code.stanziq.com/strophe/
I have formulated some ideas (enough to convince myself that your goals are very achievable) regarding this topic. I would also be interested in starting a google group on this subject if many of you are interested.
--
Sincerely,
Joshaven Potter
"No man making a profession of faith ought to sin, nor one possessed of love to hate his brother. For He that said, “Thou shalt love the Lord thy God,” said also, “and thy neighbor as thyself.” Those that profess themselves to be Christ’s are known not only by what they say, but by what they practice. “For the tree is known by its fruit.”" -- Ignatius
Ben Newman
> I have an experimental implementation of a module loader here:
>
> This particular approach requires the modules to either have a wrapper
> around them in the file or to have a little bit of wrapping applied by the
> server. That wrapping wouldn't be necessary if XHR was used, but I wanted to
> use <script> tags to ensure good debuggability.
wrapping scripts:
http://hg.mozilla.org/labs/bespin/file/2a108f9a97fc/backend/python/bespin/jsmodule.jsont
Reproduced here:
bespin.plugins.loader.moduleLoaded("{{script_name}}", function
(require,exports){ {{script}}
return exports;
});
I agree that some sort of wrapping is necessary, and I agree that
<script>s have clear advantages over XHRs. In addition to
debuggability, using <script> tags also avoids the same-origin
restriction, which is nice when you want to load a third party script
from a central location, such as Google Code, or Amazon Cloudfront, or
some other CDN.
Returning exports at the end seems unnecessary, but that's a very
minor issue.
A bigger design question is raised by
bespin.plugins.loader.moduleLoaded. If we were to standardize (or
conventionalize ;) on a global loader function, we would probably want
it to be more generically named, but what really worries me is that
defining a single global loader function such as load() would prevent
multiple independent loaders from operating on the same page. Also,
until JavaScript has widespread 'const' support, the global load()
function could be hijacked by a malicious script seeking to compromise
the module system.
For these reasons, I have a slightly different module-wrapping syntax
to propose. Instead of having scripts call a global loading function
themselves, let them define a global property, called "modules" for
lack of a better name, which has the form
modules = {
"{{script1_name}}": function(require, exports) {
{{script1}}
},
"{{script2_name}}": function(require, exports) {
{{script2}}
},
...
};
When the <script> loads, an onload handler fires, snags
window.modules, iterates over its properties, saving the name-value
pairs for lazy access via require("script_nameN"). Yes,
window.modules will get clobbered over and over again, but the trick
is that the onload handler examines it (and maybe even deletes it)
immediately after the script defines it.
As far as I can tell, this protocol addresses the multiple-independent-
loader and hijacking concerns I raised above. If that claim seems
unjustified (or unclear), please speak up.
The syntax I'm proposing supports bundling multiple modules together,
but I have not said anything about how to request multiple files. I'm
assuming the server could choose to bundle additional modules based on
the requirements of the requested module. Maybe the server sends down
an unneeded module; that's fine. If require() never asks for it, the
unnecessary function(require, exports) {...} never has to be
evaluated. It might turn out that sending down an entire library at
once is the best policy, for some applications. In such cases, lazily
evaluating required() modules could be a big win, if there are rarely-
used modules that are expensive to initialize. Server implementers
should be free to find that balance for themselves.
Let me conclude by saying: I've sketched out the implementation only
to demonstrate that this syntax has desired properties (bundleability,
multiple-loader support, security). I'm not committed to any
particular implementation, and I'm not even committed to this syntax
if there is another that could better provide these benefits.
Ben
Christoph Dorn
> I am currently looking into using JSON and or XMPP (http://xmpp.org/)
> to do what I think your talking about.
> Also I will be doing something to allow transferring of js objects
> between server & client sides.
to me.
> These may interest you (in regards to using XMPP):
> http://metajack.im/2009/03/12/awesome-demo-of-real-time-xmpp-powered-app/
> http://code.stanziq.com/strophe/
Very cool.
> I have formulated some ideas (enough to convince myself that your
> goals are very achievable) regarding this topic. I would also be
> interested in starting a google group on this subject if many of you
> are interested.
I am interested. Specifically built on top of narwhal [1] which has a
browser engine for loading modules in the works.
Christoph
[1] - http://www.narwhaljs.org/
Kris Kowal
> A bigger design question is raised by
> bespin.plugins.loader.moduleLoaded. If we were to standardize (or
> conventionalize ;) on a global loader function, we would probably want
> it to be more generically named, but what really worries me is that
> defining a single global loader function such as load() would prevent
> multiple independent loaders from operating on the same page. Also,
> until JavaScript has widespread 'const' support, the global load()
> function could be hijacked by a malicious script seeking to compromise
> the module system.
I'm using the name "require" in my efforts, since it's guaranteed to
be masked by the module constructor function. It's also usable as a
function, the sandbox object, for invoking requests for modules by
their top level identifier. In one of my prototypes, I'm using
"require.regsiter(topId, factory)" for script injection.
> For these reasons, I have a slightly different module-wrapping syntax
> to propose.
> modules = {
> "{{script1_name}}": function(require, exports) {
> {{script1}}
> },
> "{{script2_name}}": function(require, exports) {
> {{script2}}
> },
> ...
> };
>
> When the <script> loads, an onload handler fires, snags
> window.modules, iterates over its properties, saving the name-value
> pairs for lazy access via require("script_nameN"). Yes,
> window.modules will get clobbered over and over again, but the trick
> is that the onload handler examines it (and maybe even deletes it)
> immediately after the script defines it.
That's one strategy. One disadvantage of it is that the line numbers
and file names don't work out for debugging. One of my prototypes
grabs individual scripts en masse with individual requests. That's
pretty good, and also pretty fast, for development purposes when you
can collude with the server. For production, I am also going to do a
version that bundles and minifies all the transitive dependencies. It
will also use require.{{method}} machinery.
> As far as I can tell, this protocol addresses the multiple-independent-
> loader and hijacking concerns I raised above. If that claim seems
> unjustified (or unclear), please speak up.
One nice thing about Chiron is that you can have multiple independent
loaders on the same page. It has no footprint on global scope. The
only way to accomplish this is with XHR. Chiron uses sync XHR, but a
non-global asyc XHR with bundles or individual scripts can also be
constructed.
I've written up some tickets for myself on Narwhal [2]. It's clear
that no single module loader will be perfect for any particular
scenario (with/without server colusion, development/production,
with/without loader availability to later inline scripts), but we can
provide a loader for each person's requirements and a handy matrix to
help them choose. I've written up a couple tickets to this end on
Narwhal and plan to spend some time on it this weekend [1].
In any case, the proper venue for this kind of discussion is certainly
closer to implementation than specification. I welcome anyone
interested in Narwhal's implementation to join me on our mailing list
[6]
I've presently implemented two loaders for Narwhal; one that uses an
inline script [4] to inject the transitive dependencies of a given
module and then load that module, and a port [5] of Chiron's
modules.js that uses the server component to grab the module file
corresponding to its top level id. Both implementations uses a
prototype "narwhal/server" JSGI app [3].
The only issue that's relevant to standardization is that most of
these browser loaders will depend on static analysis to discover the
transitive dependencies of a module, and the trouble is that there is
no safe synchronous solution for loading "dynamic dependencies".
"require", as specified, is a synchronous operation. For dynamic
dependencies, we will need to eventually specify
a variant on "require" for the asynchronous load case. I'm presently
favoring "require.async(id)" -> promise, and *allowing* the
synchronous "require(id)" to throw an error if the module cannot be
loaded synchronously, which may only happen if the id is not a String
literal.
Kris Kowal
[1] http://github.com/tlrobinson/narwhal/issues/labels/re-modules
[2] http://github.com/tlrobinson/narwhal/issues/labels/re-modules#issue/22
Ben Newman
> Hey Ben... I am very interested in that subject...
>
> I am currently looking into using JSON and or XMPP (http://xmpp.org/) to do
> what I think your talking about.
JSON-ish schema for modules, with the slight twist that function
values are permitted (not strictly JSON). The benefit of agreeing on
a schema would be that any transport layer (<script> tags, XHRs, or
XMPP) could import any conforming module(s).
> Also I will be doing something to allow transferring of js objects between
> server & client sides.
>
Cool and cool!
> I have formulated some ideas (enough to convince myself that your goals are
> very achievable) regarding this topic. I would also be interested in
> starting a google group on this subject if many of you are interested.
eyes. If the topic seems inappropriate for this group, then, yeah, we
should probably branch off.
Kris Kowal
> After some reflection, I realize that all I'm really proposing is a
> JSON-ish schema for modules, with the slight twist that function
> values are permitted (not strictly JSON). The benefit of agreeing on
> a schema would be that any transport layer (<script> tags, XHRs, or
> XMPP) could import any conforming module(s).
Ah, yes, very apt. I believe we would have reasonable flexibility if
modules were bundled as a sequence of zero or more calls to
require.register(topId, function (require, exports) {…}); This would
be more like JSONP than JSON, as we would need (in some cases) the
callback to notify the loader that the module has arrived and been
registered.
Kris Kowal
Ben Newman
flexible enough to support almost any loader approach): you can use
"require.register" and I can use "modules=" and both be perfectly
happy. If we agree on using JSONP, then we have a lot less to
discuss.
The JSONP convention, as I understand it, is to provide a single JSON
argument to a single callback (or assignment statement, or whatever
'P' the client specifies). Did you have a particular reason for
preferring multiple calls? Does the order of the modules matter to
your system? For instance, do you have modules that modify
require.register? From its name, I imagine that require.register
merely makes modules available to require, without eagerly evaluating
them. If you agree that laziness is important, it might be worth
making that explicit.
The only objection I have to JSONP is that the scripts it requests
need to be dynamic rather than static, but caching them should be
trivial, and the rate of JSONP adoption suggests that people are not
terribly concerned about that issue.
Ben
On Aug 12, 5:50 pm, Kris Kowal <cowbertvon...@gmail.com> wrote:
Joshaven Potter
I may not be understanding you properly or may not understand JSONP well enough but I believe that a standardization is not really necessary for this.
You could include a query string in your include statement which could render your javascript however you want it.
ie: <script src="domain.tld/controller/action/loadjsonp?task=make%20me%20a%20sandwitch">
This query string could be instructions to your application to return to you a sandwich object (or a person object... etc) javascript file.
The response could be:
bread:'white',
toppings: ['Peanut Butter', 'Jelly']
}
init(sandwich);
Your page that calls the script would define the init() function that would handle the sandwich object in the browser. This way you are loading an object, function, etc. into your browser.
The beauty of this is that the script can break through domains and as long as your server can render the dynamic js file all should go smoothly. Also you should be able to grab all of the code you want through the dynamic rendering of the file on the server... And if it doesn't need to be dynamic, then it can be served statically or even cached through some proxy etc thats setting in front of your server.
Ben Newman
"tasks" to wrapped JSON in a well-defined, programmatic fashion. The
rule is simple: the JSON payload is surrounded with parentheses, and
the P(adding) argument is prepended before the opening '('. If I'm
writing a library module, I can't anticipate an arbitrary mapping like
"make me a sandwich" -> "var sandwich = {...}; init(sandwich);", but I
know how to handle arbitrary JSONP requests.
That's one part of the contract: "Give me a padding string, and I'll
craft a response for you that calls a callback, or sets a variable, or
whatever you want." Using JSONP to nail down this part makes sense
because JSONP is already in wide use. Whatever its drawbacks, people
are relatively happy with it. I should mention here that this part of
the contract doesn't matter at all if you're happy with XHRs, because
then you can wrap the raw JSON payload however you like. If you want
to use <script> injection, however, you don't have a chance to modify
the HTTP response before it executes, so you need a way to tell the
server how to do the wrapping.
The other, equally important, part of the contract is what the JSON
payload should look like. At some level, it should conform to the
serverjs spec (a sequence of statements and function declarations that
assume free variables called "require" and "exports"), but that's not
really convenient given existing web technologies. What I'm proposing
is a standard syntax for wrapping serverjs-conforming modules, so that
clients always get something they can understand. There might be some
way to give the client fine-grained control over the shape of this
data, perhaps through some sort of template language embedded in the
HTTP request, but I think standardizing that control mechanism would
be much harder than merely standardizing the wrapping syntax.
Kris Kowal
> The only objection I have to JSONP is that the scripts it requests
> need to be dynamic rather than static, but caching them should be
> trivial, and the rate of JSONP adoption suggests that people are not
> terribly concerned about that issue.
This convinces me that the JSONP approach would not be feasible for
module factory registration. JSONP is typically used for dynamic or
un-cache-able data. Modules should be cache-able. One of our goals
should be to post modules to a CDN. To that end, I think static
padding is desirable.
I concur that registration must be lazy. Module factory functions
must not be implicitly promoted to module instances. Doing so would
non-deterministically break cyclic dependencies, and even more
frequently would cause modules to be loaded before their dependencies
were registered.
My purpose for recommending that a function call for each module
factory registration is to simplify bundling. We could have the best
of both worlds with a signature of require.register(modulesMemo);
bundle : = registerCall*;
registerCall := "require.register(" + moduleFactoryMemo + ")";
moduleFactoryMemo := "{" + ((stringLiteral, ":", moduleFactory)
delimited-by ",") + "}";
moduleFactory := "function(require,exports,module){" + moduleText + "}";
So, here's my summary of our progress:
* we agree that modules must be lazy
* we agree that modules must be cache-able
* therefore, the padding or template for a module bundle must be static
* you want and i'm willing to support multi-entry module memo registration
* i want the ability to concatenate bundles (i presume you have no objection)
Kris Kowal