[CommonJS] Using eval to communicate between client-side modules
Jonathan Fine
I hope it's not off-topic, but I'm a client-side JavaScript programmer who wants to write modules. (For what it's worth, Python is my preferred programming language.)
Anyway, I've come up with something that seems to scratch my itch, and I'd like to know what you think of it. The key idea is to use 'eval' to communicate between modules. (I know 'eval' is highly disapproved of, but perhaps in this case it does no harm and a lot of good. I'm not enough of a JavaScript expert to know for sure, and besides I'm biased.)
To show you what it looks like, here's the source for a module 'mymod' that exports 'get' and 'set' methods.
===
(function(n)
{
// Here's where to place the module metadata.
var module = new Module(
{
name: 'mymod',
exports: ['get', 'set'],
imports: []
});
// Declare variables for the exports.
eval(module.start());
// Implement the module.
var x;
var get = function(){
return x;
};
var set = function(arg){
x = arg;
};
// Store the exports in the module.
eval(module.finish());
})();
===
And here's some test code:
===
var mymod = MODULES.mymod;
mymod.set(2);
mymod.get() === 2 || ddt(); // Make a noise if unequal.
===
The implementation is simple, and can be found at
<http://mathtran.svn.sourceforge.net/viewvc/mathtran/trunk/mathtran_site/help_widget/eval_module.js?revision=500&view=markup>
Please do let me know what you think of this, because I'm keen to start writing client-side JavaScript modules.
--
Jonathan
Kevin Dangoor
On Thu, Sep 17, 2009 at 10:56 AM, Jonathan Fine <jonatha...@googlemail.com> wrote:
I hope it's not off-topic, but I'm a client-side JavaScript programmer who wants to write modules. (For what it's worth, Python is my preferred programming language.)
It's off-topic, but that's okay :)
Anyway, I've come up with something that seems to scratch my itch, and I'd like to know what you think of it. The key idea is to use 'eval' to communicate between modules. (I know 'eval' is highly disapproved of, but perhaps in this case it does no harm and a lot of good. I'm not enough of a JavaScript expert to know for sure, and besides I'm biased.)
To my eyes, this module pattern seems to be bringing some indirection without any real benefit that I can see. Without using eval, or duplicate definitions of what's being exported or the name of the module or any of that, you can get modules with public and private variables pretty easily using standard JS:
http://ajaxian.com/archives/a-javascript-module-pattern
--
Kevin Dangoor
work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com
Jonathan Fine
Thank you for looking at my example.
To my eyes, this module pattern seems to be bringing some indirection without any real benefit that I can see. Without using eval, or duplicate definitions of what's being exported or the name of the module or any of that, you can get modules with public and private variables pretty easily using standard JS:
http://ajaxian.com/archives/a-javascript-module-pattern
I think its fair to say that the example I posted is syntactic sugar, and that the same could be done directly and without eval (as you say). In fact, I think the same is true of modules more generally, even in languages like Python.
Here's an example that makes the syntactic sugar clearer. To implement get and set it's my coding style to write:
==
(function(n)
{
var x;
var get = function(){
return x;
};
var set = function(arg){
x = arg;
};
==
I've now got to get them out of the closure. A common method, as in the URL you quoted, is to write
===
var MODULES.mymod = (function(){
// Body as before.
return { set:set, get:get};
})();
==
and my preferred coding style is to write
===
(function(){
// Body as before.
MODULES.mymod = {};
MODULES.mymod.set = set;
MODULES.mymod.get = get;
})();
===
Either way, there's boilerplate code here (and in a real example, lots of it). One purpose of the two eval's is to provide a syntactic sugar that reduces the amount of boilerplate.
--
Jonathan
Kevin Dangoor
So, there's boilerplate in your coding style :)
var mod = function() {
var priv = 1;
return {
get: function() { return priv; },
set: function(p) { priv = p; }
}
}();
Doesn't really have boilerplate.
Even better, though, is the CommonJS standard:
var priv = 1;
exports.get = function() { return priv; };
exports.set = function(p) { priv = p; }
Jonathan Fine
So, there's boilerplate in your coding style :)
var mod = function() {
var priv = 1;
return {
get: function() { return priv; },
set: function(p) { priv = p; }
}
}();
Doesn't really have boilerplate.
I find that this does not work for large examples. It does not allow me to switch between private and exported quantities.
To put it another way, suppose we add to the example above
var priv2 = 0;
var inc=function(){
priv2 ++;
}
It's like being forced to declare /all/ your variables in one place. (Having the choice can be useful syntactic sugar.)
Even better, though, is the CommonJS standard:
var priv = 1;
exports.get = function() { return priv; };
exports.set = function(p) { priv = p; }
Ah, now we're talking. As described on
http://jshq.org/commonjs/0.1/modules.html
(BTW, there's a broken link to http://jshq.org/commonjs/0.1/ServerJS/Modules/Secure on that page.)
So how do I manage this using client JavaScript? Clearly, 'exports' has to be defined, and also clearly we have problems if it is a global.
Oh, another 404! http://jshq.org/commonjs/0.1/ServerJS/Modules/CompiledModules
But a search finds https://wiki.mozilla.org/ServerJS/Modules/ScriptModules, which seems to give the information I need.
I'll say more in another post to this thread.
--
Jonathan
Dean Edwards
> (function(n)
> {
> // Here's where to place the module metadata.
> var module = new Module(
> {
> name: 'mymod',
> exports: ['get', 'set'],
> imports: []
> });
>
> // Declare variables for the exports.
> eval(module.start());
>
> // Implement the module.
> var x;
> var get = function(){
> return x;
> };
>
> var set = function(arg){
> x = arg;
> };
>
> // Store the exports in the module.
> eval(module.finish());
> })();
That's exactly the same pattern that I use in base2:
http://dean.edwards.name/weblog/2007/12/packages/
-dean
Jonathan Fine
So, there's boilerplate in your coding style :)
var mod = function() {
var priv = 1;
return {
get: function() { return priv; },
set: function(p) { priv = p; }
}
}();
Doesn't really have boilerplate.
According to https://wiki.mozilla.org/ServerJS/Modules/ScriptModules, to create a module that can be used as written by the browser one could write:
==
(function(){var mod=function(require,exports) {
exports.add = function() {
var sum = arguments[0];
for (var i=1; i<arguments.length; i++) {
sum += arguments[i];
}
return sum;
};
};require.install?require.install('math',mod):mod(require,exports);})();
==
Well, that's got some boilerplate. Still, the body is quite nice, so perhaps I could be persuaded to live with it. (Although I'd rather have the module name 'math' up front rather than at the end. Probably easy to fix.)
--
Jonathan
Kevin Dangoor
Ah, now we're talking. As described on
http://jshq.org/commonjs/0.1/modules.html
(BTW, there's a broken link to http://jshq.org/commonjs/0.1/ServerJS/Modules/Secure on that page.)
So how do I manage this using client JavaScript? Clearly, 'exports' has to be defined, and also clearly we have problems if it is a global.
Right.
Oh, another 404! http://jshq.org/commonjs/0.1/ServerJS/Modules/CompiledModules
Yeah, that site is not yet "done". Sorry about that. I've got a full plate at the moment with some unexpected non-work things...
Kevin Dangoor
(function(){var mod=function(require,exports) {
exports.add = function() {
var sum = arguments[0];
for (var i=1; i<arguments.length; i++) {
sum += arguments[i];
}
return sum;
};
};require.install?require.install('math',mod):mod(require,exports);})();
==
Well, that's got some boilerplate. Still, the body is quite nice, so perhaps I could be persuaded to live with it. (Although I'd rather have the module name 'math' up front rather than at the end. Probably easy to fix.)
Kris Kowal recently added some browser loading support to Narwhal which does similar server side magic to what I've been doing.
Daniel Friesen
>
> ...
>
> Ah, now we're talking. As described on
> http://jshq.org/commonjs/0.1/modules.html
>
> (BTW, there's a broken link to
> http://jshq.org/commonjs/0.1/ServerJS/Modules/Secure on that page.)
>
> So how do I manage this using client JavaScript? Clearly, 'exports'
> has to be defined, and also clearly we have problems if it is a global.
>
> Oh, another 404!
> http://jshq.org/commonjs/0.1/ServerJS/Modules/CompiledModules
>
> But a search finds
> https://wiki.mozilla.org/ServerJS/Modules/ScriptModules, which seems
> to give the information I need.
>
> I'll say more in another post to this thread.
>
> --
> Jonathan
The new wiki is at http://wiki.commonjs.org/
http://wiki.commonjs.org/wiki/Modules
The jshq stuff will probably have to be fixed to interpret links better.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]
Kris Kowal
<jonatha...@googlemail.com> wrote:
>> Even better, though, is the CommonJS standard:
>> var priv = 1;
>>
>> exports.get = function() { return priv; };
>> exports.set = function(p) { priv = p; }
>
> Ah, now we're talking. As described on
> http://jshq.org/commonjs/0.1/modules.html
>
> (BTW, there's a broken link to
> http://jshq.org/commonjs/0.1/ServerJS/Modules/Secure on that page.)
You'll find most of our work is in tact on our new wiki,
http://wiki.commonjs.org/
> So how do I manage this using client JavaScript? Clearly, 'exports' has to
> be defined, and also clearly we have problems if it is a global.
I've just completed a JSGI application that will serve any CommonJS
modules installed in Narwhal to a browser client.
http://github.com/kriskowal/narwhal/blob/master/docs/browser-api.md
This has made it finally possible for the same JavaScript modules to
run on both the client and the server for Narwhal.
The boilerplate you saw on the Wiki is the work of Peter Michaux who
did some research on ways to write modules such that they work both on
the server, and as normal global scripts on the client. The Narwhal
Browser API uses a "modules in transport" format, that is similar to
JSON in that it permits CommonJS compliant module factories and their
metadata to be transported to the client through script injection and
a common callback. The format is essentially boilerplate around a
compliant module, in addition to some JSON injected to describe the
module's shallow dependencies.
So, if you had a module:
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
It would be wrapped as:
require.register({
"factory": function (require, exports, module, system, print) {
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
},
"depends": ["gettext"]
})
There are a variety of ways that a client side module loader can be
written to make CommonJS compliant modules run in the browser. This
particular strategy is friendly to CDN's and debugging (as it can
preserve file names and line numbers). Other strategies with XHR are
more friendly to developing the JavaScript on the client with static
module files. The procedure is similar in any case: wrap the text of
the module in some sort of function that accepts via injection a
require, exports, and module object, then create a require function
that memoizes all of these "factory" functions and exports objects
based on their module identifier.
The end result is that your modules are very similar to Python, but
with certain advantages in the simplicity of module relative
identifiers and file locations.
Kris Kowal
Jonathan Fine
http://dean.edwards.name/weblog/2007/12/packages/
I've been interested in your work for a year or more, so it's very likely that I've seen that page before. (And I find it reassuring that you've already been thinking along these lines.)
I had a close look at it now, and it puzzled me for a while. It has
===
var graphics = new base2.Package(this, {
name: "graphics",
version: "1.0",
imports: "shapes",
exports: "Layout"
});
// evaluate the imported namespace
eval(this.imports);
===
If I'm right, then
new base2.Package(this,
has the side-effect of setting
this.imports
to a special value.
--
Jonathan
Kris Kowal
>
> I've just completed a JSGI application that will serve any CommonJS
> modules installed in Narwhal to a browser client.
>
> http://github.com/kriskowal/narwhal/blob/master/docs/browser-api.md
>
A word on firepower:
Here's an example server application that requires a module on the client:
http://github.com/kriskowal/narwhal/blob/master/examples/browser-deployment-jackconfig.js
This is the loader that the server puts inline in the HTML:
http://github.com/kriskowal/narwhal/blob/master/lib/narwhal/inline.js
This is the module loader bootstrapper code. This is one of the
modules that gets sent down the wire by the inline loader, and is the
first to execute. It download the sandboxing module and loads the
rest of the necessary modules.
http://github.com/kriskowal/narwhal/blob/master/lib/narwhal/client.js
And this is the server code:
http://github.com/kriskowal/narwhal/blob/master/lib/narwhal/server.js
Kris Kowal
Jonathan Fine
> So how do I manage this using client JavaScript? Clearly, 'exports' has to> be defined, and also clearly we have problems if it is a global.I've just completed a JSGI application that will serve any CommonJS
modules installed in Narwhal to a browser client.
http://github.com/kriskowal/narwhal/blob/master/docs/browser-api.md
This has made it finally possible for the same JavaScript modules to
run on both the client and the server for Narwhal.
I bet you're pleased to have that done. Pretty basic to using the same JavaScript libraries on both server and browser.
The boilerplate you saw on the Wiki is the work of Peter Michaux who
did some research on ways to write modules such that they work both on
the server, and as normal global scripts on the client.
Yes, that's exactly what I want. (Except that all I really care about is having a decent module system on the browser, and being part of a community.)
The Narwhal
Browser API uses a "modules in transport" format, that is similar to
JSON in that it permits CommonJS compliant module factories and their
metadata to be transported to the client through script injection and
a common callback. The format is essentially boilerplate around a
compliant module, in addition to some JSON injected to describe the
module's shallow dependencies.
Well, I don't think I can do so much server-side. I'm wanting a way of using modules that runs entirely client-side. (And without too much fiddling about with boilerplate.)
So, if you had a module:
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
It would be wrapped as:
require.register({
"factory": function (require, exports, module, system, print) {
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
},
"depends": ["gettext"]
})
Well, the repetition of "gettext" violates "Don't Repeat Yourself" and so for me is an accident waiting to happen.
There are a variety of ways that a client side module loader can be
written to make CommonJS compliant modules run in the browser. This
particular strategy is friendly to CDN's and debugging (as it can
preserve file names and line numbers).
Worth having, at least when developing.
Other strategies with XHR are
more friendly to developing the JavaScript on the client with static
module files. The procedure is similar in any case: wrap the text of
the module in some sort of function that accepts via injection a
require, exports, and module object, then create a require function
that memoizes all of these "factory" functions and exports objects
based on their module identifier.
OK. So it seems to me that it is perhaps possible to write JavaScript modules that
1. Load straight into the browser.
2. Don't have lots of boilerplate.
3. Don't violate DRY.
4. Are compatible with CommonJS.
And it would be nice if in addition they were
5. Compatible with Dean Edwards base2.
I'll take a look at require() and related functions.
--
Jonathan
Dean Landolt
It would be wrapped as:
require.register({
"factory": function (require, exports, module, system, print) {
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
},
"depends": ["gettext"]
})
Well, the repetition of "gettext" violates "Don't Repeat Yourself" and so for me is an accident waiting to happen.
This is just a function of how package authors structure their modules. It could just as easily be
var _ = require("i18n").gettext;
(But it may be a bit presumptuous of a package author to call their "i18n".)
Jonathan Fine
--
Jonathan
Dean Landolt
Oh -- makes sense. Still, unless I'm mistaken, that's not really repetition in the sense that you'll never have to write it -- it's just injected metadata as part of the wrapping. As Kris noted, your module would look like this:
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
Dean Edwards
> http://dean.edwards.name/weblog/2007/12/packages/
> I had a close look at it now, and it puzzled me for a while. It has
> ===
> var graphics = new base2.Package(this, {
> name: "graphics",
> version: "1.0",
> imports: "shapes",
> exports: "Layout"
> });
>
> // evaluate the imported namespace
> eval(this.imports);
> ===
>
> If I'm right, then
> new base2.Package(this,
> has the side-effect of setting
> this.imports
> to a special value.
>
Yes. The "imports" property defines a list of packages that the package
imports. The package is created inside a "new function(){}" construct so
there is a "this" object that I can use. "this.imports" is created by
adding together the exported symbols from all of the imported packages.
Other than that it is identical to the pattern that you use.
-dean
Jonathan Fine
Oh -- makes sense. Still, unless I'm mistaken, that's not really repetition in the sense that you'll never have to write it -- it's just injected metadata as part of the wrapping. As Kris noted, your module would look like this:As simple and boilerplate-free as it gets.
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
--
Jonathan
Dean Landolt
Aye, but it can if it's served from the right server [1]
[1] http://github.com/kriskowal/narwhal/blob/master/docs/browser-api.md
Kevin Dangoor
Jonathan Fine
It *can* be used directly on the client if the client loads the module via XHR and adds the boilerplate before eval'ing..On Thu, Sep 17, 2009 at 1:20 PM, Jonathan Fine <jonatha...@googlemail.com> wrote:
Yes, I agree it's quite nice. But it can't be used directly on the client, which is precisely what I want to be able to do.On Thu, Sep 17, 2009 at 6:12 PM, Dean Landolt <de...@deanlandolt.com> wrote:
Oh -- makes sense. Still, unless I'm mistaken, that's not really repetition in the sense that you'll never have to write it -- it's just injected metadata as part of the wrapping. As Kris noted, your module would look like this:As simple and boilerplate-free as it gets.
var _ = require("gettext").gettext;
var drum = 0;
exports.count = function () {
return _(++drum);
};
That requires HTML and JS to come from the same domain, so it's no good for me. (Nor is running a special server.)
--
Jonathan
Kevin Dangoor
That requires HTML and JS to come from the same domain, so it's no good for me. (Nor is running a special server.)
Just curious... because while it'd be nice to use the modules without any kind of server-side cooperation, the reality is that it's not *much* server side cooperation that's necessary.
Jonathan Fine
Is PHP too special?On Thu, Sep 17, 2009 at 2:05 PM, Jonathan Fine <jonatha...@googlemail.com> wrote:
That requires HTML and JS to come from the same domain, so it's no good for me. (Nor is running a special server.)
Just curious... because while it'd be nice to use the modules without any kind of server-side cooperation, the reality is that it's not *much* server side cooperation that's necessary.
Here's my own little example (not really suitable as it loads a big data file).
http://mathtran.svn.sourceforge.net/viewvc/mathtran/trunk/mathtran_site/help_widget/index2.html?revision=494
--
Jonathan
Kris Kowal
<jonatha...@googlemail.com> wrote:
> Yes, PHP is too special. I find it very useful to be able to run JavaScript
> straight from the SVN repository (something I saw Dean Edwards do - thank
> you Dean).
You might consider reading my draft plan for browser support.
http://github.com/kriskowal/narwhal/raw/master/docs/browser-api-plan.md
And you might want to review discourse on how we arrived at the
securable modules specification, under the headline "Related
Discussions" at http://wiki.commonjs.org/wiki/Modules
Summarily, there's no perfect solution, but there are several
solutions that address needs in various quarters and each makes a
different compromise. My Chiron project can load modules from a
single package, synchronously, over XHR, on demand. It however is
only debuggable in Firefox, is chatty, synchronous, and subject to the
same origin policy. The new technique for Narwhal is debuggable, very
fast, cross-domain, CDN-friendly, supports a module PATH, but requires
server side support to inject boilerplate and scan static
dependencies. In the next couple weeks, I'm working on a solution
that is debuggable, very fast, cross-domain, CDN-friendly, supports
the module PATH and doesn't require server side support, but does
require a build step to compose all of the modules into "factory
transport" notation (the require.register call). There are other
techniques I haven't implemented that will squeeze out more
performance using a combination of bundling, combination, and stateful
cache awareness on the server side. I hope these resources help frame
the issue.
Kris Kowal
Mike Samuel
wrote:
> perhaps in this case it does no harm and a lot of good. I'm not enough of a
> JavaScript expert to know for sure, and besides I'm biased.)
It should be avoided if any bit of the string you're evaling is not
from a source you trust.
There are a few cases where eval is a fine idea:
(1) Unpacking data from a trusted source that has taken care to format
the data correctly -- so that any embedded data from untrusted sources
is properly escaped.
(2) Code loading from a trusted source
(3) Generating on the fly expressions from user data using sound code
generation techniques, e.g. turning a mathematical expression into a
function so that you can graph it.
It should still be avoided in most cases, since the three cases above
should be done in library code since proper serializing, escaping, and
code generation are not easy to get right.
Finally, there's an odd corner case that might be relevant.
No matter how much you trust the source of the code, if you do not
trust all the code running around it, you should not use eval to load
code with secrets.
E.g. if one module does:
eval = (function (eval) {
return function (s) {
s = '' + s;
var key = /\bvar privateKey = (0x[0-9a-fA-F]+);/.exec(s);
if (key) { alert('I stole ' + key[1]); }
return eval(s);
};
})(eval);
and then you use eval to load your module
var crypto = (function () {
var privateKey = 0xABCD;
return { /* stuff that uses privateKey };
})();
then, even if you're running on an interpreter that has impenetrable
closures, an attacker can intercept secrets when you load using eval.
cheers,
mike
Mike Wilson
> The format is essentially boilerplate around a
> compliant module, in addition to some JSON injected
> to describe the module's shallow dependencies.
I remember earlier discussion where it was said that
only packages would have dependency lists. So what
you are doing here is sort of generating single-
module packages, with dependencies to other single-
module packages?
> So, if you had a module:
>
> var _ = require("gettext").gettext;
> var drum = 0;
> exports.count = function () {
> return _(++drum);
> };
>
> It would be wrapped as:
>
> require.register({
> "factory": function (require, exports, module, system, print) {
> var _ = require("gettext").gettext;
> var drum = 0;
> exports.count = function () {
> return _(++drum);
> };
> },
> "depends": ["gettext"]
> })
Interesting, I've had similar things in mind. How are
you planning to map the above module registering to a
module name that can later be "required" in the browser?
Let's say we want to get to the count function by doing:
count = require("util/math").count;
Best regards
Mike Wilson