[modules] Are we agreed on require, in general?

152 views
Skip to first unread message

Kevin Dangoor

unread,
Feb 1, 2009, 5:29:05 PM2/1/09
to serv...@googlegroups.com
Today, there were productive discussions on IRC (#serverjs on
freenode) regarding modules and scoping. The big question for module
loading was between these two choices:

1. loading the code in global scope (more or less like a <script> tag)
2. loading the code in a module-specific scope

Peter Michaux had originally been advocating the first style, and put
up the proposals on the modules page for that style:

https://developer.mozilla.org/ServerJS/Modules

He has been convinced that a variation on the "Pythonic Modules"
proposal (just having require) is the best way to go. For reference:
https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules

The biggest advantage of #2 is that accidentally leaving off a "var"
will not pollute the global namespace, which is a bigger deal on the
server than on the client because of how much code you may be loading.
And, Peter was convinced that the style did not actually lose any
useful features over the global loading style.

Ignoring syntax details for the moment, can I get a rough idea of
whether people are happy with module-scoped require()?

Kevin

--
Kevin Dangoor

email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com

Wes Garland

unread,
Feb 1, 2009, 5:49:41 PM2/1/09
to serv...@googlegroups.com
Kevin, can you provide an example of the "require" statement you're talking about?

i.e., what does it do, exactly?  Does it attach a specific module to a pre-determined name of the current 'this'?  Does that property potentially become a class constructor?

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

Kris Zyp

unread,
Feb 1, 2009, 5:52:45 PM2/1/09
to serv...@googlegroups.com
+1 for #2.

(And as I mentioned on IRC, this doesn't preclude impls from copying to
the global where appropriate, while still providing the disambiguation
that #2 allows).

Kris

Peter Michaux

unread,
Feb 1, 2009, 6:09:19 PM2/1/09
to serv...@googlegroups.com
On Sun, Feb 1, 2009 at 2:49 PM, Wes Garland <w...@page.ca> wrote:
> Kevin, can you provide an example of the "require" statement you're talking
> about?

Three libraries and one program...

// one.js

var foo = function(){};

// two.js

var asdf = require('one');

var bar = function() {
asdf.foo(); // works
};

// three.js

var baz = = function() {
asdf.foo(); // error
};

// four.js -- this is the main program started from the command line
// $ js four.js

var a = require('two');
a.bar();
var b = require('three');
b.baz();

Note that even though one.js is loaded by the interpreter before b.baz
is called, asdf.foo() is not available inside three.js. Each file that
wants to use a library function must import from that library.

> i.e., what does it do, exactly? Does it attach a specific module to a
> pre-determined name of the current 'this'?

No. It returns the otherwise unbound "this" object in the file being
imported. That is the "module scope" object.

> Does that property potentially
> become a class constructor?

No. It might have a constructor function as one of it's properties, however.

Peter

Wes Garland

unread,
Feb 1, 2009, 6:39:11 PM2/1/09
to serv...@googlegroups.com
On Sun, Feb 1, 2009 at 6:09 PM, Peter Michaux <peterm...@gmail.com> wrote:

No. It returns the otherwise unbound "this" object in the file being
imported. That is the "module scope" object.

Ah, I see what you're talking about, and it makes perfect sense. That's similar to the implementation I'm working on, but you're thinking more about modules written in JavaScript, whereas I have spent most of my time thinking about modules written in C.

Have you worked out the syntax for providing a class? i.e.  do you intend to snag Crockfords "beget()" method for making objects?  Or are you going to have something like 'new myModule.constructor()'?

(when I say "you" in this context, I meant he group, not just Peter. :) )

My personal preference (which I admit has caused some implementation snags) is to allow

var xyz = loadModule('moduleName');
var thing = new xyz();

It's particularly nice that way because it allows for easy building of trees containing modules objects and constructors as leaf nodes:
   var file = new ca.page.system.File("myfile");
   var loadavg = ca.page.system.os.loadavg;

Also, how do you plan to handle the case where many modules require the same other module?  This is relatively painless with JavaScript, but can be quite a bit more .. interesting when talking about native-language modules, either those loaded through DSO mechanisms or simple compiled into the embedding.

My current solution is to allow the module to be "loaded" multiple times, but after the first time, instead of loading/initializing the module, I simply return the same moduleObject that was created for the first load.  This means that it's possible that the same module will exist by more than one name for a given scope chain, but I don't really see any drawbacks to that.
 
Wes

Mario Valente

unread,
Feb 1, 2009, 6:46:26 PM2/1/09
to serv...@googlegroups.com
Kevin Dangoor wrote:
> Today, there were productive discussions on IRC (#serverjs on
> freenode) regarding modules and scoping. The big question for module
> loading was between these two choices:
>
> 1. loading the code in global scope (more or less like a <script> tag)
> 2. loading the code in a module-specific scope
>

> Ignoring syntax details for the moment, can I get a rough idea of


> whether people are happy with module-scoped require()?
>

/me +1 on #2

-- MV

Peter Michaux

unread,
Feb 1, 2009, 6:53:01 PM2/1/09
to serv...@googlegroups.com
On Sun, Feb 1, 2009 at 3:39 PM, Wes Garland <w...@page.ca> wrote:
> On Sun, Feb 1, 2009 at 6:09 PM, Peter Michaux <peterm...@gmail.com>
> wrote:
>>
>> No. It returns the otherwise unbound "this" object in the file being
>> imported. That is the "module scope" object.
>
> Ah, I see what you're talking about, and it makes perfect sense. That's
> similar to the implementation I'm working on, but you're thinking more about
> modules written in JavaScript, whereas I have spent most of my time thinking
> about modules written in C.

There should be no difference between loading a module written in
C/Java or loading a JavaScript module.

> Have you worked out the syntax for providing a class? i.e. do you intend to
> snag Crockfords "beget()" method for making objects? Or are you going to
> have something like 'new myModule.constructor()'?
>
> (when I say "you" in this context, I meant he group, not just Peter. :) )

I think OOP style is unrelated to module loading in the Pythonic style.

> My personal preference (which I admit has caused some implementation snags)
> is to allow
>
> var xyz = loadModule('moduleName');
> var thing = new xyz();

It would be

// moduleName.js

function xyz() {}

// anotherFile.js

var xyz = require('moduleName').xyz;


var thing = new xyz();


> It's particularly nice that way because it allows for easy building of trees
> containing modules objects and constructors as leaf nodes:
> var file = new ca.page.system.File("myfile");
> var loadavg = ca.page.system.os.loadavg;
>
> Also, how do you plan to handle the case where many modules require the same
> other module?

Every file that wants to use the xyz constructor function would need
to have the following at the top.

var xyz = require('moduleName').xyz;

If you have a common set of modules you like to use together, you
could bundle them up to reduce boiler plate.

// common.js

var abc = require('abc').abc;
var xyz = require('xyz').xyz;

// anotherFile.js

var c = require('common');

var foo = new c.abc();
var bar = new c.xyz();


> My current solution is to allow the module to be "loaded" multiple times,
> but after the first time, instead of loading/initializing the module, I
> simply return the same moduleObject that was created for the first load.

That is how the Helma/Pythonic system works. Modules are "singletons".

> This means that it's possible that the same module will exist by more than
> one name for a given scope chain, but I don't really see any drawbacks to
> that.

I think it could only make things confusing to read.

Peter

Hannes Wallnoefer

unread,
Feb 1, 2009, 7:00:22 PM2/1/09
to serv...@googlegroups.com
2009/2/1 Kevin Dangoor <dan...@gmail.com>:
>
> Today, there were productive discussions on IRC (#serverjs on
> freenode) regarding modules and scoping. The big question for module
> loading was between these two choices:
>
> 1. loading the code in global scope (more or less like a <script> tag)
> 2. loading the code in a module-specific scope
>
> Peter Michaux had originally been advocating the first style, and put
> up the proposals on the modules page for that style:
>
> https://developer.mozilla.org/ServerJS/Modules
>
> He has been convinced that a variation on the "Pythonic Modules"
> proposal (just having require) is the best way to go. For reference:
> https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules
>
> The biggest advantage of #2 is that accidentally leaving off a "var"
> will not pollute the global namespace, which is a bigger deal on the
> server than on the client because of how much code you may be loading.
> And, Peter was convinced that the style did not actually lose any
> useful features over the global loading style.

IMO the biggest advantage would be getting a language that scales to
projects with more lines of code and developers. The missing var is
just one of many things that can go wrong in such an environment.

As a proof of concept, I implemented require for spidermonkey-based
jslibs (http://code.google.com/p/jslibs/). It's very rough and my
C-skills are lousy, but it seems to work so I'm attaching the patch to
this message. There's also a little demo included, consisting of
main.js and module.js.

Hey, getting broad support for this kind of loading would be
absolutely fantastic! I can really see great stuff happening on top of
it.

Hannes
jslibs-require.diff

Chris Zumbrunn

unread,
Feb 1, 2009, 7:07:59 PM2/1/09
to serv...@googlegroups.com

On Feb 1, 2009, at 23:29 , Kevin Dangoor wrote:

> 1. loading the code in global scope (more or less like a <script> tag)
> 2. loading the code in a module-specific scope

+1 for #2, scoped modules

As mentioned on IRC, my preference would be using "modules.foo" as the
syntax instead of "require('foo')" but +1 for scoped modules either way.

Chris

Wes Garland

unread,
Feb 1, 2009, 7:19:35 PM2/1/09
to serv...@googlegroups.com
As a proof of concept, I implemented require for spidermonkey-based
jslibs (http://code.google.com/p/jslibs/). It's very rough and my
C-skills are lousy, but it seems to work so I'm attaching the patch to
this message. There's also a little demo included, consisting of
main.js and module.js.

Any chance you'd be willing to publish your code in its entirety, and under a less restrictive license?  I won't work on/with jslibs becase it's GPL only.

Here's a code-inclusion function I wrote last year which wasn't intended for module loading, but has most of the elements required for what you're describing. License is Version: MPL 1.1/GPL 2.0/LGPL 2.1.
   
Oh, in this example IIRC, the second argument to the function is an object which becomes "this" for the loaded code.

/** Load and interpreter a script in the caller's context */
static JSBool system_include(JSContext *cx, JSObject *obj, uintN argc, jsval *argv, jsval *rval)
{
  JSScript      *script;
  JSObject      *scrobj;
  JSObject      *thisObj;
  jsval         *fnArg;
  const char    *scriptFilename;
  JSString      *scriptFilename_jsstr;
  JSBool        retval;
   
  switch(argc)
  {
    default:
      return jseng_throw(cx, MODULE_ID ".include.arguments.count");
      break;
    case 1:
      thisObj = ((jseng_interpreter_t *)(JS_GetRuntimePrivate(JS_GetRuntime(cx))))->globalObj; 
      fnArg = argv + 0;
      break;
    case 2:
      thisObj = JSVAL_TO_OBJECT(argv[0]);
      fnArg = argv + 1;
 
      if (JSVAL_IS_OBJECT(argv[0]) != JS_TRUE)
        return jseng_throw(cx, MODULE_ID ".include.arguments.0.notObject");
 
      if (JSVAL_IS_NULL(thisObj) == JS_TRUE)
        return jseng_throw(cx, MODULE_ID ".include.arguments.0.isNull");
   
      break;
  }
 
  scriptFilename_jsstr = JS_ValueToString(cx, *fnArg);
  scriptFilename = JS_GetStringBytes(scriptFilename_jsstr);
  if (!scriptFilename[0])
   return jseng_throw(cx, MODULE_ID ".include.filename: Unable to determine script filename");
 
  errno = 0;
  if (access(scriptFilename, F_OK))
    return jseng_throw(cx, MODULE_ID ".include.file: %s - %s", scriptFilename, strerror(errno));
 
  JS_AddNamedRoot(cx, &scriptFilename_jsstr, "System.include.scriptFilename_jsstr");
 
  errno = 0;
  script = JS_CompileFile(cx, thisObj, scriptFilename);
  if (!script)
  {
    JS_RemoveRoot(cx, &scriptFilename_jsstr);
    return jseng_throw(cx, MODULE_ID ".include.compile: Error compiling %s (OS reports %s)", scriptFilename, strerror(errno));
  }
  JS_RemoveRoot(cx, &scriptFilename_jsstr);
     
  scrobj = JS_NewScriptObject(cx, script);
  JS_AddNamedRoot(cx, &scrobj, "include_scrobj");
  retval = JS_ExecuteScript(cx, thisObj, script, rval);
  JS_RemoveRoot(cx, &scrobj);
     
  return retval;
}

Wes

Peter Michaux

unread,
Feb 1, 2009, 7:22:12 PM2/1/09
to serv...@googlegroups.com
On Sun, Feb 1, 2009 at 4:19 PM, Wes Garland <w...@page.ca> wrote:
>
>> As a proof of concept, I implemented require for spidermonkey-based
>> jslibs (http://code.google.com/p/jslibs/). It's very rough and my
>> C-skills are lousy, but it seems to work so I'm attaching the patch to
>> this message. There's also a little demo included, consisting of
>> main.js and module.js.
>
> Any chance you'd be willing to publish your code in its entirety, and under
> a less restrictive license? I won't work on/with jslibs becase it's GPL
> only.
>
> Here's a code-inclusion function I wrote

Two examples of the Pythonic style. Could that be a strong indicator
that implementors are behind the Pythonic style?

Peter

Wes Garland

unread,
Feb 1, 2009, 7:28:58 PM2/1/09
to serv...@googlegroups.com
On Sun, Feb 1, 2009 at 6:53 PM, Peter Michaux <peterm...@gmail.com> wrote:

There should be no difference between loading a module written in
C/Java or loading a JavaScript module.

Agreed. I don't think the module user should even have to know how it was implemented.
 
> Have you worked out the syntax for providing a class? i.e.  do you intend to
> snag Crockfords "beget()" method for making objects?  Or are you going to
> have something like 'new myModule.constructor()'?
>
> (when I say "you" in this context, I meant he group, not just Peter. :) )

I think OOP style is unrelated to module loading in the Pythonic style.

I think so, too, except that it causes implementation differences.   It's actually easier to .beget() an object than it is to create non-global classes that are properly recognized by spidermonkey.  That said, I am not a fan of beget. And that is an understatement.
 
> My personal preference (which I admit has caused some implementation snags)
> is to allow
>
> var xyz = loadModule('moduleName');
> var thing = new xyz();

It would be

// moduleName.js

function xyz() {}

// anotherFile.js

var xyz = require('moduleName').xyz;
var thing = new xyz();

.....thank you for that example.  As long as that is reasonably implementable in Spidermonkey in C, you've sold me.  I'm going to try just that this week.

I think it could only make things confusing to read.

You're right. The syntax above is much better.

Wes

Kris Zyp

unread,
Feb 2, 2009, 9:01:58 AM2/2/09
to serv...@googlegroups.com
One minor recommendation/change I would like to see with #2
(pythonic/helma require) is that it shouldn't be specified in terms of
how it should be implemented, but rather in terms of the required
behavior (as any spec should be written). Currently, the proposal states
(perhaps not intentionally) that the module scope should have it's
[[Parent]] set to null, and it's [[Prototype]] set to the global scope.
I would prefer to be able to implement module scopes differently. I
think the spec should state that the module scope object that is
returned from the require should be an object where the properties
correspond to the top-level variables/function created by the evaluation
of the module. It should also state that the global scope should be in
the lookup chain for the evaluation of the module (so that all the
global functions/constructors are available to the modules).
Kris

Wes Garland

unread,
Feb 2, 2009, 9:45:43 AM2/2/09
to serv...@googlegroups.com
Hi, Kris;

On Mon, Feb 2, 2009 at 9:01 AM, Kris Zyp <kri...@gmail.com> wrote:
I think the spec should state that the module scope object that is
returned from the require should be an object where the properties
correspond to the top-level variables/function created by the evaluation
of the module.

Agreed.
 
It should also state that the global scope should be in
the lookup chain for the evaluation of the module (so that all the
global functions/constructors are available to the modules).

It's actually not possible to implement that any other way for native modules in Spidermonkey; formalizing that is not a bad idea.

Wes

Hannes Wallnoefer

unread,
Feb 2, 2009, 11:56:21 AM2/2/09
to serv...@googlegroups.com
2009/2/2 Kris Zyp <kri...@gmail.com>:
>
> One minor recommendation/change I would like to see with #2
> (pythonic/helma require) is that it shouldn't be specified in terms of
> how it should be implemented, but rather in terms of the required
> behavior (as any spec should be written). Currently, the proposal states
> (perhaps not intentionally) that the module scope should have it's
> [[Parent]] set to null, and it's [[Prototype]] set to the global scope.
> I would prefer to be able to implement module scopes differently. I
> think the spec should state that the module scope object that is
> returned from the require should be an object where the properties
> correspond to the top-level variables/function created by the evaluation
> of the module. It should also state that the global scope should be in
> the lookup chain for the evaluation of the module (so that all the
> global functions/constructors are available to the modules).

This sounds reasonable. We shoud just make sure the actual outcome is
compatible between implementations, and I'm not sure this is the case
unless module scopes are top level scopes (i.e. their parent scope is
set to null).

I'll update the proposal this evening (central european time) to drop
the include() and import() functions and a few other things that have
been suggested. I'll ping the list when I'm done.

Hannes

mob

unread,
Feb 2, 2009, 2:59:47 PM2/2/09
to serverjs
What is the definition of the Module Name? How do you prevent
collisions?

Is it a reverse domain name? com.corp....?

Just having the module name be a file name doesn't scale -- Or am I
missing something?

Michael

On Feb 2, 8:56 am, Hannes Wallnoefer <hann...@gmail.com> wrote:
> 2009/2/2 Kris Zyp <kris...@gmail.com>:

Peter Michaux

unread,
Feb 2, 2009, 3:05:23 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 11:59 AM, mob <m...@embedthis.com> wrote:
>
> What is the definition of the Module Name? How do you prevent
> collisions?
>
> Is it a reverse domain name? com.corp....?
>
> Just having the module name be a file name doesn't scale -- Or am I
> missing something?

The reason domain names don't collide is because there is a regulating
body. The same is true for Perl modules on CPAN due to Pause.

If a central repository is established for JavaScript modules then the
same system as CPAN/Pause could be used where a regulating body
approves module names. Mozilla has already stated they would host a
repository and it seems like a natural place for a central JavaScript
repository.

Your personal or company modules could be namespaced away from those
on the central repository by using your name or company name or domain
name as part of the module name.

Peter

Stuart Langridge

unread,
Feb 2, 2009, 3:11:17 PM2/2/09
to serv...@googlegroups.com
> Your personal or company modules could be namespaced away from those
> on the central repository by using your name or company name or domain
> name as part of the module name.

Or by having them on a separate path. It would be deeply handy to be
able to add folders to the path that require() looks in; I find myself
doing the equivalent in Python (sys.path manipulation) all the time in
a webserver environment. At that point you can do
file=require('file','../mycompany') and get your company's file object
rather than the one on the "standard" require path.

Actually, that's a good point. Probably all of the following could do
with being thought about for require():

1. where does require() look? can it be tweaked from inside code
(rather than from the environment)?
2. are submodules allowed? (require('foo.bar'))?
3. what's the "recommended way" to add my own personal library of
modules? Put them in a separate folder and add that folder to the
require() path? Pass the folder as a second parameter to require()?
Put my whole library of modules in the same folder as the including
file?

I suspect the way to handle my "company library" is to have one
mycompanylibrary.js file which requires all the bits of the library,
put the whole company library in mycompanylibrary/, add
mycompanylibrary/ to the require path, and then
require('mycompanylibrary'), thus giving me
mycompanylibrary.whatever() available.

sil


--
New Year's Day --
everything is in blossom!
I feel about average.
-- Kobayashi Issa

Hannes Wallnoefer

unread,
Feb 2, 2009, 3:34:06 PM2/2/09
to serverjs
On Feb 2, 9:05 pm, Peter Michaux <petermich...@gmail.com> wrote:
> On Mon, Feb 2, 2009 at 11:59 AM, mob <m...@embedthis.com> wrote:
>
> > What is the definition of the Module Name?  How do you prevent
> > collisions?
>
> > Is it a reverse domain name?  com.corp....?
>
> > Just having the module name be a file name doesn't scale -- Or am I
> > missing something?
>
> The reason domain names don't collide is because there is a regulating
> body. The same is true for Perl modules on CPAN due to Pause.

I really hate the reverse domain package naming convention in Java. It
leeds to incredibly deep package structures, like
com.company.project.module.package.Class. I think this is overkill,
even for the amount of Java code out there.

I think the sane thing is just to use the project, product or company
name as primary namespace, and then grow deeper as your project
requires. In Helma NG modules are called helma.file, helma.httpclient,
helma.webapp, helma.webapp.response etc.

> If a central repository is established for JavaScript modules then the
> same system as CPAN/Pause could be used where a regulating body
> approves module names. Mozilla has already stated they would host a
> repository and it seems like a natural place for a central JavaScript
> repository.
>
> Your personal or company modules could be namespaced away from those
> on the central repository by using your name or company name or domain
> name as part of the module name.

Agreed. I really like CPAN names (although it's a very long time since
I've done any Perl coding). They usually consist of 2-3 tokens, and
you get a good idea what they're about.

Hannes

> Peter

mob

unread,
Feb 2, 2009, 3:37:28 PM2/2/09
to serverjs
I can comment on the module scheme that Ejscript uses. Syntax is
different but the design goals seem very similar.

We support a dot notation for module names. Each module can depend on
others and thus forms a recursive dependency tree. Requiring one
module will import it and all dependent modules are loaded but
dependent modules are not introduced into scope implicitly.

To search for modules, we use an EJSPATH env var that specifies a set
of search directories for modules.

Our search strategy is then:

* Given a name "a.b.c", scan for:
*
* 1. File named a.b.c
* 2. File named a/b/c
* 3. File named a.b.c in EJSPATH
* 4. File named a/b/c in EJSPATH
* 4. File named c in EJSPATH

We don't (yet) support versioning in require directive.

mob

unread,
Feb 2, 2009, 3:39:00 PM2/2/09
to serverjs
I think you can have your cake and eat it.

If the module name is defined such that it must be unique and be
either centrally registered or a reverse domain name. That allows a
nice namespace for short, memorable names, but still allows internal
use that is guaranteed not to clash.

Michael

Ondrej Zara

unread,
Feb 2, 2009, 3:50:21 PM2/2/09
to serv...@googlegroups.com
> * Given a name "a.b.c", scan for:
> *
> * 1. File named a.b.c
> * 2. File named a/b/c
> * 3. File named a.b.c in EJSPATH
> * 4. File named a/b/c in EJSPATH
> * 4. File named c in EJSPATH
>
> We don't (yet) support versioning in require directive.

I generally agree with the idea of search path, but this seems just a
little bit too much "automagic" to my likings. GNU make comes to mind
with a billion of automatic features which make life sooo
complicated...

Ondrej

Peter Michaux

unread,
Feb 2, 2009, 3:56:14 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 12:11 PM, Stuart Langridge <s...@kryogenix.org> wrote:

> 1. where does require() look?

I think how the library path is set should be implementation
dependent. For some shell a environment variable might be good. For
some application like a mod_js, it should be set in the apache config
file.

> can it be tweaked from inside code
> (rather than from the environment)?

I like this idea but I don't think it is critical for the first
version of the spec. It can be added in a backwards compatible way at
a later time.

I've though that "require.path" would be an array of directories that
could be modified during runtime. This doesn't introduce any new
globals and makes sense to be a property of the function doing the
requiring. No unfortunately named $" global variable floating around.

Unfortunately Java requires that the classpath be set when the JVM is
started so it isn't possible to modify the lookup path successfully
for modules which need to load jar files outside the initial
classpath. A custom class loader could be written but that is a big
extra step.

I don't think that the ultimate spec should be driven by what is
difficult to do in a particular ECMAScript implementation. I do think
that the first spec should be easy to implement in all the main
ECMAScript implementations so that conforming projects (shell, web
server, etc) can appear sooner rather than later. This will help with
marketing and momentum.

I also think a "required.loaded" array of loaded module scopes could
be available but as it is not critical it can wait.

Standardizing the minimum at first is a good idea, in my opinion.


> 2. are submodules allowed? (require('foo.bar'))?

Yes but the following should be false

require('foo').bar === require('foo.bar')

The left is the bar property in the foo.js module scope. The right is
the foo/bar.js file's module scope.

It may be better to use something other than '.' as the separator for
this reason. '/' would be fine by me.


> 3. what's the "recommended way" to add my own personal library of
> modules? Put them in a separate folder and add that folder to the
> require() path?

That is my preference.

Eventually your company could set up its own repository and install
the modules with a package management client by specifying multiple
repositories to be searched.

Peter

Hannes Wallnoefer

unread,
Feb 2, 2009, 3:56:16 PM2/2/09
to serverjs
On Feb 2, 9:11 pm, Stuart Langridge <s...@kryogenix.org> wrote:
> > Your personal or company modules could be namespaced away from those
> > on the central repository by using your name or company name or domain
> > name as part of the module name.
>
> Or by having them on a separate path. It would be deeply handy to be
> able to add folders to the path that require() looks in; I find myself
> doing the equivalent in Python (sys.path manipulation) all the time in
> a webserver environment. At that point you can do
> file=require('file','../mycompany') and get your company's file object
> rather than the one on the "standard" require path.

I think I much prefer having an explicit namespace/package:
file = require('mycompany.file')

> Actually, that's a good point. Probably all of the following could do
> with being thought about for require():
>
> 1. where does require() look? can it be tweaked from inside code
> (rather than from the environment)?

In Helma NG this is called module path. By default, it consists of two
directories. If you just start the interactive shell without running a
script it's the current working directory and Helma NG's modules
directory. If you run a script, the first element is the directory
parent directory of the script instead of the current working dir.

You can add additional directories to the module search path either
via command line options or from within the application using the
helma.system.addRepository method. So in the case of a company
library, you'd probably put that into a separate directory and add
that to the module path in one way or the other.

> 2. are submodules allowed? (require('foo.bar'))?

Yes, sure. foo.bar resolves to a file called foo/bar.js somewhere in
your module search path.

> 3. what's the "recommended way" to add my own personal library of
> modules? Put them in a separate folder and add that folder to the
> require() path? Pass the folder as a second parameter to require()?
> Put my whole library of modules in the same folder as the including
> file?
>
> I suspect the way to handle my "company library" is to have one
> mycompanylibrary.js file which requires all the bits of the library,
> put the whole company library in mycompanylibrary/, add
> mycompanylibrary/ to the require path, and then
> require('mycompanylibrary'), thus giving me
> mycompanylibrary.whatever() available.

Why would you want to have the whole library in one file? I think
usually you'd just have a collections of modules, and require/import
just the ones you actually need.

Hannes

Srdjan Pejic

unread,
Feb 2, 2009, 4:00:01 PM2/2/09
to serv...@googlegroups.com
I agree with Hannes that Java-like package names are ridiculous and overkill. I also agree with him that you should start with company or project name and go deeper, ala YUI. Search path is cool, but maybe not so many options. I'd be happy with searching for "a.b.c" and "a/b/c", especially since the second one emulates Ruby requires. (With all the Pythonistas here, I thought to even it up a little ;) )

Wes Garland

unread,
Feb 2, 2009, 4:01:02 PM2/2/09
to serv...@googlegroups.com
Are you  kidding? GNU make is wonderful one you understand it well. :)

On a more serious note, it is probably worth considering the model used by the UNIX runtime linker for resolving DSOs. It will
 
 - Check the system default paths
 - Check your LD_LIBRARY_PATH (JS_LIBRARY_PATH?)
 - Check paths that were compiled into it with -R linker option (that would be an embedding compile time option in our case)

Something to note: It is unlikely that different implementations will have binary compatible native modules, certainly not at first.

Suggestion: module paths include embedding technology at head for implementation-specific modules. This allows more than one ssjs environment to be installed on a given host, and they can share javascript-language modules.

i.e.

/usr/local/ssjs/libexec/spidermonkey/system/file.so
/usr/local/ssjs/libexec/squirrelfish/system/file.so
/usr/local/ssjs/libexec/portable/parser.js

In this setup, two ssjs implementations are using /usr/local/ssjs/libexec in their lib path; parser.js can use the system.file.File object for either squirrelfish or spidermonkey, depending on which environment was using it.

Path searching is particularly important, BTW, when root installs the ssjs environment and plebes want to write their own modules.

Wes

On Mon, Feb 2, 2009 at 3:50 PM, Ondrej Zara <ondre...@gmail.com> wrote:



--

Peter Michaux

unread,
Feb 2, 2009, 4:03:20 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 12:37 PM, mob <m...@embedthis.com> wrote:

> We don't (yet) support versioning in require directive.

I think versioning should not be supported in the standard. The idea
is used in Ruby gem and it is a broken idea. Gem has been criticized
for this.

A requires B and C

B requires version 2 of D

C requires version 3 of D

Sometimes this won't matter because the Pythonic system avoids global
namespace collision; however, if module D is dealing with file locks
or database transactions, and truely needs to be a singleton, then all
hell breaks loose and you don't even know it because you didn't even
know that your module A ultimately even used module D. Other folks
wrote B and C.

Peter

Wes Garland

unread,
Feb 2, 2009, 4:04:48 PM2/2/09
to serv...@googlegroups.com
Versioning is also kind of anti-JS

We should be doing "capability testing" whenever possible.

Wes

Srdjan Pejic

unread,
Feb 2, 2009, 4:27:48 PM2/2/09
to serv...@googlegroups.com
Agreed, one thing I do hate about RubyGems is the whole versioning piece. One question only: How do you do "capability testing" on features that are stubbed out, but not yet implemented? Also, say I want to keep up on the edge of a module, how would I go about that if there's no versioning?

Peter Michaux

unread,
Feb 2, 2009, 4:28:46 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 12:56 PM, Hannes Wallnoefer <han...@gmail.com> wrote:

> You can add additional directories to the module search path either
> via command line options or from within the application using the
> helma.system.addRepository method. So in the case of a company
> library, you'd probably put that into a separate directory and add
> that to the module path in one way or the other.

Does Helma have a custom class loader so the files in the new module
path can use the jar's upon which they depend?

Peter

Wes Garland

unread,
Feb 2, 2009, 4:32:34 PM2/2/09
to serv...@googlegroups.com
Srdjan: That's a good question. And a really good reason to consider disallowing stubs.   That's actually more consistent with JS in my opinion.

Wes

Hannes Wallnoefer

unread,
Feb 2, 2009, 5:18:04 PM2/2/09
to serverjs
On Feb 2, 5:56 pm, Hannes Wallnoefer <hann...@gmail.com> wrote:
> 2009/2/2 Kris Zyp <kris...@gmail.com>:
>
> > One minor recommendation/change I would like to see with #2
> > (pythonic/helma require) is that it shouldn't be specified in terms of
> > how it should be implemented, but rather in terms of the required
> > behavior (as any spec should be written). Currently, the proposal states
> > (perhaps not intentionally) that the module scope should have it's
> > [[Parent]] set to null, and it's [[Prototype]] set to the global scope.
> > I would prefer to be able to implement module scopes differently. I
> > think the spec should state that the module scope object that is
> > returned from the require should be an object where the properties
> > correspond to the top-level variables/function created by the evaluation
> > of the module. It should also state that the global scope should be in
> > the lookup chain for the evaluation of the module (so that all the
> > global functions/constructors are available to the modules).
>
> This sounds reasonable. We shoud just make sure the actual outcome is
> compatible between implementations, and I'm not sure this is the case
> unless module scopes are top level scopes (i.e. their parent scope is
> set to null).
>
> I'll update the proposal this evening (central european time) to drop
> the include() and import() functions and a few other things that have
> been suggested. I'll ping the list when I'm done.

Ok, I removed the description of include(), import(), and export(),
leaving just require(). I also rephrased a few other parts, nothing
radical. I think it now better represents what most people can agree
on. Let me know what you think.

https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules

Hannes

Peter Michaux

unread,
Feb 2, 2009, 6:16:25 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 2:18 PM, Hannes Wallnoefer <han...@gmail.com> wrote:

> Ok, I removed the description of include(), import(), and export(),
> leaving just require(). I also rephrased a few other parts, nothing
> radical. I think it now better represents what most people can agree
> on. Let me know what you think.
>
> https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules

Can remove the following line at the bottom?

"¹) Helma NG currently only consults the exported properties for
include(), since this is the feature that is most prone to
unintentional scope pollution."

Peter

Hannes Wallnoefer

unread,
Feb 2, 2009, 6:36:29 PM2/2/09
to serv...@googlegroups.com
2009/2/3 Peter Michaux <peterm...@gmail.com>:
>
> On Mon, Feb 2, 2009 at 2:18 PM, Hannes Wallnoefer <han...@gmail.com> wrote:
>
>> Ok, I removed the description of include(), import(), and export(),
>> leaving just require(). I also rephrased a few other parts, nothing
>> radical. I think it now better represents what most people can agree
>> on. Let me know what you think.
>>
>> https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules
>
> Can remove the following line at the bottom?

Sure, makes no sense anymore.

Hannes

Peter Michaux

unread,
Feb 2, 2009, 6:46:13 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 2:18 PM, Hannes Wallnoefer <han...@gmail.com> wrote:

> Ok, I removed the description of include(), import(), and export(),
> leaving just require(). I also rephrased a few other parts, nothing
> radical. I think it now better represents what most people can agree
> on. Let me know what you think.
>
> https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules

I added a list of open issues I've seen arise on the mailing list and irc

https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules#Open_issues

Peter

ihab...@gmail.com

unread,
Feb 2, 2009, 7:04:47 PM2/2/09
to serv...@googlegroups.com
Hi folks,

On Mon, Feb 2, 2009 at 2:18 PM, Hannes Wallnoefer <han...@gmail.com> wrote:

> https://developer.mozilla.org/ServerJS/Modules/Pythonic_Modules

I am a member of the Caja group. Caja is an Open Source,
Apache-licensed Google project building a Javascript sanitizer.

As my colleague Mike Samuel pointed out, Kris Kowal (to introduce
himself) and I are working on a proposal to ECMA TC39 (ECMAScript) to
standardize an approach to module systems. Our presentation is here:

http://docs.google.com/Presentation?id=dcd8d5dk_0cs639jg8

and a more extensive document, which is in a bit of flux but goes into
more detail, is:

https://docs.google.com/Edit?tab=view&docid=dfgxb7gk_34gpk37z9v

I have a question about you folks' proposal. One thing we are
concerned about in our ECMA work is the lexical scoping of "imported"
variables. In your proposal, these are simply free variables of the
module code, and are obtained from the prototype chain of the module
scope. In our proposal, the imported variables are available as:

require.env.<name>

so we narrow the module code's free variables to only the following set:

- The name "require";

- The name "exports"; and

- The primordials ("Array", "Object", ...).

This has the following advantages:

1. It makes syntax checking of modules for dangling free references
(e.g., by IDEs) much easier.

2. It allows us to specify a clean desugaring such that the value of
the module is a *function* simply by putting the module author's text
in a function --

(function(require, exports) { <text> })

then eval'ing and returning this. This function can now be
"instantiated" as many times as needed to yield independent copies of
the module. This permits two usages of modules:

2a. Each module name defines a singleton. require("foo") returns that singleton.

2b. Each module name can be resolved to a module function via a
"loader", then the client can instantiate the module as needed to
achieve strict isolation.

This is technically possible with the current serverjs proposal but it
would require some special native magic. We have tended towards
something that is programmable in userland as much as possible and
amenable to simple scope analysis.

What do you folks think?

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

Peter Michaux

unread,
Feb 2, 2009, 8:30:21 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 4:04 PM, <ihab...@gmail.com> wrote:

> I are working on a proposal to ECMA TC39 (ECMAScript) to
> standardize an approach to module systems. Our presentation is here:
>
> http://docs.google.com/Presentation?id=dcd8d5dk_0cs639jg8

Even that presentation is a lot to absorb in one shot.

> I have a question about you folks' proposal. One thing we are
> concerned about in our ECMA work is the lexical scoping of "imported"
> variables.

Can you explain more what this means exactly. The word "lexical" has
been overloaded and I don't follow exactly your concern in comparison
with the Helma/Pythonic proposal.

----

A concrete example might help me understand. With the Helma/Pythonic
system here are several libraries and a program.

//
// print.js
// An implementation dependent library.
// This example works for Rhino. May be a .so file.
//
var print = function(str) {
java.lang.System.out.println(str);
};

//
// greet-en.js
// A pure JavaScript library
//
var print = require('print').print;
var greet = function(name) {
print('hello, ' + name);
};

//
// greet-sp.js
// Another pure JavaScript library
//
var print = require('print').print;
var greet = function(nombre) {
print('hola, ' + nombre);
};

//
// program.js
//
var greeten = require('greet-en').greet;
var greetes = require('greet-es').greet;

greeten('Ihab'); // hello, Ihab
greetes('Kris'); // hola, Kris


Can you rewrite the above four files how they would be with your proposal?

Peter

ihab...@gmail.com

unread,
Feb 2, 2009, 8:42:05 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 5:30 PM, Peter Michaux <peterm...@gmail.com> wrote:
> Can you explain more what this means exactly. The word "lexical" has
> been overloaded and I don't follow exactly your concern in comparison
> with the Helma/Pythonic proposal.

/* code code code */
function foo() {
/* code code code */
function bar() {
/* code code code */
var z = x + 1;
}
}

Lexical scope analysis is to ask: is "x" a free variable of this block
or not? This is obvious to the extent that the "code code code"
sections are short, which they will typically not be (a module will
likely contain Lots Of Stuff). The alternative is below.

> A concrete example might help me understand. With the Helma/Pythonic
> system here are several libraries and a program.

Thank you, that is a great idea. Here goes.

//
// print.js
// An implementation dependent library.
// This example works for Rhino. May be a .so file.
//

export print = function(str) {
require.env.java.lang.System.out.println(str);
};

[[ Note that the only free variables in the above are "require" and
"export". The latter you don't see because we are using syntactic
sugar, but it's there. ]]

//
// greet-en.js
// A pure JavaScript library
//

var print = require('print').print; /* NOTE: this is *not* re-exported */
export greet = function(name) {
print('hello, ' + name);
};

//
// greet-sp.js
// Another pure JavaScript library
//
var print = require('print').print;

export greet = function(nombre) {
print('hola, ' + nombre);
};

//
// program.js
//
var greeten = require('greet-en').greet;
var greetes = require('greet-es').greet;

greeten('Ihab'); // hello, Ihab
greetes('Kris'); // hola, Kris

Ihab

Peter Michaux

unread,
Feb 2, 2009, 8:59:35 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 5:42 PM, <ihab...@gmail.com> wrote:
>
> On Mon, Feb 2, 2009 at 5:30 PM, Peter Michaux <peterm...@gmail.com> wrote:
>> Can you explain more what this means exactly. The word "lexical" has
>> been overloaded and I don't follow exactly your concern in comparison
>> with the Helma/Pythonic proposal.
>
> /* code code code */
> function foo() {
> /* code code code */
> function bar() {
> /* code code code */
> var z = x + 1;
> }
> }
>
> Lexical scope analysis is to ask: is "x" a free variable of this block
> or not? This is obvious to the extent that the "code code code"
> sections are short, which they will typically not be (a module will
> likely contain Lots Of Stuff). The alternative is below.

I get lexical analysis for closures and block scope etc. What I don't
get is how your proposal is somehow completely "lexical" or that it is
more lexical than the Helma/Pythonic proposal.


>> A concrete example might help me understand. With the Helma/Pythonic
>> system here are several libraries and a program.
>
> Thank you, that is a great idea. Here goes.
>
> //
> // print.js
> // An implementation dependent library.
> // This example works for Rhino. May be a .so file.
> //
> export print = function(str) {

We cannot do that because we won't be introducing new keywords.


> require.env.java.lang.System.out.println(str);

Why the "env" level in the hierarchy?

I'm not in favor of the dot being used like this for both module
hierarchy and regular property access.

The above line is a bit of an unnecessarily complex example but is
there a file "java/lang/System.js" available to the loader with the
following content?

export out = {
println: function(){}
};

OR is their a file "java/lang/System/out.js" available with the
following content?

export println = function() {};

This is a show stopper ambiguity. Some other way of importing is needed.

> };
>
> [[ Note that the only free variables in the above are "require" and
> "export". The latter you don't see because we are using syntactic
> sugar, but it's there. ]]
>
> //
> // greet-en.js
> // A pure JavaScript library
> //
> var print = require('print').print; /* NOTE: this is *not* re-exported */
> export greet = function(name) {
> print('hello, ' + name);
> };

With the Helma/Pythonic system, the module author can avoid
re-exporting like this...

var greet;
(function() {


var print = require('print').print;

greet = function(name) {
print('hello, ' + name);
};

})();

Peter

Kris Kowal

unread,
Feb 2, 2009, 9:42:34 PM2/2/09
to serverjs
Just a primer before I dive in for real:

> > //
> > // print.js
> > // An implementation dependent library.
> > // This example works for Rhino. May be a .so file.
> > //
> > export print = function(str) {
>
> We cannot do that because we won't be introducing new keywords.

Right. And for legacy scripts, we can't either, so we propose a
transitional syntax that would work right now, and in interpreters
with new uses for the "export" and "import" keywords. That is, we
would just use the syntax that this would desugar to:

exports.print = function (str) {

> >  require.env.java.lang.System.out.println(str);
>
> Why the "env" level in the hierarchy?

"env" is meant to contain objects like "window", "document", or
"print" that are implicitly available in the loader's environment.
It's supposed to be orthogonal with the module name space since
they're conceptually distinct. Ihab meant simply to illustrate that
"java" would preferably not be another "free-variable". So, this
could be a bit of a red herring. Presumably your module loader would
integrate with Java's module's name space so "println" could be
acquired more reasonably:

var println = require("java.lang.System.out").println;

In that case, we DO presume that the corresponding module (perhaps
"java/lang/System/out.js") would look something like this, albeit
presumed auto-generated since it's a reflection of a Java module:

exports.println = function () {
return require.env.java.lang.System.out.println.apply(this,
arguments);
};

Would it be better for us to modify the spec to say that "env" is
another one of the names provided by the module function?

(function (require, env, exports) { <module text> })

- Kris Kowal

Patrick Mueller

unread,
Feb 2, 2009, 10:13:29 PM2/2/09
to serv...@googlegroups.com
On Feb 2, 2009, at 8:59 PM, Peter Michaux wrote:

> With the Helma/Pythonic system, the module author can avoid
> re-exporting like this...
>
> var greet;
> (function() {
> var print = require('print').print;
> greet = function(name) {
> print('hello, ' + name);
> };
> })();

Using an object inside the module to determine your exports makes this
kind of yuckiness go away, which is nice. For the playing I've done,
I ended up using a variable named "module" which is the module object
returned from (your version of) require(). Anything I want to export,
then, gets added as a property of the "module" variable within the
body of the module code. If there's stuff I don't exported, don't add
it to the "module" variable.

Not sure if I really like that, there is something icky about
'special' variables I tend to not like. But I also dislike just
blanket exporting everything, and requiring "hide it in a function
definition" if I want to hide it.

Other techniques could be used to "hide" otherwise exportable things,
like using a "__" prefix (just like Python).

Patrick Mueller - http://muellerware.org/

Peter Michaux

unread,
Feb 2, 2009, 10:30:43 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 6:42 PM, Kris Kowal <cowber...@gmail.com> wrote:
>
> Just a primer before I dive in for real:
>
>> > //
>> > // print.js
>> > // An implementation dependent library.
>> > // This example works for Rhino. May be a .so file.
>> > //
>> > export print = function(str) {
>>
>> We cannot do that because we won't be introducing new keywords.
>
> Right. And for legacy scripts, we can't either, so we propose a
> transitional syntax that would work right now, and in interpreters
> with new uses for the "export" and "import" keywords. That is, we
> would just use the syntax that this would desugar to:
>
> exports.print = function (str) {

I like the idea that module identifiers are private by default and
must be explicitly exported. Implementationally, this is a minor
deviation from the Helma/Pythonic proposal. Just put an empty
"exports" object on the module scope before evaluating the module in
the module scope. Then a require for the module returns that "exports"
object.

>> > require.env.java.lang.System.out.println(str);
>>
>> Why the "env" level in the hierarchy?
>
> "env" is meant to contain objects like "window", "document", or
> "print" that are implicitly available in the loader's environment.
> It's supposed to be orthogonal with the module name space since
> they're conceptually distinct. Ihab meant simply to illustrate that
> "java" would preferably not be another "free-variable". So, this
> could be a bit of a red herring. Presumably your module loader would
> integrate with Java's module's name space so "println" could be
> acquired more reasonably:
>
> var println = require("java.lang.System.out").println;

No I wouldn't do that. I would write just
"java.lang.System.out.println" as I did in the example and isolate
this platform-dependent code in it's own module and call it from a
platform-independent module. This sort of platform-dependent stuff is
mostly out of scope for this group anyway.


> In that case, we DO presume that the corresponding module (perhaps
> "java/lang/System/out.js") would look something like this, albeit
> presumed auto-generated since it's a reflection of a Java module:

I wouldn't presume auto-generated.

> exports.println = function () {
> return require.env.java.lang.System.out.println.apply(this,
> arguments);

The dot notation is problematic. A module "java.lang.System" with an
exported "out" is confused with a module "java.lang.System.out" with
an exported "println". This sort of ambiguity cannot exist.

This system of dots seems inspired by Java. In Java there is not the
same problem due to lowercase namespace convention. In a Java import
statement a.b.C cannot be confused with a.b.c.D because of
capitalization. In Java each file is a class and classes start with
capital letters. Everything works out ok. It won't workout in
JavaScript because a module will not have just a constructor function
(captialized by convention). A module will potentially just be a
collection of objects with lowercase names and another module file may
have one of those object names as its filename.


> };


>
> Would it be better for us to modify the spec to say that "env" is
> another one of the names provided by the module function?
>
> (function (require, env, exports) { <module text> })

I don't think so. That isn't the main problem. Less free variables is
better. The ambiguity above is a problem.

----

I do like the private-by-default with explicit export as a
modification to the Helma/Pythonic proposal.

Peter

Peter Michaux

unread,
Feb 2, 2009, 10:43:14 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 7:13 PM, Patrick Mueller <pmu...@yahoo.com> wrote:
>
> On Feb 2, 2009, at 8:59 PM, Peter Michaux wrote:
>
>> With the Helma/Pythonic system, the module author can avoid
>> re-exporting like this...
>>
>> var greet;
>> (function() {
>> var print = require('print').print;
>> greet = function(name) {
>> print('hello, ' + name);
>> };
>> })();
>
> Using an object inside the module to determine your exports makes this
> kind of yuckiness go away, which is nice. For the playing I've done,
> I ended up using a variable named "module" which is the module object
> returned from (your version of) require(). Anything I want to export,
> then, gets added as a property of the "module" variable within the
> body of the module code. If there's stuff I don't exported, don't add
> it to the "module" variable.

I like the name "module" as it is analogous to "global" for the global
scope. I usually have a property called "global" of the global object
(analogous to "window" in the browser) so that my JavaScript isn't
browser-only.

> Not sure if I really like that, there is something icky about
> 'special' variables I tend to not like.

I'm the same.

> But I also dislike just
> blanket exporting everything, and requiring "hide it in a function
> definition" if I want to hide it.
>
> Other techniques could be used to "hide" otherwise exportable things,
> like using a "__" prefix (just like Python).

Using a prefix convention makes sharing code with the browser more
difficult because browsers now wouldn't protect the __ prefixed ones.

Peter

ihab...@gmail.com

unread,
Feb 2, 2009, 11:07:29 PM2/2/09
to serv...@googlegroups.com
You said:

On Mon, Feb 2, 2009 at 7:30 PM, Peter Michaux <peterm...@gmail.com> wrote:
>> var println = require("java.lang.System.out").println;
> No I wouldn't do that. I would write just

> "java.lang.System.out.println" ...

but then you said:

> The dot notation is problematic. A module "java.lang.System" with an
> exported "out" is confused with a module "java.lang.System.out" with
> an exported "println". This sort of ambiguity cannot exist.

so I'm confused. The way Kris wrote it, the stuff inside the require()
is a module ID, and the stuff outside is dereferencing the structure
of the instance of the module. It seems like your choice of:

"java.lang.System.out.println"

is what caused this problem. What am I missing?

Peter Michaux

unread,
Feb 2, 2009, 11:20:39 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 8:07 PM, <ihab...@gmail.com> wrote:
>
> You said:
>
> On Mon, Feb 2, 2009 at 7:30 PM, Peter Michaux <peterm...@gmail.com> wrote:
>>> var println = require("java.lang.System.out").println;
>> No I wouldn't do that. I would write just
>> "java.lang.System.out.println" ...
>
> but then you said:
>
>> The dot notation is problematic. A module "java.lang.System" with an
>> exported "out" is confused with a module "java.lang.System.out" with
>> an exported "println". This sort of ambiguity cannot exist.
>
> so I'm confused.

Sorry. For the specific example of the java package on the Rhino
implementation, I would just do java.lang.System.out.println because
it is available in Rhino and there isn't confusion because of the Java
naming convention where namespaces have lowercase and classes start
uppercase. I wouldn't use require() to load a java package class in
Rhino. What I thought Kris was implying was to somehow make require()
work for both modules and for java packages. I wouldn't overload it
like that. I knew this Java example was going to be confusing.

The important part is the following has trouble

require.A.B.C

as the way to get the identifiers exported in A/B/C.js

It needs to be something else like

require('A.B.C')

so the dots are not confused.

Does that make sense?

Peter

ihab...@gmail.com

unread,
Feb 2, 2009, 11:38:47 PM2/2/09
to serv...@googlegroups.com
Remarking in reverse order ;) --

On Mon, Feb 2, 2009 at 8:20 PM, Peter Michaux <peterm...@gmail.com> wrote:
> The important part is the following has trouble
> require.A.B.C
> as the way to get the identifiers exported in A/B/C.js
> It needs to be something else like
> require('A.B.C')
> so the dots are not confused.
> Does that make sense?

Absolutely. That is definitely what we propose.

> For the specific example of the java package on the Rhino
> implementation, I would just do java.lang.System.out.println because
> it is available in Rhino and there isn't confusion because of the Java
> naming convention where namespaces have lowercase and classes start
> uppercase. I wouldn't use require() to load a java package class in
> Rhino.

For that specific case, that makes a lot of sense.

That all said, there is a sense in which the module system we propose
is very "hermetic": it assumes that there are no free variables. Like,
really *really* no free variables in a module, apart from the minimum
possible (we have chosen 'require' and 'exports'). In your case,
'java' is a free variable. I don't know how that would play out in our
proposal but, again, this is Rhino and it is clearly a binding to a
specific feature of the embedding in Java....

If I had to do it myself, I would:

require("packages.java.lang.System").println;

Peter Michaux

unread,
Feb 2, 2009, 11:54:09 PM2/2/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 8:38 PM, <ihab...@gmail.com> wrote:
>
> Remarking in reverse order ;) --
>
> On Mon, Feb 2, 2009 at 8:20 PM, Peter Michaux <peterm...@gmail.com> wrote:
>> The important part is the following has trouble
>> require.A.B.C
>> as the way to get the identifiers exported in A/B/C.js
>> It needs to be something else like
>> require('A.B.C')
>> so the dots are not confused.
>> Does that make sense?
>
> Absolutely. That is definitely what we propose.

Because of the require.env example, I thought you were proposing

require.A.B.C

----

I think the only difference between your proposal and the
Helma/Pythonic proposal is adding the "exports" object. That is a much
smaller difference than the difference between the global eval
proposal and the Helma/Pythonic proposal.

Peter

Kris Kowal

unread,
Feb 3, 2009, 1:04:51 AM2/3/09
to serverjs

> I think the only difference between your proposal and the
> Helma/Pythonic proposal is adding the "exports" object. That is a much
> smaller difference than the difference between the global eval
> proposal and the Helma/Pythonic proposal.

That's very, very awesome.

So, speaking for at least Ihab and myself, I'd like to enumerate our
general objectives:

* interoperability: scripts should work in more environments
* client to server
* insecure to secure
* present to future
* lexical analyzability
* sovereignty: scripts should be able to determine the names that
they import, although adopting names is expected to be the common case
* scalability: no coordination with other developers required to
maintain a single, global API
* verifiability: static tools can identify reference errors
* more difficult to subvert through modifications to the global
scope
* produce smaller optimized forms (YUICompressor variable renaming)
* tells the IDE what names are in scope

So, at this point, I have several nice things to say about the
"exports" object if there are hold-outs for the approach of scraping
the module scope for exportable variables. I'm not sure how much of
this is in question, but I'll give it a go anyway.

One of the questions we've discussed is whether a module ought to
return its exports, produce its exports in its last evaluated
expression, or receive its exports object as a parameter (our current
proposal). The former are behaviorally identical. We went with the
'exports' parameter because it enables us to resolve cyclic
dependencies without infinite recursion and without banning cycles
outright. Consider a module A that depends on B, and a module B that
depends on A. B would require A before A returns its complete
exports. However, if B is willing to require a partially constructed
version of A, using the "exports" object, A can build some of its
exports, then import B. The exports object would be memoized by the
loader before calling the corresponding module factory function, so
infinite recursion does not occur.

The next discussion, which is unresolved (between Ihab and me, at
least), is whether "exports" should be in scope, or a member of the
unique "require" object provided to the module. The "pro" for the
latter strategy is that it reduces the number of messy, implicit
variables in the module scope. The "pro" for the former strategy,
which involves passing the exports object as an argument to the module
function, is that the "exports.foo = ..." syntax closely resembles the
speculated "export foo = ..." syntax that we hope to make available in
future versions of ECMAScript. This is also the reason why we're
favoring "exports" over "module" as the name of this parameter, but
that, like all other parts of our proposal, is still up for
discussion.

As for whether to use an "exports" object versus the local scope of
the module, there are several facets:

- Pro for using the local scope:
* strongly resembles global scope, so migration is easier
- Con for using local scope:
* not interoperable since there's no analogous strategy for
scraping locals in a cross-browser fashion with current JavaScript
implementations.
- Pro for using an "exports" object:
* does not require any masking strategy for "closed" or "private"
data. The module function is like a constructor function since locals
are hidden.
- Con for using an "exports" object:
* consumes "exports" name in module scope, making it implicitly
"special".

Speaking to the point about whether it's possible to scrape local
scope in client-side JavaScript in a cross-browser fashion, the only
strategy I know that can accomplish this takes advantage of a bug in
Safari 2, where "var" declarations inside a "with" block add
attributes to the "with" object. In standards-compliant
implementations of JavaScript, including Safari 3 at least to this
specification point, the declaration aspect of a "var" statement adds
a name to the next function block scope in the scope chain, skipping
all other blocks, including "with" blocks. The initialization aspect
occurs separately and on the original line of the "var" statement,
whether the name refers to the declared object or not. This
particularly eccentric aspect of the specification is the reason why
"let" is encouraged in future versions of JavaScript. I don't recall
how function declaration statements behave in this context.
Meanwhile, I'm told that privileged JavaScript in Gecko engines does
have the ability to scrape its local scope. In either case, the point
is moot. There isn't a known interoperable way to scrape the local
scope.

Kris Kowal

Hannes Wallnoefer

unread,
Feb 3, 2009, 7:41:52 AM2/3/09
to serv...@googlegroups.com
Thanks for explaining, Ihab and Kris.

My impression is that your propsal shares one goal with the pythonic
modules proposal, which is script isolation. But in the pythonic scope
scheme, protection is more about accidental collisions and less the
evil attacker I think your hermetic setup is addressing. Also, for me,
two really important goas were simplicity and backwards compatibility.

Simplicity means that I wanted to be able to explain the whole loading
scheme in one or two sentences. So beyond the special setup of the
scope object, it's just an ordinary script being evaluated. At one
time, I considered wrapping a function closure around each module
source, or put all exported properties into a separate object, like
you do in the salty versions of your proposal. But it seemed and still
seems like too much magic going on behind the scenes. I think it's ok
to leave it to the programmer to use closures when public/private
variables are required.

Backwards compatibility means that I wanted to preserve the
functionality of JavaScript as we know it, meaning the global object
and all its standard objects such as Object, Array, Date etc should be
there. I put the global object in the prototype chain, so everything
is there like you expect, but application code is kept apart. So we
can run existing code, and there's no steep relearning curve for
developers. Again, different goals, different result.

One question: are you able to implement this proposal in today's
browsers? I'm aware of Kris' module.js project - is that implementing
it, or going to implement it?

Hannes

2009/2/3 <ihab...@gmail.com>:

Patrick Mueller

unread,
Feb 3, 2009, 10:06:07 AM2/3/09
to serv...@googlegroups.com
On Feb 3, 2009, at 1:04 AM, Kris Kowal wrote:

Great information in general, thanks.

> As for whether to use an "exports" object versus the local scope of
> the module, there are several facets:
>
> - Pro for using the local scope:
> * strongly resembles global scope, so migration is easier
> - Con for using local scope:
> * not interoperable since there's no analogous strategy for
> scraping locals in a cross-browser fashion with current JavaScript
> implementations.
> - Pro for using an "exports" object:
> * does not require any masking strategy for "closed" or "private"
> data. The module function is like a constructor function since locals
> are hidden.
> - Con for using an "exports" object:
> * consumes "exports" name in module scope, making it implicitly
> "special".


The "Cons" here are the kicker for me. The first, "locals not
scrapeable" is a big deal, so more or less forces an "exports" object
in my mind. The second, "special variable", I'm not terribly happy
about, but isn't a big deal to me either. So an "exports" object wins
in my mind.

Eugene Lazutkin

unread,
Feb 3, 2009, 1:26:08 PM2/3/09
to serverjs
I agree with you on long package names. I like how python handles this
problem --- it allows to alias package names locally during the
import.

Thanks,

Eugene

On Feb 2, 2:34 pm, Hannes Wallnoefer <hann...@gmail.com> wrote:
> On Feb 2, 9:05 pm, Peter Michaux <petermich...@gmail.com> wrote:
>
> > On Mon, Feb 2, 2009 at 11:59 AM, mob <m...@embedthis.com> wrote:
>
> > > What is the definition of the Module Name?  How do you prevent
> > > collisions?
>
> > > Is it a reverse domain name?  com.corp....?
>
> > > Just having the module name be a file name doesn't scale -- Or am I
> > > missing something?
>
> > The reason domain names don't collide is because there is a regulating
> > body. The same is true for Perl modules on CPAN due to Pause.
>
> I really hate the reverse domain package naming convention in Java. It
> leeds to incredibly deep package structures, like
> com.company.project.module.package.Class. I think this is overkill,
> even for the amount of Java code out there.
>
> I think the sane thing is just to use the project, product or company
> name as primary namespace, and then grow deeper as your project
> requires. In Helma NG modules are called helma.file, helma.httpclient,
> helma.webapp, helma.webapp.response etc.
>
> > If a central repository is established for JavaScript modules then the
> > same system as CPAN/Pause could be used where a regulating body
> > approves module names. Mozilla has already stated they would host a
> > repository and it seems like a natural place for a central JavaScript
> > repository.
>
> > Your personal or company modules could be namespaced away from those
> > on the central repository by using your name or company name or domain
> > name as part of the module name.
>
> Agreed. I really like CPAN names (although it's a very long time since
> I've done any Perl coding). They usually consist of 2-3 tokens, and
> you get a good idea what they're about.
>
> Hannes
>
> > Peter

Peter Michaux

unread,
Feb 3, 2009, 1:50:32 PM2/3/09
to serv...@googlegroups.com
On Mon, Feb 2, 2009 at 10:04 PM, Kris Kowal <cowber...@gmail.com> wrote:
>
>
>> I think the only difference between your proposal and the
>> Helma/Pythonic proposal is adding the "exports" object. That is a much
>> smaller difference than the difference between the global eval
>> proposal and the Helma/Pythonic proposal.
>
> That's very, very awesome.

Your proposal has no global object, correct?

The Helma/Pythonic proposal has the shared global object on the module
scope's prototype chain.

That would be a big difference.

Peter

Kris Kowal

unread,
Feb 3, 2009, 3:06:45 PM2/3/09
to serverjs

> Your proposal has no global object, correct?
>
> The Helma/Pythonic proposal has the shared global object on the module
> scope's prototype chain.
>
> That would be a big difference.
>
> Peter

For the purpose of this conversation, let's assume that the phrases
"in the scope chain", or "in the prototype chain" are generalized to
mean "provides names to the lexical scope".

Our proposal says that a security sandbox would have a deeply frozen
global object that only contains standard primordials.

We also say that in order for a module to be interoperable with a
security sandbox, it must not have free variables (missing var's)
apart from names of primordials.

I'm presently suggesting that for the purposes of serverjs, it would
be acceptable for some modules to take advantage of eccentricities of
their platform provided that we acknowledge that they would not work
in a security sandbox.

That being said, Ihab and I are strong proponents of encouraging
modules to be interoperable. To that end, we recommend that
environments agree to place and take cross-platform capabilities with
the "require.env" object.

This is to say that we have several degrees of specification that
we're working on.

1. ask ECMA TC39 to add tools to the ECMAScript standard such that
native client-side security sandboxes can be created with pure
JavaScript.
2. define how module systems must be implemented in order to provide
security assertions.
3. define how modules ought to be written such that they are
interoperable, including secure, insecure, client-side, server-side,
current, and future modules.
4. ask ECMA TC39 to give us some sugar with the hope that ten years
out, certain import and export patterns are more expressive.

Our present discussion is entirely about 3, how to write modules,
which is modestly informed by 2, the constraints on module system
implementations in security environments.

So, the parts of our proposal that are relevant to this discussion are
the syntax we recommend for importing and exporting.

1. a "require" function that accepts absolute or relative module
identifiers and returns objects that contain module exports.
2. an "exports" object, (which we also propose be bound to "this"),
that the module augments as it's interpreted to expose its API.
3. the guarantee that "var" and "function" declarations are private
to the module file.
4. an "env" object, currently proposed to be "require.env", that
contains the limited set of voodoo powers bestowed upon the module and
it's dependencies.
5. that the global object is only guaranteed to contain primordials
and may be deeply frozen.

The fourth and fifth points address interoperability with secure
sandboxes. If a module intends to work inside a sandbox, it really
has to use such objects at least as if they were a deeply frozen in
the assumption that they would be in some platforms, eventually. We
strongly encourage this practice. We also make no bones about how
these requirements are supported by the module system, but we do know
that there's at least one way to do it. We also have some thoughts
about how a sandbox can instantiate a new sandbox with an unprivileged
or adapted environment, and how module factory functions can be shared
by multiple sandboxes without leaking.

Beyond our specification, you guys would still need to reach an accord
about:

1. what's in the interoperable environment
2. what is the grammar for module identifiers

We intend to leave these options open to the curators of libraries and
package managers.

Kris

Peter Michaux

unread,
Feb 3, 2009, 4:00:36 PM2/3/09
to serv...@googlegroups.com
I think we can cut down on the things about which we need to agree
which will help the process move along.


On Tue, Feb 3, 2009 at 12:06 PM, Kris Kowal <cowber...@gmail.com> wrote:

> That being said, Ihab and I are strong proponents of encouraging
> modules to be interoperable. To that end, we recommend that
> environments agree to place and take cross-platform capabilities with
> the "require.env" object.

I don't think it is necessary to agree on this. The platform-dependent
code is a layer below the standardized API that this project seeks.


> 1. a "require" function that accepts absolute or relative module
> identifiers and returns objects that contain module exports.

How a particular domain of applications interprets the string sent to
"require" would be application dependent. For example, serverjs may
standardize require('a.b.c') and those are resolved relative to the
path directories.


> 2. an "exports" object, (which we also propose be bound to "this"),
> that the module augments as it's interpreted to expose its API.

I don't think mandating its binding it to "this" is necessary.


> 3. the guarantee that "var" and "function" declarations are private
> to the module file.

yes

> 4. an "env" object, currently proposed to be "require.env", that
> contains the limited set of voodoo powers bestowed upon the module and
> it's dependencies.

I don't think that is necessary for this project to agree about this.
Everything can be loaded through "require" including "env".

var env = require('env');


> 5. that the global object is only guaranteed to contain primordials
> and may be deeply frozen.

The global object must also have the "require" function, or at least
it should appear to be global just like the "exports" object is
already there when the module starts executing. since it is not
possible to do

var require = require('require').require;

Whether or not "require" is actually global or just appears in the
module scope could be up to the implementation.

----

I think your proposal is a generalized version of the helma/pythonic
proposal with the addition of the exports object (and private by
default). Also a standard library should not be allowed to mutate the
global object.

Perhaps you can write up a proposal for the wiki? There would be no
need to describe all the various environments in which the code would
be able to run etc. Just the minimal subset of your grand proposal
that would matter for the writing of the serverjs standard library.

https://developer.mozilla.org/ServerJS/Modules

Peter

ihab...@gmail.com

unread,
Feb 3, 2009, 4:06:24 PM2/3/09
to serv...@googlegroups.com
On Tue, Feb 3, 2009 at 1:00 PM, Peter Michaux <peterm...@gmail.com> wrote:
>> 4. an "env" object, currently proposed to be "require.env", that
>> contains the limited set of voodoo powers bestowed upon the module and
>> it's dependencies.
>
> I don't think that is necessary for this project to agree about this.
> Everything can be loaded through "require" including "env".
>
> var env = require('env');

We considered this and, so far, we have rejected it. The reason is that:

a. We do not wish to conflate the namespace of modules identifiers
with the namespace of environment names provided to the module;
because

b. The specification of the environment and the specification of the
module identifier space are orthogonal concerns; and in practice

c. The module identifier resolution and the resolution of environment
names are properly the responsibility of separate objects in the
system, and this separation allows sandboxes to be constructed
piecewise (e.g., mixing and matching module identifier conventions
within the same sandbox), which is important for interoperability.

Peter Michaux

unread,
Feb 3, 2009, 4:25:49 PM2/3/09
to serv...@googlegroups.com
On Tue, Feb 3, 2009 at 1:06 PM, <ihab...@gmail.com> wrote:
>
> On Tue, Feb 3, 2009 at 1:00 PM, Peter Michaux <peterm...@gmail.com> wrote:
>>> 4. an "env" object, currently proposed to be "require.env", that
>>> contains the limited set of voodoo powers bestowed upon the module and
>>> it's dependencies.
>>
>> I don't think that is necessary for this project to agree about this.
>> Everything can be loaded through "require" including "env".
>>
>> var env = require('env');
>
> We considered this and, so far, we have rejected it. The reason is that:
>
> a. We do not wish to conflate the namespace of modules identifiers
> with the namespace of environment names provided to the module;
> because

I don't think of the environment and modules as different things.
Anything that needs to be exposed by the host can be part of the env
module.


> b. The specification of the environment and the specification of the
> module identifier space are orthogonal concerns; and in practice

I don't see them as orthogonal. To the programmer, both expose
functionality, objects, etc.


> c. The module identifier resolution and the resolution of environment
> names are properly the responsibility of separate objects in the
> system, and this separation allows sandboxes to be constructed
> piecewise (e.g., mixing and matching module identifier conventions
> within the same sandbox), which is important for interoperability.

A sandbox's require could be overloaded and treat require('env') specially.

Peter