RFC reqiure.paths behaviour

447 views
Skip to first unread message

Ash Berlin

unread,
May 25, 2009, 12:15:13 PM5/25/09
to serv...@googlegroups.com
So trying to make flusspferd behave as set out in <https://wiki.mozilla.org/ServerJS/Modules/SecurableModules
> but be slightly more permissive (Oh and this is not something that
I think necessarily needs to be speced, I just want a few more eyes on
the behaviour to make sure its sane, and still follows the ServerJS
spec.)

The use case is basically this:

15:58 < ashb> 2: require('ThingThatLoadsPlugins'); which does
require('./Some/Plugin')
15:59 < ashb> to me it would be nice if
./lib/ThingThatLoadsPlugins/Some/Plugin.js would work
for loading
application specific plugins
16:00 < ashb> Wes--: i can see an argument for looking in the same dir
as the
loaded module first, but i think falling back to
searching other
include paths is fairly sane

The easiest way to demonstrate this is with code: http://gist.github.com/117595

I've also included the content of that gist below in case people
prefer to read it in their mail client.

So, does what I've proposed seem like a sane way for flusspferd to
operate?

-ash

/opt/my_app/main.js:
> // THIS IS THE LAUCNHED PROGRAM
>
> /* system config gives this
> require.paths = [
> '/usr/lib/js/',
> '/home/ash/js_libs/'
> ];
>
> // To work with the spec we also add this conceptually
> require.paths.unshift('/opt/my_app/');
> */
>
>
> // This should probably be automatically expanded to an abs path.
> // Only question is relative to $CWD or to require._dir
> require.paths.unshift('./lib');
>
> // require.id = ""
> // require._dir; // Currently not a prop, should be "/opt/my_app/"
>
> require('./myApp/main');
> /*
> This builds up a module ID based on require.id, so
>
> myApp/main.js
>
> Then uses require._dir and looks for the following files in order
>
> /opt/my_app/lib/myApp/main.js // runtime changes to require.paths
> take precedence, obv.
> /opt/my_app/myApp/main.js
> /usr/lib/js/myApp/main.js
> /home/ash/js_libs/myApp/main.js
>
> */


/opt/my_app/lib/myApp/main.js:
> // See /opt/my_app/main.js for main program.
>
> // require.id = "/myApp"
> // require._dir; // Currently not a prop, should be "/opt/my_app/lib/
> myApp"
>
> require('./corge');
> /*
>
> Looks for:
>
> /opt/my_app/lib/myApp/corge.js
> /opt/my_app/myApp/corge.js
> /usr/lib/js/myApp/corge.js
> /home/ash/js_libs/myApp/corge.js
> */
>
>
>
> require('../xyzzy');
>
> /*
>
> Looks for:
>
> /opt/my_app/lib/xyzzy.js
> /opt/my_app/xyzzy.js
> /usr/lib/js/xyzzy.js
> /home/ash/js_libs/xyzzy.js
>
> */
>
>
> const DBI = require('dbi').DBI;
>
> /*
>
> Looks for:
>
> /opt/my_app/lib/dbi.js
> /opt/my_app/dbi.js
> /usr/lib/js/dbi.js
> /home/ash/js_libs/dbi.js
>
> */
>
> var dbi = new DBI('SQLite');
> /*
>
> Causes the DBI constructor to look for (note order change):
>
> /usr/lib/js/lib/dbi/driver/sqlite.js
> /opt/my_app/lib/dbi/driver/sqlite.js
> /opt/my_app/dbi/driver/sqlite.js
> /home/ash/js_libs/lib/dbi/driver/sqlite.js
>
> */

/usr/lib/js/dbi.js:
> // require.id = "/dbi"
> // require._dir; // Currently not a prop, should be "/usr/lib/js/"
>
> function DBI(driver) {
> return require('./dbi/driver/' + driver.toLower());
> }
>
> exports.DBI = DBI;

Aristid Breitkreuz

unread,
May 25, 2009, 12:56:27 PM5/25/09
to serverjs
Ash Berlin schrieb:
> So trying to make flusspferd behave as set out in <https://wiki.mozilla.org/ServerJS/Modules/SecurableModules

I think it is already compliant.

Kris Kowal

unread,
May 25, 2009, 3:28:35 PM5/25/09
to serv...@googlegroups.com

Yeah, I agree. I'm not seeing a deviation from the spec. You do have
some extensions which we might want to converge on as a group though
(and I welcome you to clarify if you feel that there is in fact a
deviation from spec that I'm not seeing).

- require.id
I don't think this is presently specified, but it should be required, I think.

- require.fileName
You mention require._dir; I think that this should be called
require.fileName and that it should only exist in modules that are
stored as files locally. You would have to feature-test for it, if
(require.fileName). I think it should also be the fully qualified
path to the file. Getting its directory would be
require("file").dirname(require.fileName).

- require.path
Presently, I made this require.loader.getPath and
require.loader.setPath in Narwhal so that the loader could defend its
integrity even in low-tech clients, and so that it would be attached
to the loader object that uses it instead of the sandbox which only
indirectly uses it by calling loader.load(id). The point on security
I think can be relaxed for the sanity of our users; require.path
should not be available at all in secure sandboxes. So, after you've
feature tested or know somehow otherwise that you're not in a sandbox,
you would be able to modify but not replace require.path with
operations like unshift. The point about the loader; presently this
falls under the category of implementation details; there hasn't been
any motion to standardize the relationship between loaders and sandbox
objects, and there probably doesn't need to be, yet. So, I think for
now I would be content to expose require.loader.path and make a copy
of that reference as require.path for convenience.

- require.extensions
In Narwhal we have this presently set to ["", ".js"] so that when we
look for "module" we can find both shell scripts and module files
along the require.path.

- absolute module identifiers
In Narwhal these have been handy. At the moment we have "./relative"
and "top-level" identifiers specified, but nothing for "/absolute"
identifiers, mostly because they won't and should not be be possible
in all environments. But for loaders that do load files from the
local file system, or from URL's within the loader's domain, absolute
identifiers are very useful and probably should work consistently
across platforms.

So how about some prose.

=== Module Context ===

# The "require" function must have an "id" property that must be the
current module's top-level identifier.

=== File Loader Extension ===

In permissive (not securely sandboxed) module systems that have access
to the entire file system with the privileges of the user:

# The "require" function may have a "require.fileName" property that
must have no other name and must be equal to the fully-qualified path
of the current module file. "require.fileName" may be "undefined" or
not set for any reason.
# The "require" function may have a "require.paths" property that must
be an array of strings referring to the file-system search path for
modules from highest priority to lowest priority. "require.paths" may
be "undefined" or not set for any reason.
# The "require" function may have a "require.extensions" property that
must be an array of strings identifying all of the file extensions
that may be considered modules from highest priority to lowest
priority. An extension must correspond to any file name that ends
with that string, thus file name extensions may include the null
string, "", that will match against any file including those with
other extensions, and ".js" that would match any file ending with
".js". File name extensions must have higher precedence than file
paths, such that the loader considers all extensions in each directory
before proceeding to a lower priority directory.
# The "require" function may accept "absolute" identifiers,
identifiers that begin with a "/". These identifiers must be resolved
as file names from the root of the file system, circumventing the
search for a containing path, but not circumventing the search for a
corresponding extension.

Kris Kowal

Ash Berlin

unread,
May 25, 2009, 4:10:29 PM5/25/09
to serv...@googlegroups.com

On 25 May 2009, at 20:28, Kris Kowal wrote:

>
> On Mon, May 25, 2009 at 9:56 AM, Aristid Breitkreuz
> <aristid.b...@gmx.de> wrote:
>> Ash Berlin schrieb:
>>> So trying to make flusspferd behave as set out in <https://wiki.mozilla.org/ServerJS/Modules/SecurableModules
>> I think it is already compliant.
>
> Yeah, I agree. I'm not seeing a deviation from the spec. You do have
> some extensions which we might want to converge on as a group though
> (and I welcome you to clarify if you feel that there is in fact a
> deviation from spec that I'm not seeing).

Currently flusspferd doesn't re-order the search path as shown on line
49 .../myApp/main.js of http://gist.github.com/117595 I think that
this is where we diverge from the intent of the spec, if not the
letter of it. Just to be clear, let me explain how flusspferd
currently behaves when given a relative module:

Lets assume (as per the gist) that we are in a module that was loaded
with reqiure('myApp/main') and that this module was found at /opt/
my_app/lib/myApp/main.js. This module then executes

const DBI = require('dbi').DBI;

var dbi = new DBI('SQLite');

// which in turn causes /usr/lib/js/dbi.js to do:
// require('./dbi/sqlite');

Currently flusspferd would search for a file called dbi/sqlite.js
under all its include paths, starting again at the first path. So if
the include paths were

* /opt/my_app/lib/
* /opt/my_app/
* /usr/lib/js/
* /home/ash/js_libs/

If /opt/my_app/lib/dbi/sqlite.js exists, it will (currently) be loaded
in preference to /usr/lib/js/dbi/sqlite.js, despite the fact that /usr/
lib/js/dbi.js requested './dbi/sqlite'.

Wes thinks this diverges from the spec (I think - we were getting a
bit crossed lines trying to explain our PoVs to each other), Aristid
doesn't. And I can see both of thier points.

>
> - require.id
> I don't think this is presently specified, but it should be
> required, I think.
>
> - require.fileName
> You mention require._dir; I think that this should be called
> require.fileName and that it should only exist in modules that are
> stored as files locally. You would have to feature-test for it, if
> (require.fileName). I think it should also be the fully qualified
> path to the file. Getting its directory would be
> require("file").dirname(require.fileName).

Probably shouldn't exist (in JS space, or not to insecure code) to
avoid info leakage in sandboxes?

>
> - require.path
> Presently, I made this require.loader.getPath and
> require.loader.setPath in Narwhal so that the loader could defend its
> integrity even in low-tech clients, and so that it would be attached
> to the loader object that uses it instead of the sandbox which only
> indirectly uses it by calling loader.load(id). The point on security
> I think can be relaxed for the sanity of our users; require.path
> should not be available at all in secure sandboxes. So, after you've
> feature tested or know somehow otherwise that you're not in a sandbox,
> you would be able to modify but not replace require.path with
> operations like unshift. The point about the loader; presently this
> falls under the category of implementation details; there hasn't been
> any motion to standardize the relationship between loaders and sandbox
> objects, and there probably doesn't need to be, yet. So, I think for
> now I would be content to expose require.loader.path and make a copy
> of that reference as require.path for convenience.

the ||paths|| property on flusspferd require function is ||
DontDelete||. Probably does make sense to ensure its also ||ReadOnly||
such that is always an array.

>
> - require.extensions
> In Narwhal we have this presently set to ["", ".js"] so that when we
> look for "module" we can find both shell scripts and module files
> along the require.path.

Hmmm. This gets a bit messy when you start dealing with naitve
(compiled) modules. Since on unix both GPSEE and Flusspferd name them
using the native DSO conventions: libfoo.so which would get loaded by
require('foo').

>
> - absolute module identifiers
> In Narwhal these have been handy. At the moment we have "./relative"
> and "top-level" identifiers specified, but nothing for "/absolute"
> identifiers, mostly because they won't and should not be be possible
> in all environments. But for loaders that do load files from the
> local file system, or from URL's within the loader's domain, absolute
> identifiers are very useful and probably should work consistently
> across platforms.
>
> So how about some prose.
>
> === Module Context ===
>
> # The "require" function must have an "id" property that must be the
> current module's top-level identifier.

Yes, but we need to further clarify exactly how this is constructed,
since https://wiki.mozilla.org/ServerJS/Modules/
SecurableModules#Module_Identifiers doesn't quite seem right yet,
particularly with regard to relatively required modules (I think they
should be turned into an absolute id somehow.)


>
> === File Loader Extension ===
>
> In permissive (not securely sandboxed) module systems that have access
> to the entire file system with the privileges of the user:
>
> # The "require" function may have a "require.fileName" property that
> must have no other name and must be equal to the fully-qualified path
> of the current module file. "require.fileName" may be "undefined" or
> not set for any reason.
> # The "require" function may have a "require.paths" property that must
> be an array of strings referring to the file-system search path for
> modules from highest priority to lowest priority. "require.paths" may
> be "undefined" or not set for any reason.

*not set or ignored for any reason.

> # The "require" function may have a "require.extensions" property that
> must be an array of strings identifying all of the file extensions
> that may be considered modules from highest priority to lowest
> priority. An extension must correspond to any file name that ends
> with that string, thus file name extensions may include the null
> string, "", that will match against any file including those with
> other extensions, and ".js" that would match any file ending with
> ".js". File name extensions must have higher precedence than file
> paths, such that the loader considers all extensions in each directory
> before proceeding to a lower priority directory.

(See above)

> # The "require" function may accept "absolute" identifiers,
> identifiers that begin with a "/". These identifiers must be resolved
> as file names from the root of the file system, circumventing the
> search for a containing path, but not circumventing the search for a
> corresponding extension.

so require("/A/B.txt") would die then?

Also some prose about how relative module IDs to require behave is
needed, and what exactly the behavior of that should be.

Kris Kowal

unread,
May 25, 2009, 5:07:22 PM5/25/09
to serv...@googlegroups.com
On Mon, May 25, 2009 at 1:10 PM, Ash Berlin <ash_g...@firemirror.com> wrote:
> Currently flusspferd doesn't re-order the search path as shown on line
> 49 .../myApp/main.js of http://gist.github.com/117595 I think that
> this is where we diverge from the intent of the spec, if not the
> letter of it. Just to be clear, let me explain how flusspferd
> currently behaves when given a relative module:

Would you mind reducing this to a test case?

>
> Lets assume (as per the gist) that we are in a module that was loaded
> with reqiure('myApp/main') and that this module was found at /opt/
> my_app/lib/myApp/main.js. This module then executes
>
>     const DBI = require('dbi').DBI;
>     var dbi = new DBI('SQLite');
>     // which in turn causes /usr/lib/js/dbi.js to do:
>     // require('./dbi/sqlite');
>
> Currently flusspferd would search for a file called dbi/sqlite.js
> under all its include paths, starting again at the first path. So if
> the include paths were
>
> * /opt/my_app/lib/
> * /opt/my_app/
> * /usr/lib/js/
> * /home/ash/js_libs/
>
> If /opt/my_app/lib/dbi/sqlite.js exists, it will (currently) be loaded
> in preference to /usr/lib/js/dbi/sqlite.js, despite the fact that /usr/
> lib/js/dbi.js requested './dbi/sqlite'.

> Wes thinks this diverges from the spec (I think - we were getting a
> bit crossed lines trying to explain our PoVs to each other), Aristid
> doesn't. And I can see both of thier points.

It does not conflict with the spec. It simply illustrates a hazard
and feature of fiddling with the top-level paths, one that I'm content
to leave.

>> - require.fileName
>> You mention require._dir; I think that this should be called
>> require.fileName and that it should only exist in modules that are
>> stored as files locally.  You would have to feature-test for it, if
>> (require.fileName).  I think it should also be the fully qualified
>> path to the file.  Getting its directory would be
>> require("file").dirname(require.fileName).
>
> Probably shouldn't exist (in JS space, or not to insecure code) to
> avoid info leakage in sandboxes?

There are two reasons why it might not exist in a particular module:
the module isn't stored in the file system, or as you say, to avoid
leaking information to sandboxes.

>> - require.path (correction: paths)


> the ||paths|| property on flusspferd require function is ||
> DontDelete||. Probably does make sense to ensure its also ||ReadOnly||
> such that is always an array.

I think it makes sense to say that the loader "may" set the "paths"
property to ReadOnly and DontDelete.

>> - require.extensions
>> In Narwhal we have this presently set to ["", ".js"] so that when we
>> look for "module" we can find both shell scripts and module files
>> along the require.path.
>
> Hmmm. This gets a bit messy when you start dealing with naitve
> (compiled) modules. Since on unix both GPSEE and Flusspferd name them
> using the native DSO conventions: libfoo.so which would get loaded by
> require('foo').

Good point. Any ideas on how to generalize this feature to
accommodate that kind of situation? Perhaps a regular expression
instead? "require.pattern"?

>> # The "require" function must have an "id" property that must be the
>> current module's top-level identifier.
>
> Yes, but we need to further clarify exactly how this is constructed,
> since https://wiki.mozilla.org/ServerJS/Modules/
> SecurableModules#Module_Identifiers doesn't quite seem right yet,
> particularly with regard to relatively required modules (I think they
> should be turned into an absolute id somehow.)

"absolute" is not defined for non-file-name module-identifiers, such
that it must be the "top-level" identifier of the module, which is
trivial to construct from a relative module id from the top-level id
of its requiring module.

>> # The "require" function may have a "require.fileName" property that
>> must have no other name and must be equal to the fully-qualified path
>> of the current module file.  "require.fileName" may be "undefined" or
>> not set for any reason.
>> # The "require" function may have a "require.paths" property that must
>> be an array of strings referring to the file-system search path for
>> modules from highest priority to lowest priority.  "require.paths" may
>> be "undefined" or not set for any reason.
>
> *not set or ignored for any reason.

>> # The "require" function may have a "require.extensions" property that
>> must be an array of strings identifying all of the file extensions
>> that may be considered modules from highest priority to lowest
>> priority.  An extension must correspond to any file name that ends
>> with that string, thus file name extensions may include the null
>> string, "", that will match against any file including those with
>> other extensions, and ".js" that would match any file ending with
>> ".js". File name extensions must have higher precedence than file
>> paths, such that the loader considers all extensions in each directory
>> before proceeding to a lower priority directory.
>
> (See above)

Okay and okay.

>> # The "require" function may accept "absolute" identifiers,
>> identifiers that begin with a "/".  These identifiers must be resolved
>> as file names from the root of the file system, circumventing the
>> search for a containing path, but not circumventing the search for a
>> corresponding extension.
>
> so require("/A/B.txt") would die then?

Not if "" is one of the accepted extensions.

> Also some prose about how relative module IDs to require behave is
> needed, and what exactly the behavior of that should be.

Could you propose an explanation to illustrate what is not clearly specified?

Kris Kowal

Hannes Wallnoefer

unread,
May 25, 2009, 5:21:23 PM5/25/09
to serv...@googlegroups.com
2009/5/25 Ash Berlin <ash_g...@firemirror.com>:

Helma NG originally implemented ./ paths as strictly relative to the
local module, then we switched to resolving against the whole module
path to support some construct in Narwhal I think.

I'm not sure I can read one or the other behaviour out of the
securable modules spec. But IMO our current implementation is ok as a
means of "virtualization", although it's very dangerous if used
thoughtlessly. In fact, I think I would advice against splitting up
functionality over multiple module repositories in jS just like (and
for the same reasons) I would advice against excessive use of class
inheritance in Java.

>>
>> - require.id
>> I don't think this is presently specified, but it should be
>> required, I think.

Please don't. In Helma NG, require is plain old global javascript
function that is the same for all scopes. Because we have
pythonic/top-level modules scopes there's no need to set up per-module
closures, and I'd very much leave it that way.

Besides, require.id is not really a very intuitive name, is it? FWIW,
in Helma NG we set a __name__ property in the module scope to the
value of the canonical module name, or "__main__" if the module was
called as command line script.

Hannes

Kris Kowal

unread,
May 25, 2009, 6:52:33 PM5/25/09
to serv...@googlegroups.com

Ah, that would pose a problem for us, wouldn't it. I redact my
original suggestion.

>
> Besides, require.id is not really a very intuitive name, is it? FWIW,
> in Helma NG we set a __name__ property in the module scope to the
> value of the canonical module name, or "__main__" if the module was
> called as command line script.

I'd like to steer clear of wunderbars wherever possible.

Since we don't have a module-scope in Chiron or Narwhal, the
alternative would be to put names in the module factory function
arguments. While this is…odd…and makes it harder on the library
writer, It's certainly not impossible and I've never liked library
writers anyway.

I've had some success experimenting with the idioms:

if(require.name == require.id)
exports.main(system.args);

And:

var config = require(require.main);

…wherein the first is slightly different than the Pythonic version,
and enables the second idiom. It also would get rid of double-loading
bugs that come up in Python when you attempt to load your main module
from another module.

__file__, __name__, and __main__ are options, but again, I'd like to
steer clear of the wunderbars if we can think of something better. It
would make things easier for us if there were some module-specific
variable to attach them to. I'm reluctant, but perhaps we could
inject "module.file", "module.name", and "module.main". I feel like
I'm reopening an old wound here, so I wouldn't mind postponing the
issue indefinitely.

Kris Kowal

Ash Berlin

unread,
May 25, 2009, 7:56:06 PM5/25/09
to serv...@googlegroups.com

On 25 May 2009, at 23:52, Kris Kowal wrote:


__file__, __name__, and __main__ are options, but again, I'd like to
steer clear of the wunderbars if we can think of something better.  It
would make things easier for us if there were some module-specific
variable to attach them to.  I'm reluctant, but perhaps we could
inject "module.file", "module.name", and "module.main".  I feel like
I'm reopening an old wound here, so I wouldn't mind postponing the
issue indefinitely.

Kris Kowal

Why don't we just define the require behaviour in terms of 'private state variables' and leave it as up to the implementation how or if these get exposed?

Kris Kowal

unread,
May 26, 2009, 3:41:59 AM5/26/09
to serv...@googlegroups.com
On Mon, May 25, 2009 at 4:56 PM, Ash Berlin <ash_g...@firemirror.com> wrote:
> Why don't we just define the require behaviour in terms of 'private state
> variables' and leave it as up to the implementation how or if these get
> exposed?

I don't know; what does everyone else think?

1.) Should we standardize a way to know whether the current module is
the main program?

2.) Should we standardize a way to discover the top-level identifier
of the main program?

3.) Should we standardize a way to discover the location on the file
system of the current module, so as to find neighboring resources like
images, in the cases where modules are in fact stored on a file
system?

Kris Kowal

Ondrej Zara

unread,
May 26, 2009, 3:50:26 AM5/26/09
to serv...@googlegroups.com
I don't know; what does everyone else think?

1.) Should we standardize a way to know whether the current module is
the main program?

I don't see any need for this.
 

2.) Should we standardize a way to discover the top-level identifier
of the main program?

Dtto, when is this information useful?
 

3.) Should we standardize a way to discover the location on the file
system of the current module, so as to find neighboring resources like
images, in the cases where modules are in fact stored on a file
system?

This can be passed as an argument to the module? Several ideas here:

- "modulePath",
- "exports.__modulePath",
- "__path"

and many others...


O.



Kris Kowal



Kris Kowal

unread,
May 26, 2009, 4:32:13 AM5/26/09
to serv...@googlegroups.com
On Tue, May 26, 2009 at 12:50 AM, Ondrej Zara <ondre...@gmail.com> wrote:
>> 1.) Should we standardize a way to know whether the current module is
>> the main program?
> I don't see any need for this.

Ah, there are certainly uses. In Python at the very least, it's a
common idiom for running unit tests or for any module that provides a
feature as a module but also as a script:

if __name__ == '__main__':
from doctest import testmod
testmod()

That would run all of the unit tests in that module.

>> 2.) Should we standardize a way to discover the top-level identifier
>> of the main program?
> Dtto, when is this information useful?

Particularly in the implementation of doctest.testmod, where it is a
necessary piece of information to allow the doctest module's testmod
function to inspect the "__main__" module for unit tests in its
"__doc__" properties. In Python this is accomplished by importing the
"sys" module, and grabbing "sys.modules["__main__"]", but I've
mentioned that calling the main module "__main__" instead of using its
canonical top-level identifier means that the main module can be
accidentally loaded twice. I think the far more useful idiom would be
to tell the module what it's identifier is and what the main module's
identifier is separately.

>> 3.) Should we standardize a way to discover the location on the file
>> system of the current module, so as to find neighboring resources like
>> images, in the cases where modules are in fact stored on a file
>> system?
>
> This can be passed as an argument to the module? Several ideas here:
>
> - "modulePath",
> - "exports.__modulePath",
> - "__path"
> and many others...

- "__file__"
- "module.file"
- "require.file" which is already deemed infeasible in Helma NG.

So, Ondrej, given that there are known uses for these features, do you
think it would be worth going through the effort of negotiating a
consensus, or would it be better to leave the issue alone and allow
each system to do it differently, if at all?

The operative choice here is not whether every module loader should
implement these features, (in fact there are some system for which
they definitely should not be available like security sandboxes, and
web and database modules) but whether every module loader that opts to
implement these features should conform to the same interface.

Kris Kowal

Ondrej Zara

unread,
May 26, 2009, 5:43:51 AM5/26/09
to serv...@googlegroups.com
Ah, there are certainly uses.  In Python at the very least, it's a
common idiom for running unit tests or for any module that provides a
feature as a module but also as a script:

if __name__ == '__main__':
   from doctest import testmod
   testmod()

Hm, yes. In Python architecture, this surely makes sense. On the other hand, I always tend to distinguish between a "program" and a "library" (correct me if I am mistaken, but every other language - except Python - makes this differentiation?).
 

Particularly in the implementation of doctest.testmod, where it is a
necessary piece of information to allow the doctest module's testmod
function to inspect the "__main__" module for unit tests in its
"__doc__" properties.  In Python this is accomplished by importing the
"sys" module, and grabbing "sys.modules["__main__"]", but I've
mentioned that calling the main module "__main__" instead of using its
canonical top-level identifier means that the main module can be
accidentally loaded twice.  I think the far more useful idiom would be
to tell the module what it's identifier is and what the main module's
identifier is separately.

My Python knowledge is very limited, but does not this introduce some kind of a circular dependency? Main module needs some submodules, while these depend on the main module... as far as I can say from my (rather quick and vague) observation, this is not something I would want...



>> 3.) Should we standardize a way to discover the location on the file
>> system of the current module, so as to find neighboring resources like
>> images, in the cases where modules are in fact stored on a file
>> system?
>
> This can be passed as an argument to the module? Several ideas here:
>
> - "modulePath",
> - "exports.__modulePath",
> - "__path"
> and many others...

- "__file__"
- "module.file"
- "require.file" which is already deemed infeasible in Helma NG.

So, Ondrej, given that there are known uses for these features, do you
think it would be worth going through the effort of negotiating a
consensus, or would it be better to leave the issue alone and allow
each system to do it differently, if at all?


I believe that the correct way here is a "need to have" basis. Let's standardize a feature as soon (not sooner) as it is necessary. For example, I have no motivation to solve your first two points. But the third one, "let each module know where it is located in a filesystem", looks like a feature which will be 100% necessary (captcha module needs to have a font file available), so I opt for a small voting here to get into a concensus. 


The operative choice here is not whether every module loader should
implement these features, (in fact there are some system for which
they definitely should not be available like security sandboxes, and
web and database modules) but whether every module loader that opts to
implement these features should conform to the same interface.


Yes, this makes perfect sense to me.



Ondrej


 

Kris Kowal



Wes Garland

unread,
May 26, 2009, 8:14:43 AM5/26/09
to serv...@googlegroups.com
3) certainly has tremendous value.  I would go as far as to say that both the current module's location and the program module's name and location should be discoverable.  That would allow general-purpose libraries transparent access to program-specific resources (like, I dunno, stylesheets or config parameters).

If we have all that stuff, then 1) and 2) fall out by comparing current-module and program-module location.

Wes
--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

Patrick Mueller

unread,
May 26, 2009, 8:57:35 AM5/26/09
to serv...@googlegroups.com
On May 26, 2009, at 4:32 AM, Kris Kowal wrote:

> On Tue, May 26, 2009 at 12:50 AM, Ondrej Zara
> <ondre...@gmail.com> wrote:
>
>>> 3.) Should we standardize a way to discover the location on the file
>>> system of the current module, so as to find neighboring resources
>>> like
>>> images, in the cases where modules are in fact stored on a file
>>> system?
>>
>> This can be passed as an argument to the module? Several ideas here:
>>
>> - "modulePath",
>> - "exports.__modulePath",
>> - "__path"
>> and many others...
>
> - "__file__"
> - "module.file"
> - "require.file" which is already deemed infeasible in Helma NG.

Would prefer this be in a property of an object, as opposed to a bare
variable. ie, "module.file" is the style I would prefer. As we add
more of these geegaws, it would be nice to keep the namespace
pollution down. Also would prefer to keep down on wunderbars (nice
term!), which will get even worse if we don't give them a container to
live in.

> So, Ondrej, given that there are known uses for these features, do you
> think it would be worth going through the effort of negotiating a
> consensus, or would it be better to leave the issue alone and allow
> each system to do it differently, if at all?
>
> The operative choice here is not whether every module loader should
> implement these features, (in fact there are some system for which
> they definitely should not be available like security sandboxes, and
> web and database modules) but whether every module loader that opts to
> implement these features should conform to the same interface.

I concur with the last bit: "every module loader that opts to
implement these features should conform to the same interface". And
as you indicate, these should be optional.

Patrick Mueller - http://muellerware.org/

Wes Garland

unread,
May 26, 2009, 9:40:30 AM5/26/09
to serv...@googlegroups.com
> Would prefer this be in a property of an object, as opposed to a bare
> variable.  ie, "module.file" is the style I would prefer.

+1000

(and I really, really, really hope that they don't wind up as a property of require)

Wes

Daniel Friesen

unread,
May 26, 2009, 10:01:55 AM5/26/09
to serv...@googlegroups.com
Ondrej Zara wrote:
>
> Ah, there are certainly uses. In Python at the very least, it's a
> common idiom for running unit tests or for any module that provides a
> feature as a module but also as a script:
>
> if __name__ == '__main__':
> from doctest import testmod
> testmod()
>
>
> Hm, yes. In Python architecture, this surely makes sense. On the other
> hand, I always tend to distinguish between a "program" and a "library"
> (correct me if I am mistaken, but every other language - except Python
> - makes this differentiation?).
I agree. IMHO things like tests and actual program files inside
libraries (Like Rails where half is a library and half is a program)
should be handled with extra files outside the main module file. cli
files for running tests, or a nice Rake like system (my Jake idea) so
things like `rake test.run` can be used to quickly run tests by jumping
into the library's installed location.
I have one potential side idea. Rather than something like module.file
module.getResource(); which would grab a resource.
Then again now that I think of it, the File spec isn't exactly kind
towards that idea. It's just one of those advantages to an instance
based api.

> The operative choice here is not whether every module loader should
> implement these features, (in fact there are some system for which
> they definitely should not be available like security sandboxes, and
> web and database modules) but whether every module loader that opts to
> implement these features should conform to the same interface.
>
>
>
> Yes, this makes perfect sense to me.
>
>
>
> Ondrej
>
>
>
>
>
> Kris Kowal
>

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

Kris Kowal

unread,
May 26, 2009, 12:43:12 PM5/26/09
to serv...@googlegroups.com
On Tue, May 26, 2009 at 7:01 AM, Daniel Friesen
<nadir.s...@gmail.com> wrote:
> I have one potential side idea. Rather than something like module.file
> module.getResource(); which would grab a resource.
> Then again now that I think of it, the File spec isn't exactly kind
> towards that idea. It's just one of those advantages to an instance
> based api.

How so? Seems equitable to write:

var file = require("file");
module.getResource = function (/*path components*/) {
return new file.Path(file.join.apply(file, arguments));
};

var path = module.getResource('font.ttf')
var stream = path.open("b");
var content = stream.read();
stream.close();

(Would you mind trimming the quoted to text to relevant excerpts in the future?)

Kris Kowal

Daniel Friesen

unread,
May 26, 2009, 1:13:09 PM5/26/09
to serv...@googlegroups.com
Kris Kowal wrote:
> On Tue, May 26, 2009 at 7:01 AM, Daniel Friesen
> <nadir.s...@gmail.com> wrote:
>
>> I have one potential side idea. Rather than something like module.file
>> module.getResource(); which would grab a resource.
>> Then again now that I think of it, the File spec isn't exactly kind
>> towards that idea. It's just one of those advantages to an instance
>> based api.
>>
>
> How so? Seems equitable to write:
>
> var file = require("file");
> module.getResource = function (/*path components*/) {
> return new file.Path(file.join.apply(file, arguments));
> };
>
> var path = module.getResource('font.ttf')
> var stream = path.open("b");
> var content = stream.read();
> stream.close();
>
Oh right Path. Though I'm referring more to the alternative
possibilities, of storage. ie: Weren't people making comments about
storage in the database?

In terms of instance based (where a File is an instance, and you can
open() a stream from it; ie: My plans in MonkeyScript)
module.getResource = function(identifier) {
var content = db.queryOne("SELECT content FROM resources WHERE
module = '?' AND identifier = '?'", [module.id, identifier]).content;
return new FauxFile({name: identifier, contents: content}); //
Something which mimics the file interface
};

To the module it would just use the normal f.open('r'); or f.contents;
like normal completely ignorant of the fact that the data doesn't even
exist on the filesystem.

Although myself, I have no actual plans to let bananas be stored
anywhere else other than the filesystem. I have yet to see other
languages do it, nor any advantages to it for MonkeyScript (However
don't get me wrong, I DO see the advantages for other environments to
store modules outside of the filesystem) and for MonkeyScript itself I
don't see how that will be possible with all the native C/C++ code (it
would be more possible for a Rhino base environment of course).


> (Would you mind trimming the quoted to text to relevant excerpts in the future?)
>

Sure

Kris Kowal

unread,
May 26, 2009, 7:19:16 PM5/26/09
to serv...@googlegroups.com
On Tue, May 26, 2009 at 10:13 AM, Daniel Friesen
<nadir.s...@gmail.com> wrote:
> Oh right Path. Though I'm referring more to the alternative
> possibilities, of storage. ie: Weren't people making comments about
> storage in the database?

Ah, I see. This is a great idea. We could specify something like an
optional module.resource(id) that would use the same namespace
nomenclature as module identifiers and would return an object that
implements .open(mode, options) which would be more portable.

Kris Kowal

Hannes Wallnoefer

unread,
May 26, 2009, 7:50:33 PM5/26/09
to serv...@googlegroups.com
2009/5/27 Kris Kowal <cowber...@gmail.com>:
That sounds very much like the resource framework we have in Helma NG:

https://dev.helma.org/trac/helma/browser/helma-ng/trunk/src/org/helma/repository

This is what the module loader is built upon. And there's also a
getResource() function so you can retrieve arbitrary file resources
using the same naming and lookup rules as used for modules. Currently
we have implementations for plain files, .war (java web archive) and
.zip archives.

Hannes

> Kris Kowal
>
> >
>

Kris Kowal

unread,
May 26, 2009, 10:38:08 PM5/26/09
to serv...@googlegroups.com
On Tue, May 26, 2009 at 4:50 PM, Hannes Wallnoefer <han...@gmail.com> wrote:
>> Ah, I see.  This is a great idea.  We could specify something like an
>> optional module.resource(id) that would use the same namespace
>> nomenclature as module identifiers and would return an object that
>> implements .open(mode, options) which would be more portable.
>
> That sounds very much like the resource framework we have in Helma NG:
>
> https://dev.helma.org/trac/helma/browser/helma-ng/trunk/src/org/helma/repository
>
> This is what the module loader is built upon. And there's also a
> getResource() function so you can retrieve arbitrary file resources
> using the same naming and lookup rules as used for modules. Currently
> we have implementations for plain files, .war (java web archive) and
> .zip archives.

Sounds good; I think we should converge on our interfaces. Would you
recommend/describe your API from within modules? Do you like the idea
of injecting a "module" name into your module scope to host the API
beside exports? I think that injecting a module variable for these
optional features would work with both module factory functions and
module context runners.

Kris Kowal

Hannes Wallnoefer

unread,
May 27, 2009, 4:09:40 AM5/27/09
to serv...@googlegroups.com
2009/5/27 Kris Kowal <cowber...@gmail.com>:
I think injecting a "module" name for module meta information and
functionality is a good idea. I also think "module.id" and
"module.path" for the current module id and path are much more
intuitive than "require.id" and "require.path". So +1 from my side to
an additional "module" object. I'm not even sure whether it should be
optional. Implementation-wise it shouldn't be too much of a burden,
and it would definitely help interoperability if at least some basic
parts were just required, wouldn't it?

Hannes

> Kris Kowal
>
> >
>

Daniel Friesen

unread,
May 27, 2009, 5:40:40 AM5/27/09
to serv...@googlegroups.com
Since this is starting to get closer and closer to my plans for banana
(of course never close enough for merge) I'll make a quick note of what
I have planned in MonkeyScript's Banana system.

There are three free properties in module scope. banana, self, and exports.
banana is the same as it is globally (well ok, not actually, recently I
decided to make the banana inside a banana a special transparent version
which acts precisely like the normal banana but will only allow loading
of bananas which have been listed as required within the banana.json
metadata).
self is the metadata object for the module. This is basically the same
as running banana.info(q); for the module instead of banana(q);
exports is... well that's fairly obvious. The only difference between
ServerJS' exports and MonkeyScript Banana's exports is that I support
`exports = {};` inside of a module. And I don't allow anything but what
the banana has defined on that object to be there, like `exports.PATH`,
or anything which would pollute the`with(banana(q)) { ... }` pattern.

module.id is roughly self.namespace; module.path is roughly self.path;
require('./???'); is roughly the self.exec('foo.js'); I was thinking of
to replace exec( __DIR__ + 'foo.js' ); or exec( self.path + 'foo.js' );

In my terms.
banana is globalish; Other than the fact that it's restricted to only
loading required things from inside a module banana is the same everywhere.
self is local;
exports is local until the banana is loaded when of course it becomes
public.

Btw, I never actually thought of practically using it in my own stuff...
But I do have room for someone to program using the idom.
-- somebanana.js
with(exports) {
exports.a = function a() 'b';
a(); // because we with'ed our exports we can use anything we define
as an export ourselves.
}
----

Kris Kowal

unread,
May 27, 2009, 1:30:27 PM5/27/09
to serv...@googlegroups.com
On Wed, May 27, 2009 at 1:09 AM, Hannes Wallnoefer <han...@gmail.com> wrote:
> I think injecting a "module" name for module meta information and
> functionality is a good idea. I also think "module.id" and
> "module.path" for the current module id and path are much more
> intuitive than "require.id" and "require.path". So +1 from my side to
> an additional "module" object. I'm not even sure whether it should be
> optional. Implementation-wise it shouldn't be too much of a burden,
> and it would definitely help interoperability if at least some basic
> parts were just required, wouldn't it?

Proposal:

module -- required object
module.id -- required, top-level id of current module, String
module.path -- optional, absolute path of current module file, String
module.resource(id).open(…) -- optional, conforms to file.open
mode and options.
require.main -- top-level id of the program module, String

Kris Kowal

Ash Berlin

unread,
May 27, 2009, 1:59:33 PM5/27/09
to serv...@googlegroups.com

Does "module.path" include the file itself, or is it just the dir (or
other suitable URI) where the file can be found?

require.main: could there be a case when module.id == require.main but
its not the same module? And I'm not quite sure how to build up a ID
for a module that isn't in the search path. Say i have:

my_serverjs_interpreter /opt/my_app/my_app.js

What would module.id (and require.main) be for this? The reason I ask
this is because right now flusspferd calculates module IDs based upon
its location under the include path, and so currently we can't create
an id for anything not in the include path (which only happens for the
main module.) Do we need to change how we do things? If so then I dont
see how module.id is different to module.path?

Is there a need for knowing what the main module is explicitly, vs
just the case where 'the current module is the main entry point'?
Since right the only usage of require.main that springs to mind is

if (require.main == module.id) { }

But i suspect i'm just missing something here.

-ash

Kris Kowal

unread,
May 27, 2009, 2:23:21 PM5/27/09
to serv...@googlegroups.com
On Wed, May 27, 2009 at 10:59 AM, Ash Berlin <ash_g...@firemirror.com> wrote:
> On 27 May 2009, at 18:30, Kris Kowal wrote:
>> On Wed, May 27, 2009 at 1:09 AM, Hannes Wallnoefer
>
> Does "module.path" include the file itself, or is it just the dir (or
> other suitable URI) where the file can be found?

The full file name, dirname, basepath, and extension all together.
"path" is two kinds of ambiguous. Can we go with "file" or "fileName"
instead?

> require.main: could there be a case when module.id == require.main but
> its not the same module?

require(require.main) must grab the singleton exports of that module.
That means that require.main must == module.id for that module.

> And I'm not quite sure how to build up a ID
> for a module that isn't in the search path. Say i have:
> my_serverjs_interpreter /opt/my_app/my_app.js

In Narwhal, we're presently accepting absolute file names as module
identifiers for modules outside the top-level module path. This is
presently outside the standard, and probably does not need to be
standardized since interoperable modules must not take advantage of
this kind of feature. It's merely a convenience for shell scripting
and could be safely handled in a variety of ways and we wouldn't
suffer for differences among our implementations on that point.

> Is there a need for knowing what the main module is explicitly, vs
> just the case where 'the current module is the main entry point'?
> Since right the only usage of require.main that springs to mind is
>
> if (require.main == module.id) { }

Right, the other usage is require(require.main), which I've found
handy in Jack configuration files.

1 #!/usr/bin/env jackup
2
3 var ContentLength = require('jack/contentlength').ContentLength;
4
5 exports.app = new ContentLength(function (env) {
6 return [200, {"content-type": "text/plain"}, ["Hello, World!"]];
7 });
8
9 if (module.id == require.main) {
10 system.args.push(module.fileName);
11 require('jackup').main(system.args);
12 }
13

Kris Kowal

Kris Kowal

unread,
May 27, 2009, 2:25:23 PM5/27/09
to serv...@googlegroups.com
>  1 #!/usr/bin/env jackup
>  2
>  3 var ContentLength = require('jack/contentlength').ContentLength;
>  4
>  5 exports.app = new ContentLength(function (env) {
>  6     return [200, {"content-type": "text/plain"}, ["Hello, World!"]];
>  7 });
>  8
>  9 if (module.id == require.main) {
>  10     system.args.push(module.fileName);
>  11     require('jackup').main(system.args);
>  12 }
>  13

Oh, right. Not used in this example. Here's the concept though: line
10 explicitly adds the main module's fileName to the arguments. If it
were omitted jackup.main could require(require.main) instead of
require(system.args.shift()) to get that config file.

Kris Kowal

Hannes Wallnoefer

unread,
May 28, 2009, 4:24:55 AM5/28/09
to serv...@googlegroups.com
2009/5/27 Kris Kowal <cowber...@gmail.com>:
>
> Proposal:
>
> module -- required object
> module.id -- required, top-level id of current module, String
> module.path -- optional, absolute path of current module file, String
> module.resource(id).open(…) -- optional, conforms to file.open
> mode and options.
> require.main -- top-level id of the program module, String

+1 on everything, except I'd prefer require.main to contain the
program path instead of the program's module id, see rationale below.

2009/5/27 Kris Kowal <cowber...@gmail.com>:
>
> On Wed, May 27, 2009 at 10:59 AM, Ash Berlin <ash_g...@firemirror.com> wrote:
>> On 27 May 2009, at 18:30, Kris Kowal wrote:
>>> On Wed, May 27, 2009 at 1:09 AM, Hannes Wallnoefer
>>
>> Does "module.path" include the file itself, or is it just the dir (or
>> other suitable URI) where the file can be found?
>
> The full file name, dirname, basepath, and extension all together.
> "path" is two kinds of ambiguous.  Can we go with "file" or "fileName"
> instead?

I actually like "path" and think "file" and "fileName" have their own
problem of hinting to the pure file name.

>> require.main: could there be a case when module.id == require.main but
>> its not the same module?
>
> require(require.main) must grab the singleton exports of that module.
> That means that require.main must == module.id for that module.
>
>> And I'm not quite sure how to build up a ID
>> for a module that isn't in the search path. Say i have:
>> my_serverjs_interpreter /opt/my_app/my_app.js

This is why I think it would be better if require.main contained the
program path. I think it is safer to assume that the main program has
a path than a module id, and checking if a module is the main program
would still be simple:

if (require.main == module.path) { ... }

Admitted, this introduces the risk of getting confused by
non-canonical paths, but I think it would be easy to guarantee that if
require.main and module.path use non-canonical paths, at least they
use the same one.

> In Narwhal, we're presently accepting absolute file names as module
> identifiers for modules outside the top-level module path.  This is
> presently outside the standard, and probably does not need to be
> standardized since interoperable modules must not take advantage of
> this kind of feature.  It's merely a convenience for shell scripting
> and could be safely handled in a variety of ways and we wouldn't
> suffer for differences among our implementations on that point.

In Helma NG, the directory containing the main script is added to the
front of the module search path by default, and the modules directory
in the helma-ng home directory to the end. Thus, applications
consisting of a single app directory can be run without worrying about
the module search path. You only have to set the module search path if
your application consists of several script repositories, or you're
using libraries that aren't installed in helma-ng's modules directory.
I think this is working pretty well for us.

Hannes

Ash Berlin

unread,
May 28, 2009, 4:33:34 AM5/28/09
to serv...@googlegroups.com

On 28 May 2009, at 09:24, Hannes Wallnoefer wrote:

>
> 2009/5/27 Kris Kowal <cowber...@gmail.com>:
>>
>> Proposal:
>>
>> module -- required object
>> module.id -- required, top-level id of current module, String
>> module.path -- optional, absolute path of current module file,
>> String
>> module.resource(id).open(…) -- optional, conforms to file.open
>> mode and options.
>> require.main -- top-level id of the program module, String
>
> +1 on everything, except I'd prefer require.main to contain the
> program path instead of the program's module id, see rationale below.

I was thinking about this too.

>
> 2009/5/27 Kris Kowal <cowber...@gmail.com>:
>>
>> On Wed, May 27, 2009 at 10:59 AM, Ash Berlin <ash_g...@firemirror.com
>> > wrote:
>>> On 27 May 2009, at 18:30, Kris Kowal wrote:
>>>> On Wed, May 27, 2009 at 1:09 AM, Hannes Wallnoefer
>>>
>>> Does "module.path" include the file itself, or is it just the dir
>>> (or
>>> other suitable URI) where the file can be found?
>>
>> The full file name, dirname, basepath, and extension all together.
>> "path" is two kinds of ambiguous. Can we go with "file" or
>> "fileName"
>> instead?
>
> I actually like "path" and think "file" and "fileName" have their own
> problem of hinting to the pure file name.

>>> And I'm not quite sure how to build up a ID
>>> for a module that isn't in the search path. Say i have:
>>> my_serverjs_interpreter /opt/my_app/my_app.js
>
> This is why I think it would be better if require.main contained the
> program path. I think it is safer to assume that the main program has
> a path than a module id, and checking if a module is the main program
> would still be simple:
>
> if (require.main == module.path) { ... }
>
> Admitted, this introduces the risk of getting confused by
> non-canonical paths, but I think it would be easy to guarantee that if
> require.main and module.path use non-canonical paths, at least they
> use the same one.

Can i suggest module.uri instead of .path. That way it still makes
sense when the module is not stored on the filesystem. (Not that any
of us are doing this now, but I can easily see it happening.)


Kris Kowal

unread,
May 28, 2009, 12:45:06 PM5/28/09
to serv...@googlegroups.com
On Thu, May 28, 2009 at 1:24 AM, Hannes Wallnoefer <han...@gmail.com> wrote:
> I actually like "path" and think "file" and "fileName" have their own
> problem of hinting to the pure file name.

Alright, module.path.

>>> And I'm not quite sure how to build up a ID
>>> for a module that isn't in the search path. Say i have:
>>> my_serverjs_interpreter /opt/my_app/my_app.js
> This is why I think it would be better if require.main contained the
> program path. I think it is safer to assume that the main program has
> a path than a module id, and checking if a module is the main program
> would still be simple:

I think that it's okay to use the absolute path as the id for scripts,
as long as absolute paths are not used as ids within modules. When
possible, it would be better to reconstruct the id of a module within
the search path, but I haven't done so yet, and I understand that it's
onerous. It would be troublesome for the require.main to be the path
though for modules that do not have paths. Also, while Ash Berlin's
idea of using a URI has merit, I would like to leave open the
possibility of a later standard that extends the module identifier
domain to include URL's. Doing so right now would leave too many open
variables and more discussion than is necessary in the short term.

> if (require.main == module.path) { ... }
>
> Admitted, this introduces the risk of getting confused by
> non-canonical paths, but I think it would be easy to guarantee that if
> require.main and module.path use non-canonical paths, at least they
> use the same one.

I agree to the extent that for practical purposes (beyond the
specification) that absolute paths are an acceptable strict-superset
of module identifiers, and that URI's would be an acceptable
strict-superset thereof when need arises and the time comes for
carving out that namespace in excruciating detail.

> In Helma NG, the directory containing the main script is added to the
> front of the module search path by default, and the modules directory
> in the helma-ng home directory to the end. Thus, applications
> consisting of a single app directory can be run without worrying about
> the module search path. You only have to set the module search path if
> your application consists of several script repositories, or you're
> using libraries that aren't installed in helma-ng's modules directory.
> I think this is working pretty well for us.

Python does something similar, and I think that it causes problems in
the very long term. I'd recommend that scripts use relative id's to
refer to neighboring resources instead of top-level id's. Otherwise
you run into the problem where your application's name space may mask
a module on the top-level identifier space. For example, if your
application has an "file" module, the standard library's "file" module
would become inaccessible.

Kris Kowal

Wes Garland

unread,
May 28, 2009, 12:56:33 PM5/28/09
to serv...@googlegroups.com
> I'd recommend that scripts use relative id's to
> refer to neighboring resources instead of top-level id's.  Otherwise
> you run into the problem where your application's name space may mask
> a module on the top-level identifier space.

I agree, but for different reasons.

In enterprise computing, it is important to be able to containerize images, from a snapshot/testing point of view.

It is often actually cheaper to run 10 versions of a library, all in different containers (e.g. directory trees) than it is to maintain a central library repository.  It is much better than the be able to push a patch at a specific subsystem than it is to test 1,000 other pieces which might have unexpected regressions.

Daniel Friesen

unread,
May 29, 2009, 12:33:22 AM5/29/09
to serv...@googlegroups.com
Interesting note.

It's probably not require() applicable, but what do you think of modules
that have actual metadata specifying version and a system that lets you
have multiple versions installed and picks the best version based on
what version you tell it to limit to?

So, libraries you've made local patches to? Interesting point. I might
throw some thought into that when I think about things.

Huh... wait... agh, I already forgot. I already made my system able to
handle that kind of use case. That's what the local rc script was for,
so you could build registries of libraries local to an app to have them
override the normal global stuff.
And here I was thinking of setting up some sort of way of tagging
modules you've modified with something marking them as a
branch/fork/whatever of the component.
Then again, that might not be half bad.
banana('language.javascript.narcissus[branch=js1.8support]');

I think it would be nice to start a wiki page for listing all these
kinds of special cases to think about. Like needing to maintain a local
patch to a library because of an issue with it you just can't wait for
the project to fix, but not wanting that to potentially break other
things using the library on your system.

Hannes Wallnoefer

unread,
Jun 4, 2009, 9:10:54 AM6/4/09
to serv...@googlegroups.com
2009/5/28 Kris Kowal <cowber...@gmail.com>:
>
> On Thu, May 28, 2009 at 1:24 AM, Hannes Wallnoefer <han...@gmail.com> wrote:
>> I actually like "path" and think "file" and "fileName" have their own
>> problem of hinting to the pure file name.
>
> Alright, module.path.

So it seems we agree on this. The only potential problem I see is if
the "module" object itself is going to be optional. If this is so,
checking for any module.* property is going to be quite painful. So
while having the id, path, loader properties optional might be fine,
having the module object itself optional would be a deal breaker IMO.

https://wiki.mozilla.org/ServerJS/Modules/Loaders

(The same is true for the "system" free variable vs module question, btw.)

Hannes

Kris Kowal

unread,
Jun 4, 2009, 12:44:04 PM6/4/09
to serv...@googlegroups.com
On Thu, Jun 4, 2009 at 6:10 AM, Hannes Wallnoefer <han...@gmail.com> wrote:
> So it seems we agree on this. The only potential problem I see is if
> the "module" object itself is going to be optional. If this is so,
> checking for any module.* property is going to be quite painful. So
> while having the id, path, loader properties optional might be fine,
> having the module object itself optional would be a deal breaker IMO.
>
> https://wiki.mozilla.org/ServerJS/Modules/Loaders
>
> (The same is true for the "system" free variable vs module question, btw.)
>
> Hannes

Yes. I'm in favor of "module" being a required free variable, and
"require.id" being a required property. "require.path" would have to
be optional, and "require.resource(id)" would be more universalizable
but I think it can be optional or specified later.

I don't recall what the final sentiment for the "system" variable was,
but I believe that we last settled that it would not be specified
since it was only necessary for the secure platforms, so it is safely
relegated to being "platform specific". That being said, if it's
available generally (as it is in Narwhal at the moment, as it would be
inconvenient to have separate evaluators for secure and non-secure
platforms) it's likely to be used. I'm indifferent at this point
since there are advantages both ways; for one, it would be nice to
give people contriving security platforms a lot of liberty with the
namespace they use to communicate capabilities into a sandbox.

Kris Kowal

Kris Kowal

unread,
Jun 8, 2009, 2:21:05 AM6/8/09
to serverjs
I've posted an amendment proposal.

https://wiki.mozilla.org/ServerJS/Modules/Meta

The only voiced preference that I did not follow was that one where
module.main would be most logically a fully-qualified path rather than
a compliant module identifier.

I think there are two ways to go with that issue. We could require
module.main to be a fully-qualified path, in which case module.main
would not be possible in secure sandboxes and non-filesystem module
storage systems (URL's, databases, archives).

The latter route would be to allow the main module id to be a fully-
qualified file-system path, a URL, or some other presently non-
specified module identifier notation. I recommend that we make this a
de-facto behavior. The program module and any module transitively
required by way of relative ids could have fully-qualified paths for
their "module.id", and therefore the main module's "require.main" id.
Some effort should probably be made to discover the canonical path of
the module and attempt to fit it inside the top-level module paths,
but that is not necessary to specify at this stage, in my opinion.

The danger to specifying that fully qualified paths be permissible
module identifiers is that we might paint ourselves into a corner. We
may want to extend the specification with a particular super-set of
the present module identifier name space, like absolute paths and
perhaps eventually full URL's, and fully-qualified file-system paths
might have to be refactored to accommodate that later, especially on
Windows. This is an area that I feel requires more experimentation.

But, I think we can either ratify or refine this amendment we've
already started converging on and incorporate it into the
SecurableModules spec.

Kris Kowal
Reply all
Reply to author
Forward
0 new messages