Module meta data Proposal.

[Ash Berlin]

So i was discussing on IRC the http://wiki.commonjs.org/wiki/Modules/Meta
proposal (require.main and module free variable) (see the Talk
version of that page for the chat log) - tidying it up and making the
intentions more explicit.

And I was all happy with it and the "require.main == module.id"
proposal but then I had a thought: you will probably also want to get
hold of the path of the main module which is only available in
"module.main" when you are in the main module, but otherwise not
available.

So how about this one change:

Instead of "require.main" being the id of the main module, make it the
main module's "module" object. This lets you do the following.

if (require.main === module) { /* I am the main */ }
require(require.main.id);

as you could before, but you can now also do

loadResourceFrom(new Path(require.main.path).parent());
// That is if the FS spec had a 'parent' method, which seems to be
lacking.

Show of hands for making this change?

* Ash Berlin (obviously)

[Daniel Friesen]

+1, implementations may want to recursively seal the module object though

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

[Kris Kowal]

On Fri, Sep 18, 2009 at 4:49 PM, Ash Berlin
<ash_flu...@firemirror.com> wrote:
> Instead of "require.main" being the id of the main module, make it the
> main module's "module" object. This lets you do the following.
>
> if (require.main === module) { /* I am the main */ }
> require(require.main.id);

This is strictly more useful. It'll be a mild pain to migrate, but so be it.

+1

Kris Kowal

[Ash Berlin]

Wiki updated. Does anyone have any problem with the proposal as it now
stands on the wiki. Staying silent counts as no problem.

Looking back over the thread linked there, the other issue was
"module.path" vs ".fileName" vs ".uri". It is my understanding that at
some future point we will add a "module.resource()" function, so
storing the path/directory of the module shouldn't be needed, so I
therefore think we should have "module.uri". If we can reach an
agreement on this last point, this proposal can offically be ratified.

Which would be nice.

-ash

[Daniel Friesen]

I kinda like .uri
I was debating adding uri support to some of my inclusion methods so
data: javascript: file: jar: etc... can be used on them.
After all, I already support `monkeyscript javascript:"print(1+1);"`
(Instead of an -e or whatever to execute code I just adopted javascript
urls; This ended up expanding into my rc script and hook setup to form
the ability to do some very nice things from the cli without an
extensive list of arguments for slightly different uses)

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

Ash Berlin wrote:

[Neville Burnell]

On Sep 19, 8:52 pm, Daniel Friesen <nadir.seen.f...@gmail.com> wrote:
> I kinda like .uri

+1
me too!

[Hannes Wallnoefer]

2009/9/19 Ash Berlin <ash_flu...@firemirror.com>:

>
>
> On 19 Sep 2009, at 01:01, Kris Kowal wrote:
>
>>
>> On Fri, Sep 18, 2009 at 4:49 PM, Ash Berlin
>> <ash_flu...@firemirror.com> wrote:
>>> Instead of "require.main" being the id of the main module, make it
>>> the
>>> main module's "module" object. This lets you do the following.
>>>
>>> if (require.main === module) { /* I am the main */ }
>>> require(require.main.id);
>>
>> This is strictly more useful.  It'll be a mild pain to migrate, but
>> so be it.
>>
>> +1
>>
>
> Wiki updated. Does anyone have any problem with the proposal as it now
> stands on the wiki. Staying silent counts as no problem.

I know I'm a bit late, but I have growing doubts about changing
require.main to the main module's "module" object instead of just the
module id. I think it adds a lot of complexity that isn't apparent at
first sight, and I don't think it's worth it.

First, there's implementational complexity. The requirement of
referential identity means the setup of require.main and instantiation
of the main module have to be tightly coupled.

Second, it implies that there will only be one instance of the main
module, which may be the case for simple, single-threaded scripts. But
if a main module just starts a server or otherwise launches some
threads/workers and then exits, other threads requiring the main
module will create new instances (at least in Helma NG, modules are
singletons only within one thread/context, and I really think that's
the way it should be).

Finally, I think there's a conceptual problem. The way I see it, the
module meta-object is meant to be private to the module it belongs to.
Exposing it as require.main would call for the object to be sealed
IMO. But that clashes with the requirement for referential identity
with the actual module meta-object, which is needed for the
require.main == module check to work.

So all points considered I would prefer for require.main to remain the
string id of the main module. Has anybody made major investments in
this feature already? As far as I can tell, neither Narwhal nor
Flusspferd are implementing it yet.

Hannes

[Kris Kowal]

On Sat, Oct 17, 2009 at 3:53 PM, Hannes Wallnoefer <han...@gmail.com> wrote:
> I know I'm a bit late, but I have growing doubts about changing
> require.main to the main module's "module" object instead of just the
> module id. I think it adds a lot of complexity that isn't apparent at
> first sight, and I don't think it's worth it.

I have the feature in my "refactor" branch with a shim that Ash Berlin
contrived to make it backward compatible with the module.id idiom
(providing a .toString() on the module object). The shim works if
you've used the idiom:

if (require.main == module.id) {
}

but not if you used the idiom:

if (require.main === module.id) {
}

And provides support for migration to:

if (require.main == module) {
}

The implementation was a little weird, but not insurmountable.

So, the pros are:

* You can use require.main.path and other metadata of the main module
from within any module.

The cons (and counter arguments where applicable) are:

* You can use other metadata on the main module, that might
conceivably be private to that module.
** Modules within a sandbox are not presently security membranes
*** But they should be treated as such anyway as often as possible for safety.
* The idiom of a "main" only works if there's only one "main". This
could be a problem if there's a different "main" in each thread.
** We can specify that there may be no "main" if it would be ambiguous.
** This isn't really a problem; the idiom of a "main" module works as
long as there's one "main" for each "sandbox" (system of modules, a
module memo and some other state like which module was the main
module) and one "sandbox" per "thread", "worker", "vat", "process" or
whatever you call your shared-nothing async-message-passing system.
* Coupling: the process of requiring the "main" module must be
special and the usual module loader machinations must accommodate it,
which introduces coupling.
** It's not too bad; the coupling remains in the sandboxing system's
logic at least for Narwhal, so no new interfaces are needed. [1] [2]

I'm indifferent on the issue. It would not be hard to avoid merging
these changes.

Kris Kowal

[1] http://github.com/kriskowal/narwhal/blob/refactor/lib/sandbox.js#L61-66
[2] http://github.com/kriskowal/narwhal/blob/refactor/lib/sandbox.js#L166-170

[Ash Berlin]

On 17 Oct 2009, at 23:53, Hannes Wallnoefer wrote:

2009/9/19 Ash Berlin <ash_flu...@firemirror.com>:

Wiki updated. Does anyone have any problem with the proposal as it now

stands on the wiki. Staying silent counts as no problem.

I know I'm a bit late,

A little bit late yes :)

First, there's implementational complexity. The requirement of
referential identity means the setup of require.main and instantiation
of the main module have to be tightly coupled.

Is this much different to the module exports object being a singleton object?

Second, it implies that there will only be one instance of the main
module, which may be the case for simple, single-threaded scripts. But
if a main module just starts a server or otherwise launches some
threads/workers and then exits, other threads requiring the main
module will create new instances (at least in Helma NG, modules are
singletons only within one thread/context, and I really think that's
the way it should be).

The way you handle module exports is fine if you only support shared nothing threads (I guess is what most platforms will do) in which case the thread is basically another sandbox, so can have another 1) main module, or as Kris pointed out, not main object at all.

If we change the spec to say that this object must be referentially equal where ever the module exports would also be identical, i.e. within the same sandbox/shared-nothing thread, or within a platform with shared everything threads. And also have the spec say that there might not be a main module, in which case require.main should be undefined (that is, the property exists, but is set to undefined?)

Finally, I think there's a conceptual problem. The way I see it, the
module meta-object is meant to be private to the module it belongs to.
Exposing it as require.main would call for the object to be sealed
IMO.

Well as it stands there is nothing really private about the info it has, right now it has an .id and a .uri. Neither of these are private. In future I can see us adding .resource('x') to this (to get at resources/files relative to the module.) In flusspferd these properties are dont-delete and read-only.

But that clashes with the requirement for referential identity
with the actual module meta-object, which is needed for the
require.main == module check to work.

Sealing an object shouldn't break any kind of equality tests...? In fact the sealed object is still the same Object, so I dont see how this affects equality tests.

So all points considered I would prefer for require.main to remain the
string id of the main module. Has anybody made major investments in
this feature already? As far as I can tell, neither Narwhal nor
Flusspferd are implementing it yet.

Flusspferd 0.8 is shipping with this behaviour on Tuesday - you've left if to late after no one complained to complain :P

Hannes

The reason I wanted this change is it is, as Kris pointed out 'This is strictly more useful.' Having require.main as just an ID limits you to testing 'am I the main module' or getting its exports. Having it as the module object lets you get the hold of the path to the module via .uri and in the future call module.resource('../x') once we've worked out a proposal for that (or do it manually by munging .uri which I currently do)

-ash

[Hannes Wallnoefer]

2009/10/18 Kris Kowal <cowber...@gmail.com>:

>
> On Sat, Oct 17, 2009 at 3:53 PM, Hannes Wallnoefer <han...@gmail.com> wrote:
>> I know I'm a bit late, but I have growing doubts about changing
>> require.main to the main module's "module" object instead of just the
>> module id. I think it adds a lot of complexity that isn't apparent at
>> first sight, and I don't think it's worth it.
>
> I have the feature in my "refactor" branch with a shim that Ash Berlin
> contrived to make it backward compatible with the module.id idiom
> (providing a .toString() on the module object).  The shim works if
> you've used the idiom:
>
> if (require.main == module.id) {
> }
>
> but not if you used the idiom:
>
> if (require.main === module.id) {
> }
>
> And provides support for migration to:
>
> if (require.main == module) {
> }

I wouldn't want to do that. Backwards compatibility and migration
support are not bit concerns for Helma NG at this time. And once you
add a feature, you have to keep it or you'll just delay the breakage.
In the long term, I think this is more confusing than helpful.

Hannes

[Hannes Wallnoefer]

2009/10/18 Ash Berlin <ash_flu...@firemirror.com>:

>
> On 17 Oct 2009, at 23:53, Hannes Wallnoefer wrote:
>
> 2009/9/19 Ash Berlin <ash_flu...@firemirror.com>:
>
> Wiki updated. Does anyone have any problem with the proposal as it now
>
> stands on the wiki. Staying silent counts as no problem.
>
> I know I'm a bit late,
>
> A little bit late yes :)
>
> First, there's implementational complexity. The requirement of
> referential identity means the setup of require.main and instantiation
> of the main module have to be tightly coupled.
>
> Is this much different to the module exports object being a singleton
> object?

For Helma NG yes, because module loading (including all concurrency
constraints) is implemented in core, while require is just a plain JS
function that sits on top. Also, require is created before the main
module is instantiated. In short, the module meta-object is owned by
the module scope[1] which doesn't exist at the time require() is
created[2], so I guess implementing this will require some amount of
head scratching and, in the end, uglification.

[1] http://github.com/hns/helma-ng/blob/master/src/org/helma/engine/ModuleScope.java
[2] http://github.com/hns/helma-ng/blob/master/modules/helmaglobal.js#L17-28

> Second, it implies that there will only be one instance of the main
> module, which may be the case for simple, single-threaded scripts. But
> if a main module just starts a server or otherwise launches some
> threads/workers and then exits, other threads requiring the main
> module will create new instances (at least in Helma NG, modules are
> singletons only within one thread/context, and I really think that's
> the way it should be).
>
> The way you handle module exports is fine if you only support shared nothing
> threads (I guess is what most platforms will do) in which case the thread is
> basically another sandbox, so can have another 1) main module, or as Kris
> pointed out, not main object at all.

Actually, Helma NG implements shared-nothing by default, but allows
individual modules to opt-in to be shared and multithreaded.
Accidentally, this is done by setting a property called "shared" to
true on the module meta-object. The module spec doesn't require the
module meta-object to be read-only or sealed, so I think this it's
legitimate to use it for implementation specific extensions like this.
However, exactly for this reason, it the module object needs to be
private to its owning module.

> If we change the spec to say that this object must be referentially equal
> where ever the module exports would also be identical, i.e. within the same
> sandbox/shared-nothing thread, or within a platform with shared everything
> threads. And also have the spec say that there might not be a main module,
> in which case require.main should be undefined (that is, the property
> exists, but is set to undefined?)
>
> Finally, I think there's a conceptual problem. The way I see it, the
> module meta-object is meant to be private to the module it belongs to.
> Exposing it as require.main would call for the object to be sealed
> IMO.
>
> Well as it stands there is nothing really private about the info it has,
> right now it has an .id and a .uri. Neither of these are private. In future
> I can see us adding .resource('x') to this (to get at resources/files
> relative to the module.) In flusspferd these properties are dont-delete and
> read-only.

In Helma NG they are currently writable. However, that's not an issue
- it wouldn't be a problem to define them as read-only and
dont-delete.

> But that clashes with the requirement for referential identity
> with the actual module meta-object, which is needed for the
> require.main == module check to work.
>
> Sealing an object shouldn't break any kind of equality tests...? In fact the
> sealed object is still the same Object, so I dont see how this affects
> equality tests.

It does if you want it to be unsealed for its owning module, as
explained above. You can't have an unsealed version and a sealed
version of an object and have them compare as equal.

> So all points considered I would prefer for require.main to remain the
> string id of the main module. Has anybody made major investments in
> this feature already? As far as I can tell, neither Narwhal nor
> Flusspferd are implementing it yet.
>
> Flusspferd 0.8 is shipping with this behaviour on Tuesday - you've left if
> to late after no one complained to complain :P

Well, Helma NG 0.4 will be out in a few days as well with require.main
as string id, unless I can sort out these problems, which seems
unlikely. We'll see how it all turns out.

Hannes

[Hannes Wallnoefer]

After some deliberation, I think I can come up with solutions to the
problems I'm having with this.

Ash provided some insight into how Flusspferd implements this feature
on IRC today that may be helpful, and I think I can solve the
module.shared issue using a setter that protects against changing the
property once it's been set. So while I'm still not enthusaistic about
it, but I can see how it's useful and I don't want to spoil it for
everybody (provided it is still broadly supported, which seems to be
the case).

Hannes


2009/10/19 Hannes Wallnoefer <han...@gmail.com>: