Module System feedback

[Kris Kowal]

In other news, Tom Robinson has recommended that the "environment"
I've spoken about in the past, which would inject system dependencies
in a module sandbox, be called "sys", so that "sys.print", "sys.argv",
"sys.stdin", "sys.stdout", "sys.stderr", "sys.environ", "sys.fs", and
such be brief and concise, without conflict with the conventional HTTP
env in Jack or the system environment. I think this is an equitable
name choice, assuming that people don't mind them not being global
variables, which I argue would harm interoperability with secure
environments.

If it is generally acceptable for module loaders to provide a "sys"
variable, I would like to revise the Interoperability unit tests to
use "sys.print" instead of "environment.print" for test result
reporting. Please reply your objections if this isn't a good idea.

I think "sys" should be used instead of global variables, in general,
for adding capabilities to our platforms. For one, it's replaceable
in security sandboxes. Secondly, it would make the syntax clean for
testing for the availability of platform and sandbox features. Thus,
I think we should carefully manage what variables mean if they're
members of "sys" for all of our platforms.

Also, on IRC Ondras opined that "absolute" and "relative" were a poor
choice of words in the module system specification. He recommended
"global", "local". Wes recommended that "absolute" be changed to
"top-level". I've revised the spec to say "top-level", since I think
that satisfies Ondras's objection with "absolute": that it's an
overload of the same term applied to paths but lacking the same
semantics with regard to initial "/"; and this change avoids the
alternate overload on "global".

Kris Kowal

[ihab...@gmail.com]

+1 to all.

Ihab

[Kevin Dangoor]

+1 to all. Great suggestions.

--
Kevin Dangoor

work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com

[George Moschovitis]

+1

[solo kihara]

+1

--
Solloh Kihara
Software Developer / Programmer
Tel: +254 733 567 797; +254 726 567 797
EMail: solo...@gmail.com
Blog: http://soloincc.blogspot.com/
==============================
Najivunia Kuwa Mkenya

[Hannes Wallnoefer]

2009/4/2 Kris Kowal <cowber...@gmail.com>:

>
> If it is generally acceptable for module loaders to provide a "sys"
> variable, I would like to revise the Interoperability unit tests to
> use "sys.print" instead of "environment.print" for test result
> reporting.  Please reply your objections if this isn't a good idea.

Why not make sys (or system) itself a module?

I think the simpler the module spec is (and the less special
cases/named objects it defines) the better. By making sys/system a
module it will be part of the common/standard ServerJS API instead of
the module specification.

Hannes

[Wes Garland]

Hannes:

Why not make sys (or system) itself a module?

 FWIW, if "sys" is ratified, my implementation will probably be to execute "const sys=require('sys');" in program-module scope.

What do you think of that as a compromise?

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Hannes Wallnoefer]

2009/4/2 Wes Garland <w...@page.ca>:

> Hannes:
>
>> Why not make sys (or system) itself a module?
>
>  FWIW, if "sys" is ratified, my implementation will probably be to execute
> "const sys=require('sys');" in program-module scope.
>
> What do you think of that as a compromise?

You mean you would use require() internally to provide the sys constant?

I think that misses my point. My point being: if we already implement
this wonderful module system we should use it to provide
common/standard functionality _to_ modules.

Hannes

[Kris Zyp]

Hannes Wallnoefer wrote:
> 2009/4/2 Kris Kowal <cowber...@gmail.com>:
>
>> If it is generally acceptable for module loaders to provide a "sys"
>> variable, I would like to revise the Interoperability unit tests to
>> use "sys.print" instead of "environment.print" for test result
>> reporting. Please reply your objections if this isn't a good idea.
>>
>
> Why not make sys (or system) itself a module?
>

And would defining "sys" as a top level variable mean that we will be
defining other (presumably important) modules as required to be
available as top-level globals? I thought the original intent was that
ServerJS was only going to define "require" and everything else would be
brought into an environment by require calls. Or is there a security or
logistical reason that "sys" would be also be included? I don't mind
having ServerJS define top-level globals, I think that can actually be
convenient, but it seems like a deviation from the original path.
Kris

[Kris Kowal]

Thanks, Hannes, Wes, Kris. You do bring out good points, and I'd like
to frame this dilemma in a fashion that should be equitable. We can
take a hint from Tom's stack in Narwhal and Jack and think about
security as a "platform", and securability as a requirement for a
module to work in the "secure" platform. There's a bit of a
chicken+egg paradox regarding authority here, and I think illuminating
where authority comes from may help us resolve this issue.

Jack/Narwhal is particularly well suited for illustrating this because
it's architected to work on multiple platforms. For example, the
"file" module draws its authority to manipulate files in different
ways for each platform. In all cases, the program has all the same
capabilities as the owner of the process, but through a different
channel. In Rhino and Helma NG, authority flows through the
"Packages" global variable. In Wes's GPSEE, authority flows through
dynamically linked modules and the Moz File object.

In a secure platform, all authority must _ultimately_ flow through a
single, controllable channel. This is where the "sys" variable would
come in, and for security purposes, Ihab and I believe that this could
be a "platform" specific variable. The "file" module would get its
authority to manipulate files from the "sys.fd" object. The "system"
module would get its authority to read and write from standard input
and output streams from "sys.stdin", "sys.stdout", and "sys.stderr".
A "jquery" module would get its limited authority to manipulate a
region of a page through a "sys.dom" object. Since all authority
flows through "sys", the person constructing a sandbox can create a
sandbox with any "sys" they craft.

It's our intent, and I have a functioning prototype as of last night,
to construct secure module systems _within_ the permissive platforms
you all have constructed. So, we would use the standard library's
"file" module to construct a module loader within your module loader.
The "file" module within the "secure" platform would get its authority
from "sys", while your platform's "file" module would get its
authority from a "Packages", the Moz File, or your linked module.

The upshot is that "sys", or whatever we call it, does not need to be
supported by permissive platforms; it can be constructed for the
"secure" platform within each of your platforms, and used exclusively
by modules targeting security.

However
=======

If, for the elegance of the system, it is desirable to have a set of
objects like "stdin", "stdout", "stderr", "environ", and "args" in an
object that does not need to be loaded with "require", it is
imperative for the securability of interoperable JavaScript modules
(which includes those modules in the standard library that can be
"platform" agnostic) that these not be global variables because it is
not tractable for a secure module loader to attenuate global
variables. It would be _convenient_ if that were the same variable
name as what we would be using to attenuate authority in the "secure"
platform, that being the "sys" object, or whatever we call it. It
would furthermore be acceptable for that variable to be a global, even
though in a secure system it would be an argument to the module
factory, since the module would be none the wiser about the
implementation.

----

So, behind door A:
We add a "sys" or likewise named variable that must be available in
all "platforms" which can contain "stdin", "stdout", "stderr". Secure
platform modules would likely use this variable for dependency
injection, while permissive platforms would provide it as a global
variable.

Door B:
We require("system") or some likewise named module which provides
"stdin", "stdout", "stderr", and so on. The "sys" variable would be
provided exclusively by the "secure" platform and only be used by
"secure" platform modules.

Thanks,
Kris Kowal

[Kris Kowal]

On Thu, Apr 2, 2009 at 4:03 PM, Kris Kowal <cowber...@gmail.com> wrote:
> In a secure platform, all authority must _ultimately_ flow through a
> single, controllable channel.  This is where the "sys" variable would
> come in, and for security purposes, Ihab and I believe that this could
> be a "platform" specific variable.  The "file" module would get its
> authority to manipulate files from the "sys.fd" object.  The "system"
> module would get its authority to read and write from standard input
> and output streams from "sys.stdin", "sys.stdout", and "sys.stderr".
> A "jquery" module would get its limited authority to manipulate a
> region of a page through a "sys.dom" object.  Since all authority
> flows through "sys", the person constructing a sandbox can create a
> sandbox with any "sys" they craft.

I've attached an illustration of the flow of
authority-to-use-the-file-system between a permissive Rhino platform
and a secure platform, as I'm proposing. In this case, there are two
platforms: Rhino and secure; and three architecture layers: the
platform interface, the standard library, and the application. The
application uses the standard library's sandbox module to construct a
secure sub-platform with managed authority to access modules from the
file system, and a subtree of the file system directly. This brings
together ideas about security, packaging, and the file system API.

In the diagram, all authority to use the file system flows from the
Java Packages object: because you have Java packages, you are
implicitly granted the authority to do anything your process owner can
do with the file system. The platform then provides modules that
adapt those objects to a standard file system interface module that
can be used in any platform. The application then uses the standard
"sandbox" module to construct a sandbox where the only channel of
authority is the "sys" object (so there is not Java Package object,
and only pure JavaScript modules can be imported). The flow of
authority within the sandbox is similar to that in the Rhino platform:
the authority is channeled through the variable, "sys" instead of
"Packages", through a platform interface layer that produces the
standard file system API. That module then provides access to the
file system for the restricted application.

Kris Kowal

[Hannes Wallnoefer]

Thanks for explaining your motivation in such great detail, Kris. I do
now better understand what you're trying to achieve, and it does make
sense. I'm still not convinced this couldn't be done by the require
function passed to the secured environment, either by providing the
sys module on request, or by making sure itself the file module it
returns is properly chroot'ed.

However, having the sys object just there is certainly not a terrible
thing. So in terms of vote that would be +/-0 I guess.

I'm still digesting all of this, though. I implemented my own
sandboxing today in Helma NG. It provides a separate engine with its
own global scope and module search path, and you can optionally inject
global properties, define a class shutter to control java access, and
seal the global object.

http://dev.helma.org/trac/helma/browser/helma-ng/trunk/modules/helma/system.js?rev=9571#L29

By injecting the require function from the host environment you get a
minimal securable modules conformant sandbox environment:

include("helma/system");
var sandbox = createSandbox("sandbox/code", {require: require});
sandbox.runScript("test.js");

This code will set up a sandbox with the directory sandbox/code/ as
only module path, try to evaluate a file called global.js and then
test.js.

Or the same without access to java objects and global object sealing
after global.js has been evaluated:

var classShutter = function(classname) { return false; };
var sandbox = createSandbox('sandbox/code', {require: require},
classShutter, true);

Of course right now you can escape the sandbox module root using "..",
so it's not really that secure yet :-) But interesting stuff for sure.

Hannes

[Kris Kowal]

On Fri, Apr 3, 2009 at 8:59 AM, Hannes Wallnoefer <han...@gmail.com> wrote:
> I'm still digesting all of this, though. I implemented my own
> sandboxing today in Helma NG. It provides a separate engine with its
> own global scope and module search path, and you can optionally inject
> global properties, define a class shutter to control java access, and
> seal the global object.
>
> http://dev.helma.org/trac/helma/browser/helma-ng/trunk/modules/helma/system.js?rev=9571#L29
>
> By injecting the require function from the host environment you get a
> minimal securable modules conformant sandbox environment:
>
> include("helma/system");
> var sandbox = createSandbox("sandbox/code", {require: require});
> sandbox.runScript("test.js");
>

The implementation I've thrown together sets up a context and a
sealed/frozen global scope and removes the Packages object. This
makes that context and global tree safely sharable among sandboxes.
Secure module loaders then use that context to evaluate modules, so
you only need to create a single module loader for any number of
sandboxes; the module factory functions are "inert". When you create
a sandbox, you select a main module, and options. The options include
an alternate "sys" for dependency injection (uses your own "sys" by
default), a pre-initialized module memo (so, you could pre-populate a
"system" module instead of using "sys"), an alternate loader (uses
"require.loader" by default, since that's likely to be the only loader
available within a sandbox if it wishes to isolate one of its
components), and a couple other convenience options. The mechanics of
"require" are pretty consistent, so it's more useful to parameterize
the loader and sandbox than create custom require methods.

Here's the use case program:
http://github.com/kriskowal/chiron/blob/636e05de067ca43ea61b83f4d810b21e640c3552/src/test/sandbox/program

The SecureLoaderMixin validates module identifiers and throws errors
if the program tries to break out of the box. By mixing that with a
FileLoader, it is a SecureFileLoader.

Here's the sandbox test it runs:
http://github.com/kriskowal/chiron/blob/636e05de067ca43ea61b83f4d810b21e640c3552/src/test/sandbox/secured.js

Here's the evaluator:
http://github.com/kriskowal/chiron/blob/636e05de067ca43ea61b83f4d810b21e640c3552/src/sandbox.js#L371

Here's the SecureLoaderMixin:
http://github.com/kriskowal/chiron/blob/636e05de067ca43ea61b83f4d810b21e640c3552/src/sandbox.js#L217

Here's the FileLoader:
http://github.com/kriskowal/chiron/blob/636e05de067ca43ea61b83f4d810b21e640c3552/src/sandbox.js#L72

Which is based on AbstractLoader, that defines a bunch of stuff that
most loaders are likely to need:
http://github.com/kriskowal/chiron/blob/636e05de067ca43ea61b83f4d810b21e640c3552/src/sandbox.js#L4

While it's desirable to craft all kinds of loaders, the Sandbox type
is suitable for all kinds. The sandbox implementation in modules.js
and narwhal.js are almost identical. Here's the one for use inside a
sandbox:
http://github.com/kriskowal/chiron/blob/636e05de067ca43ea61b83f4d810b21e640c3552/src/sandbox.js#L276

Kris Kowal

[Peter Michaux]

On Thu, Apr 2, 2009 at 4:57 PM, Hannes Wallnoefer <han...@gmail.com> wrote:
>
> 2009/4/2 Kris Kowal <cowber...@gmail.com>:
>>
>> If it is generally acceptable for module loaders to provide a "sys"
>> variable, I would like to revise the Interoperability unit tests to
>> use "sys.print" instead of "environment.print" for test result
>> reporting.  Please reply your objections if this isn't a good idea.
>
> Why not make sys (or system) itself a module?
>
> I think the simpler the module spec is (and the less special
> cases/named objects it defines) the better. By making sys/system a
> module it will be part of the common/standard ServerJS API instead of
> the module specification.

I agree completely. Sys doesn't need to be part of the module loader
spec. It would be better to have the functionality of sys in its own
module.

Peter