Hi all,

In a different mailing list there was some question brought up regarding how
does module collections are handled and I think it makes sense to have some
standard regarding on how the collection of modules can be packaged, what
topid's should they get and what metadata should they include about a
packages. I have quoted original post below to get a better idea what am I
about.

I would like to stick to the 2) option as it's a way currently narwhal works
and my goal is to make narwhal packages completely compatible with bespin
plugins. (It might be a good idea to discuss module packaging and module
path resolution on the CommonJS)
example:
*
1) foobar-plugin/
**lib/
**foo/
**bar.js
*
*2) foobaz-plugin
* *lib/
**bla.js
foo/
**baz.js
***
****
**

In this example three modules will be registered with the topid's:

*foo/bar
bla
foo/baz

*Some more details on packaging or on how it's done in narwhal can be found
here.
http://narwhaljs.org/packages-howto.html

Think it's very important to have a standard for packaging so that the same
libs can be reused between different commonjs implementations.

Regards
--
Irakli Gozalishvili
Web: http://rfobic.wordpress.com/
Phone: +31 614 205275
Address: Taksteeg 3 - 4, 1012PB Amsterdam, Netherlands

Hi folks,

I have a 3rd proposal. Perhaps I should call it "hermetic packages"
(with a nod towards Kris Kowal's naming of "hermetic eval").

A package is a directory tree (which can be distributed as a tarball,
zipfile, ...) containing:

  * a hierarchy of directories and modules (as *.js files)
  * a manifest

Implementation is not important, but let's say we have:

  <pkg_root>/
      MANIFEST.json
      foo/
          bar.js
          baz.js
      boo/
          fribble.js

Modules within the package can refer to each other by short names. So
bar.js can say:

  require('./baz');
  require('../boo/fribble');

as would be expected. However, to refer to a foreign module from
another package, bar.js would say something like:

  require('#someOther/foo/frob');

where #someOther is a reference to a foreign package specification in
the MANIFEST.json. This would contain entries like this suggesting a
list of places to look for that package and asking for a specific
checksum to be verified:

  someOther: {
    locations: [
      'http://allmysoftware.com/archive/bar/bar_2.4.8.zip',
      'http://barsoft.com/distros/bar_2.4.8.zip'
    ]
    checksum: '24f7c8cbf635f4c03bad2dde7bf07531'
  }

and like this asking for anything signed by a given principal:

  yetAnother: {
    locations: [
      'http://foosoft.com/public/latest/foo.zip'
    ]
    signed-by: 'c1:b1:30:29:d7:b8:de:6c:97:77:10:d7:46:41:63:87'
  }

The advantages are: strict hygiene of module names, and ability to
specify very clearly what foreign modules are being asked for. The
"installation" of a module tramples over nobody else's namespace.

The disadvantages are: a somewhat more complicated packaging scheme,
and less flexibility to go in and hack dependencies around until
something works.

The upshot, I think, is that this scheme is oriented towards
non-administered environments, where an end-user may ask for a given
piece of code to be installed and there is no local "expert" to mess
with the module configuration. As a side effect, it reduces or
eliminates the *need* for this local "expert" since the client modules
are now the ones who do the up-front work. As such, it enables
3rd-party hosted server environments where applications can be dropped
into a running server and just adapt themselves to whatever is around
them and yet still download their dependencies on demand and share
them where possible.

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

Why can't we just use absolute URLs in requires:
require("http://allmysoftware.com/archive/bar/bar_2.4.8");
This seems far simpler and more intuitive. Plus users can add complexity
as they desire. If you wish to have some redirection, they could add that:
require(myPackages["someOther"].location);

For some of the other features you are asking for, it seems like a much
more flexible and composable solution would be to allow for custom
loaders for require (that can do checksum, multiple server checks, and
signing). Users could create as sophisticated as schemes as they desire
then.

Kris

That's what I was originally thinking, but it ends up being (a) a
potential security hole, without hooks for verifying that what you're
getting is what you really want; and (b) fragile since your package is
going out into the wild and you want to give it the best possible hope
of finding what it needs out into the future. Also, all those URLs end
up getting unmanageable, and it's nice to have a central place where
they can be edited....

That said, your point is a good one, and a degenerate case of the
scheme I propose (with one URL and no verification) is very similar to
what you describe.

But --

Yeah, perhaps that is the real end result here. It would be nice to
have some commonality -- in other words, just one or two sophisticated
schemes rather than a plethora -- since that would allow greater
sharing of packages which is, after all, what we're all aiming for.

So I guess we're talking about 2 things:

1. Standardize the way require() relies on custom loaders; and

2. Standardize a "hermetic" package scheme for security wonks / zero
admin installations.

?

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

That breaks when the user wants to make local changes. Package-relative
addressing is much friendlier in this regard.

Additionally, including version numbers in packages is problematic, as it
means you need to change source code as foreign module versions increase.

On the flip side, using "whatever's on the web" is not a good situation for
those wishing to create safely-versioned trees of apps, particularly in a
revision-controlled corporate environment.  Relative addressing is
absolutely mandatory in an environment like that, unless you want to QA and
entire server everytime you make a minor bugfix to a common library.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

I currently use "../libexec/libraryName.js" for my JS programs that depend
on non-system-wide libraries. Module loading relative to the script is easy
and predictable; you can untar a package anywhere and it will work.  This is
not unlike the loader options that Apple added to its C products, where you
can specify that certain DSOs load relative to the application binary.
That's a big reason why why you can actually usefully use .dmg files on Mac
OS/X.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

Agreed, that is definitely a component of what I propose.

Within your own package, "short names" relative to the current module
(or to the root of the package) are definitely the common use case.
They would be used to create dependencies among the modules in a
package, which are all assumed to be managed by one [virtual]
development team and versioned together.

To make a reference to another package, which contains code versioned
and maintained separately by some other team, I propose the package
manifest and the "#foreignAlias/dir/module" notation.

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

Of course, by all means use relative addressing wherever possible.
If you want to freeze a version, you can copy it. Any decent package
repository should also offer version-specific URLs as well as
current-version URLs.
Kris

Why is additional notation required here?  If your manifest looks like a
list of where to find foreign packages, they could just as easily be
represented by variables rather than another stage of lookup/parse.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

Sorry, Kris -- when you said,  "why don't we just use absolute URIs", did
you mean "why don't we only use absolute URIs" (my interpretation), or "why
don't we simply adopt absolute URIs"?

If the latter, I see no problem with adding require(fullURI) as a way of
loading modules. The initial require() discussion in fact pointed out that
we could potentially use URIs or even database indices down the road to load
modules -- but I think that the core, required functionality should be
exactly what we have now -- and if we add require(URI) that
require("./moreStuff") also be able to be URI-relative.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

My apologies, my grammar was ambiguous. My intention was adopt absolute
URIs in addition to our current path mechanisms. I meant "just" as in
simpler form of package loading, not as in the only way. And yes, I
agree, that our current syntax should be URI-relative as well.
Kris

This would be an enormous boon to rapid application development. Plus, as
Kris alluded to, once you're ready for release all your packages could be
frozen and a manifest like Ihab suggests generated.

Agreed in principle.

This scheme was suggested by a ServerJS participant in a thread that,
unfortunately, I am unable to find in the archives. It is a way to
shoehorn the package mechanism into a require() that takes only a
string argument, thus reducing the proliferation of APIs.

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

Interesting. There are some details to be ironed out, but yeah, it
would be great to have the option to say, "just grab this URL and
don't bother me any more".

The details that bug me are, what do I do if the module at the
specified URL is part of a package? Let's say the package is a jarfile
(say). Perhaps I would do:

  require('jar:http://example.com/package.jar!foo/bar/baz');

? In that case would the loader know to look at the file:

  jar:http://example.com/package.jar!MANIFEST.json

in order to create a proper package loader context for the module:

  jar:http://example.com/package.jar!foo/bar/baz.js

? Hm.

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

Maybe you were referring to this [1] thread.

Christoph

[1] -
http://groups.google.com/group/commonjs/browse_thread/thread/978c3e8f...

On Wed, Sep 9, 2009 at 12:25 PM, Christoph Dorn

I was! Thank you (for the original idea and for the ref).

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

I think there needs to be a separation between *package management* and
*module loading*.

A package manager such as tusk is designed to download archives into a
cache/local repository, extract them and prepare them for the runtime
system. These steps do not belong into the module loader which should
operate only on a purely local prepared cache/working copy.

I would like to see the source of a module stay untouched when wanting
to switch the version or implementation of a dependency. This should be
done via a manifest file.

In my simplified ideal world a module always belongs to a package and a
package manifest defines a universe of modules and packages for that
package. To load an *external* module (read a module not shipped with a
given package) the third party package can be referenced via a
*#packageName* alias which is mapped to a dependency in the package
manifest file.

Using this approach the above require statement would be:

    require('#packageName/foo/bar/baz');

And the manifest file could be something like:

    {
      "dependencies": [      
         ["packageName", {
           uri: "jar:http://example.com/package.jar",
           maxVersion: "1.3"
         }]
      ]
    }

tusk can then download the defined package and invoke the *jar*
packaging plugin to prepare the modules for simple inclusion by the loader.

One important point to note here. The *#packageName* alias is defined
only for modules shipped with the package. An external module can map
the same alias to a different package/version via it's package manifest
file.

There is one major point to consider when implementing this. When
loading an external module, a separate loader must be initialized based
on the external package manifest in order to resolve the aliases
accordingly.

Assuming this can be implemented efficiently it can provide complete
namespace isolation for packages and modules and allow for multiple
versions of the same package/module to co-exist depending on the calling
code. This may not be required for *core* packages and modules that
provide a common foundation (which can still be loaded the same way as
now) but IMHO it is critical for userland packages and modules where
namespace collisions are guaranteed.

Christoph

I don't want that separation.. An elegant solution that completely
eliminates the need for package management would be ideal, IMO. Or at
least, provide a means that those that don't want to use a package
manager could avoid doing so, and package management tools could be a
purely additive tool, and not essential.
Kris

Kris:

I agree -- if it is possible to design a simple system that does not depend
on a package manager, you are much better off in the long run. It eliminates
both bootstrapping and deployment issues.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

I do not think we need to set ourselves up with a false dilemma.

I think that, for different purposes, we will need both managed
package systems and hosted packages.

We need to break this into two discussions, not a debate about which
discussion to have.

1.) What should the layout of a package be?  What format should it be
stored in (zip, tar)?  Where should its metadata (like dependencies)
be hosted (e.g., package.json)?  What should the schema and format of
the package metadata file be? Would it be possible for such packages
to be used by both a managed package system like "tusk" and a hosted
URL package loader, or would it be necessary to package projects
differently for these two purposes?  What are the common goals for
hosted packages?

2.) For a hosted package loader, what should the module identifier
domain be?  For example, require(url, id) or
require("{url}{delimiter}{topId}").  How should the loader system
indicate that such identifiers are supported?

I recommend that we begin new threads for these topics, perhaps even
down to individual questions.

Kris Kowal

On Wed, Sep 9, 2009 at 1:21 PM, Christoph Dorn

I agree with a proviso that "install time" and "run time" might get
blurred sometimes.

If there is a run-time API for invoking a "foreign" package, and/or if
I load a package into a running system and that package has "foreign"
package dependencies, then the "package manager" tool (whatever that
is) might have to get fired up pronto puppy while the client code is
waiting.

That said, using a/the promise API, and making dynamic operations
return promises, we should be able to implement whatever we need.

Yepper. :)

Yes. With perhaps the possibility of a "SYSTEM" (or whatever) package
representing primordial stuff that you expect to get with any CommonJS
runtime?

Agreed.

Agreed.

With the caveat that the download may happen while client code is
twiddling its fingers, yes.

Absolutely!

Yes. So, when loading something that comes from a package, regardless
of how addressed, either via:

  #packageName/foo/bar

or via:

  jar:http://example.com/thepkg.jar!foo/bar.js

the calling context has to construct for the newly loaded code a
suitable loader.

Agreed.

Ihab

--
Ihab A.B. Awad, Palo Alto, CA

On 9 Sep 2009, at 21:40, Kris Kowal wrote:

Right, let me try to impart some of the lessons of perl/CPAN, since  
CPAN is one of perl's strong points.

1) Version the meta data format. VERY IMPORTANT. If we miss something  
or make a mistake (which we inevitably will)

2) Be strict about the format of the metadata file. I think everyone  
so far has mentioned that this will be a JSON file. Good, lets be  
strict about the format of this in the consumers. (Perl wasn't, and  
this has lead to all sorts of headaches, in part due to the fact that  
perl uses a 'META.yml' which is YAML, except when its not.)

3) Some flag or field should indicate if a module is a pure JS module,  
or a native module for Engine Z (by Engine here i mean  
flusspferd,gpsee, v8vgi, rhino, helma etc.) This lets us have *one*  
central package repo. Without a central package repo, packages are  
almost doomed. (We can talk about where this central repo is, what  
indexes it maintains, upload policies later. Actually the basic upload  
policy is 'first come to a module can upload it. No vetting of uploads')

4) as well as listing dependancies on other modules, we need the  
ability to list external system deps, such as 'this Module needs make  
to install', or we need this java class installed in the classpath, or  
this .dll etc.. Oh and 'We need jake' etc.

5) Be able to specify (in the meta file) the command needed to build/
test/install the module. It wont always be i) straight forward even  
for pure JS modules, ii) native modules might have special build  
requirements.

Fields that should go in the metadata:

* Module's SCM repo is at $URL
* License of the module (either a url, or one of the 'GPL2', 'MIT',  
'Artistic' etc.)
* Module homepage
* (Original) Author
* list of module deps (with versions)
* the version of hte package/module/bundle itself

I have other things to impart from perl, but those aren't about  
metadata. Main one being managers should run tests at install time to  
make sure it works with particular combination of modules installed on  
$users env.

Yes absolutely and it may in some cases completely eliminate the need
for the package install step as voiced by Kris Zyp and Wes Garland.

Right, however I see two main problems with using URL's without a
mapping via a manifest file:

  * It does not allow transparency into the dependency tree
  * It makes it more cumbersome to overwrite dependencies

Christoph

I've codified the items that I'm using in Narwhal.  I think this is a
good place to start.

http://narwhaljs.org/packages-howto.html

+

1.) "schema" for the package.json schema version.  Perhaps a URL or
just a dewey decimal number.  In general, I think we should accept [1,
0, 1] and "1.0.1" as interchangeable in such places and normalize them
to the array notation in our tools.

We should standardize the license names.  An array of licenses should be "ored".

We need a format for dependency versions.  I think they should be
optional in the format, and enforced by managed package catalogs.
That would go for our jshq.org package catalog.

2 and 3 I think I've addressed.  I've got an "engine" flag.
Alternately we can use engines/ directories for engine specific
components.  We do both in Narwhal so far, and it's still a discussion
point.

4. and 5. yes.  It's my hope that we need not distinguish system
package dependencies from our own dependencies.  So, "jake", for
example can be installed as a dependency, requisite for the "make"
system.  I also would like us to be able to build things with "jake"
without going to a subprocess.  That could be denoted by a "jake" flag
instead of "make".

Kris Kowal

On Wed, Sep 9, 2009 at 2:14 PM, Christoph Dorn

Agreed; if supported, it would be for quick hacks only.

Ihab

--
Ihab A.B. Awad, Palo Alto, CA