modules packaging
Irakli Gozalishvili
In a different mailing list there was some question brought up regarding how does module collections are handled and I think it makes sense to have some standard regarding on how the collection of modules can be packaged, what topid's should they get and what metadata should they include about a packages. I have quoted original post below to get a better idea what am I about.
The question here becomes whether collections of modules are handled Python style or if they're handled differently (I think Ruby works the other way, but I'm not sure). What I mean is:
1. Python-style: a given top-level name determines where everything under that name will be. For example, if there's a directory called "foo", then "foo/bar" will only be found under that known directory foo.
2. The other style: the search path is searched for all module lookups. So, foo/bar might be found in one directory, whereas foo/baz may be found somewhere else entirely.
I would like to stick to the 2) option as it's a way currently narwhal works and my goal is to make narwhal packages completely compatible with bespin plugins. (It might be a good idea to discuss module packaging and module path resolution on the CommonJS)
example:
1) foobar-plugin/
foo/
In this example three modules will be registered with the topid's:
foo/bar
bla
foo/baz
Some more details on packaging or on how it's done in narwhal can be found here.
http://narwhaljs.org/packages-howto.html
Think it's very important to have a standard for packaging so that the same libs can be reused between different commonjs implementations.
Regards
--
Irakli Gozalishvili
Web: http://rfobic.wordpress.com/
Phone: +31 614 205275
Address: Taksteeg 3 - 4, 1012PB Amsterdam, Netherlands
ihab...@gmail.com
I have a 3rd proposal. Perhaps I should call it "hermetic packages"
(with a nod towards Kris Kowal's naming of "hermetic eval").
A package is a directory tree (which can be distributed as a tarball,
zipfile, ...) containing:
* a hierarchy of directories and modules (as *.js files)
* a manifest
Implementation is not important, but let's say we have:
<pkg_root>/
MANIFEST.json
foo/
bar.js
baz.js
boo/
fribble.js
Modules within the package can refer to each other by short names. So
bar.js can say:
require('./baz');
require('../boo/fribble');
as would be expected. However, to refer to a foreign module from
another package, bar.js would say something like:
require('#someOther/foo/frob');
where #someOther is a reference to a foreign package specification in
the MANIFEST.json. This would contain entries like this suggesting a
list of places to look for that package and asking for a specific
checksum to be verified:
someOther: {
locations: [
'http://allmysoftware.com/archive/bar/bar_2.4.8.zip',
'http://barsoft.com/distros/bar_2.4.8.zip'
]
checksum: '24f7c8cbf635f4c03bad2dde7bf07531'
}
and like this asking for anything signed by a given principal:
yetAnother: {
locations: [
'http://foosoft.com/public/latest/foo.zip'
]
signed-by: 'c1:b1:30:29:d7:b8:de:6c:97:77:10:d7:46:41:63:87'
}
The advantages are: strict hygiene of module names, and ability to
specify very clearly what foreign modules are being asked for. The
"installation" of a module tramples over nobody else's namespace.
The disadvantages are: a somewhat more complicated packaging scheme,
and less flexibility to go in and hack dependencies around until
something works.
The upshot, I think, is that this scheme is oriented towards
non-administered environments, where an end-user may ask for a given
piece of code to be installed and there is no local "expert" to mess
with the module configuration. As a side effect, it reduces or
eliminates the *need* for this local "expert" since the client modules
are now the ones who do the up-front work. As such, it enables
3rd-party hosted server environments where applications can be dropped
into a running server and just adapt themselves to whatever is around
them and yet still download their dependencies on demand and share
them where possible.
Ihab
--
Ihab A.B. Awad, Palo Alto, CA
Kris Zyp
Why can't we just use absolute URLs in requires:
require("http://allmysoftware.com/archive/bar/bar_2.4.8");
This seems far simpler and more intuitive. Plus users can add complexity
as they desire. If you wish to have some redirection, they could add that:
require(myPackages["someOther"].location);
For some of the other features you are asking for, it seems like a much
more flexible and composable solution would be to allow for custom
loaders for require (that can do checksum, multiple server checks, and
signing). Users could create as sophisticated as schemes as they desire
then.
Kris
ihab...@gmail.com
> Why can't we just use absolute URLs in requires:
> require("http://allmysoftware.com/archive/bar/bar_2.4.8");
That's what I was originally thinking, but it ends up being (a) a
potential security hole, without hooks for verifying that what you're
getting is what you really want; and (b) fragile since your package is
going out into the wild and you want to give it the best possible hope
of finding what it needs out into the future. Also, all those URLs end
up getting unmanageable, and it's nice to have a central place where
they can be edited....
That said, your point is a good one, and a degenerate case of the
scheme I propose (with one URL and no verification) is very similar to
what you describe.
But --
> For some of the other features you are asking for, it seems like a much
> more flexible and composable solution would be to allow for custom
> loaders for require (that can do checksum, multiple server checks, and
> signing). Users could create as sophisticated as schemes as they desire
> then.
Yeah, perhaps that is the real end result here. It would be nice to
have some commonality -- in other words, just one or two sophisticated
schemes rather than a plethora -- since that would allow greater
sharing of packages which is, after all, what we're all aiming for.
So I guess we're talking about 2 things:
1. Standardize the way require() relies on custom loaders; and
2. Standardize a "hermetic" package scheme for security wonks / zero
admin installations.
?
Wes Garland
> require("http://allmysoftware.com/archive/bar/bar_2.4.8");
Additionally, including version numbers in packages is problematic, as it means you need to change source code as foreign module versions increase.
On the flip side, using "whatever's on the web" is not a good situation for those wishing to create safely-versioned trees of apps, particularly in a revision-controlled corporate environment. Relative addressing is absolutely mandatory in an environment like that, unless you want to QA and entire server everytime you make a minor bugfix to a common library.
Wes
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
Wes Garland
> admin installations.
ihab...@gmail.com
>> 2. Standardize a "hermetic" package scheme for security wonks / zero
>> admin installations.
>
> I currently use "../libexec/libraryName.js" for my JS programs that depend
> on non-system-wide libraries. Module loading relative to the script is easy
> and predictable; you can untar a package anywhere and it will work.
Agreed, that is definitely a component of what I propose.
Within your own package, "short names" relative to the current module
(or to the root of the package) are definitely the common use case.
They would be used to create dependencies among the modules in a
package, which are all assumed to be managed by one [virtual]
development team and versioned together.
To make a reference to another package, which contains code versioned
and maintained separately by some other team, I propose the package
manifest and the "#foreignAlias/dir/module" notation.
Kris Zyp
Wes Garland wrote:
> > Why can't we just use absolute URLs in requires:
> > require("http://allmysoftware.com/archive/bar/bar_2.4.8");
>
> That breaks when the user wants to make local changes.
> Package-relative addressing is much friendlier in this regard.
Of course, by all means use relative addressing wherever possible.
>
> Additionally, including version numbers in packages is problematic, as
> it means you need to change source code as foreign module versions
> increase.
>
> On the flip side, using "whatever's on the web" is not a good
> situation for those wishing to create safely-versioned trees of apps,
> particularly in a revision-controlled corporate environment. Relative
> addressing is absolutely mandatory in an environment like that, unless
> you want to QA and entire server everytime you make a minor bugfix to
> a common library.
If you want to freeze a version, you can copy it. Any decent package
repository should also offer version-specific URLs as well as
current-version URLs.
Kris
Wes Garland
> and maintained separately by some other team, I propose the package
> manifest and the "#foreignAlias/dir/module" notation.
Wes Garland
If you want to freeze a version, you can copy it. Any decent package
repository should also offer version-specific URLs as well as
current-version URLs.
Sorry, Kris -- when you said, "why don't we just use absolute URIs", did you mean "why don't we only use absolute URIs" (my interpretation), or "why don't we simply adopt absolute URIs"?
If the latter, I see no problem with adding require(fullURI) as a way of loading modules. The initial require() discussion in fact pointed out that we could potentially use URIs or even database indices down the road to load modules -- but I think that the core, required functionality should be exactly what we have now -- and if we add require(URI) that require("./moreStuff") also be able to be URI-relative.
Wes
Kris Zyp
Wes Garland wrote:
> On Wed, Sep 9, 2009 at 1:51 PM, Kris Zyp <kri...@gmail.com
> <mailto:kri...@gmail.com>> wrote:
>
> If you want to freeze a version, you can copy it. Any decent package
> repository should also offer version-specific URLs as well as
> current-version URLs.
>
>
> Sorry, Kris -- when you said, "why don't we just use absolute URIs",
> did you mean "why don't we only use absolute URIs" (my
> interpretation), or "why don't we simply adopt absolute URIs"?
>
> If the latter, I see no problem with adding require(fullURI) as a way
> of loading modules. The initial require() discussion in fact pointed
> out that we could potentially use URIs or even database indices down
> the road to load modules -- but I think that the core, required
> functionality should be exactly what we have now -- and if we add
> require(URI) that require("./moreStuff") also be able to be URI-relative.
My apologies, my grammar was ambiguous. My intention was adopt absolute
URIs in addition to our current path mechanisms. I meant "just" as in
simpler form of package loading, not as in the only way. And yes, I
agree, that our current syntax should be URI-relative as well.
Kris
Dean Landolt
This would be an enormous boon to rapid application development. Plus, as Kris alluded to, once you're ready for release all your packages could be frozen and a manifest like Ihab suggests generated.
ihab...@gmail.com
> Why is additional notation required here? If your manifest looks like a
> list of where to find foreign packages, they could just as easily be
> represented by variables rather than another stage of lookup/parse.
Agreed in principle.
This scheme was suggested by a ServerJS participant in a thread that,
unfortunately, I am unable to find in the archives. It is a way to
shoehorn the package mechanism into a require() that takes only a
string argument, thus reducing the proliferation of APIs.
ihab...@gmail.com
> This would be an enormous boon to rapid application development. Plus, as
> Kris alluded to, once you're ready for release all your packages could be
> frozen and a manifest like Ihab suggests generated.
Interesting. There are some details to be ironed out, but yeah, it
would be great to have the option to say, "just grab this URL and
don't bother me any more".
The details that bug me are, what do I do if the module at the
specified URL is part of a package? Let's say the package is a jarfile
(say). Perhaps I would do:
require('jar:http://example.com/package.jar!foo/bar/baz');
? In that case would the loader know to look at the file:
jar:http://example.com/package.jar!MANIFEST.json
in order to create a proper package loader context for the module:
jar:http://example.com/package.jar!foo/bar/baz.js
? Hm.
Christoph Dorn
>> Why is additional notation required here? If your manifest looks like a
>> list of where to find foreign packages, they could just as easily be
>> represented by variables rather than another stage of lookup/parse.
>>
> Agreed in principle.
>
> This scheme was suggested by a ServerJS participant in a thread that,
> unfortunately, I am unable to find in the archives. It is a way to
> shoehorn the package mechanism into a require() that takes only a
> string argument, thus reducing the proliferation of APIs.
>
Maybe you were referring to this [1] thread.
Christoph
[1] -
http://groups.google.com/group/commonjs/browse_thread/thread/978c3e8f00d2a513
ihab...@gmail.com
<christ...@christophdorn.com> wrote:
> Maybe you were referring to this [1] thread.
I was! Thank you (for the original idea and for the ref).
Christoph Dorn
> Interesting. There are some details to be ironed out, but yeah, it
> would be great to have the option to say, "just grab this URL and
> don't bother me any more".
>
I think there needs to be a separation between *package management* and
*module loading*.
A package manager such as tusk is designed to download archives into a
cache/local repository, extract them and prepare them for the runtime
system. These steps do not belong into the module loader which should
operate only on a purely local prepared cache/working copy.
I would like to see the source of a module stay untouched when wanting
to switch the version or implementation of a dependency. This should be
done via a manifest file.
> The details that bug me are, what do I do if the module at the
> specified URL is part of a package? Let's say the package is a jarfile
> (say). Perhaps I would do:
>
> require('jar:http://example.com/package.jar!foo/bar/baz');
>
> ? In that case would the loader know to look at the file:
>
> jar:http://example.com/package.jar!MANIFEST.json
>
> in order to create a proper package loader context for the module:
>
> jar:http://example.com/package.jar!foo/bar/baz.js
>
> ? Hm.
>
In my simplified ideal world a module always belongs to a package and a
package manifest defines a universe of modules and packages for that
package. To load an *external* module (read a module not shipped with a
given package) the third party package can be referenced via a
*#packageName* alias which is mapped to a dependency in the package
manifest file.
Using this approach the above require statement would be:
require('#packageName/foo/bar/baz');
And the manifest file could be something like:
{
"dependencies": [
["packageName", {
uri: "jar:http://example.com/package.jar",
maxVersion: "1.3"
}]
]
}
tusk can then download the defined package and invoke the *jar*
packaging plugin to prepare the modules for simple inclusion by the loader.
One important point to note here. The *#packageName* alias is defined
only for modules shipped with the package. An external module can map
the same alias to a different package/version via it's package manifest
file.
There is one major point to consider when implementing this. When
loading an external module, a separate loader must be initialized based
on the external package manifest in order to resolve the aliases
accordingly.
Assuming this can be implemented efficiently it can provide complete
namespace isolation for packages and modules and allow for multiple
versions of the same package/module to co-exist depending on the calling
code. This may not be required for *core* packages and modules that
provide a common foundation (which can still be loaded the same way as
now) but IMHO it is critical for userland packages and modules where
namespace collisions are guaranteed.
Christoph
Kris Zyp
Christoph Dorn wrote:
>
>> Interesting. There are some details to be ironed out, but yeah, it
>> would be great to have the option to say, "just grab this URL and
>> don't bother me any more".
>>
>>
>
> I think there needs to be a separation between *package management* and
> *module loading*.
>
I don't want that separation.. An elegant solution that completely
eliminates the need for package management would be ideal, IMO. Or at
least, provide a means that those that don't want to use a package
manager could avoid doing so, and package management tools could be a
purely additive tool, and not essential.
Kris
Wes Garland
I agree -- if it is possible to design a simple system that does not depend on a package manager, you are much better off in the long run. It eliminates both bootstrapping and deployment issues.
Kris Kowal
I think that, for different purposes, we will need both managed
package systems and hosted packages.
We need to break this into two discussions, not a debate about which
discussion to have.
1.) What should the layout of a package be? What format should it be
stored in (zip, tar)? Where should its metadata (like dependencies)
be hosted (e.g., package.json)? What should the schema and format of
the package metadata file be? Would it be possible for such packages
to be used by both a managed package system like "tusk" and a hosted
URL package loader, or would it be necessary to package projects
differently for these two purposes? What are the common goals for
hosted packages?
2.) For a hosted package loader, what should the module identifier
domain be? For example, require(url, id) or
require("{url}{delimiter}{topId}"). How should the loader system
indicate that such identifiers are supported?
I recommend that we begin new threads for these topics, perhaps even
down to individual questions.
Kris Kowal
ihab...@gmail.com
<christ...@christophdorn.com> wrote:
> I think there needs to be a separation between *package management* and
> *module loading*.
>
> A package manager such as tusk is designed to download archives into a
> cache/local repository, extract them and prepare them for the runtime
> system. These steps do not belong into the module loader which should
> operate only on a purely local prepared cache/working copy.
I agree with a proviso that "install time" and "run time" might get
blurred sometimes.
If there is a run-time API for invoking a "foreign" package, and/or if
I load a package into a running system and that package has "foreign"
package dependencies, then the "package manager" tool (whatever that
is) might have to get fired up pronto puppy while the client code is
waiting.
That said, using a/the promise API, and making dynamic operations
return promises, we should be able to implement whatever we need.
> I would like to see the source of a module stay untouched when wanting
> to switch the version or implementation of a dependency. This should be
> done via a manifest file.
Yepper. :)
> In my simplified ideal world a module always belongs to a package and a
> package manifest defines a universe of modules and packages for that
> package.
Yes. With perhaps the possibility of a "SYSTEM" (or whatever) package
representing primordial stuff that you expect to get with any CommonJS
runtime?
> To load an *external* module (read a module not shipped with a
> given package) the third party package can be referenced via a
> *#packageName* alias which is mapped to a dependency in the package
> manifest file.
Agreed.
> Using this approach the above require statement would be:
>
> require('#packageName/foo/bar/baz');
>
> And the manifest file could be something like:
>
> {
> "dependencies": [
> ["packageName", {
> uri: "jar:http://example.com/package.jar",
> maxVersion: "1.3"
> }]
> ]
> }
Agreed.
> tusk can then download the defined package and invoke the *jar*
> packaging plugin to prepare the modules for simple inclusion by the loader.
With the caveat that the download may happen while client code is
twiddling its fingers, yes.
> One important point to note here. The *#packageName* alias is defined
> only for modules shipped with the package. An external module can map
> the same alias to a different package/version via it's package manifest
> file.
Absolutely!
> There is one major point to consider when implementing this. When
> loading an external module, a separate loader must be initialized based
> on the external package manifest in order to resolve the aliases
> accordingly.
Yes. So, when loading something that comes from a package, regardless
of how addressed, either via:
#packageName/foo/bar
or via:
jar:http://example.com/thepkg.jar!foo/bar.js
the calling context has to construct for the newly loaded code a
suitable loader.
> This may not be required for *core* packages and modules that
> provide a common foundation (which can still be loaded the same way as
> now) but IMHO it is critical for userland packages and modules where
> namespace collisions are guaranteed.
Agreed.
Ash Berlin
On 9 Sep 2009, at 21:40, Kris Kowal wrote:
>
> I do not think we need to set ourselves up with a false dilemma.
>
> I think that, for different purposes, we will need both managed
> package systems and hosted packages.
>
> We need to break this into two discussions, not a debate about which
> discussion to have.
>
> 1.) What should the layout of a package be? What format should it be
> stored in (zip, tar)? Where should its metadata (like dependencies)
> be hosted (e.g., package.json)? What should the schema and format of
> the package metadata file be? Would it be possible for such packages
> to be used by both a managed package system like "tusk" and a hosted
> URL package loader, or would it be necessary to package projects
> differently for these two purposes? What are the common goals for
> hosted packages?
Right, let me try to impart some of the lessons of perl/CPAN, since
CPAN is one of perl's strong points.
1) Version the meta data format. VERY IMPORTANT. If we miss something
or make a mistake (which we inevitably will)
2) Be strict about the format of the metadata file. I think everyone
so far has mentioned that this will be a JSON file. Good, lets be
strict about the format of this in the consumers. (Perl wasn't, and
this has lead to all sorts of headaches, in part due to the fact that
perl uses a 'META.yml' which is YAML, except when its not.)
3) Some flag or field should indicate if a module is a pure JS module,
or a native module for Engine Z (by Engine here i mean
flusspferd,gpsee, v8vgi, rhino, helma etc.) This lets us have *one*
central package repo. Without a central package repo, packages are
almost doomed. (We can talk about where this central repo is, what
indexes it maintains, upload policies later. Actually the basic upload
policy is 'first come to a module can upload it. No vetting of uploads')
4) as well as listing dependancies on other modules, we need the
ability to list external system deps, such as 'this Module needs make
to install', or we need this java class installed in the classpath, or
this .dll etc.. Oh and 'We need jake' etc.
5) Be able to specify (in the meta file) the command needed to build/
test/install the module. It wont always be i) straight forward even
for pure JS modules, ii) native modules might have special build
requirements.
Fields that should go in the metadata:
* Module's SCM repo is at $URL
* License of the module (either a url, or one of the 'GPL2', 'MIT',
'Artistic' etc.)
* Module homepage
* (Original) Author
* list of module deps (with versions)
* the version of hte package/module/bundle itself
I have other things to impart from perl, but those aren't about
metadata. Main one being managers should run tests at install time to
make sure it works with particular combination of modules installed on
$users env.
Christoph Dorn
> I agree with a proviso that "install time" and "run time" might get
> blurred sometimes.
>
> If there is a run-time API for invoking a "foreign" package, and/or if
> I load a package into a running system and that package has "foreign"
> package dependencies, then the "package manager" tool (whatever that
> is) might have to get fired up pronto puppy while the client code is
> waiting.
>
Yes absolutely and it may in some cases completely eliminate the need
for the package install step as voiced by Kris Zyp and Wes Garland.
> Yes. So, when loading something that comes from a package, regardless
> of how addressed, either via:
>
> #packageName/foo/bar
>
> or via:
>
> jar:http://example.com/thepkg.jar!foo/bar.js
>
> the calling context has to construct for the newly loaded code a
> suitable loader.
>
Right, however I see two main problems with using URL's without a
mapping via a manifest file:
* It does not allow transparency into the dependency tree
* It makes it more cumbersome to overwrite dependencies
Christoph
Kris Kowal
good place to start.
http://narwhaljs.org/packages-howto.html
+
1.) "schema" for the package.json schema version. Perhaps a URL or
just a dewey decimal number. In general, I think we should accept [1,
0, 1] and "1.0.1" as interchangeable in such places and normalize them
to the array notation in our tools.
We should standardize the license names. An array of licenses should be "ored".
We need a format for dependency versions. I think they should be
optional in the format, and enforced by managed package catalogs.
That would go for our jshq.org package catalog.
2 and 3 I think I've addressed. I've got an "engine" flag.
Alternately we can use engines/ directories for engine specific
components. We do both in Narwhal so far, and it's still a discussion
point.
4. and 5. yes. It's my hope that we need not distinguish system
package dependencies from our own dependencies. So, "jake", for
example can be installed as a dependency, requisite for the "make"
system. I also would like us to be able to build things with "jake"
without going to a subprocess. That could be denoted by a "jake" flag
instead of "make".
Kris Kowal
ihab...@gmail.com
<christ...@christophdorn.com> wrote:
> Right, however I see two main problems with using URL's without a
> mapping via a manifest file:
>
> * It does not allow transparency into the dependency tree
> * It makes it more cumbersome to overwrite dependencies
Agreed; if supported, it would be for quick hacks only.
Christoph Dorn
> 1.) "schema" for the package.json schema version. Perhaps a URL or
> just a dewey decimal number. In general, I think we should accept [1,
> 0, 1] and "1.0.1" as interchangeable in such places and normalize them
> to the array notation in our tools.
>
well? How do we indicate stability?
> We should standardize the license names. An array of licenses should be "ored".
>
+1
> We need a format for dependency versions. I think they should be
> optional in the format, and enforced by managed package catalogs.
> That would go for our jshq.org package catalog.
>
+1
Would dependency versions be precisely defined or can we constrain them
with ranges?
i.e. Use the latest x.x.beta until a x.x.stable version is available up
to version x.y
Also what do you think about giving each package a UUID? The UUID would
be unique for every *fork* of the package to uniquely identify the
source/package combination. It would allow us to easily correlate
packages between repositories. You would then only need a uuid + version
combo to reference a package.
Christoph
ryan dahl
<christ...@christophdorn.com> wrote:
>
>> 1.) "schema" for the package.json schema version. Perhaps a URL or
>> just a dewey decimal number. In general, I think we should accept [1,
>> 0, 1] and "1.0.1" as interchangeable in such places and normalize them
>> to the array notation in our tools.
>>
> Would the [1,0,1] or "1.0.1" convention be used to version packages as
> well? How do we indicate stability?
Cabal has a "stability" entry in their package description file.
in general there are a lot of ideas to be borrowed from them.
>> We should standardize the license names. An array of licenses should be "ored".
>>
> +1
Ash Berlin
On 11 Sep 2009, at 16:49, Christoph Dorn wrote:
>
>
>> 1.) "schema" for the package.json schema version. Perhaps a URL or
>> just a dewey decimal number. In general, I think we should accept
>> [1,
>> 0, 1] and "1.0.1" as interchangeable in such places and normalize
>> them
>> to the array notation in our tools.
>>
> Would the [1,0,1] or "1.0.1" convention be used to version packages as
> well? How do we indicate stability?
>
>> We should standardize the license names. An array of licenses
>> should be "ored".
>>
> +1
>
>> We need a format for dependency versions. I think they should be
>> optional in the format, and enforced by managed package catalogs.
>> That would go for our jshq.org package catalog.
>>
> +1
>
> Would dependency versions be precisely defined or can we constrain
> them
> with ranges?
>
> i.e. Use the latest x.x.beta until a x.x.stable version is available
> up
> to version x.y
I don't personally like the idea of version ranges on deps, since I
worry that it would breed a culture of module authors not worrying
about back-compat.
>
> Also what do you think about giving each package a UUID? The UUID
> would
> be unique for every *fork* of the package to uniquely identify the
> source/package combination. It would allow us to easily correlate
> packages between repositories. You would then only need a uuid +
> version
> combo to reference a package.
I think the most successful route for packages is where there is a
single central repository. The ruby approach of 'oh just install it of
github' is a bad one i think. 1) What if the master repo is no longer
the most up-to-date since the author has vanished etc, 2) the HEAD of
the repo might not actually be suitable (this could be addressed by
tags)
I'm not saying using such a uuid is a bad idea, in fact its probably a
good idea, but I'd like suggest that any module that is not from the
central module server gets treated as a dev/beta release and only gets
installed when explicitly specified.
-ash
Kris Kowal
> I think the most successful route for packages is where there is a
> single central repository. The ruby approach of 'oh just install it of
> github' is a bad one i think. 1) What if the master repo is no longer
> the most up-to-date since the author has vanished etc, 2) the HEAD of
> the repo might not actually be suitable (this could be addressed by
> tags)
To be clear, these are issues that are not so much about module
metadata, but about another file format that specifies where to grab
packages, which I think that we can safely leave implementation
dependent for the time being. Tusk has a sources.json, that *should*
look like:
{
"schema": "0.2",
"version": "0.1.2",
"sources": [
"http://hqdb.com/sources.json",
"http://github.com/tlrobinson/narwhal/1.0/blob/sources.json"
],
"packages": {
"narwhal": {
"type": "github",
"user": "tlrobinson",
"ref": "1.0"
},
"objj": {
"type": "github",
"user": "tlrobinson",
"name": "capppucccinnno",
"ref": "1.0"
},
"appengine-java-sdk": {
"type": "inline",
"package.json": {
"name": "gae",
"title": "Google App Engine Java SDK",
"description": "Java components necessary for
deploying applications on Google's App Engine.",
"author": "Google",
"version": "1.2.2",
"dependencies": ["jack-servlet", "jack", "narwhal"],
"lib": [],
"type": "zip",
"location":
"http://googleappengine.googlecode.com/files/appengine-java-sdk-1.2.2.zip"
}
}
}
}
We use the "inline" source to bring in installable archives into the
Narwhal package system involuntarilly. Any zip file on the web will
do, if we provide a package.json for them, either on our own URL, or
placed inline as here. We also want to eventually support other
sources.
So, summarily, I think it's our intention to address both of your
concerns in both of the ways you've mentioned. We want to support
central, co-versioned package management on jshq.org, and we also want
to support version control sources like Github and Subversion over
Webdav, with refs and tags.
But, also, I don't think we need to have interoperability on the
package sources layer since it's not really a published format and
it's likely to coevolve with a package manager implementation. We
definitely *do* need interoperability in the package.json format, and
the package.zip layout so we can share packages on jshq.org.
Kris
Christoph Dorn
Would dependency versions be precisely defined or can we constrain them with ranges? i.e. Use the latest x.x.beta until a x.x.stable version is available up to version x.yI don't personally like the idea of version ranges on deps, since I worry that it would breed a culture of module authors not worrying about back-compat.
1) The dependencies are installed into a central repo and used by multiple packages. In this case a version range would make sense as it is up to the user to decide when/if to upgrade a dependency.
2) The dependencies are installed into a repo specific to a package. In this case the same dependency may exist with different versions for different packages. To ensure interoperability the API of the dependency must be stable across versions.
3) A mix of central and package-specific repos.
While the implementation of a module can differ between versions, the exported API should probably be consistent based on a major/minor version convention to support all three scenarios.
I think the most successful route for packages is where there is a single central repository. The ruby approach of 'oh just install it of github' is a bad one i think. 1) What if the master repo is no longer the most up-to-date since the author has vanished etc, 2) the HEAD of the repo might not actually be suitable (this could be addressed by tags) I'm not saying using such a uuid is a bad idea, in fact its probably a good idea, but I'd like suggest that any module that is not from the central module server gets treated as a dev/beta release and only gets installed when explicitly specified.
Christoph
Kris Kowal
> Cabal has a "stability" entry in their package description file.
>
> http://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/Distribution-PackageDescription.html#v%3Astability
>
> in general there are a lot of ideas to be borrowed from them.
>
Lots of good ideas here. We should rip most of this off wholesale. Thanks!
* "license" and their list of license names.
* "licenseFile"
* "copyright" we already have
* "maintainer", already have
* "author", already have
* "stability" we should definitely do
* "testedWith" isn't really applicable
* "homepage": let's use this instead of the "url" I proposed earlier.
* "pkgUrl", I'm presently using "location" for this. Thoughts?
* "bugReports", sure
* "sourceRepos", let's revisit what this means for us
* "synopsis", we're presently using "description". Let's keep "description".
* "category", I don't really want to manage a taxonomy. Let's stick
with "keywords".
* "buildDepends". We need to choose between "depends" and
"dependencies". We're using "dependencies" in package.json presently
on Narwhal, and "depends" in Jake. Perhaps it's okay to be different.
We definitely need to distinguish build dependencies from normal
dependencies, for lean package installation in certain contexts like
Google App Engine.
* "descCabalVersion" is not really applicable. I would be fine with
leaving the "x" prefix for such things as "xNarwhalVersion" for
Narwhal versions or ranges.
* "buildType" I think should be addressed with a "build" attribute
that is a path to a build module. This would let us use os.system, or
a Jakefile. The module should accept the package.json data object as
an argument so we have maximum flexibility to configure various build
systems.
I'm not sure about the applicability of the remaining args. If
someone wants one, I'd recommend making a case here.
Kris Kowal
Dean Landolt
Sounds great, but why not really base it on how DNS works and use URLs? They can be a whole lot more human readable, plus, unlike UUIDs, you get resolution for free.
Christoph Dorn
I envision a central *registry*, not a central repository. The registry tracks uuid's and directs requests to a repository similar to how DNS works. This approach would allow multiple repositories (public and private). If a specific package (based on uuid) is no longer maintained the registry can be updated to point to a new maintained package (uuid) and all users can optionally or automatically be redirected.
Sounds great, but why not really base it on how DNS works and use URLs? They can be a whole lot more human readable, plus, unlike UUIDs, you get resolution for free.
Can we use "id" for that?
Christoph
Ash Berlin
On 11 Sep 2009, at 22:02, Kris Kowal wrote:
>
> On Fri, Sep 11, 2009 at 11:18 AM, ryan dahl <coldre...@gmail.com>
> wrote:
>> Cabal has a "stability" entry in their package description file.
>>
>> http://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/Distribution-PackageDescription.html#v%3Astability
>>
>> in general there are a lot of ideas to be borrowed from them.
>>
>
> Lots of good ideas here. We should rip most of this off wholesale.
> Thanks!
Agreed. Comments inline
> * "license" and their list of license names.
> * "licenseFile"
> * "copyright" we already have
> * "maintainer", already have
> * "author", already have
> * "stability" we should definitely do
> * "testedWith" isn't really applicable
> * "homepage": let's use this instead of the "url" I proposed earlier.
> * "pkgUrl", I'm presently using "location" for this. Thoughts?
> * "bugReports", sure
bugReports: one of: a url to the bug tracker or an email address?
> * "sourceRepos", let's revisit what this means for us
To me this is just a URL to the repo, i.e. git://foo or http://foo.bar/hg
. maybe a { url: "", type: ""} is needed?
> * "synopsis", we're presently using "description". Let's keep
> "description".
sure.
> * "category", I don't really want to manage a taxonomy. Let's stick
> with "keywords".
I'm not really sure that either of these are needed at all :)
> * "buildDepends". We need to choose between "depends" and
> "dependencies". We're using "dependencies" in package.json presently
> on Narwhal, and "depends" in Jake. Perhaps it's okay to be different.
> We definitely need to distinguish build dependencies from normal
> dependencies, for lean package installation in certain contexts like
> Google App Engine.
buildDepends and depend{s,encides} are different things to me. build
depends are things like make/jake/cmake etc that are only needed to
compile/build the module, but not at runtime.
It could easily be a module has utility only at build/test time. An
example that comes to mind is something that gives a 'can_run'
function for presence of optional binary.
Deps are naturally modules that this module uses at runtime. I'd go
for 'depends' personally. "This module depends on X,Y and Z"
> * "descCabalVersion" is not really applicable. I would be fine with
> leaving the "x" prefix for such things as "xNarwhalVersion" for
> Narwhal versions or ranges.
Maybe place these under an engines keyword: i.e. engines: { narwhal:
{ version: some_version_spec } }. Otherwise the consumer of the
metadata has to look over every key of the top level.
> * "buildType" I think should be addressed with a "build" attribute
> that is a path to a build module. This would let us use os.system, or
> a Jakefile. The module should accept the package.json data object as
> an argument so we have maximum flexibility to configure various build
> systems.
What do you mean when you say 'path to a build module'? I guess
needing to wrap what ever build system is in use (cmake, make,
autoconf, et al.) in a module isn't too much of a pain, but is it
worth. Perhaps to unify the build API to the javascript layer for the
package manager.
-ash
Kris Kowal
<ash_flu...@firemirror.com> wrote:
> bugReports: one of: a url to the bug tracker or an email address?
Yeah, Narwhal internally converts "Name (url) <email>" to {name, url,
email} objects instanceof Author. Perhaps something similar.
>> * "sourceRepos", let's revisit what this means for us
>
> To me this is just a URL to the repo, i.e. git://foo or http://foo.bar/hg
> . maybe a { url: "", type: ""} is needed?
Yeah, let's go with "repository: {url, type}.
>> * "category", I don't really want to manage a taxonomy. Let's stick
>> with "keywords".
>
> I'm not really sure that either of these are needed at all :)
Obviously not necessary, but as an option to help out with
"package-manager apropos" or "package-manager search".
> buildDepends and depend{s,encides} are different things to me. build
> depends are things like make/jake/cmake etc that are only needed to
> compile/build the module, but not at runtime.
>
> It could easily be a module has utility only at build/test time. An
> example that comes to mind is something that gives a 'can_run'
> function for presence of optional binary.
>
> Deps are naturally modules that this module uses at runtime. I'd go
> for 'depends' personally. "This module depends on X,Y and Z"
Okay. "buildDepends": [], "depends": []
>> * "descCabalVersion" is not really applicable. I would be fine with
>> leaving the "x" prefix for such things as "xNarwhalVersion" for
>> Narwhal versions or ranges.
>
> Maybe place these under an engines keyword: i.e. engines: { narwhal:
> { version: some_version_spec } }. Otherwise the consumer of the
> metadata has to look over every key of the top level.
Sure.
I also think that we need an "engine" keyword if it's an
engine-specific package. We should enumerate valid engine strings.
>> * "buildType" I think should be addressed with a "build" attribute
>> that is a path to a build module. This would let us use os.system, or
>> a Jakefile. The module should accept the package.json data object as
>> an argument so we have maximum flexibility to configure various build
>> systems.
>
> What do you mean when you say 'path to a build module'? I guess
> needing to wrap what ever build system is in use (cmake, make,
> autoconf, et al.) in a module isn't too much of a pain, but is it
> worth. Perhaps to unify the build API to the javascript layer for the
> package manager.
I think using a module would give us maximum flexibility to plug in
these things without necessarily resorting to a subprocess call. I
could buy "build" and "buildModule" to distinguish these, as I concur
that a shell command is probably the most common use case. "test" and
"testModule" too.
Kris Kowal
Ash Berlin
On 11 Sep 2009, at 23:20, Kris Kowal wrote:
> [all agreed]
>
>>> * "buildType" I think should be addressed with a "build" attribute
>>> that is a path to a build module. This would let us use
>>> os.system, or
>>> a Jakefile. The module should accept the package.json data object
>>> as
>>> an argument so we have maximum flexibility to configure various
>>> build
>>> systems.
>>
>> What do you mean when you say 'path to a build module'? I guess
>> needing to wrap what ever build system is in use (cmake, make,
>> autoconf, et al.) in a module isn't too much of a pain, but is it
>> worth. Perhaps to unify the build API to the javascript layer for the
>> package manager.
>
> I think using a module would give us maximum flexibility to plug in
> these things without necessarily resorting to a subprocess call. I
> could buy "build" and "buildModule" to distinguish these, as I concur
> that a shell command is probably the most common use case. "test" and
> "testModule" too.
>
I'd be tempted to not have testModule. you can have buildModule which
is then used for build, test and install. or you can have build, test
and install properties. If you've got a module, it makes sense to me
to use that for everything.
Especially if that module can live in the zipball itself for added
flexability. Hmmm maybe not, for the reason given below.
(Note: this para not about meta, but about yet another lesson from
perl). Though if we do end up letting the buildModule live in the
{zip,tar}ball, I'd like to make a strong case for it comparing
versions between that and the system installed version (if there is
one) and using that version for preference. Perl's two builder modules
get this wrong in different ways. Mobuld::Build (MB) doesn't bundle
itself in the dist, and there was no way of saying 'I need MB to
build' leading to problems. Module::Install does bundle it self with
the dist, but it doesn't self update, meaning if there is a bug in a
version that you ship a dist with, you have to ship a new dist to fix
it.
Both of these problems are fixed by buildDeps/buildModule. (I guess
buildModule is an implicit buildDep)
Kevin Dangoor
I envision a central *registry*, not a central repository. The registry tracks uuid's and directs requests to a repository similar to how DNS works. This approach would allow multiple repositories (public and private). If a specific package (based on uuid) is no longer maintained the registry can be updated to point to a new maintained package (uuid) and all users can optionally or automatically be redirected.
This is actually one area in which Python's Cheeseshop actually works pretty well. It acts as a registry, keeping the metadata for various versions of a package (and a package can have multiple owners that are allowed to manage that metadata). It provides a "download URL" field to point people to the place where they can download it....
But it's *also* a central repository for packages. It's not always convenient for people to host their own packages, and it's hard to get much easier than "python setup.py sdist upload" for releasing a new version of a package.
Kevin
--
Kevin Dangoor
work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com
Christoph Dorn
>>> * "sourceRepos", let's revisit what this means for us
>> To me this is just a URL to the repo, i.e. git://foo or http://foo.bar/hg
>> . maybe a { url: "", type: ""} is needed?
>
> Yeah, let's go with "repository: {url, type}.
Can we add "path" to reference a package within a repo in cases where a
repo contains multiple packages.
Christoph
Ash Berlin
Is path needed? Can you not just provide that in the url? i.e.
git://github.com/ashb/opensource.git/foo/bar (or what ever the correct
url is for github)