Towards a Simplified Packages Spec
Charles Jolley
In the 9 months that I've been a part of CommonJS, we haven't really converged on a common package format. In fact, if anything they seem to have diverged. We've added new features (like mappings) that increase complexity. Also several new package managers have come out in this time; all of them claim to support CommonJS packages and yet they are all distinct and incompatible.
I've been thinking about ways to solve this problem for the last few weeks. Finally, I came to the realization that, actually, our current package spec is trying to address address three questions at once:
1. How is a package structured?
i.e. how do I specify configuration, etc.? Most people seem to agree that a package should be a directory with a 'package.json' file and its source code should be in a 'lib' directory.
2. How do I load a module from a package?
i.e. if I want to load the 'bar' module from the 'foo' package, how do I describe that in code? It sounds like most people want this to be something like require('foo/bar'); though there are a lot of different opinions on how this should be implemented.
There are also lots of additional divergent ideas here such as mappings for arbitrary transforms, or directories to remap the "lib" dir, etc.
3. How are packages discovered, installed, managed, etc?
This is where we have the most divergence. Some want to use github to store everything. Others want to have a dedicated web server. And let's not even get into all the disagreement over how the tools themselves should be structured or function.
--
The fact is, if we are going to achieve wide adoption of packages in JavaScript (which we desperately need), we need to start with something much much simpler. Like CommonJS modules - which I think we can all agree have had great success.
In this light, I've put together what I think it would take to build the bare minimum package-aware system possible. [Note: I wrote "package AWARE" not a full package manager. i.e. I want a CommonJS runtime that can load modules from packages; regardless of how they are installed or managed.]
This would be sort of a "Packages Level-0". It turns out, this can be quite simple. Here is what I would propose:
----
PACKAGE LEVEL-0 SPEC
1. A CommonJS package is a directory with a 'package.json' file at its root. Any normal source code should appear in the 'lib' directory. Unit tests and other supporting files go outside of the lib directory.
2. The package.json contains a JSON hash which may contains any properties that are useful to the specific package manager. This spec requires the hash to contain only two properties: "PACKAGE-FORMAT" and "dependencies". It would have the following structure:
{
"PACKAGE-FORMAT": "1.0",
"dependencies": {
"packageId": {
"src": "http://path/to/package"
}
}
}
The PACKAGE-FORMAT is used by the package system to identify the spec version this package complies with. It should be "1.0" for now.
"dependencies" describes a mapping of a packageId to a source URL. The packageId can be any valid name chosen by the developer. The value is a hash that contains at least a "src" property which is the URL where the package can be found. The package runtime just uses this URL to uniquely identify the package you want to use.
As a simplification, the runtime should also allow you to just name the source URL. e.g.
"foo": "http://github.com/bar/foo"
is equivalent to:
"foo": {
"src": "http://github.com/bar/foo"
}
Note also that although dependencies requires a 'src' URL, the URL could be something custom like "pkg://foo:1.2.3" which the package manager may use to automatically discover the package/version from its own system. This is a custom extension outside the scope of the spec.
3. When code inside of a package calls require(), the CommonJS runtime should attempt to map the first term of the moduleId to a packageId named in the dependencies hash. e.g.
require("foo/bar");
in a package with a package.json like:
{
"dependencies": {
"foo": {
"src": "http://github.com/mystuff/foo"
}
}
}
would load the module at "lib/bar" inside the package found at http://github.com/mystuff/foo
---
That's it. This basically covers points #1 and #2 that I described at the top of the email. It leaves #3 open so that package managers can vary in how they implement while still remaining compatible.
I think breaking out into a level-0 like this is critically important because at least part of this functionality must be implemented into the CommonJS runtimes themselves (i.e. as part of the node.js runtime, or narwhal runtime, etc.)
As an optional "Level-1" version of this spec, we could also specify the optional keys in the package.json useful for package discovery like name, version, description, author, etc. These are generally not required though to simply load and run a package at runtime. Which is why I think they should be separated out.
--
My question is: can we break out our package spec with a level-0 like this one and then build on top of it? If so, is there anything missing here that means this absolutely will not work without it. We should try to strip aways all that we can...
-Charles
Irakli Gozalishvili
I still have some inline comments though
Hi everyone,
In the 9 months that I've been a part of CommonJS, we haven't really converged on a common package format. In fact, if anything they seem to have diverged. We've added new features (like mappings) that increase complexity. Also several new package managers have come out in this time; all of them claim to support CommonJS packages and yet they are all distinct and incompatible.
I've been thinking about ways to solve this problem for the last few weeks. Finally, I came to the realization that, actually, our current package spec is trying to address address three questions at once:
1. How is a package structured?
i.e. how do I specify configuration, etc.? Most people seem to agree that a package should be a directory with a 'package.json' file and its source code should be in a 'lib' directory.
2. How do I load a module from a package?
i.e. if I want to load the 'bar' module from the 'foo' package, how do I describe that in code? It sounds like most people want this to be something like require('foo/bar'); though there are a lot of different opinions on how this should be implemented.
There are also lots of additional divergent ideas here such as mappings for arbitrary transforms, or directories to remap the "lib" dir, etc.
3. How are packages discovered, installed, managed, etc?
This is where we have the most divergence. Some want to use github to store everything. Others want to have a dedicated web server. And let's not even get into all the disagreement over how the tools themselves should be structured or function.
--
The fact is, if we are going to achieve wide adoption of packages in JavaScript (which we desperately need), we need to start with something much much simpler. Like CommonJS modules - which I think we can all agree have had great success.
In this light, I've put together what I think it would take to build the bare minimum package-aware system possible. [Note: I wrote "package AWARE" not a full package manager. i.e. I want a CommonJS runtime that can load modules from packages; regardless of how they are installed or managed.]
This would be sort of a "Packages Level-0". It turns out, this can be quite simple. Here is what I would propose:
----
PACKAGE LEVEL-0 SPEC
1. A CommonJS package is a directory with a 'package.json' file at its root. Any normal source code should appear in the 'lib' directory. Unit tests and other supporting files go outside of the lib directory.
2. The package.json contains a JSON hash which may contains any properties that are useful to the specific package manager. This spec requires the hash to contain only two properties: "PACKAGE-FORMAT" and "dependencies". It would have the following structure:
{
"PACKAGE-FORMAT": "1.0",
"dependencies": {
"packageId": {
"src": "http://path/to/package"
}
}
}
The PACKAGE-FORMAT is used by the package system to identify the spec version this package complies with. It should be "1.0" for now.
Hate bikeshading, but wold prefer lower case
"dependencies" describes a mapping of a packageId to a source URL. The packageId can be any valid name chosen by the developer. The value is a hash that contains at least a "src" property which is the URL where the package can be found. The package runtime just uses this URL to uniquely identify the package you want to use.
As a simplification, the runtime should also allow you to just name the source URL. e.g.
"foo": "http://github.com/bar/foo"
is equivalent to:
"foo": {
"src": "http://github.com/bar/foo"
}
Note also that although dependencies requires a 'src' URL
I do believe there were objections on weather url being identifier is a good idea, specially because it would encode versioning in it and I partially agree on versioning part.
the URL could be something custom like "pkg://foo:1.2.3" which the package manager may use to automatically discover the package/version from its own system. This is a custom extension outside the scope of the spec.
I think it's a bad idea cause it would mean no cross platform packages at all.
3. When code inside of a package calls require(), the CommonJS runtime should attempt to map the first term of the moduleId to a packageId named in the dependencies hash. e.g.
require("foo/bar");
in a package with a package.json like:
{
"dependencies": {
"foo": {
"src": "http://github.com/mystuff/foo"
}
}
}
would load the module at "lib/bar" inside the package found at http://github.com/mystuff/foo
---
That's it. This basically covers points #1 and #2 that I described at the top of the email. It leaves #3 open so that package managers can vary in how they implement while still remaining compatible.
I think breaking out into a level-0 like this is critically important because at least part of this functionality must be implemented into the CommonJS runtimes themselves (i.e. as part of the node.js runtime, or narwhal runtime, etc.)
As an optional "Level-1" version of this spec, we could also specify the optional keys in the package.json useful for package discovery like name, version, description, author, etc. These are generally not required though to simply load and run a package at runtime. Which is why I think they should be separated out.
--
My question is: can we break out our package spec with a level-0 like this one and then build on top of it? If so, is there anything missing here that means this absolutely will not work without it. We should try to strip aways all that we can...
-Charles
--
You received this message because you are subscribed to the Google Groups "CommonJS" group.
To post to this group, send email to comm...@googlegroups.com.
To unsubscribe from this group, send email to commonjs+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/commonjs?hl=en.
Irakli Gozalishvili
"dependencies" describes packages that on the current one happens to depend on. The value is a hash that contains metadata about the dependcy package, what metadata is out of the scope of the specification only requirement is support for vendor specific prefixes for all the keys and with a higher priority.
In that case some package managers will use just versions others will use urls some might even put overlay data, but the vendor specific prefixes will allow cross platform model.
"dependencies": {
"packageId": {
"seedjs": {
"src": "http://path/to/package",
}
"npm": {
"version":"0.1"
},
"narwhal": {
"location": "http://github.com/280north/jack/zipball/master",
"descriptor": {
"index": "lib/jack.js",
"directories": {
"lib": "lib/jack"
}
},
"verify": {
"signature": "c1:b1:30:29:d7:b8:de:6c:97:77:10:d7:46:41:63:87",
"algorithm": "sha1"
}
}
}
}
this way everyone is happy inclusive me cause my package works everywhere. Platform specific implementers will just put properties without prefixes.
If we're lucky and we'll manage to agree on some keys like version or src we will standardize them in version 2
Thoughts ?
Torrey Rice
3. How are packages discovered, installed, managed, etc?
This is where we have the most divergence. Some want to use github to store everything. Others want to have a dedicated web server. And let's not even get into all the disagreement over how the tools themselves should be structured or function.
Mikeal Rogers
specification to handle. Specs aren't the best place to solve every
problem and in some cases diversity isn't just beneficial it's
required.
For instance, package discovery and installation do not belong in a
CommonJS specification. Package management tools are targeted at
platforms and will require platform specific directives, those package
managers will have their own opinions and requirements related to
discovery and installation and that's a necessity. The existing
Modules implementations are incredibly diverse and it's not
conceivable that a package manager could support them all.
I actually don't find the current Packages specification that
deficient. It's very close to providing a way for complex packages to
be created, defined, and installed by any number of package mangers on
a variety of platforms.
There are a few points we need to work out, like require('name')
resolves the package's "main" directly but I don't believe we have
done a good job of defining require("name/sub") as importing a module
from the "lib" directory. Nor do I necessarily think those semantics
are a good idea even although they seem to be in use.
But to your points, I think #1 is covered well already, #2 is not and
needs to be defined even if defined as to say "this is not part of the
specification" and #3 does *not* belong in any CommonJS specification.
I'll send a separate email about the points in "package level 0".
-Mikeal
Mikeal Rogers
>
> In this light, I've put together what I think it would take to build the bare minimum package-aware system possible. [Note: I wrote "package AWARE" not a full package manager. i.e. I want a CommonJS runtime that can load modules from packages; regardless of how they are installed or managed.]
>
> This would be sort of a "Packages Level-0". It turns out, this can be quite simple. Here is what I would propose:
>
> ----
> PACKAGE LEVEL-0 SPEC
>
> 1. A CommonJS package is a directory with a 'package.json' file at its root. Any normal source code should appear in the 'lib' directory. Unit tests and other supporting files go outside of the lib directory.
>
Why are you defining the required directory where the source goes?
This is why we have a directive for "lib" so that it can be structured
differently. Also, "directory" assumes more about a Module than is
currently defined, directories and files are not assumed in the
Modules specification and it doesn't require them (CouchDB for
instance uses a design doc sandbox with strings as "files" and an
object as the "directory" structure).
> 2. The package.json contains a JSON hash which may contains any properties that are useful to the specific package manager. This spec requires the hash to contain only two properties: "PACKAGE-FORMAT" and "dependencies". It would have the following structure:
>
> {
> "PACKAGE-FORMAT": "1.0",
> "dependencies": {
> "packageId": {
> "src": "http://path/to/package"
> }
> }
> }
>
> The PACKAGE-FORMAT is used by the package system to identify the spec version this package complies with. It should be "1.0" for now.
First off, I hate ALL CAPS and I'm pretty sure I'm not alone.
Secondly, a version declaration for the packages specification version
is not necessary. It's JSON, so we can add and remove properties at
will. Package mangers can decide which properties they require and
will fail on them regardless of the "version". Defining an explicit
version means that we're encouraging version checking over what might
be considered "capabilities" checking ( looking for a field that is
used or required and then branching the behavior for that field rather
than just failing or branching behavior based on a static version).
Also, dependencies should not be required and I'm pretty sure in the
latest draft that I worked on we made it no longer required.
And I'm pretty sure you will need a "name" as part of the requirements :)
>
> "dependencies" describes a mapping of a packageId to a source URL. The packageId can be any valid name chosen by the developer. The value is a hash that contains at least a "src" property which is the URL where the package can be found. The package runtime just uses this URL to uniquely identify the package you want to use.
>
> As a simplification, the runtime should also allow you to just name the source URL. e.g.
>
> "foo": "http://github.com/bar/foo"
>
> is equivalent to:
>
> "foo": {
> "src": "http://github.com/bar/foo"
> }
>
> Note also that although dependencies requires a 'src' URL, the URL could be something custom like "pkg://foo:1.2.3" which the package manager may use to automatically discover the package/version from its own system. This is a custom extension outside the scope of the spec.
We can't define a canonical url to a service that we haven't actually
specified. GitHub is great but it's not clear how a github url can get
me a package.json. Also, again, different platforms and package
managers will have exceeding requirements for package discovery so
this needs to not be in the spec.
>
> 3. When code inside of a package calls require(), the CommonJS runtime should attempt to map the first term of the moduleId to a packageId named in the dependencies hash. e.g.
>
> require("foo/bar");
>
> in a package with a package.json like:
>
> {
> "dependencies": {
> "foo": {
> "src": "http://github.com/mystuff/foo"
> }
> }
> }
>
> would load the module at "lib/bar" inside the package found at http://github.com/mystuff/foo
Why not use the existing "main" behavior?
We're fairly close to what might be considered the best securable
cross-platform module system that has been developed in terms of
security and sandboxing. If we allowed the "main" directive to be the
only publicly facing module import then we keep the rest of the system
private and only accessible via the main module. This would be a lot
better for security especially once you start thinking about having
these modules in the browser.
>
> ---
>
> That's it. This basically covers points #1 and #2 that I described at the top of the email. It leaves #3 open so that package managers can vary in how they implement while still remaining compatible.
>
> I think breaking out into a level-0 like this is critically important because at least part of this functionality must be implemented into the CommonJS runtimes themselves (i.e. as part of the node.js runtime, or narwhal runtime, etc.)
>
> As an optional "Level-1" version of this spec, we could also specify the optional keys in the package.json useful for package discovery like name, version, description, author, etc. These are generally not required though to simply load and run a package at runtime. Which is why I think they should be separated out.
>
> --
>
> My question is: can we break out our package spec with a level-0 like this one and then build on top of it? If so, is there anything missing here that means this absolutely will not work without it. We should try to strip aways all that we can...
>
> -Charles
>
>
>
Kris Zyp
This seems to be very close to the current mappings proposal. So you are
suggesting that we rename "mappings" to "dependencies" and rename
"location" to "src"? I am not sure how these renames make packages any
simpler (except maye "src" is a few less characters than "location").
Or are suggesting some type of "simplification" to the meaning source
URL? There doesn't really seem to be any clear definition of source URL
in this proposal. In the mappings proposal, a URL can point to an
archive of a package, a JavaScript file, or a "folder" (for runtime
mapping of ids to URLs). Restricting the options for a target reduces
the possible packages and modules (many existing modules aren't in
proper packages) that can be utilized by packages, which seems to defeat
the goal of wider adoption.
As far as the names go, "mappings" seems more clear than "dependencies"
to me, as this proposal actually describes it as "mapping of a packageId
to a source URL". I don't care about "location" vs "src" or calling the
whole thing "package mappings" vs "package level-0".
--
Thanks,
Kris
Isaac Schlueter
Wow, I disagree with pretty much all of this. Charles, let me say at
the outset that while I plan to thoroughly thrash your idea here, it's
not personal, and I hope we can still be friends after. Going easy on
it would be disrespectful.
> all of them claim to support CommonJS packages and yet they are all
> distinct and incompatible.
It's not terribly hard to write a package.json that works across
different platforms. The "overlay" hash helps :)
Those differences are slowly being reduced.
> 1. How is a package structured?
The package spec should not address this question, except to specify
how the author is to *communicate* this information. That is, through
the "main" module and "directories" fields, the author can tell the
package manager how his package is structured. Beyond that, it's just
a "folder" (or maybe some object, or collection of code, or a single
file with fields, or whatever) with a package.json at the "root"
(whatever that means). That's it.
> i.e. how do I specify configuration, etc.? Most people seem to agree
> that a package should be a directory with a 'package.json' file and
> its source code should be in a 'lib' directory.
Yes, but that a convention, and is specified per-package, not in the
spec. "lib" is just a sensible default.
> 2. How do I load a module from a package? i.e. if I want to load the
> 'bar' module from the 'foo' package, how do I describe that in code?
Again, this is something that must necessarily differ between
platforms. The author tells the package manager how his code is
structured, and the package manager decides how best to expose that
structure to the platform it's concerned with.
> 3. How are packages discovered, installed, managed, etc?
That's what package managers are *for*. Leaving this open lets them
act as proxies to the platforms upon which they run, so that package
author can be less bothered. The spec must not cover this.
> we need to start with something much much simpler. Like CommonJS
> modules...
They've had great success because that spec picks one discrete scope,
and solves just that problem, without specifying any implementation
details, like folder structures or installation procedures or module
resolution. It *just* outlined one specific software contract, and
left the rest alone.
That's what the packages.json spec does. It provides a way for a
package author to advertise how his package is structured, and what it
works with.
> 1. A CommonJS package is a directory with a 'package.json' file at its
> root. Any normal source code should appear in the 'lib' directory.
> Unit tests and other supporting files go outside of the lib directory.
Let the author put code wherever he pleases, so long as it's specified
correctly in the package.json file.
> 2. The package.json contains...
So, in a nutshell:
1. There's an arbitrary version number identifying the spec version
2. dependencies are identified by a url rather than a version range.
3. everything except "dependencies" will not be in the spec.
1 is not a good idea. Version numbers on specs encourage breaking
changes. They seem like they help, but they make it worse. Specs
should be simple enough that they don't need version numbers. And
plus, we'll never stop arguing over the caps thing, or what should go
in which version. Isn't that why Kris Kowal stopped calling things
"ratified" and instead we just say who supports what now? I like
that.
2 has interesting possibilities in some cases, but introduces really
big problems for npm and any other package manager that would try to
use a similar "install ahead of time, then get out of the way"
approach.
Again, this is covered in a *lot* of detail in the mappings thread,
and I suspect that we're actually close to a workable solution there.
Suffice it to say, it's not a clear simplification to replace a
version range with a URL.
> 3. require("foo/bar");
That's already how npm works, but not because the spec says so. This
is a platform-specific decision that does not belong in the packages
spec.
> My question is: can we break out our package spec with a level-0 like
> this one and then build on top of it?
The packages spec as it stands today is not much more than the
"level-0" already. In a nutshell, the packages spec is concerned with
giving authors and package managers a common vocabulary for
communicating the following bits of information:
1. What's it called?
2. What version is it?
3. How is it structured?
4. What does it depend on?
5. How do I install it?
6. Various meta info.
1, 2, and 6 are trivial, and solved, and super handy.
3 is not trivial, but is mostly resolved. (I don't hear anyone
arguing *against* the "main" field, and that seems to be the only
point remaining.)
4 is even less trivial, and about half-resolved. (Qv "mappings" thread.)
5 is not trivial, and pretty much impossible to specify in a general
way, but it is handled well by the "scripts" hash for systems where
that's relevant, on a case-by-case basis.
> In the 9 months that I've been a part of CommonJS, we haven't really
> converged on a common package format.
Tusk, Nodules, and npm (that I know of, at least; Ash, isn't something
in flusspferd using package.json, too?) are using package.json in
different ways, and with relatively little divergence in the meaning
of the various fields in that specification. Some of the finer points
are still being worked out, but that's normal.
If you think this is "not really converging" on a package spec, you
should've seen the 9 months prior! Incidentally, it's been about 9
months since I started writing npm, and in that time (especially the
last two months or so) I've seen a lot of people actually using the
package.json file to describe their projects in the nodejs community.
It works. It's converging. This is a very simple spec with real and
proven utility. It's easy to use, and versatile enough to handle a
lot of not-so-simple stuff that comes up. Throwing it out and
starting over is a Bad Idea.
--i
Ash Berlin
On 16 Jun 2010, at 23:25, Isaac Schlueter wrote:
> Ash, isn't something in flusspferd using package.json, too?)
Nope - but I've been writing a number of modules with packages.json
>
> It works. It's converging. This is a very simple spec with real and
> proven utility. It's easy to use, and versatile enough to handle a
> lot of not-so-simple stuff that comes up. Throwing it out and
> starting over is a Bad Idea.
^^ that.
> --i
Mikeal Rogers
> proven utility. It's easy to use, and versatile enough to handle a
> lot of not-so-simple stuff that comes up. Throwing it out and
> starting over is a Bad Idea.
To this point, I would wager that Packages is actually the second most
widely used specification from CommonJS (second to Modules of course).
The latest version flushed out a lot of issues and reduce there
requirements surface area a lot and once the mapping thread is
resolved we should probably stamp the latest rev.
-Mikeal
Charles Jolley
with a package.json but I have yet to find a package on github that is
not somehow dependent on a specific package manager and/or specific js
runtime.
have I missed something? Can anyone point me to packages that just
work everywhere? I would be happy to treat those as canonical.
-Charles
On Jun 16, 2010, at 4:20 PM, Mikeal Rogers <mikeal...@gmail.com>
wrote:
Charles Jolley
reference code from other packages. Or for code not in a package to
reference packages. Again have I missed something?
I have implemented (or attempted to plement) a package aware runtime
that complies with this spec several times in both the web browser and
command line environments. It takes an incredible amount of code (for
the browser) and actually I can't find any way to implement it that
makes packages I find on github just work.
What am I missing here. I just want to write packages and put them
out there and let anyone who wants to use them use them regardless of
package mgr or js runtime.
I also want to be able to load and run in the browser in the most fast
and efficient way possible. I happen to know a lot about this area
and I can say with confidence I have not seen a package aware system
for the browser that does this right yet.
So again, maybe I am wrong but my impression is that most people write
packages for their own pet platform and say they are compatible when
in fact they won't run on anything else.
-Charles
On Jun 16, 2010, at 4:20 PM, Mikeal Rogers <mikeal...@gmail.com>
wrote:
>> It works. It's converging. This is a very simple spec with real and
Kris Kowal
deploying packages to web browsers with this specification.
It is a separable concern. It's my impression we are
talking about how to share zip archives of code, assets, and
metadata in source form. There's a relationship between the
Modules proposal and the Modules/Transport proposal. I
think that there will eventually need to be a
Packages/Transport proposal, and I think that we'll have a
lot more luck with that than we have the Modules/Transport.
I do think it would be healthy to trim a little fat here and
there. I would not mind cutting the proposal down to zip
archive packages (there are presently provisions for
unarchived packages with individually retrieved files, which
may be onerous to some package managers). We could also add
provisions for more loosely referencing packages. In the
presence of a catalog/registry, a package name and optional
version would be sufficient for divining a URL.
Perhaps we need to make the descriptor more of a
feature-testable object. We could take the buffet approach,
where package providers can add progressively support a
wider variety of package managers.
{
"mappings": {
"foo": {
// narwhal, nodules, & al
"archive": "http://example.com/foo.zip",
// nodules, & al
"location": "http://example.com/foo",
// npm, & al
"name": "foo", // default to same as mapping
"version": "1.0.0"
}
}
}
We could mandate all fields, or make them feature-testable.
"name" and "version" presume the existence of a catalog. We
could require that the catalog URL be explicated at the
top-level. For the interim, we could just use the catalog
URL to verify that the catalog matches the catalog/registry
of the target package manager, but eventually we can specify
a CommonJS registry format with all the information to map a
name and version to a full descriptor. Or we could do that
now, but I think that's a separate spec.
In all cases, we mandate that require("foo") would grab the
"main" module and require("foo/bar") grabs the "bar" module
of the package, using "lib" by convention or
"directories.lib" by configuration. That much I think we've
already agreed, addressing Charles's #2, and is codified in
Mappings/B.
Summarily, I would encourage Charles and Iraliki to draft
Packages/Transport proposals based on their experience with
Tiki and NarwhalRunner, and get this specification narrowly
focused on distributing source archives so we can separate
the web browser and esoteric loader cases. Making it easy
to publish and consume packages with source code and assets
should be the scope of this proposal. We can build a huge
tool ecosystem around such a proposal.
Kris Kowal
Charles Jolley
But still my original question remains: I haven't seen any packages out there that actually work cross-package manager (except for maybe one or two simple libraries with no other dependencies). In my book this means that the packages spec has largely failed even if people are publishing with the format.
I think a simplification like what Kris describes below would actually work quite well.
It would support CommonJS runtimes to support packages; probably though a plugin format to locate packages from a package name.
-Charles
Charles Jolley
Kris Kowal
> But still my original question remains: I haven't seen any packages out there that actually work cross-package manager (except for maybe one or two simple libraries with no other dependencies).
I don't think there are enough compliant implementations (maybe 1 so
far, right?) of mappings yet to conclude that it hasn't worked. I
would agree that before mappings, the packages specification was not
sufficient to guarantee portability of packages across package
managers, but it did forestall variance in package layouts and
conventions that have assisted us in keeping a narrow scope to the
mappings amendment. It was a good foundation, and I think we were all
aware that it wasn't a finished project. When we stamped a version
number on Packages/1.0, I think that the consensus was that we needed
to do at least two more versions before it was prime-time. So, let's
keep going.
Kris Kowal
Charles Jolley
I will be adapting Tiki + Seed to use this in July.
-C
Irakli Gozalishvili
What do you think of this:
"dependencies" describes packages that current one happens to depend on. The value is a hash that contains package specific data about the dependency, what data is out of the scope of this specification only requirement for implementers is support for vendor specific prefixes for all the properties but with a higher priority.
This way some package managers will put there versions others urls and some might even put overlays. Prefixing will address cross platform packages issue. In example it will be:
"dependencies": {
"packageId": {
"seedjs": {
"src": "http://path/to/package",
}
"npm": {
"version":"0.1"
},
"narwhal": {
"location": "http://github.com/280north/jack/zipball/master",
"descriptor": {
"index": "lib/jack.js",
"directories": {
"lib": "lib/jack"
}
},
"verify": {
"signature": "c1:b1:30:29:d7:b8:de:6c:97:77:10:d7:46:41:63:87",
"algorithm": "sha1"
}
}
}
}
Once we'll agree on some properties like version, src, location etc we will standardize them in version 2.
Thoughts ?
--
Irakli Gozalishvili
Web: http://www.jeditoolkit.com/
Phone: +31 614 205275
Address: Taksteeg 3 - 4, 1012PB Amsterdam, Netherlands
Charles Jolley
Kris Kowal
> Thoughts ?
Having package-manager-specific semantics defeats the purpose of
converging on a specification. It does carve up the name space well,
and there certainly will need to be some custom properties, but we do
need to converge on some minimal set of properties so that package
providers can follow the specification instead of following O(n)
package manager implementations. Although the implementations vary,
it seems clear that there are certain properties that have common
semantic values; we should specify those so they don't need to be
explicated for every package manager that uses them.
Kris Kowal
Irakli Gozalishvili
--
Irakli Gozalishvili
Web: http://www.jeditoolkit.com/
Phone: +31 614 205275
Address: Taksteeg 3 - 4, 1012PB Amsterdam, Netherlands
Kris Kowal
Nathan White
I think this kind of progressive enhancement design would be awesome.
Christoph Dorn
> Perhaps we need to make the descriptor more of a
> feature-testable object. We could take the buffet approach,
> where package providers can add progressively support a
> wider variety of package managers.
>
> {
> "mappings": {
> "foo": {
> "archive": "http://example.com/foo.zip",
> // nodules,& al
> "location": "http://example.com/foo",
> // npm,& al
> "name": "foo", // default to same as mapping
> "version": "1.0.0"
> }
> }
> }
>
> We could mandate all fields, or make them feature-testable.
>
> "name" and "version" presume the existence of a catalog. We
> could require that the catalog URL be explicated at the
> top-level. For the interim, we could just use the catalog
> URL to verify that the catalog matches the catalog/registry
> of the target package manager, but eventually we can specify
> a CommonJS registry format with all the information to map a
> name and version to a full descriptor. Or we could do that
> now, but I think that's a separate spec.
I have been basing all my work on a feature-testable "package locator"
object which has turned out to be very handy. It allows you to reference
packages via catalogs during development and lock it to specific
versions during deployment for example.
It has been my assumption that we would have such a facility specified
in the spec as it has been floating around for a while and makes a lot
of sense.
It is also essential that the package.json spec contains properties that
have primarily object values instead of arrays to allow for overwriting
of most objects/properties and merging of arrays.
Christoph
Charles Jolley
Christoph Dorn
The following are examples from the PINF[1] project I am working on. I
will be refactoring things to comply with the new packages spec once it
stabilizes a bit more.
PINF keeps all packages in a "workspaces" folder where packages declare
their dependencies with:
package.json ~ {
"using": {
"domplate": {
"catalog":
"http://registry.pinf.org/cadorn.org/github/catalog.json",
"name": "domplate",
"revision": "master"
}
}
}
The catalog file referenced above
(http://registry.pinf.org/cadorn.org/github/catalog.json) is
automatically updated on the registry server via a github post-commit hook.
Locally PINF remaps catalog entries to workspace directories:
sources.json ~ {
"http://registry.pinf.org/cadorn.org/github/catalog.json": {
"domplate": {
"master": {
"@": {
"path": "/pinf/workspaces/github.com/cadorn/domplate"
}
}
}
}
}
During development the loader pulls packages from the mapped paths.
When building a release the dependencies are locked to specific versions:
package.json ~ {
"using": {
"domplate": {
"catalog":
"http://registry.pinf.org/cadorn.org/github/catalog.json",
"name": "domplate",
"revision": "0.0.0rev-538297080cad2571781963407f75da894b0e97a3"
}
}
}
or
package.json ~ {
"using": {
"domplate": {
"catalog":
"http://registry.pinf.org/cadorn.org/github/catalog.json",
"name": "domplate",
"version": "3.5.7"
}
}
}
In the case where a "version" is specified it is assumed that the latest
available 3.x release (based on semver) is used.
All this is completely automated and I can publish a project with:
pinf switch-workspace github.com/cadorn/<name>
pinf build-program --target distribution .
pinf publish-program --revision master .
Where the builders and publishers are pluggable CommonJS modules.
Christoph
[1] - http://github.com/cadorn/pinf
Isaac Schlueter
What you're suggesting is very similar to the "overlay" field. While
"overlay" does provide an escape-hatch for developers who bump into
incompatible package features, it's sort of a bad smell that we should
try to get away from, a mark of an unfinished spec. Also, it belongs
at the root, rather than nested inside another field.
I don't agree that a URL is necessarily better or more specific than a
version number. Ideally, both would be provided, because the thing
sitting at a URL can change, and you should have some kind of way to
check and see if it'll still be ok.
A URL is not necessarily a permanent or reliable identity. It is a
helpful hint. A "name" is not arbitrary; it's chosen by the package
author. The fact that the registry uses it as a key is secondary.
But we probably don't have to agree on any of that.
I think we CAN agree that it's A Good Thing to enable an author to
provide the name, expected version, a URL to the source, URL to an
archive, and desired location to expose the dependency, and let the
package manager deal with whatever it knows how to handle.
I am not particular about spelling. If it makes sense to call it
"mapping", or even make the data a tad more wordy for the sake of some
other benefit, then that's fine. I'd love to have a way to install
stuff that's not in the registry and not local. I want to make
developers' lives easier. I want interoperability. But as Ryan
quipped a while back, "Operability > Interoperability".
The Mappings proposal doesn't give npm what it needs. As it stands,
there is no way that I'm going to implement that. If package authors
want to use it, and also want their stuff installable by npm, then
they'll have to still provide a dependency hash. That's not ideal,
but that's where we're at. If there was *overwhelming* demand for
this in the nodejs community, I'd change npm to work that way, and
just accept the cost as a necessary overhead. It still probably
wouldn't happen for quite a while unless someone wanted to provide a
patch.
I think there is a way to do things so that it works for npm as well
as satisfying these other ideas and requirements, without giving
package authors any false expectations.
So, what's that look like?
--i
Dean Landolt
Irakli: Apologies, I saw your suggestion, but got sidetracked.
What you're suggesting is very similar to the "overlay" field. While
"overlay" does provide an escape-hatch for developers who bump into
incompatible package features, it's sort of a bad smell that we should
try to get away from, a mark of an unfinished spec. Also, it belongs
at the root, rather than nested inside another field.
I don't agree that a URL is necessarily better or more specific than a
version number. Ideally, both would be provided, because the thing
sitting at a URL can change, and you should have some kind of way to
check and see if it'll still be ok.
A URL is not necessarily a permanent or reliable identity. It is a
helpful hint.
A "name" is not arbitrary; it's chosen by the package
author. The fact that the registry uses it as a key is secondary.
But we probably don't have to agree on any of that.
I think we CAN agree that it's A Good Thing to enable an author to
provide the name, expected version, a URL to the source, URL to an
archive, and desired location to expose the dependency, and let the
package manager deal with whatever it knows how to handle.
I am not particular about spelling. If it makes sense to call it
"mapping", or even make the data a tad more wordy for the sake of some
other benefit, then that's fine. I'd love to have a way to install
stuff that's not in the registry and not local. I want to make
developers' lives easier. I want interoperability. But as Ryan
quipped a while back, "Operability > Interoperability".
The Mappings proposal doesn't give npm what it needs. As it stands,
there is no way that I'm going to implement that. If package authors
want to use it, and also want their stuff installable by npm, then
they'll have to still provide a dependency hash. That's not ideal,
but that's where we're at. If there was *overwhelming* demand for
this in the nodejs community, I'd change npm to work that way, and
just accept the cost as a necessary overhead. It still probably
wouldn't happen for quite a while unless someone wanted to provide a
patch.
I think there is a way to do things so that it works for npm as well
as satisfying these other ideas and requirements, without giving
package authors any false expectations.
So, what's that look like?
Christoph Dorn
> On Thu, Jun 17, 2010 at 10:14 PM, Isaac Schlueter <i...@izs.me
> version number. Ideally, both would be provided, because the thing
> sitting at a URL can change, and you should have some kind of way to
> check and see if it'll still be ok.
>
> A URL is not necessarily a permanent or reliable identity. It is a
> helpful hint.
>
> this combination is the /only/ thing in this world (wide web) that
> /could/ be considered permanent. So if your semver is part of your URL
> and the response ETag is it's hash you're now dealing with a /permanent,
> reliable/ identity. I know noone wants to marry the package spec to HTTP
> -- I'm just trying to point out how URLs /can/ be used unambiguously.
I believe using hostnames + paths (URLs) is the best approach for a
distributed package system.
hostnames go away when someone does not maintain their domain
registration but that does not mean the resources attached to it need to
be lost.
I believe it is sufficient for someone to claim a namespace via a
hostname by having access to it at one point in time. If you are serious
about a hostname you are going to maintain it longterm. If you let a
hostname with resources attached lapse someone else should not be able
to take it over without your permission (or the permission of the
community using the resources).
All this can be done via publicly hosted registry servers that track
hostnames + paths similar to how DNS works now.
For example, I put all my code under cadorn.org which I have registered
with the PINF registry server [1]. My namespace URL is:
http://registry.pinf.org/cadorn.org/
I have asked registry.pinf.org to be my primary registry server where
other registry servers may mirror my entries. That means going to a
mirror will yield the same results:
http://reg.mirror.org/cadorn.org/
It is important to note that the purpose of the hostname is to verify
ownership [2] and *imply* an "owner" but does not require the resources
to be hosted at the same paths on the actual domain. This works similar
to how java packages are namespaced using the reverse hostname dot notation.
Say you have a bunch of packages on github you want to make available
via the registry.
Your packages:
If github.com is a participant of the registry system your packages
would show up at:
http://registry.pinf.org/github.com/cadorn/
If you choose to control your own namespace you could place your
packages at:
http://registry.pinf.org/cadorn.org/github/
When referencing packages/resources, instead of specifying the full URL
including HTTP you simply start with the hostname desginating your
namespace:
github.com/cadorn/...
cadorn.org/github/...
The package manager resolves the "ID" by contacting a hierarchy of
registry servers similar to how DNS works until it finds the
authorative/primary server or a mirror where it finds something like:
http://registry.pinf.org/cadorn.org/github/fireconsole/packages/catalog.json
The "catalog" is auto-generated based on package.json files and
knowledge of source control information provided by post-commit hooks.
I have an overview of this design here [3] and am using it with good
success so far in practice.
We are some time away from having this work in a CommonJS specified
manner needing an evolved package and catalog spec but I think it is
time to agree or disagree on using URL's as a resource identifier for
packages.
Christoph
[1] - http://www.pinf.org/
[2] - http://github.com/cadorn/pinf/blob/master/docs/Registry.md
[3] - http://github.com/cadorn/pinf/blob/master/docs/Design/Foundation.md
Isaac Schlueter
> Ah but a URL + ETag (or Last-Modified) is. In fact, I'd argue that this
> combination is the only thing in this world (wide web) that could be
> considered permanent.
So, I'd need to get the ETags of all my dependencies? Yikes.
What if the archive is hosted on a CDN with multiple origin servers
running Apache, which generates ETags based on the inode, so identical
files at identical URLs can get different ETags?
ETags are great for a lot of things, but they're generally not fit for
human consumption. Remember, package.json files are for humans, too.
>> I think we CAN agree that it's A Good Thing to enable an author to
>> provide the name, expected version, a URL to the source, URL to an
>> archive, and desired location to expose the dependency, and let the
>> package manager deal with whatever it knows how to handle.
>
> That would be a world I'd be very happy to live in.
Clarification: I'd assume that package authors would provide some of
that information, not likely all of it, unless they're really into
that sorta thing (which of course some are).
>> So, what's that look like?
> I wish I knew, but I do believe that name:version and URL:ETag could live
> harmoniously.
It would be good to refresh the Mappings proposal to specifically
address the goal of conveying these different kinds of information in
a way that makes it easy for developers to give different systems what
they need to be successful. It's not a trivial problem.
--i
Mikeal Rogers
norms. And even we did define these semantics we're far too dependent
on HTTP implementations being good and complete which very few are.
People really love to fuck this stuff up, I've never used an HTTP
server or caching proxy that didn't break this in some odd and
unexpected way. This stuff all sounds simple but there a lot of wild
edge cases that you don't understand until you come across them.
-1
-Mikeal
Ryan Riley
what about using a URN instead of a URL and sticking it I an Atom feed? Then
you could reference the URN of the package with the URN or date of the Atom
feed. Linking from your feed would then lead wherever you like.
~ Ryan Riley