Preview: npm, the node package manager

[Isaac Z. Schlueter]

I think node needs a package manager. There are a lot of very useful
modules out there, but it's tricky right now to actually use more than
one of them together.

Here's a proposal for a very lightweight and simple way to alleviate
the situation. I'm calling it npm, and it should be able to install
itself fairly soon. :) In fact, calling this a "preview" is a bit
disingenuous, as it's rather heavy on "pre" and not so much "view"
just yet.


Goals

1. npm should be able to pull in all or most of the current node
projects that are on github today, by leveraging what is common, and
exposing it in a consistent way, without requiring that those projects
change to accommodate npm's needs.

2. npm should make it easy to use any package it can install with a
predictable and consistent inclusion method. No matter how the
package is set up, this should work:

$ npm install some-node-package
(then in my code)
var thing = require("/some-node-package.js");

3. npm should require a minimum of metadata in order to pull in code.
The more effort it is to create a package, the less likely that people
will do it. Small interfaces tend to be both more stable, and easier
to grok.

4. The npm package catalog should be something that we can all
contribute to, without any one person having to manage or know about
all the different packages out there. The package catalog should be
separate from npm, and both should be separate from node, so that they
can grow and change without being necessarily tied to one another.
Some degree of intermingling is inevitable, but it should be
minimized, not encouraged.

5. You should be able to tell npm to look at a local dataset, or
override the "public" catalog's version of your project with a local
development version.

6. Code is more valuable than words. Get something small working asap,
with minimal features, and then start adding to it as necessary. So,
at first, that means no support for: versions, checksums, per-package
settings, configurable installation locations, and so on. Any non-
obvious bikeshedable questions should be postponed if at all
possible. (Fairly soon, version numbers will be an issue. Please
read on.)

7. npm should encourage good practices. Currently, if your project
uses ejs, you have to include your own ejs.js script in your project.
That means that a bug we find someday in ejs will have to be fixed in
multiple places. It'd be much better to have a single "ejs" package
that npm knows about, make your project require it, and we can all
share. (This is why not having version numbers will suck soon.
Changing something like ejs could easily break things that depend on
it.)


What do you think about these goals? Do they seem valid? Does the
feature set I've sketched out seem like enough to be useful? Too big
for a first release?

http://github.com/isaacs/npm-data is the project on github with the
package data. I tried to get all the projects that are linked in the
node documentation, but I'm sure I missed several of them. I didn't
get anyone's author/description/etc. info included, so if you maintain
a node project, please feel free to fork/update/pullreq, and I'll get
your info included. There's not much in it for you just yet, but if
people show interest, that'll provide a lot of motivation to get npm
over the finish line.

The npm-data readme describes how the data works right now. It's very
larval, containing just the bare minimum for now. (See goal #6
above.) How do you think it should grow? Specifically, it'd be good
to get some thinking around how to express version-related things like
this in an elegant manner:

A) different versions of a package.
B) package foo requires versions 1.* through 2.* of package bar.
C) package foo requires at least version 1.0.2 of package bar.
D) foo-0.1.12 is "stable". foo-0.1.15 is "beta".

The metadata should describe the package, and this should in turn
guide npm's api.


Please let me know what you think. Talk about features you'd like to
see is great, but it's less valuable to me than figuring out a way to
structure npm so that it can be expanded in the future without
cluttering the API.

It's very early, so now's the time to provide input on what you'd like
it to be, if you have opinions.


--i

[Ryan Dahl]

On Thu, Oct 1, 2009 at 12:14 AM, Isaac Z. Schlueter <i...@foohack.com> wrote:
>
> I think node needs a package manager.  There are a lot of very useful
> modules out there, but it's tricky right now to actually use more than
> one of them together.

Great - I think so too. I'm glad you're working on this.

> 1. npm should be able to pull in all or most of the current node
> projects that are on github today, by leveraging what is common, and
> exposing it in a consistent way, without requiring that those projects
> change to accommodate npm's needs.

You don't need to worry too much about legacy code. Everyone here
knows that this is developmental and the API is fluid - at least in
the near future. If making some requirement on "packages" makes sense,
then do it :-)

> 2. npm should make it easy to use any package it can install with a
> predictable and consistent inclusion method.  No matter how the
> package is set up, this should work:
>
> $ npm install some-node-package
> (then in my code)
> var thing = require("/some-node-package.js");

What about namespace collisions?

> 3. npm should require a minimum of metadata in order to pull in code.
> The more effort it is to create a package, the less likely that people
> will do it.  Small interfaces tend to be both more stable, and easier
> to grok.

Agreed, as long as it has room to grow.

> 4. The npm package catalog should be something that we can all
> contribute to, without any one person having to manage or know about
> all the different packages out there.  The package catalog should be
> separate from npm, and both should be separate from node, so that they
> can grow and change without being necessarily tied to one another.
> Some degree of intermingling is inevitable, but it should be
> minimized, not encouraged.

I would like to work towards a future where there is a specialized
authorization web app / REST interface to the catalog. Perhaps even
multiple concurrent catalogs. It would be nice if access to your flat
JSON catalog was abstracted in a way that the flat file database could
be replaced with some specialized internet database but not affect the
code. In particular, assumptions about having the entire catalog
locally stored should be avoided.

> 7. npm should encourage good practices.  Currently, if your project
> uses ejs, you have to include your own ejs.js script in your project.
> That means that a bug we find someday in ejs will have to be fixed in
> multiple places.  It'd be much better to have a single "ejs" package
> that npm knows about, make your project require it, and we can all
> share.  (This is why not having version numbers will suck soon.
> Changing something like ejs could easily break things that depend on
> it.)

Dealing with namespace collision should be a main design goal. I'm not
sure of the best way to handle this.

> Please let me know what you think.  Talk about features you'd like to
> see is great, but it's less valuable to me than figuring out a way to
> structure npm so that it can be expanded in the future without
> cluttering the API.

In addition to the above comments, I hope the system does not get too
heavily tied to git or github.

May I recommend that packages need not be loaded by require(), perhaps
you can just add a new function: requirePackage("ejs"). Then require()
and include() can be used specifically for loading files and not for
loading entire software. This will allow you more freedom.

I imagine it going like this:

include("/npm.js");
ejs = requirePackage("ejs", "0.10.2+")
ejs.parse(x)

[Urban Hafner]

Isaac Z. Schlueter wrote:
> I think node needs a package manager. There are a lot of very useful
> modules out there, but it's tricky right now to actually use more than
> one of them together.
>
> Here's a proposal for a very lightweight and simple way to alleviate
> the situation. I'm calling it npm, and it should be able to install
> itself fairly soon. :) In fact, calling this a "preview" is a bit
> disingenuous, as it's rather heavy on "pre" and not so much "view"
> just yet.

Even though I don't have the time right now to really look at this, I
think it's really great that you started it! We really need something
like this :)

Urban

[Joshaven Potter]

I have a few recommendations:

1) I would like to see a package registry, something like DNS.  I think the registry should not actually host the package but rather point to package regardless of where it is hosted: github, or other public or private servers.  Furthermore I think that the registry should be distributed across multiple servers. (I have the resource to host a registry server if a server is needed.)

2) Packages should be simple.  I don't like the way rubygems are packed nor do I like having to study for days before I can fully understand the workings of a package.  Packages should be a folder that contains a standard layout of files like ./bin ./lib ./src etc.  The packages when loaded can look for ./lib/init.js (or something of the like) which is a file that will install, include, require, etc.  

3) Packages will also need some type of local registry.  They could be located by looking at a directory tree or through some type of registry application.  A lot of thought should be put into package version management & lookup.  I would be very happy if node packages call something like npm.locate('cool_package') which could use the default locate method unless I overwrite/extend the method inside my application.  This way I could use: yaml, mysql, sqlite3 or a directory tree, etc.

4) I would also really like to see a way of having signed certificates accompany packages.  

I am in the process of making a simple method of managing package certificates with rubygems.  I am expecting that my solution will not be only for rubygems but rather for any package.  I am the author of a not-yet-public package certificate authority which will be located at: http://trustca.org/. If node packages end up supporting ssl certs then I will be very happy to have my root ca distributed with the node package manager client and then any packages signed through trustca.org will be automatically trusted. TrustCA.org will be a FREE and simple certificate authority which targets secure package distribution.

--
Sincerely,
Joshaven Potter

"No man making a profession of faith ought to sin, nor one possessed of love to hate his brother. For He that said, “Thou shalt love the Lord thy God,”  said also, “and thy neighbor as thyself.”  Those that profess themselves to be Christ’s are known not only by what they say, but by what they practice. “For the tree is known by its fruit.”" -- Ignatius

[Isaac Z. Schlueter]

Thanks for the feedback.

On Sep 30, 3:46 pm, Ryan Dahl <coldredle...@gmail.com> wrote:
> You don't need to worry too much about legacy code. Everyone here
> knows that this is developmental and the API is fluid - at least in
> the near future. If making some requirement on "packages" makes sense,
> then do it :-)

Sure. So far it hasn't been necessary. Most node projects have quite
a bit in common. I'd like to try to build npm without making any
requirements on package maintainers, and only start making requests of
them if that effort fails or is overly difficult for some reason.

Obviously, if they do something like node.http instead of require("/
http.js"), then npm isn't going to step in and make that work.

> I would like to work towards a future where there is a specialized
> authorization web app / REST interface to the catalog. Perhaps even
> multiple concurrent catalogs. It would be nice if access to your flat
> JSON catalog was abstracted in a way that the flat file database could
> be replaced with some specialized internet database but not affect the
> code. In particular, assumptions about having the entire catalog
> locally stored should be avoided.

A flat JSON file at a URL is a pretty good analog for a "specialized
internet database" (since many expose a GETable JSON interface, after
all), and is easy to get started with. Storing the whole thing
locally is a step towards storing just the parts that you care about.
While the catalog is small, it'll work, but you're right, it doesn't
really scale as well as I'd like. At the very least, I can see that
I'll almost certainly have to break the catalog.json into multiple
smaller per-package json files.

In any event, I agree that this is a definite area for improvement.
Joshaven Porter has offered to throw some server resources at the
problem, and I think we'll come to something that works.


> What about namespace collisions?
> ...

> Dealing with namespace collision should be a main design goal. I'm not
> sure of the best way to handle this.

+1. Namespace management is the biggest value that npm should add to
the equation.

I'm handling it by requiring that every package has a unique name. If
you have package "foo" that looks like this:

src/
├── foo.js
├── include.js
└── root.js

and a package "bar" that looks like this:

src/
├── bar.js
├── foo.js
└── root.js

then npm would create something like this:

.node_libraries/
├── bar/
│ └── src/
│ ├── bar.js
│ ├── foo.js
│ └── root.js
└── foo/
└── src/
├── foo.js
├── include.js
└── root.js

My initial thought was to have npm write a foo.js and bar.js file
there with some logic inside to pull in the appropriate library and
expose it. But that may turn out to be a bad approach, we'll see.

So, the "foo" dir, and "foo.js" file, would both be managed by npm,
and based on the package name. As long as there are no two packages
with the same name, we should be safe. With version numbers, perhaps
there'd be a foo-0.1.2 and a foo-0.1.3, with a bit of symlink magic to
select the appropriate folder.

> May I recommend that packages need not be loaded by require(), perhaps
> you can just add a new function: requirePackage("ejs"). Then require()
> and include() can be used specifically for loading files and not for
> loading entire software. This will allow you more freedom.

Having an advanced require/include with a version check is a neat
idea, and the extra freedom might turn out to be very useful.

In any event, I'm looking forward to playing with the options. The
advantage of just putting a foo.js somewhere that can do the right
thing is that other code wouldn't necessarily need to know that foo.js
is put there by npm, whereas npm.requirePackage("foo") would mean that
the consumer code is tied to npm. Maybe that's not such a bad thing.

This mechanism of putting code in a folder based on package name also
leaves the door open for grouping packages by category, like "db/
couch" and "db/postgres".

> In addition to the above comments, I hope the system does not get too
> heavily tied to git or github.

I agree. That's why I put a url to a tarball in the catalog, not a
url to a git project. (The fact that the tarballs are on github is
just a coincidence.) The package manager shouldn't require anything
more than node, tar, and gzip. After all, WE might all use github,
but if I write a useful utility, I don't necessarily want my users to
have to be git experts.

That's not to say that it'd be terrible for npm-data to eventually use
git under the hood if it makes sense, since that gives us tracking,
security, easy distributability, and accounability. But the client
itself shouldn't be too tied to git, and package maintainers shouldn't
be forced to use git.

On Oct 1, 10:07 am, Joshaven Potter <yourt...@gmail.com> wrote:

> 1) I would like to see a package registry, something like DNS.

The registry does currently just host a list of metadata, no actual
package contents. (Part of the metadata is a link to a tarball hosted
somewhere else.)

I'm not sure how "like DNS" you're suggesting. It's not very DNS-like
right now ^_^

> 2) Packages should be simple.

Agreed. I'm actually not even requiring that they have any kind of
lib/init.js. Each package can have its own internal structure, so
long as the entry point is specified in the metadata.

> 3) Packages will also need some type of local registry.

Currently that's done by downloading the entire catalog, which is a
Bad Idea (see above).

But yes, local caching is important. I'd prefer to just put json
files in a folder, as sqlite/etc would certainly be overkill in this
case.

> 4) I would also really like to see a way of having signed certificates
> accompany packages.

post-beta. But yes, important. Eventually, if npm is successful, and
grows, then it will be the target of attacks.


--i

[Felix Geisendörfer]

Just wanted to try NPM but had trouble. Below is my output. I don't
think it's a permissions issue as ~/.npm/ gets created and so does
~/.npm/optparse-staging

Any ideas?

./install.js
npm: bootstrapping
npm: Fetching package data from http://github.com/isaacs/npm-data/raw/master/catalog/npm.json
npm: adding: npm
npm: Fetching package data from http://github.com/isaacs/npm-data/raw/master/catalog/optparse.json
npm: adding: optparse

npm: install: optparse
npm: fetch: http://github.com/jfd/optparse-js/tarball/master
npm: fetch: http://waitdownload.github.com/jfd-optparse-js-915cf2f.tar.gz
npm: fetch: http://download.github.com/jfd-optparse-js-915cf2f.tar.gz
npm: failure: step: cd /Users/felix/.npm/optparse
npm: failure: /usr/bin/cd: line 4: cd: /Users/felix/.npm/optparse: No
such file or directory

npm bootstrap failed: Failed installing npm with npmnpm: failure:
Failed to install
npm: failure: Failed to install package set. @TODO: rollback
npm: failure: Failed to unpack optparse

> http://github.com/isaacs/npm-datais the project on github with the

[Felix Geisendörfer]

The patch below seems to fix it.Does anybody know how to upload files
to an existing conversation? Can I just put "Re: <thread>" in the my
message subject and google groups will "get it"?

From ea78d55cb6bb149b10a222d3989f65813f247351 Mon Sep 17 00:00:00 2001
From: =?utf-8?q?Felix=20Geisend=C3=B6rfer?= <fe...@debuggable.com>
Date: Wed, 7 Oct 2009 17:09:09 +0200
Subject: [PATCH] Fixing install.js for OSX

---
npm.js | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/npm.js b/npm.js
index f6eed7b..f5c80f8 100755
--- a/npm.js
+++ b/npm.js
@@ -149,6 +149,7 @@ function unpack (name, data, opt) {
"rm -rf "+targetFolder,
"mkdir -p "+staging,
"cd "+staging + " && tar -xf "+target,
+ "mkdir "+targetFolder,
"cd "+staging + " && mv * "+targetFolder,
"cd " + targetFolder,
"rm -rf "+staging,
--
1.6.3.3

On Oct 7, 4:29 pm, Felix Geisendörfer <fe...@debuggable.com> wrote:
> Just wanted to try NPM but had trouble. Below is my output. I don't
> think it's a permissions issue as ~/.npm/ gets created and so does
> ~/.npm/optparse-staging
>
> Any ideas?
>
> ./install.js
> npm: bootstrapping

> npm: Fetching package data fromhttp://github.com/isaacs/npm-data/raw/master/catalog/npm.json
> npm: adding: npm
> npm: Fetching package data fromhttp://github.com/isaacs/npm-data/raw/master/catalog/optparse.json

> >http://github.com/isaacs/npm-dataisthe project on github with the

[Felix Geisendörfer]

Ok, forget my stupid patch. As we figured out in IRC the problem came
into life by utils.exec() not invoking sh anymore and thus loosing all
of it's nice features.

Ryan's commit e46a83 fixes that and the installer is working great for
me again.

-- Felix Geisendörfer aka the_undefined

> > >http://github.com/isaacs/npm-dataistheproject on github with the