Default package manager

[816...@gmail.com]

There are some good package managers at github, but in my opinion
there are should be only one, main, which will be maintained with
node, and has an standard interface, like:

node install haml

or:

node-install haml

This would make node more friendly because if some beginner will try
to make something like:

var h = require('haml')

he, will get:

Error: Cannot find module 'haml'

but should get smth like:

Error: Cannot find module 'haml', try 'node-install haml'

also default package manager will provide updating of node, and single
standard of package description for modules.

I think we should design an console interface for package manager, and
select the best one, to include into node.

[Ryan Dahl]

On Thu, Mar 11, 2010 at 12:32 PM, ol...@emby.ru <816...@gmail.com> wrote:
> There are some good package managers at github, but in my opinion
> there are should be only one, main, which will be maintained with
> node, and has an standard interface, like:

Maybe there will be in the future - it's much too early now. I'm happy
to have different people, with different ideas working on it.

[Rasmus Andersson]

I'll take this opportunity to discuss a little around what such a
program should do.

In my opinion a module manager should manage node modules. Not a
package (like Debian Aptitude or Redhat Package Manager). The reason
being node modules are simply node modules. They do (should) not
install files anywhere on your system.

A node module should:

- Only add one new branch in the tree of modules (i.e. only install
one "root" module).
- This is wrong: require("foo"); require("foo_utils")
- This is correct: require("foo"); require("foo/utils")

- Not use hungarian notation. The name should reflect what the module
is, given the environment of node modules (not all software in the
world).
- This is wrong: "node-foo", "js-foo"
- This is correct: "foo"

- Not modify the host system at install-time (e.g. add configuration files etc)

I have been given this a lot of time and my interpretation of how a
module manager for node should work is mode[1]:

a. Distributed. Let's face it, this community is way too speedy for us
to be able to keep a monolithic repository of all modules. They are
already semantically structured on GitHub (and elsewhere on the web)
along with the implied versioning (and thus dependencies) of git.

b. Client-local only. Should not use a server model because of a.

c. Simple to use. Should have a set of simple commands: search,
install, activate, deactivate, uninstall, list, update. Should
gracefully take care of displaying error messages and shall not be
able to produce a broken state (i.e. each step in the process of
installation should be atomic and idempotent).

d. Should be written in node and not have any dependencies (on node
modules, obviously).

There are currently three different module and/or package managers
being developed[2]. I agree this is a good thing — because we're
acting in a totally free "software economy" the most suiting manager
will probably take more ground that the others (well, you could do
some fancy advertising or Ruby-community-like rallies, but let's
imagine a beautiful even world :P).

[1] http://github.com/rsms/mode
[2] http://wiki.github.com/ry/node/modules#package-management

[Michael J. I. Jackson]

Coupling the package manager with any kind of SCM doesn't make sense. Like you said, the whole thing should be written in node itself, and coupling the thing with Git or Mercurial or Subversion essentially means rewriting large portions of those systems in JavaScript so the package manager can use them natively. Otherwise, you're issuing exec commands to the shell and hoping the client has git installed.

A much more sensible solution would be to create a small "tar.js" (or just use zip.js) module, vendor it (that is, distribute it with your package manager, much like node vendors v8), and use that to create node "packages". Then, instruct the user to stick the resulting package somewhere accessible via HTTP (the project "downloads" page, presumably, but not necessarily) and let node handle the actual package retrieval. RubyGems uses this model (.gem files are just tarballs with a special informational file at the beginning IIRC), and it works well.

Michael

--
Michael J. I. Jackson
http://mjijackson.com/
@mjijackson

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Rasmus Andersson]

On Sat, Mar 13, 2010 at 15:31, Michael J. I. Jackson
<mjija...@gmail.com> wrote:
> Coupling the package manager with any kind of SCM doesn't make sense. Like
> you said, the whole thing should be written in node itself, and coupling the
> thing with Git or Mercurial or Subversion essentially means rewriting large
> portions of those systems in JavaScript so the package manager can use them
> natively. Otherwise, you're issuing exec commands to the shell and hoping
> the client has git installed.

Being pragmatic, node currently changes so rapidly I don't see the
point in making something so simple as copying a js file (that's what
"installing a node module" means in most cases) into a complex task of
packaging, distributing (uploading somewhere) etc. I'm walking out on
a branch here and say people who are installing node modules have
access to git.

> A much more sensible solution would be to create a small "tar.js" (or just
> use zip.js) module, vendor it (that is, distribute it with your package
> manager, much like node vendors v8), and use that to create node "packages".

While at ti, "vendor" is a word which started to be mis-used by Ruby
on Rails. "Vendor" means a reseller. I recommend avoiding that word in
this context as. IMHO "bundle" is a more descriptive and correct word.

> Then, instruct the user to stick the resulting package somewhere accessible
> via HTTP (the project "downloads" page, presumably, but not necessarily) and
> let node handle the actual package retrieval. RubyGems uses this model (.gem
> files are just tarballs with a special informational file at the beginning
> IIRC), and it works well.

Agreed this model works (pretty) well, works OK, for Ruby. But node is
not Ruby. What applies to X might not apply directly to Y just because
they are both upper case characters.

--
Rasmus Andersson

[sste...@gmail.com]

On Mar 13, 2010, at 9:56 AM, Rasmus Andersson wrote:

> Otherwise, you're issuing exec commands to the shell and hoping
>>
>> the client has git installed.

If the user wants to use git based repositories, they have to have git installed.

I don't see any problem with that.

I think packages should be able to be installed in any type of storage.

Let's not make a big deal about writing a simple shim to retrieve them.

S

[Kris Kowal]

On Sat, Mar 13, 2010 at 7:29 AM, sste...@gmail.com
<sste...@gmail.com> wrote:
> If the user wants to use git based repositories, they have to have git installed.

This is not necessarily true. For the case of Github hosted Git
repositories, you can download a tarball or zip archive of any ref of
a repository. Tom just finished a pure-js Zip archiving package.
Case and point:

http://github.com/tlrobinson/zipjs/zipball/v0.1

Kris Kowal

[Stephen Belanger]

I'm not in disagreement on any particular points in this thread, however; I would recommend that for modules that depend on external software such as db wrappers, a configurable message should be available to spit out after installing the module to make sure the user is aware that only the module was installed, and not the db system itself.

On Sat, Mar 13, 2010 at 7:29 AM, sste...@gmail.com <sste...@gmail.com> wrote:

--

[sste...@gmail.com]

On Mar 13, 2010, at 3:45 PM, Kris Kowal wrote:

> On Sat, Mar 13, 2010 at 7:29 AM, sste...@gmail.com
> <sste...@gmail.com> wrote:
>> If the user wants to use git based repositories, they have to have git installed.
> This is not necessarily true.

I understand that. The point was that if a user wants to use a particular back-end, it's not unreasonable to expect them to have the tools to use it. Especially with developer tools. Installing git is just not that big a deal.

> For the case of Github hosted Git
> repositories, you can download a tarball or zip archive of any ref of
> a repository. Tom just finished a pure-js Zip archiving package.
> Case and point:
>
> http://github.com/tlrobinson/zipjs/zipball/v0.1

Cool.

S

[Michael J. I. Jackson]

You're right, it's not a big deal. Nobody ever said it was a big deal. It's just bad design.

--
Michael J. I. Jackson
http://mjijackson.com/
@mjijackson

S

[Isaac Schlueter]

I agree with a lot of what you have to say here.

On Sat, Mar 13, 2010 at 03:54, Rasmus Andersson <ras...@notion.se> wrote:
> In my opinion a module manager should manage node modules. Not a
> package (like Debian Aptitude or Redhat Package Manager). The reason
> being node modules are simply node modules. They do (should) not
> install files anywhere on your system.

I'm not convinced of that. For instance, quite a few node packages
may in fact want to add a command-line program. That command line
program must go somewhere, right?

npm addresses this by allowing you to set up a hash of command-name to
file-name in the "bin" object in your package.json file. If it's a
node module (ie, if it ends in .node or .js, or shebangs to node),
then a shim is created (so that relative includes will work as
expected in that file.) If it's something else, then it's symlinked
(so other kinds of executables can be installed as well.)

> - Only add one new branch in the tree of modules (i.e. only install
> one "root" module).
>  - This is wrong: require("foo"); require("foo_utils")
>  - This is correct: require("foo"); require("foo/utils")

+1!!

npm links the "lib" directory as the package name, and creates a
single entry-point shim for the "main" field.

In the case above, you might do something like this:

{ "directories" : {"lib":"lib"}, "main" : "lib/foo"}

and your folder structure might look like this:

lib/
foo.js
utils.js

This prevents an explosion of require.paths, and also makes it easy to
just require a single file.

>  - This is wrong: "node-foo", "js-foo"
>  - This is correct: "foo"

Agreed. The name on github can be something like node-foo, but the
"name" field in the package.json should assume that it's js and node,
etc.

> - Not modify the host system at install-time (e.g. add configuration files etc)

I don't necessarily agree. Installation is necessarily a change, and
some things might require a configuration file to work properly.
However, such changes should ideally be specified to the package
manager and done by the package manager itself, and should be
reversible on uninstallation. The "scripts" field in the package.json
is a last resort, in my opinion, and should be avoided.

> a. Distributed. Let's face it, this community is way too speedy for us
> to be able to keep a monolithic repository of all modules. They are
> already semantically structured on GitHub (and elsewhere on the web)
> along with the implied versioning (and thus dependencies) of git.

+1. In fact, even the "index" in mode is, imo, opening the door for a
lot of unnecessary work for you. I'm working with Mikeal Rogers on a
package registry that will allow users to publish and tag their
projects without anyone else being involved.

If you'd like to take a look, it's at
http://github.com/mikeal/js-registry. If you'd like to have mode use
it, that could be cool. Let me know if you're interested, and if
there are any changes that would make it more amenable to mode's use
cases. My goal is to have a way for people to publish their stuff
with a package.json file, and have it Just Work no matter which
package manager their users are using. If a package.json contains a
"repositories" hash, then that should give you what you need for mode,
right?

> b. Client-local only. Should not use a server model because of a.

I'm not sure what you mean by this. Do you mean that using modules
shouldn't depend on a server, or that installing them shouldn't depend
on a server?

> c. Simple to use.

+1. npm really needs more work in this area, but I've been focusing
on core functionality over prettiness. Fairly soon, prettiness will
be a core functionality issue, however ;)

> d. Should be written in node and not have any dependencies (on node
> modules, obviously).

Agreed. Although, I don't think it's unreasonable to depend on tar
and gzip. At some point, it may make sense to implement those things
in js and include them with npm, but maybe not.

> I agree this is a good thing — because we're
> acting in a totally free "software economy" the most suiting manager
> will probably take more ground that the others (well, you could do
> some fancy advertising or Ruby-community-like rallies, but let's
> imagine a beautiful even world :P).

+1. The choice to not put "the" in npm's description was very intentional.

I'd add a few things to this list:

e. Should "play nice" with other modules that may already be
installed. So far, it looks like mode and kiwi play nice enough with
npm, but we could make it a little sweeter. We're not Microsoft and
Apple and Adobe here, there's no need to try to increase the switching
cost ;) (It seems like mode is a bit of an autocracy. Any chance of
letting me link .node_libraries to mode/active rather than the other
way around?)

f. Should not be involved at the module require() stage. While I'm
all for alternative module loaders, imo the package manager should be
as light as possible, and bundling the two together is an improper
muddling of concerns. Clearly this is not a unanimous opinion, qv
kiwi.require().

g. Should allow dependency specification, and allow for resolving
dependencies without ending up in "dependency hell".

h. Should not be coupled to a specific SCM. Again, not unanimous (qv
mode), but it seems to me that tar and gzip are a bit more universally
supported than git.

i. Should not be coupled to a specific source control community. It'd
be nice to support hg+BitBucket or svn+googlecode, or even
cvs+sourceforge, or any random host where code can be downloaded from.
Popular projects may want to host their code on s3 or something for
greater reliability.

--i

[Mikeal Rogers]

>> a. Distributed.  Let's face it, this community is way too speedy for us
>> to be able to keep a monolithic repository of all modules. They are
>> already semantically structured on GitHub (and elsewhere on the web)
>> along with the implied versioning (and thus dependencies) of git.
>
> +1.  In fact, even the "index" in mode is, imo, opening the door for a
> lot of unnecessary work for you.  I'm working with Mikeal Rogers on a
> package registry that will allow users to publish and tag their
> projects without anyone else being involved.
>
> If you'd like to take a look, it's at
> http://github.com/mikeal/js-registry.  If you'd like to have mode use
> it, that could be cool.  Let me know if you're interested, and if
> there are any changes that would make it more amenable to mode's use
> cases.  My goal is to have a way for people to publish their stuff
> with a package.json file, and have it Just Work no matter which
> package manager their users are using.  If a package.json contains a
> "repositories" hash, then that should give you what you need for mode,
> right?
>

A few notes on js-registry. It's a couchapp (all the code and
semantics are in a design document that goes in to any CouchDB).

This means that all the data can be distributed and continuously
synced using replication. Anyone who wants to keep an intranet mirror
can use continuous replication. Anyone who wants to keep a private
registry on their intranet can take the code and stick it on an
intranet CouchDB.

One of the goals was to make it easily distributable and CouchDB
replication solves that.

-Mikeal

[Joe Developer]

my 2c:

There shouldn't be a separate package manager per se.

Consider the usage of yui loader: http://developer.yahoo.com/yui/3/examples/yui/yui-loader-ext.html

If we use a similar mechanism then we can setup node servers simply by running the app on it, which will in turn installs dependencies. Sure this adds to the 'first run' time, but for subsequent runs it can rely on local caches. 

Unless configured to do otherwise all binaries and payload should be kept in a node sandbox.  

--

[Rasmus Andersson]

But then you'll just have the one tarball.

Mode takes an approach of keeping a certain branch of a git repository
up-to-date. Because of the distributed nature and the speed of
development, modules are updated frequently.

mode upgrade foo

Would possibly perform a git pull from a "stable" branch. The
developer of "foo" can simply patch the "stable" branch to publish new
versions. At the same time multiple checkouts of the same repo might
co-exist — tags, explicit trees, etc (to support dependencies).

Of course a tarball could do the same, but you would loose the implied
"difference knowledge" of git (e.g. an updated version of the sqlite
module might not require you to reconfigure, rebuild and reinstall it
but simply update a js file).

Agreed that a module manager should support different kinds of
"transport" — git, tarballs, etc.

>
> Kris Kowal

[Rasmus Andersson]

On Sun, Mar 14, 2010 at 00:44, Isaac Schlueter <i...@izs.me> wrote:
> I agree with a lot of what you have to say here.
>
> On Sat, Mar 13, 2010 at 03:54, Rasmus Andersson <ras...@notion.se> wrote:
>> In my opinion a module manager should manage node modules. Not a
>> package (like Debian Aptitude or Redhat Package Manager). The reason
>> being node modules are simply node modules. They do (should) not
>> install files anywhere on your system.
>
> I'm not convinced of that.  For instance, quite a few node packages
> may in fact want to add a command-line program.  That command line
> program must go somewhere, right?

Then it's not a node module anymore, in my book. Such a thing is a
"program" and belongs in the operating systems' package manager
(ports, aptitude, et al). I think a node module should be a node
module and not a "blob of things which to some extent concern node".

Anyhow, this is a very interesting discussion in general! Where goes
the border between a os package manager (ports, aptitude...) and a
framework/platform "plug-in" (module) manager?

[Isaac Schlueter]

On Sun, Mar 14, 2010 at 03:44, Rasmus Andersson <ras...@notion.se> wrote:
> On Sun, Mar 14, 2010 at 00:44, Isaac Schlueter <i...@izs.me> wrote:
>> a few node packages may in fact want to add a command-line program.
>

> Then it's not a node module anymore, in my book. Such a thing is a
> "program" and belongs in the operating systems' package manager

There's definitely precedents in both directions. For instance, "gem
install dicks" installs a command-line "dicks" program (which is in
fact a ruby script shim that loads and runs the installed program.)
On the other extreme, the YUI gallery/library and loader strictly
concern themselves with YUI modules, which just attach themselves onto
an instance of the YUI object, and that's it. (Of course, in that
case, it's a DOM, and not a file system, so things are necessarily
different.)

I've been envisioning npm as closer to the "gem" side of the scale.
You could conceivably have a "package" in npm that does almost
anything, and this certainly does open the door for some interesting
(if ill-advised) abuses. I would not recommend creating an "nginx"
npm package. (But, perhaps a package that checks for nginx and then
does some config magic to set up proxying or something, that might be
handy.)

I'm sort of hoping that the fact that it's tricky to do these crazy
things, and easy to use npm for vanilla node modules, will make it
easy to "fall into the pit of success", so to speak.

> Mode takes an approach of keeping a certain branch of a git repository
> up-to-date. Because of the distributed nature and the speed of
> development, modules are updated frequently.

> ...

> Of course a tarball could do the same, but you would loose the implied
> "difference knowledge" of git

These are all good points, and touch on some of the trade-offs that
I've considered in the design of npm.

My gut feeling about all this is that mode and npm target somewhat
different niches, and it is very unclear to me which is going to turn
out to be more useful/popular in the long run. I think we'll only
know that once they're both a bit more fully fleshed out, so I'm glad
we're both in this. Code > words.

I do like how mode manages to be a bit simpler by putting most of the
"hard stuff" on git. Otoh, I also really want to write command line
programs in JavaScript and install them easily, which goes outside of
mode's intentionally limited scope.

> Anyhow, this is a very interesting discussion in general!

INORITE!!?

> Where goes
> the border between a os package manager (ports, aptitude...) and a
> framework/platform "plug-in" (module) manager?

It's a very fuzzy border, indeed!

On Sun, Mar 14, 2010 at 03:39, Joe Developer <joe.d.d...@gmail.com> wrote:
> There shouldn't be a separate package manager per se.
> Consider the usage of yui
> loader: http://developer.yahoo.com/yui/3/examples/yui/yui-loader-ext.html

Have you seen this? http://github.com/davglass/nodejs-yui3

It turns YUI into basically just another custom require()
implementation. Add some local caching and registry-updating, and
you've got pretty much exactly what you're talking about.

--i

[816...@gmail.com]

I think that git based package manager is a good practice for node,
because it very comfortable for developers, and git installing isn't
an problem for users, there only *nix users here, "apt-get install
git" or something like this, its even easier then install package
manager module..

The topic of post is about an default cli interface for PM (package
manager). I'm agree with Ryan, it's to early now, but with one
exception. Its early for users, all of them can use one of existed
PMs, without any troubles, but,its time for developers, one example:

I want to use in my module another one from other developer,
and I don't want to put into my project folder and update every time.

There are 3 different PMs (may be more) what format for package file
should I use to indicate dependings, package.json or seed.yml, maybe
another one?

So I'll take this opportunity to discuss what package format is
better,
As for me, my vote is for package.json

[tjholowaychuk]

Kiwi is 100% usable at the moment, however features and refinement
will certainly continue. http://github.com/visionmedia/kiwi

Features include:

* No GIT dependency
* Lightweight and written in bash (good for the moment since node
continues to change it's public API)
however should be re-written eventually in plain JS
* Very fast
* Intuitive and clean CLI
* Dependency resolution
* Version resolution
* Interactive REPL (console) -- auto-detects rlwrap
* Multiple environment support
* Full test coverage
* Kiwi "seed" tarballs
* Kiwi server already implemented and generously hosted by
Slicehost

The only thing I would like to see from node in this regard is
perhaps tapping into require(), to prevent explicit use of these
PMs within libraries.

[Camilo Aguilar]

+1 for kiwi :)

[sste...@gmail.com]

On Mar 14, 2010, at 10:18 PM, Camilo Aguilar wrote:

+1 for kiwi :)

I'd like it even better with package.json rather than YAML.  

Let's stay native if we can.  

JSON will work just fine for what this requires; there's no need for a foreign data format.

S

[Rasmus Andersson]

On Mon, Mar 15, 2010 at 03:54, sste...@gmail.com <sste...@gmail.com> wrote:
>
> On Mar 14, 2010, at 10:18 PM, Camilo Aguilar wrote:
>
> +1 for kiwi :)
>
> I'd like it even better with package.json rather than YAML.
> Let's stay native if we can.
> JSON will work just fine for what this requires; there's no need for a
> foreign data format.

Why not just javascript? It's easier to write and you can embed
"scripts" (pre-configure, post-install, etc).

--
Rasmus Andersson

[816...@gmail.com]

If you use javascript you give to developer an possibility to write
working code in package file, it's bad, Also javascript package
description will be hard to read from other languages.

kiwi is a nice one, but i'm not agree with idea of yaml, my arguments:

1) Its not comfortable to work with it from JS, for example i'm want
to write script, which will update my package file, every time I run
it - "upgrade_version.js".

2) Yaml is Pythonic style format, but node is for JS

3) Sometimes I need to load package file to client throw AJAX, with
YAML I should seach an library for browsers

[Jon Gretar Borgthorsson]

If I may add my 2 cents to the package management discussion. Don't make some default package manager. If people want to use one then fine but I find little need for it personally.

It's the same in the Erlang world. There are loads of package management solutions and many of them are stable and have years of refinement behind them. But all of them have it in common that no one uses them really. It's an unwritten rule that system wide packages are bad practice and that the best way is to include the dependencies with the application itself. Either copied inside or included as git/svn externals. This really is a best practice that should be encouraged especially in a language/framework that aims to be scalable and highly concurrent. 

I really think your app should be self reliant and exclude external system dependencies. As soon as you run 2 different apps on a server you can hit a problem with package management as all Rails developers know. Which is the reason for why Rails is moving away from system wide gems.

I would much rather see Node.js leave this open for users themselves but itself should encourage the convention of including the libraries in-place by for example adding "./libs/**/lib/" in the load path.

But that's just my opinion.

-- Jón Grétar Borgþórsson

--

[tjholowaychuk]

I agree with you guys in regards to Kiwi's YAML usage. That
being said IMO YAML is a much better documentation language,
although I can understand the want for JSON. Kiwi itself if needed
can and will be re-written in JavaScript when we feel the time is
write.

One of the main reasons I choose YAML aside from it being easier
to read, is that it is easier to parse using a scripting language
like bash.

Thanks for the feedback :)

[sste...@gmail.com]

On Mar 15, 2010, at 7:12 AM, Jon Gretar Borgthorsson wrote:
> I would much rather see Node.js leave this open for users themselves but itself should encourage the convention of including the libraries in-place by for example adding "./libs/**/lib/" in the load path.

Including the packages within an application doesn't preclude an intelligent tool for finding packages, automatically installing dependencies, and helping to manage upgrades.

Finding & installing packages and deciding where to install them are completely separate issues, though both should be available from any package manager worthy of the name.

I'd love to be able to say 'get me package xxx and its dependencies and install them all completely within myappxyz project.'

S

[David Parker]

On Mar 15, 5:12 am, Jon Gretar Borgthorsson

<jon.borgthors...@gmail.com> wrote:
> If I may add my 2 cents to the package management discussion. Don't make
> some default package manager. If people want to use one then fine but I find
> little need for it personally.
>
> It's the same in the Erlang world. There are loads of package management
> solutions and many of them are stable and have years of refinement behind
> them. But all of them have it in common that no one uses them really. It's
> an unwritten rule that system wide packages are bad practice and that the
> best way is to include the dependencies with the application itself. Either
> copied inside or included as git/svn externals. This really is a best
> practice that should be encouraged especially in a language/framework that
> aims to be scalable and highly concurrent.
>
> I really think your app should be self reliant and exclude external system
> dependencies. As soon as you run 2 different apps on a server you can hit a
> problem with package management as all Rails developers know. Which is the
> reason for why Rails is moving away from system wide gems.
>
> I would much rather see Node.js leave this open for users themselves but
> itself should encourage the convention of including the libraries in-place
> by for example adding "./libs/**/lib/" in the load path.
>
> But that's just my opinion.
>
> -- Jón Grétar Borgþórsson
>

+1. I've had this happen on several different occasions (in Rails).
In Java-land, you always package dependencies within the app itself,
and it generally works out really well.

David Parker

[sste...@gmail.com]

On Mar 15, 2010, at 9:36 AM, tjholowaychuk wrote:

> I agree with you guys in regards to Kiwi's YAML usage. That
> being said IMO YAML is a much better documentation language,
> although I can understand the want for JSON. Kiwi itself if needed
> can and will be re-written in JavaScript when we feel the time is
> write.

The time is right now, before there is 'too much code' to be rewritten.

Right now, it's only really supporting your own packages, so it's a one-shot, one programmer effort.

> One of the main reasons I choose YAML aside from it being easier
> to read, is that it is easier to parse using a scripting language
> like bash.

Since parsing JSON is unlikely to change any time soon, this may be a good time to start moving parts over to node itself.

S

[Dean Landolt]

A nice alternative to packaging all dependencies is virtual environments. Narwhal has a nice setup in "sea" that lets you do this. It certainly makes a nice compliment to package managers and allows you to maintain control over how much package reuse vs. isolation.

[Ciaran]

On Mon, Mar 15, 2010 at 1:36 PM, tjholowaychuk <tjholo...@gmail.com> wrote:
> I agree with you guys in regards to Kiwi's YAML usage. That
> being said IMO YAML is a much better documentation language,
> although I can understand the want for JSON. Kiwi itself if needed
> can and will be re-written in JavaScript when we feel the time is
> write.

;) As you know, I've already re-written 70% of the client, my isn't it
verbose <g>
-cj

[MilesTogoe]

What ever happened to KISS ?

it is far simpler to have a bunch of required libraries at the head of a
program - then every developer knows exactly what are the dependancies
and where they are. No more "hidden" magic links. And certainly no
package mgmt code embedded in a program. Anyone who has used gems over
multiple systems and versions knows what a nightmare gem locations can
be. Now we see Node making this same error.

Less "elegant", less over engineering, more kiss.

[Mikeal Rogers]

>> Where goes
>> the border between a os package manager (ports, aptitude...) and a
>> framework/platform "plug-in" (module) manager?
>
> It's a very fuzzy border, indeed!

Once there is a popular package manager for node an os package manager
could be integrated the way the ports collection integrates with CPAN.

-Mikeal

[Mikeal Rogers]

Is there a reason you can't just support the package.json format that
everyone else is trying to standardize on?

-Mikeal

[tjholowaychuk]

Not once kiwi's JS implementation is finished no

[tjholowaychuk]

GIT submodules just enforce a git dependency which is no good,
and as I have mentioned before maintaining a large tree of
dependencies
within other libraries is not only a huge pain in the ass it bloats
the repos
with code not directly related to the library itself.

I have never had any issues with gems to be honest... I agree that
the PM should be able to vendorize libs, but its not hard to
understand
where the gems live... How would you feel if every mac app had its own
localized opengl etc lol good luck

[tjholowaychuk]

I dont disagree with package.json, aside from JSON being a nice
serialization schema
its less friendly to a reader (you cant deny that), however for the
sake of the JSON
parser being native this gets a +1 from me

[sste...@gmail.com]

On Mar 15, 2010, at 2:14 PM, tjholowaychuk wrote:

> where the gems live... How would you feel if every mac app had its own
> localized opengl etc lol good luck

Many Mac apps *do* include specific versions of specific libraries and frameworks to avoid version incompatibilities.

OpenGL is a specious example; it's an integral and stable part of the OS and wouldn't need to be duplicated except in extreme circumstances.

Fast moving node.js (or Rails, or Django) modules are an entirely different kettle of fish.

S

[tjholowaychuk]

I know but its a very debatable thing. I personally have never had an
issue
with package management in nearly any shape or form. When it comes to
deployment I like things to be bundled but that is easy to support

On Mar 15, 12:25 pm, "sstein...@gmail.com" <sstein...@gmail.com>
wrote:

[sste...@gmail.com]

On Mar 15, 2010, at 6:15 PM, tjholowaychuk wrote:

> I know but its a very debatable thing. I personally have never had an
> issue with package management in nearly any shape or form.

Reminds me of the recent graduate I hired one time who told me he had never had a bug in one of his programs. He was instantly nick-named "No Bugs Doug." Needless to say, this "state of perfection" didn't last long. Maybe you've just been lucky.

> When it comes to deployment I like things to be bundled but that is easy to support

Yes, bundling within an 'application' should be an option as should a 'global' install.

At this point, there seem to be several different ideas of where "global" means.

S

[Dean Landolt]

On Mon, Mar 15, 2010 at 7:03 PM, sste...@gmail.com <sste...@gmail.com> wrote:

On Mar 15, 2010, at 6:15 PM, tjholowaychuk wrote:

> I know but its a very debatable thing. I personally have never had an
> issue with package management in nearly any shape or form.

Reminds me of the recent graduate I hired one time who told me he had never had a bug in one of his programs.  He was instantly nick-named "No Bugs Doug."  Needless to say, this "state of perfection" didn't last long.  Maybe you've just been lucky.

Heh. But to be fair he did say "in nearly any shape or form." Honestly, once virtualenv came along in the python world -- and I learned how to use it -- I can say the same thing.

[tjholowaychuk]

I am not disagreeing with anyone, I just thing they both have pros and
cons,
and I prefer the distributed approach overall. There is nothing really
magic about it,
install something (you would do it manually anyway for node), unshift
to the require path,
and then roll on like normal.

Besides even a simple library like Express having 3-4 dependencies, I
would much rather
not update them and bump the version every time a little bug that is
not even related
to Express pops up

On Mar 15, 4:43 pm, Dean Landolt <d...@deanlandolt.com> wrote:

> On Mon, Mar 15, 2010 at 7:03 PM, sstein...@gmail.com <sstein...@gmail.com>wrote:
>
> > On Mar 15, 2010, at 6:15 PM, tjholowaychuk wrote:
>
> > > I know but its a very debatable thing. I personally have never had an
> > > issue with package management in nearly any shape or form.
>
> > Reminds me of the recent graduate I hired one time who told me he had never
> > had a bug in one of his programs.  He was instantly nick-named "No Bugs
> > Doug."  Needless to say, this "state of perfection" didn't last long.  Maybe
> > you've just been lucky.
>

> Heh. But to be fair he did say "in *nearly *any shape or form." Honestly,

[sste...@gmail.com]

Yes, but virtualenv packages things up in a neat, isolated sort of way.   tjholowaychuk was giving someone shit for wanting to isolate their installs with a specious argument about "installing OpenGL in every app."

I was pointing out that, without virtualenv type isolation, you're *going* to have packaging problems in a fast moving environment like node.js.

node.env anyone?

S

[tjholowaychuk]

calm down lol holy crap... all I am saying is if we did that with
every library / app out there
it would be a ridiculous mess of unrelated code in every repo out
there. They BOTH have pros
and cons, fuck lol get that in your head. Me forcing kiwi on express
is/was a test, not
something definite, I wanted feedback, so far lots of people like it,
lots dont.

You cant praise RubyGems and then smash Kiwi because they are very
similar ATM

On Mar 15, 6:06 pm, "sstein...@gmail.com" <sstein...@gmail.com> wrote:
> On Mar 15, 2010, at 7:43 PM, Dean Landolt wrote:
>

[sste...@gmail.com]

On Mar 15, 2010, at 9:37 PM, tjholowaychuk wrote:

> calm down lol holy crap... all I am saying is if we did that with every library / app out there it would be a ridiculous mess of unrelated code in every repo out there.

All I've ever said is that any package manager has to be able to *install* into an isolated environment, like a Python virtualenv; not just into the global environment.

Global installs can be useful, like OpenGL, for example ;-).

S

[tjholowaychuk]

I have not tried virtualenv, but could you be constructive instead of
just dissing people lol,
good developers fix, not complain so contribute. So now onto business,
what aspects do you
like / dislike about kiwi if any? In regards to environments kiwi can
and does allow any number
of custom environments with:

$ kiwi switch some-env

however you keep talking about packaging code / tarballs directly in
the source tree that is
probably not what you are referring to.

[Mikeal Rogers]

That functionality is not provided by the package manager, it's
provided by virtualenv. pip contains some convenience options that
will install in a virtualenv but most of the time you simply install
modules in to the isolated environment by running the setuptools/pip
that exists inside the env instead of using the installer in the
global env.

Once someone writes a virtualenv equivalent for node they can make
that work but it's not the job of the package manager/install to
provide it, nor should it be. The package manager should simply
install in whatever environment it's being run from.

-Mikeal

[Ciaran]

On Tue, Mar 16, 2010 at 4:01 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:
> That functionality is not provided by the package manager, it's
> provided by virtualenv. pip contains some convenience options that
> will install in a virtualenv but most of the time you simply install
> modules in to the isolated environment by running the setuptools/pip
> that exists inside the env instead of using the installer in the
> global env.
>
> Once someone writes a virtualenv equivalent for node they can make
> that work but it's not the job of the package manager/install to
> provide it, nor should it be. The package manager should simply
> install in whatever environment it's being run from.

Hmm, kiwi *as I understand it* does offer a 'virtual environment'
equivalent, it is possible to install different dependencies/versions
of packages into different grouped environments on the disk and then
select the environment (although I'm unsure if you can select the
'active' environment at runtime, however from what I know of kiwi that
would be simple to achieve)

However there are two issues;

i) Some packages (inevitably) require installing command line tools
that go on the system path and I'm not sure how one would deal with
that (how is this dealt with virtualenv, as it *appears* to virtualise
the python directory not the system paths ? Or is this a non-issue
with python)

ii) To use the kiwi packages you have to have a
require('kiwi').seed(<package_name>) somewhere in your app, which
immediately couples you to kiwi, if node had a package manager built
in this would not be a problem, but as it doesn't we have to make an
explicit choice of which package manager to support
at-the-time-of-writing our app :(

I've not used npm since I first joined the community and re-wrote it
to support whatever that week's breaking change was and I understand
it has since grown from an ugly caterpillar into something more
butterfly orientated so I can't comment directly on that, but I do
feel it is a shame that we have multiple smart people all funnelling
their efforts down seperate channels :(

I'm still keen on using node.js in windows (but these days lack the
C++/Win32 compilation environment fu to make it happen) so it is a big
+ for me that the package manager is written in Javascript, this also
underlines the point that I don't personally believe that package
management is an OS level responsibility, some Operating Systems don't
have package managers built into them! :)

I'm genuinely surprised that some people feel that package managers
are a bad thing, I totally accept that you can get into a mess when
you have multiple dependencies that all depend on each other and
common packages, but some of those packages are incompatible versions,
however I don't know how one would resolve that using the 'native'
copy + paste approach ?

Using Git submodules as 'package management' only works reliably in
the current lifecycle of node.js: Where everything is breaking on a
weekly basis and doing a git submodule update; regularly is pretty
safe, but in a controlled production environment that is the *last*
thing you want to be doing, you want to be able to carefully control
each package version bump you do :) submodules also lack a reasonable
mechanism of determining inter-dependencies meaning you have to always
be aware of them yourself and deal with the consequences should your
update automatically require some other un-specified dependency.
Currently to find a package to ensure I'm not re-hashing someone
else's effort I have to search the wiki, then github eventually give
up, write my own module then 3 weeks later discover someone else has
already done it better, a package registry completely avoids that
issue (assuming you can search it effectively!)

It depends on how Ryan wants node.js to continue I guess, if he
doesn't want it to be adopted more widely then the current approaches
I guess are fine, but if he wants a wider community able to deploy it
into more strictly controlled environments then I can't see how one
could avoid having some form of package management.

I'm tired of copying and pasting the same JS files over and over into
each of my little projects, finding an issue in it, then having to
copy+paste that fix across all the rest just because I don't have
access to Git on one of my servers, a relatively locked down package
manager with clearly defined target urls is easier to get installed
than 'Git' which (rightly so) can access anything in the outside world
:)

We all have opinions I guess <g>

-cj.

[Jon Gretar Borgthorsson]

Hehh. Yeah people mabey should not take this discussion personally. :)

All we can do is state preferences. And mine is that package managers for this kind of thing is not needed. And that more quality code can come out if none are used. I just think that making the developer himself manage these things is a good thing.

But *if* we go for a default package managers I would prefer if they were optional. I'm always frustrated at some rubygems that set them selves up so that they can *only* be used as gems. And although TJ disagrees with this in an earlier post I have no problem at all with having Git dependency.

The worst case scenario would be to end up like in the java world. Where the developers code is little more than a glue for hundreds of library deps. I compiled a small Apache project the other day and that Maven2 piece of crap that they use had first to spend 20 minutes downloading and compiling dependencies. :D How they can work with that crap is beyond me.

-- Jón Grétar Borgþórsson

[Aaron Heckmann]

It depends on how Ryan wants node.js to continue I guess, if he
doesn't want it to be adopted more widely then the current approaches
I guess are fine, but if he wants a wider community able to deploy it
into more strictly controlled environments then I can't see how one
could avoid having some form of package management.

Ryan didn't say no to a pm, he just said not yet. 

> "Maybe there will be in the future - it's much too early now. I'm happy

to have different people, with different ideas working on it."

 

--
Aaron

[Ciaran]

;) Yeah I got that , sorry re-reading that sentence it came across
rather more black'n'white than I had intended, I should've really
suffixed that with 'at the moment' .i.e. if the feeling is that he
doesn't really want node.js being used more widely *right-now* ..
which I can totally understand given the rate of change :)

-cj.

[sste...@gmail.com]

On Mar 15, 2010, at 11:14 PM, tjholowaychuk wrote:

> I have not tried virtualenv, but could you be constructive instead of
> just dissing people lol,

The only things I've said are that I prefer JSON to YAML, and that installation to an isolated environment is essential. I also pointed out that you were giving someone (maybe me) crap about wanting isolated environments with a specious argument about "copying OpenGL". I haven't dissed anyone.

> So now onto business, what aspects do you like / dislike about kiwi if any? In regards to environments kiwi can and does allow any number of custom environments with:
>
> $ kiwi switch some-env

The way that virtualenv works, since it's been mentioned and I use it every day, is to isolate the entire library environment including the Python executable itself. You can include or exclude the system libraries, site packages as they're called in Python, and have a completely clean environment to load with specific versions of things. The isolation is not perfect; system packages are symlinked in so global changes bleed in but, once you've switched to an environment, all installations 'just work' and go into that environment.

Kiwi has its own partial solution with its 'seeds', but the environment isolation should be either part of the language environment, or handled by a package like virtualenv. The installer should just install into the current environment; the installation tool shouldn't try to manage that, too.

> however you keep talking about packaging code / tarballs directly in
> the source tree that is probably not what you are referring to.

I think we've gotten cross threaded here; I never said anything about that. My desire is only to have something like virtualenv that keeps things bundled together without relying on a package manager to attempt to do something that's not really its function.

S

[tjholowaychuk]

Ah I see. I dont think kiwi is to far off from being both (even if
that is not really it's "job").
I wanted it's own bin dir so that we could safely keep all bins out as
well, my
main issue with that is that we would need to automate or require that
a
user adds this to PATH, which is just kind of a pain in the ass and
causes one
more road block.

Kiwi's YAML will become JSON, and the source itself will become
javascript. The
one benefit I see with it not being js is that we can install node
without node ever
previously being installed which may be nice for people getting
started.

[sste...@gmail.com]

On Mar 16, 2010, at 9:49 AM, tjholowaychuk wrote:

> Ah I see. I dont think kiwi is to far off from being both (even if
> that is not really it's "job").

Maybe pull the env switching out to a different project? I would love to be able to have any installer (or even manual installs, whatever that means) just go to the 'right place' as they do in virtualenv.

> I wanted it's own bin dir so that we could safely keep all bins out as
> well, my main issue with that is that we would need to automate or require that
> a user adds this to PATH, which is just kind of a pain in the ass and
> causes one more road block.

The bin directory path is handled by virtualenv by creating a 'bin' directory into which things go, just for this virtual environment.

> Kiwi's YAML will become JSON, and the source itself will become
> javascript. The one benefit I see with it not being js is that we can install node
> without node ever previously being installed which may be nice for people getting
> started.

Yay for JSON, +0 on making it javascript, at the moment. I think the way it is works just fine; let things stabilize before going crazy converting and, as you point out, there should always be a 'bootstrap' part that doesn't require node at all.

S

[tjholowaychuk]

>The bin directory path is handled by virtualenv by creating a 'bin' directory into which things go, just for this virtual environment.

For sure, it would still have to be in your PATH though, which is
means it wont work out of the box without some config,
wish I would like to avoid if possible.

> Yay for JSON, +0 on making it javascript, at the moment.  I think the way it is works just fine; let things stabilize before going crazy converting and, as you point out, there should always be a 'bootstrap' part that doesn't require node at all.

This is an issue since bash is useless if we want to parse JSON in any
sort of manor,
YAML is actually pretty easy if its just bits and pieces that you
want, but since JSON's
whitespace is completely arbitrary it more or less wont work with bash
(without busting out another scripting language).

Aside from that I dont think bash is a bad choice, eventually it would
be ideal to have it written in JS,
I like some benefits of it NOT being JS such as auto-detecting rlwrap
when available to provide a really slick
REPL, installing node without node previously being installed, less
code, bypassing node's insane breaking of
public APIs etc

thanks for the suggestions

[Ciaran]

On Tue, Mar 16, 2010 at 3:22 PM, tjholowaychuk <tjholo...@gmail.com> wrote:
>
>>The bin directory path is handled by virtualenv by creating a 'bin' directory into which things go, just for this virtual environment.
>
> For sure, it would still have to be in your PATH though, which is
> means it wont work out of the box without some config,
> wish I would like to avoid if possible.
>
>> Yay for JSON, +0 on making it javascript, at the moment.  I think the way it is works just fine; let things stabilize before going crazy converting and, as you point out, there should always be a 'bootstrap' part that doesn't require node at all.
>
> This is an issue since bash is useless if we want to parse JSON in any
> sort of manor,
> YAML is actually pretty easy if its just bits and pieces that you
> want, but since JSON's
> whitespace is completely arbitrary it more or less wont work with bash
> (without busting out another scripting language).
>
> Aside from that I dont think bash is a bad choice, eventually it would
> be ideal to have it written in JS,
> I like some benefits of it NOT being JS such as auto-detecting rlwrap
> when available to provide a really slick
> REPL,

REPL I've done, but yeah the lack of rlwrap does suck, is there anyway
to get more fine-grained access to stdio other than line-ending
detction ?

>installing node without node previously being installed,

I still think there should be a bash script to provide this support
(ideally actually a platform-independent set of scripts inc. a bash
one for unix platforms)

> less code,
;) if node.js provides native support for things like mkdirs etc. then
the code won't be too much worse.

> bypassing node's insane breaking of
> public APIs etc

meh, I think the touch points to 'dangerous' apis (those which are
likely to mutate rapidly) are fairly minimal, its mostly logic in
there.

>
> thanks for the suggestions
;)

=- cj

[tjholowaychuk]

It does not seem like NPM is following commonjs standards as far as
package.json goes
either, some of it but not all of it. I dont find it ideal either
really but I will try and make kiwi
conform as much as possible

[sste...@gmail.com]

On Mar 16, 2010, at 11:22 AM, tjholowaychuk wrote:

>
>> The bin directory path is handled by virtualenv by creating a 'bin' directory into which things go, just for this virtual environment.
>
> For sure, it would still have to be in your PATH though, which is
> means it wont work out of the box without some config,
> wish I would like to avoid if possible.

The way virtualenv handles that is by having an "activate" shell script within the virtualenv that munges the environment in a couple of different ways like setting the prompt to indicate the ve you're in, and creating a few simple commands that are handy within a virtualenv. I really think a node.virtualenv is in order and I'd be happy to start/contribute when I've got a little more time (and experience with node.js!).

>> Yay for JSON, +0 on making it javascript, at the moment. I think the way it is works just fine; let things stabilize before going crazy converting and, as you point out, there should always be a 'bootstrap' part that doesn't require node at all.
>
> This is an issue since bash is useless if we want to parse JSON in any
> sort of manor,
> YAML is actually pretty easy if its just bits and pieces that you
> want, but since JSON's
> whitespace is completely arbitrary it more or less wont work with bash
> (without busting out another scripting language).

Stupid computers! Perl and Python are ubiquitous so a quick call to either wouldn't be out of line though installing a JSON library might be (or not, what the heck!).

> Aside from that I dont think bash is a bad choice, eventually it would
> be ideal to have it written in JS, I like some benefits of it NOT being JS such as auto-detecting rlwrap
> when available to provide a really slick REPL, installing node without node previously being installed, less
> code, bypassing node's insane breaking of public APIs etc

I think the last is the most important. Last thing you want is for a package manager to fall on its head when trying an experimental version of the host language. I think the way it is is actually *better* than depending on such a fast moving target. That's kind of the whole argument for "why we need virtual environments" in the first place; everything's moving much too fast to get a stable place to work!

S

[Camilo Aguilar]

I think that we should speak less and do more. I think that kiwi is going well, if someone have suggestions just simply contribute with a patch or write another package manager. That's the beauty of opensource :)

S

[tjholowaychuk]

I will look into the json parsing issue so we can support package.json
even though
most others are kinda using their own variant right now it seems

On Mar 16, 9:53 am, Camilo Aguilar <cam...@cloudescape.com> wrote:
> I think that we should speak less and do more. I think that kiwi is going
> well, if someone have suggestions just simply contribute with a patch or
> write another package manager. That's the beauty of opensource :)
>

> On Tue, Mar 16, 2010 at 11:11 AM, sstein...@gmail.com
> <sstein...@gmail.com>wrote:

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>

[Mikeal Rogers]

I'm actually not worried about the variations right now, eventually
it'll all calm down.

js-registry only cares about two properties, versions and
version-tags, and validates that they are proper semver. Other than
that, it just stores JSON and links people to tarballs. Ideally this
registry could be used by all package managers that support
package.json.

http://github.com/mikeal/js-registry

Standardizing on package.json and semver should provide some solid
ground between the package managers so that they can focus on bigger
problems like the ones we've been discussing.

I'll say it again, at this time it's a great thing that there is no
blessed pm and that none of this is included in node. There is more
good work happening around packaging and distribution right now than
there has ever been for Python and we should keep that up as long as
we can.

-Mikeal

> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

[Dean Landolt]

On Tue, Mar 16, 2010 at 1:50 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:

I'm actually not worried about the variations right now, eventually
it'll all calm down.

js-registry only cares about two properties, versions and
version-tags, and validates that they are proper semver. Other than
that, it just stores JSON and links people to tarballs. Ideally this
registry could be used by all package managers that support
package.json.

http://github.com/mikeal/js-registry

Standardizing on package.json and semver should provide some solid
ground between the package managers so that they can focus on bigger
problems like the ones we've been discussing.

Exactly -- just the filename alone is enough to get us some useful interop between PMs. Hopefully more useful fields of the package.json schema can be agreed upon sooner rather than later...

 

I'll say it again, at this time it's a great thing that there is no
blessed pm and that none of this is included in node. There is more
good work happening around packaging and distribution right now than
there has ever been for Python and we should keep that up as long as
we can.

+1!

And the more PMs there are the more of an incentive there will be to conform (at least in part) to a package.json schema. And of course nothing stops PMs from extending package.json any way they please, but the package manager market should keep PMs from stomping all over otherwise-agreed-upon package.json fields.