local private npm repository

[Osher El-Netanany]

Hi all,

'ok, this is kind'a embarrassing a little, 

we've tried few times in few ways but still did not get it right:

• we're trying to set a local npm repo for our packages
• we don't want to host an entire replica of the npm (which is HUGE these days, and growing)
  i.e - the private replica should hold only our packages, the rest can come from the public repo
• publishing should direct to our repo
• install should take our packages from our repo, and public packages from the global repo

What's the best way to do it?

What's the best contemporary tutorial/doc/explanation about it on the web?

Thanks...

      Don

[Osher El-Netanany]

oups, sorry for the resending, 

but I forgot to mention another problem we had - 

• users in local repo should match users in global repo
  

it sounds so simple, but we had problems with that
(all the part with npm adduser and stuff)

[Andy Ennamorato]

We had similar questions and issues (especially with setting up a local npm repo) so if you search back you'll see some recommendations from the list.

In the end we did get a replica working (couchdb 1.3 on Windows no less) and at ~50GB it wasn't too bad to host on a local machine. 

FWIW the StrongLoop folks are hosting a webinar tomorrow about private npm repos so that might be worth attending: http://t.co/WcQDkHqLf5 

I think they are going to talk about the work they've done with their node distro and/or changes to node-reggie a lightweight repo option (https://github.com/mbrevoort/node-reggie). I'm hopeful that in the near future multiple repos (and npm's ability to use them) becomes a more realistic option.

Andy

Sent from my iPhone

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Miroslav Bajtoš]

Hi Osher,

There are multiple options for setting up a private repo, unfortunately none of them is ideal.

1. Create a replica of npmjs.org

2. Setup your own registry using CouchDB and npmjs code, but don't replicate the public registry

3. Use a different registry server, e.g. reggie [1] mentioned by Andy

Let's check your requirements:

we don't want to host an entire replica of the npm (which is HUGE these days, and growing)
i.e - the private replica should hold only our packages, the rest can come from the public repo

This is true for options 2 and 3.

publishing should direct to our repo

This can be achieved with all three options. Either add the following line to your package.json

   "publishConfig": { "registry": "http://private/registry/url" }

or use the reggie CLI client for publishing the packages.

install should take our packages from our repo, and public packages from the global repo

The npm client does not support multiple repositories at the moment, even though there are plans to get this implemented in the next few weeks/months. Unless you have a replica of npmjs.org (option 1), you must use different way for installing and referencing packages from public and private repository.

With couchdb-based registry, you can use absolute URL to the tarball of the desired version. (This means you can't specify a fuzzy version like ~0.5.2.)

Node reggie implements several kinds of package URLs to support fuzzy version specification. See project README [1] for more details.

It may be possible to overcome this limitation by sticking an npm proxy [2] in front of your private registry.

users in local repo should match users in global repo

You will get this for free with npmjs.or replica.

If its possible to setup CouchDB replication in such way that only users are replicated, then you can get this functionality with non-replicated CouchDB-based registry too.

Reggie does not implement authentication and authorization at the moment - it will happily accept (and ignore) whatever credentials are sent by the npm client.

I hope you will find this helpful, don't hesitate to ask if there is anything else not clear.

Miroslav

[1] https://github.com/mbrevoort/node-reggie

[2] https://npmjs.org/package/npm-proxy

[Adam Crabtree]

Miroslav,

Thanks for the link to npm-proxy. I didn't realize that project existed. 

Osher,

There is Nodejitsu's new "Enterprise Private npm" initiative, though I'm not sure if it offers the same private/public fallback of npm-proxy, which to me is the key feature for smaller companies not wanting to (like you said) replicate the whole npm repo. I would opine that main npm repo can get very slow at times (though usually not), so there is the additional benefit of speed wrt hosting a replica even if it all seems rather wasteful.

I personally find myself wanting to publish forks of public modules to a private repo to avoid the need for npm_shrinkwrap for nested dependencies needing fixes.

Cheers,

Adam

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Better a little with righteousness
       than much gain with injustice.
Proverbs 16:8

[Santiago Gimeno]

Hi,

Probably a dumb question, but what are the advantages of having a private npm repository over getting the modules from your private git repository with npm using: "my_module": "git+ssh://git@my_repo:my_module.git#version" ?

Cheers,

Santi

2013/8/2 Adam Crabtree <atcra...@gmail.com>

[Adam Crabtree]

I can think of 3:

1. Semver.

Pushing out a critical bug fix can be a liability if every project that consumes that module, and all its parents must be updated. This is exacerbated by node.js/npm's default behavior of deeply nested static dependencies. 

2. Private Forks of Public Modules

Ability to publish private versions of public modules. Recently we wanted a unique fix to forever-agent[1] which request depends upon. You could achieve this via npm_shrinkwrap (I believe), but that would need to be done on a per-project basis, whereas being able to publish a private forked version, while still consuming the public request module is a little more elegant because now everyone at your company who depends upon request, doesn't need to remember to include an npm_shrinkwrap.

3. Speed

I don't know if the debate over whether you should checkin your dependencies and run `npm rebuild` on deploy, or use npm_shrinkwrap, or prebuild and deploy a tar is a solved issue or not, but it definitely seems the easiest way to deploy is to deploy your application code, run npm install (w/ or w/o a shrinkwrap) and go on your merry way. A private npm repo will speed this process up significantly, especially as your dependency list grows (which can be partially mitigated by breaking your application into several services).

Cheers,

Adam

[1] https://github.com/mikeal/forever-agent

[Santiago Gimeno]

Hi,

2013/8/2 Adam Crabtree <atcra...@gmail.com>

I can think of 3:

1. Semver.

Pushing out a critical bug fix can be a liability if every project that consumes that module, and all its parents must be updated. This is exacerbated by node.js/npm's default behavior of deeply nested static dependencies. 

2. Private Forks of Public Modules

Ability to publish private versions of public modules. Recently we wanted a unique fix to forever-agent[1] which request depends upon. You could achieve this via npm_shrinkwrap (I believe), but that would need to be done on a per-project basis, whereas being able to publish a private forked version, while still consuming the public request module is a little more elegant because now everyone at your company who depends upon request, doesn't need to remember to include an npm_shrinkwrap.

I hadn't thought of these two and now I can see the advantages.

Thanks for the thorough response!

Cheers,

Santi

[pablo borrazas]

Hi there,

We have tried to install an npm local registry to improve our development workflow taking advantage of this, but we had some issues.

First, we have found that our linux  CentOS 5 server installation was not properly because it needs a lot of software updates, and we have to use at least CentOs 6.

We still are looking for a good solution to easely switch to our private repos or global ones in order to install & publish the modules.

Please, let us now of any update to this issue.   

--

[George Stagas]

Anyone reading this thread may also want to try
https://github.com/stagas/npm-server

It's a much more minimalistic approach, its only function is to be
able to install packages from a local dir. It will use the real npm
registry to fetch foreign packages and run all other commands (publish
etc). It's more of a development tool than an actual registry but it
may be handy to some.

2013/8/2 pablo borrazas <pablo.b...@gmail.com>:

[Alex Kocharin]

Hi Osher,

I'm trying to implement such server in this project: https://github.com/rlidwka/sinopia

It's very much unstable right now, but seem to work good enough for us so far.

But a basic idea seem to be a right one: we proxy all package requests that haven't been found locally to a npm repository this way avoiding replication.