Need an "offline" npm-style repository - what's a good approach?

[andy e]

Apologies in advance because I've only glanced at this problem, but we work in a unique environment where we have no Internet connectivity.

So, with our Java apps, we run an instance of Artifactory on our LAN and load it by running an instance that is connected, which we then export and bring into the 'offline' instance. That gives us a sort of mirror of Java dependencies for maven and what not when we're developing.

Is there anything like Artifactory for npm? Do I need to roll my own somehow (i.e. would a simple WebDAV server work or is it more complex)? 

I've glanced at Mike's node-reggie idea so maybe that is a place to start (https://github.com/mbrevoort/node-reggie).

We don't need anything fancy - just a way to add npm modules to a project without having to check them in or pass around a giant .zip copy with all possible repos...etc. (Right now I just have a "node_modules_for_work" folder where I load up a ton of modules, then I zip that up and bring it in.)

I'm happy to go off and do some reading/digging, so links to similar ideas/attempts are appreciated. 

Thanks,

Andy

[Luke Arduini]

You could set up git on a server on your lan and just specify dependencies in your projects' package.json files pointing to the git address. Seems like it would be the easiest way to go for private modules not on the internet.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Caswell]

If you want a mirror of the public npm repo, one option is to replicate the couch database.  It's pretty big, but once replicated would give you a full mirror.  I'm pretty sure you can do delta updates later on by putting the couch database back online and syncing again.

[Luke Arduini]

Touching on what Tim said, if you create a document for replication in /_replicator (not /_replicate) on couch you can turn couch off/on and have replication resume whenever. I do it all the time for my private npm and it works well, example:

https://gist.github.com/st-luke/4165831

[andy e]

Cool, thanks for the suggestion(s).

Two quick questions:

1) Can anyone replicate w/ the npm couch repo? I take it that's a yes but maybe you need permission (then again maybe people are smart enough to not want to have to do this, unlike us...)

2) How big? 10GB? 100GB? 1TB+? 

Thanks,

Andy

[Bradley Meck]

Right now without compaction you are looking at 45~ GB of data it looks like on the current npm public registry, and it can be replicated without needing any permission. 

[andy e]

OK, great, thanks for the info. I'll chat it over and maybe give a shot at replicating the couch instance.

Thanks for the help!

Andy

[Tim Dickinson]

You might want to read https://blog.caurea.org/2012/01/31/local-npm-registry-mirror.html

[nwhite]

I wish it was trivial to have a sparse npm server. One where I could publish my private modules to and replicate only the packages I'm interested in. Then have all npm package requests that are not found automatically forwarded to the official server.

[Mark Hahn]

>  and replicate only the packages I'm interested in. 

It is pretty easy to filter replications in general.  I don't know what the npm couch docs look like so I don't know how easy it would be in this case.  And, as a bonus, the filtering is coded in JS.

[andy e]

This would be ideal in our scenario. It's what Artifactory does for us in Java land (if a module doesn't exist in our local repo, it grabs it from maven central) and would be awesome to see in npm. 

If someone does an "Enterprise NPM Repo" Kickstarter I'll certainly kick in a few bucks to support it. :)

andy

[Floby]

You can set up a npm repository of your own pretty easily [1]. It doesn't have to replicate the public npm repository.

I don't know how hard it would be to patch it so it does "if package not locally availbale then fetch from the net" but it could be another approach

[1] http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository

[Floby]

another link here

https://npmjs.org/doc/registry.html#Can-I-run-my-own-private-registry

[Andy Ennamorato]

Wow, that link is awesome. Thank you.

Andy

Sent from my iPhone

[Jackson Tian]

When your build your NPM registry, you can replicate single package with sync_package, but replicate all packages(It's pretty big). 

https://github.com/TBEDP/sync_package 

在 2013年2月19日星期二UTC+8上午2时23分05秒，andy写道：

[Brian Vanderbusch]

I've run into a similar situation, where my company has governance around acquisition of Open Source Software.  So even thought we have interenet connections, we really can't use them for npm.  We also have an artifactory instance running with support for Maven, and I'm actually wondering if it's possible to set up an npm mirror (with our own governance layer on top of the couchDB) inside artifactory.  (It's a budget/precedent thing... don't ask). 

i have built the code for our specific brand of (Justice!) governance around accessing/acquiring from npm where our registries are configured to our local instance of an npm mirror, but it's only deployed at a level where a few teams can access it and befit from it.  In order for me to allow all the devs in our enterprise to use it, I have to have another team that has that scope of support sign off on it.... and they just so happen to have artifactory stood up with Maven.  So if I can find a way for artifactory to pipe in updates from the global npm registry, then it's considered a solution we already support, and I can actually stand up.

so yeah... anyone taken a crack at, or have any thoughts on whether a marriage between a couch based npm mirror and artifactory is possible?

Thanks OP for this question!

[Shubhra Kar]

Hi Andy, I posted a detailed answer, but did not show up here. Hope this one does.


Here is a quickstart : http://docs.strongloop.com/display/DOC/Setting+up+a+private+package+registry

We are also building a self-contained lightweight repo server not depending on external services, i.e. using embedded db or filesystem instead of CouchDB + Redis. We use artifactory today and are trying to bridge Reggie to Artifactory for a comprehensive private solution. Let me know if you want to collaborate

-Shubhra Kar
Product Manager, StrongLoop

Using the current version of reggie as a private/on-premise NPM registry

Setup:

# run on the server

npm install -g reggie

reggie-server -d {store-directory}

Publish a package:

# run on developer's machine

npm install -g reggie

reggie -u http://reggie:8080/ publish # run inside your package/module folder

Add the package as a dependency

# package.json:

dependencies: {

 "foo": "http://reggie:8080/package/foo/0.1.0"

}

# Command-line version

npm install --save http://localhost:8080/package/foo/0.1.0

See reggie documentation for the description of all possible version strings (e.g. latest or 0.1.x).

[altsang]

Hey Brian et al,

We've looked into this quite a bit both for what we needed internally for our own CI at StrongLoop and as a product because a bunch of our prospects have asked about a private npm registry with "enterprise" features.  We use artifactory internally because it has great versioning, plays well with other CI (e.g. Jenkins) and does a decent job of storing binaries.

For folks who don't have artifactory from a Java based infrastructure who wanted a private registry we looked at www-npm and Reggie.  www-npm had a lot of moving pieces including CouchDB, ElasticSearch, Redis.  Reggie seemed to fit the bill for the use case of a private registry but it was deficient on a couple of key features centered around a subset of the npm protocol.  One of our engineers Miroslav added this subset and folks can read it about it here - http://strongloop.com/strongblog/deploy-a-private-npm-registry-without-couchdb-or-redis/.

Reggie has simple flat file support for storing the packages, but what we're looking at is an integration to publish from artifactory into Reggie and for Reggie to have a URL hosted by artifactory.  This idea of building in npm support into a artifactory plugin is pretty intriguing though ;)

-a-