(npm downtime) npm public mirror / private mirror / caching / multi server list solution

[Dan Jenkins]

Hi All,

I'm not aiming to bash anyone or anything that someone has put in place, just trying to work out a solution to an ever growing issue - so let's be constructive!

So we all know that npm goes down occasionally, it's only happening more and more as popularity grows. 

And then guess what, we all get mighty annoyed and blame "npm" and say it shouldn't happen. Well guess what, we are a community, so let's try and sort it out as a community.

So there's many different options out there:

1. Public mirror of the couchdb install
• But how do people access this, they can't currently unless you change some settings in your npmrc file
2. Private mirror of the couchdb install
• As a business, I want stability on something I rely on, so let's take a private mirror and push our private modules to it while we're at it - then we know our replica will be available
3. Cache Proxy
• Using something like nexus oss to proxy npm - (https://issues.sonatype.org/browse/NEXUS-5852?page=com.atlassian.streams.streams-jira-plugin:activity-stream-issue-tab)
• Or something like Artifactory - (https://www.jfrog.com/jira/browse/RTFACT-5930)
4. Add more servers, more official npm mirrors around the world (under registry.npmjs.org)
• Is there an official way for people to donate to the cause?
• Can you donate money?
• Can you donate a server + bandwidth?
5. Make it easy to have a list of public mirrors 

There's probably more but that's all I can think of right now.

But these all have their own issues.

When we talk about making a replica, whether it be public or private - you've got a 100gb replica (last time i checked it wasn't far off this), it may even be more than this now. That's alot of data; I can completely understand why npm "goes down" so often, it's having to store and transfer all this data...

So there's solutions to the issue of size - take all of the artifacts out of couchdb for instance and put them on a cdn, then npm just becomes meta data, and wouldn't be so massive, which would solve replication issues, with the option of a private npm replica storing the artifact in couchdb - with a backup going back to github/bitbucket's tag artifact or whatever.

How many public npm replicas are there out on the net? If there really are a load, then why not allow a list of npm servers in the npmrc file? If this all gets agreed, then let's put it into action. Seems like it would be a small-ish change to npm?

Is there anything we can do to help these other projects like nexus/artifactory/a tonne more out there to move along quicker?

Are there any other solutions out there? Let's get together and try and sort it, or at least come up with a plan on how to tackle this and make it public so that people know what's going on,

I know people have talked about these things before but I've never seen anything come from them, so what's come of them?

I was also wondering, are there any public stats on downtime? Like a pingdom report looking at registry.npmjs.org?

EDIT - After looking for status page for npm i saw there is pingdom reports - http://stats.pingdom.com/d50hxzpzk7x4/650599

Could we also get a public status page for npm? Something like https://www.statuspage.io/ - the last thing you want when npm goes down is for people to carry on trying to access npm and make the issue worse. I'm sure they'd donate an account to npm...

EDIT - I've just remembered about http://status.npmjs.org/

Node.js is taking off, we've seen countless times how package growth is just growing and growing, it's absolutely amazing,

Now let's all try and help, as a community, if there is already something out there on the net, about plans etc can you point me to it and I'll close this topic down,

If anyone has any other ideas, bring them up!

Dan

[Luke Arduini]

Dan,

Check this blog post for some upcoming changes that should help with npm availability

http://blog.nodejitsu.com/npm-innovation-through-modularity

Go to "What's next for npm?"

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dan Jenkins]

Awesome, thanks for that

So what can the community do to help here?

Can these kinds of things go somewhere more public in terms of npm? I mean, I did a google search but didn't find this...

It's great that iriscouch and nodejitsu do so much for npm - but should it be up to you guys to handle it all as such?

If for example, multiple npm registries has been agreed upon, is this currently under development? If it's not, is there anything stopping the community submitting a PR? And even if this happens, how do we go about finding public replicas?

Thanks for replying!

[Alain Mouette]

Em 13-11-2013 16:16, Dan Jenkins escreveu:

So there's many different options out there:

1. Public mirror of the couchdb install
• But how do people access this, they can't currently unless you change some settings in your npmrc file

npm itself could come configured with a list of available servers.
The list would be updated whenever npm itself is updated...

just my 2¢
Alain

[Steve Mason]

This is last I heard WRT multiple registries: https://github.com/mbrevoort/node-reggie/issues/14#issuecomment-22586570

There doesn't seem to be a public place where anything like this is being co-ordinated (unless it is this list? The google group seems dead) and some kind of update about what's generally would be nice, especially given this comment.

[Alex Kocharin]

There are quite a few existing caches out there (sinopia, reggie), including npm cache itself (--no-registry option afair). They'll help you access packages you downloaded before, but if you need a new one and registry is down, you're in trouble.

List of public mirrors will be nice. Maybe even go crazy and think about p2p here. Sadly, npm packages aren't signed, and trust issue comes out.

Another solution would be to use github like bower does, but it can go down as well. :)