Polling the npm registry for changes

[Isaac Schlueter]

If you're repeatedly fetching the JSON data from the npm registry for
some reason (it, for http://nodul.es or http://npm.mape.me or
@nodemodules or whatever other crazy thing you may be considering),
then you should leverage the _changes feed on the npm registry's
underlying couchdb instance. It's at:

http://isaacs.couchone.com/jsregistry/_changes

That'll give you a "results" array, and a "last_seq" id. Each of the
objects in the "results" array has an "id" field which is the name of
the package. To get the data for that package (and see what changed)
you can fetch

http://registry.npmjs.org/<id>

and do whatever you want to do with it.

Then, hang onto that "last_seq" id, and make a request to:

http://isaacs.couchone.com/jsregistry/_changes?since=<last_seq>&feed=longpoll

This connection will hang open until the db has new data to push to
you. When it returns, process the new things, get the last_seq id,
and go again.

That way, you'll only request what you need, and only when you know you need it.

Also! You can replicate the npm registry, and do other creative
things with it. It's just a couchapp. In fact, you can even go to
http://isaacs.couchone.com/_utils and log in with your npm username
and password (if you've ever done "npm adduser") and manually modify
your packages or your user record. I'd definitely suggest doing
things through the npm client whenever possible, because that will
prevent most kinds of data integrity issues.

If I end up eventually moving off of couchone (not likely), I'll give
you all ample warning.

It's been very interesting to see the creative things that people have
done with this data. Noders ftw! :)

--i

[Isaac Schlueter]

Oh, I forgot to mention, if you do replicate the database, then you
might be interested in playing around with the couchapp:

http://github.com/isaacs/js-registry

If you want to play around with adding views or other fun things to
the couchapp itself, I'd be very excited to see what any of you come
up with.

Thanks!

--i

[Isaac Schlueter]

chewbranca in #couchdb just clued me into some more couchdb _changes magic:

Here's Jan talking about it:
http://www.youtube.com/watch?v=WaH4_Hq33PI

So, for node apps, what you want is this:

http://isaacs.couchone.com/jsregistry/_changes?include_docs=true&feed=continuous

That has all the info, and it'll keep dumping out data as it comes in.
Still give it a &last_seq=### if you have one, but then you can sit
there waiting, and respond to body events that give you the data about
what changed.

CouchDB is quite winful.

--i

[fuzzy spoon]

Thanks for this info as well, nice work Isaac.

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

[Tane Piper]

+1 on the thank you

[fuzzy spoon]

As noted on IRC by digitalspaghetti

http://isaacs.couchone.com/jsregistry/_changes?include_docs=true&feed=continuous&last_seq=995

^^^ This is incorrect, its not last_seq its since=

http://isaacs.couchone.com/jsregistry/_changes?include_docs=true&feed=continuous&since=995