node-modules.com - a clever search engine for node modules (and a paradigm shift?)

[Mathias Buus Madsen]

On and off over the last six months we've built a search engine for finding modules, http://node-modules.com. It uses several parameters and some cleverness to give you great results quickly.

While using it ourselves we have noticed a paradigm shift in the way we work, which has been really interesting. The last part of this post will go over that.

Node-modules.com and what it does

There are already two search engines in the node community (apart from npmjs.org), http://eirikb.github.com/nipster and http://nodetoolbox.com. However, node-modules.com have something new contribute with. We set out to give you the absolute most relevant results at the very top. Node-modules.com uses different parameters to rank all modules and give better results:

* Name, keywords, and description of the module (cleverly done so that 'parse' and 'parsing' are alike)

* Number of other modules dependant on this module

* Number of stars on github

* If you follow the author

* If the module is used by an author you follow

* If you have starred another module by the same author

Some of these parameters require you to log in with github, but even without it you get great results. By starring more modules and following your favorite authors on github you'll improve your search results.

That's about all there is to it. Why not try some of these:

The top router modules, http://node-modules.com/search?q=router

The top mongo modules, http://node-modules.com/search?q=mongo

Or all the modules substack has created, http://node-modules.com/search?q=@substack

Command-line version

Of course there's also a cli tool for node-modules.com

$ npm install -g node-modules

$ node-modules search web framework

A paradigm shift?

But what was this with a paradigm shift? To avoid this post being too lengthy, and miss the point, we wrote it about on our blog, http://www.reddit.com/r/node_modules/comments/1msh9n/a_paradigm_shift/

Thoughts?

Feedback is much appreciated, @node_modules or the blog, http://reddit.com/r/node_modules

Pull requests accepted, http://github.com/mafintosh/node-modules

// Mathias Buus, http://github.com/mafintosh

// Tobias Baunbæk, http://github.com/freeall

[Jake Verbaten]

npm-modules is amazing.

It's indexing is pretty good and I love the fact it shows me modules by unknown authors because people I know on github have used that module. 

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Arnout Kazemier]

In addition to stars on github it should really take the amount of issues, open pull requests commit activity in consideration when recommedating modules. So it can filter out all the dead and unmaintained projects which are likely to die soon.

--

[Mark Hahn]

>  open pull requests commit activity

There are some modules so simple that the code will never change.  I have several of my own that are untouched over a year but I rely on them heavily.  I do look at commit activity myself.  I'm just saying it isn't always accurate.

The opposite example is socket.io.  The project died and my app's communication got worse and worse and changing to sockjs worked miracles.

[Arnout Kazemier]

Indeed, it shouldn't focus indication but a combination of all, open pull requests, lots of issues and nearly no commits is a good indication that the project is dead and should not be promoted actively. And as much as it hurts me to say this, socket.io is indeed a good example of this.

[Mathias Buus Madsen]

@Raynos:

Glad you like it! (assuming you mean node-modules)

We actually find ourselves starring a lot more stuff on Github now since it'll mean our search results will become better (that's actually how I found your xtend module).

@3rdEden

Currently it adds a penalty to the module rank if the module hasn't been updated within a year.

I think what you're talking about would be a better approach - will look into implementing something like that

[Marco Rogers]

I would also be concerned about false positives when trying to determine if a project is abandoned. If you do try and implement this, please create a way for people to see how the scoring works, so that they can help you refine the algorithm.

:Marco

[Mathias Buus Madsen]

The entire search engine is open-source. You can find it here

https://github.com/mafintosh/node-modules

All contributions are welcome.
The ranking system is working really well as is, so we'll probably just add a 'last published' date and similar info to the results that can give you an indication of whether or not a module is abandoned without changing to much in the scoring algorithm itself.

[Brian Lalor]

On Sep 20, 2013, at 2:35 PM, Mathias Buus Madsen <mathi...@gmail.com> wrote:

On and off over the last six months we've built a search engine for finding modules, http://node-modules.com. It uses several parameters and some cleverness to give you great results quickly.

Very interesting, but the first search for "json parser" is for a shell script. :-) http://node-modules.com/search?q=json+parser

--
Brian Lalor
bla...@bravo5.org
http://github.com/blalor

[Mathias Buus Madsen]

Very interesting, but the first search for "json parser" is for a shell script. :-) http://node-modules.com/search?q=json+parser

JSON.sh is on npm and is very popular so this is actually intended behaviour - the other top results are actually quite good.. In my personalized search jsonparse by Tim Caswell is actually on top for that query since I star a lot of his stuff on github. If there is an easy way to filter out non node/javascript stuff then perhaps we should look into that.

But everyone can help by opening issues on the repository (https://github.com/mafintosh/node-modules) if you think something needs improvement :)

/m

[freeall]

That's definitely a real concern.

One idea was to calculate an activity level for all modules, and then show it with a pixel that changes color based on how active it is. Kind of like the 'pixels' in the Contributions box on your github account, https://github.com/freeall.

It should be visible to the user as to help decide which module to use, but it's harder to make it count in the ranking.

[greelgorke]

awesome!

would be cool if it can handle typos aswell: http://node-modules.com/search?q=json+parser+straem gives me an error messages instead of ignoring (or asuming) the misspelled term

[greelgorke]

and what about http://nodezoo.com/#q=web%20framework ?

[Tobias Baunbæk]

I didn't know nodezoo, but looking into.

As for changing misspelled words, it may be something we'll look into. We're already making sure that similar words like 'parse', 'parser', 'parsing' are the same.

http://node-modules.com/search?q=spell. I see that mscdex have a module for getting suggestions. Seems simple enough.

--

Kind regards,

Tobias Baunbæk,

Co-founder and CEO, Ge.tt

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/FPz3zEqaIig/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.