What's faster: Mongo, Ruby or Postgres?

[JoshL]

I currently have the following architecture:

- raw data in mongo as documents
- rollup process, written in Ruby, that runs a few times a day to pull
from Mongo, rollup and stick in Postgres for full-text search.
- ruby routine to do a tsearch against Postgres to pull out data
- ruby code against results to do some post-processing, hash to array
conversions, sorting, etc.

So my question is - is this a good idea or should I try to stick as
much as possible in one place, like Mongo?

I believe I can use MapReduce to create the rollup table I need but
you cant really do a ranked, weighted full text search in Mongo I
don't believe to pull out data.

I figured the group would have some thoughts. So what's faster for
processing data? Mongo, Ruby or Postgres?

[Mitch Pirtle]

Have you looked at ElasticSearch?

-- Mitch

[Robert McIntosh]

or SOLR, which is also based on Lucene.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Stephen Eley]

On Fri, Apr 2, 2010 at 2:09 PM, JoshL <jlip...@gmail.com> wrote:
>

> I figured the group would have some thoughts.  So what's faster for
> processing data?  Mongo, Ruby or Postgres?

The Magic 8-Ball says: "YES."

Your situation is idiosyncratic. Of course *everyone's* situation is
idiosyncratic, but nobody in the peanut gallery is going to be able to
tell you where you should optimize. Particularly when you're talking
about using three _totally_ different classes of technology -- you're
comparing apples and oranges and bananas, and asking which is
fruitier.

But there are some important questions you should consider first.

First: Is your workflow actually complete and working? Are your use
cases being met? (If not, changing things for speed is premature
optimization, which many will tell you is the root of most programming
evil.)

Second: Do you have a performance problem? Are your users
complaining, are you unable to keep up with the flow of data, can you
feel yourself getting older as you wait for results, etc.? (If not,
changing things for speed is a waste of resources.)

Third: Are you profiling and benchmarking your current system to see
where the time is really being spent? (If not, changing things for
speed is just guesswork. You're just as likely to optimize the wrong
thing)

Fourth... Well, there is no 'fourth.' Once you've isolated your
biggest problem area, there'll be a hundred different ways to fix it.
No one way is the 'right' way, although simplifying would probably
help. But anything you were told here could be a bad fit for your
situation, since everything depends on other factors.

Good luck.

(I like oranges, myself.)

--
Have Fun,
Steve Eley (sfe...@gmail.com)
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org

[JoshL]

Ha! Thanks steve.

Well after tinkering I decided to move basically the ENTIRE thing to
mongo. Im using mapReduce to aggregate into a permanent collection on
a regular basis and then using a custom function that does a
conditional group by to pull the data I need from the collection.
It's pretty damn fast and feels a hell of a lot cleaner.

Im not sure if javascript loops in mongo or ruby loops in ruby are
faster, guess I could benchmark, but my guess is the difference is
trivial.

On Apr 2, 6:12 pm, Stephen Eley <sfe...@gmail.com> wrote:

[Ryan Angilly]

FWIW, my guess is that javascript loops in MongoDB's mapReduce blow Ruby out of the water.  I would be SUPER interested to see any benchmarks you put together.

Thanks,

Ryan

[Jon Hancock]

I've been using ruby sunsopt http://outoftime.github.com/sunspot/ for
full text search of my mongo docs. Its a turnkey solr/lucene system.
Unless you really need tsearch/postgres for other reasons, you may
find sunspot to be less trouble.

~Jon

[Chris Heald]

Chances are good that the fastest piece of the puzzle is going to be
MongoDB - Javascript is a very fast language under the right
circumstances, and MongoDB is a "right circumstance". However, for
fulltext search, I'd recommend using something like Solr, Xapian, or
Sphinx; they have fulltext search "solved", and integrate well with
MongoDB by returning a list of ObjectIds for a given query, so you'll
query, say, Xapian with your search text, receive a list of ObjectIds,
and then query your MongoDB collections for those ObjectIds. The
result is a very snappy search without much additional work at all.

[Grig Gheorghiu]

My initial tests with MapReduce in MongoDB show that's it's pretty
slow (see also other threads related to its slowness on this mailing
list). I understand that the issue is being addressed, and that it's
due to javascript code being still slower than desired on the server
side.

In fact, I also experimented with running queries with the 'where'
clause specifying javascript snippets executed on the server, versus
doing my own aggregation in my python code -- and my own aggregation
was faster.

Grig