Very large nosql systems - any experiences?

[eprpa...@gmail.com]

I'm looking for examples of NoSql systems that support 10's of millions
of users, and if you know of any that support 100's of millions so much
the better. Does anyone have experience with these very large systems?
If you are, can you tell us (me) about the deployment of them? How about
performance issues - do you know how many requests can be handled per
second? And so on...

Chaz

[Thomas Vial]

You might want to go through http://highscalability.com/
(though it's not only nosql-related ;-)

--
..........................................................
Thomas VIAL
OCTO Technology
..........................................................
50, Avenue des Champs-Elysées
75008 Paris
Tél : (33) 1 58 56 10 00
Fax : (33) 1 58 56 10 01
GSM : (33) 6 28 50 07 64
Web : http://www.octo.com/
..........................................................

[Tracy Reed]

On Thu, Nov 05, 2009 at 12:00:48PM -0500, eprpa...@gmail.com spake thusly:

> I'm looking for examples of NoSql systems that support 10's of millions
> of users, and if you know of any that support 100's of millions so much
> the better. Does anyone have experience with these very large systems?

Not to be a wise-guy but... http://google.com

And I don't mean google for the answer to your problem. I mean they
ARE exactly what you are looking for. Read their BigTable papers
etc. Best estimates are that Google does hundreds of millions of
search queries per day.

> If you are, can you tell us (me) about the deployment of them? How about
> performance issues - do you know how many requests can be handled per
> second? And so on...

They tend to hold their cards rather closely but they have
occasionally opened up and published papers about their architecture.

http://research.google.com/pubs/papers.html

--
Tracy Reed
http://tracyreed.org

[eprpa...@gmail.com]

Thanks Tracy. I know of all the technology out there what I was hoping
for was some of the other nosql solutions and some people with hands-on
experience.

Chaz

[Rob Adams]

Hbase/hadoop is used widely by yahoo (and it's based on bigtable used by google); cassandra is used by facebook; voldemort is used by linkedin. 

[Asaf Ary]

If you want some more information about the google infrastructure I
found this lecture to be interesting:

http://videolectures.net/wsdm09_dean_cblirs/

It's kinda long (over an hour), but it really gives you some insight
into how Google handle millions of request per day.

Where I work we (obviously) don't reach that volume, but we do handle
many requests per second.
Could you provide a more detailed description of your problem?
What kind of latency demands do you have and how much information
needs to be associated with each user?
Also, are you doing mostly reads, writes or say 90-10 ration?

If you looking at very simple key / value read operations, then you
probably need to know what sort of latency is acceptable.
We use MongoDB for offline data management it successfully holds
against thousands of requests / updates per second and has acceptable
latency for an offline process.
The main reason we chose it was because we had a need for a write-
intensive store which can still handle significant load, another
reason was the relatively simple query language and the fact that it
is written is C.
For the kind of needs you are describing you might need to shard the
MongoDB (which is supported but I haven't tried it) or define swim-
lanes through the load balancer and aggregate the query yourself.
Since the result is in JSON aggregating is really simple.
MongoDB is memory intensive (using memory-mapped files) so on a high-
memory ec2 instance it should really fly.

If you are looking into MongoDB, here are some of the downsides.
1) The drivers are really immature: I use the Java client driver and
it is kinda buggy don't trust it's JSON parsing abilities and use as
little of it as possible, at least until it matured
2) MongoDB limits the number of indexes that may be defined on a table
(used to be 10 now 40) so keep that in mind if you have a need for
many indexes.
3) MongoDB requires a 64bit server: since MongoDB operates in-memory
(it is persistent) it is limited by the amount of memory it can
represent (like your PC) so on a 32bit server you will only get ~3Gb
of data before it maxes out.
If you need to handle massive amounts of data you will need to install
it on a 64bit machine.

I can tell you I'm very pleased with the MongoDB as a NoSql solution
since it providew exactly what we need at scale.
If you can describe your data that would help in tailoring a
solution...

--
Asaf Ary
http://atlarge.tumblr.com

[Swapnil Bawaskar]

Well, not just open source, there are nosql solutions like GemFire you can buy from companies like www.gemstone.com. I work for them and will be happy to help out with specific questions you may have.

-Swapnil.

On Thu, Nov 5, 2009 at 12:04 PM, <eprpa...@gmail.com> wrote:

[K.S. Bhaskar]

What do you mean by 10s of millions of users? Can you estimate in
terms of database sizes and how may accesses per second and what mix
of reads and writes?

Regards
-- Bhaskar