Voldemort for 100 TB data with 1 billion key-value pairs

[Srikanth]

Is it possible to scale voldemort for key-value store with approx. 1-
billion key-value pairs where each value is around 50KB in size and
cumulative data is in the range of 100TB?

Can we achieve a random-read response time of ~100-200ms?
How many nodes will one need in the cluster?

Any information on indicative figures for number of nodes or
feasibility of such as setup will be highly helpful.

Thanks,
Srikanth

[OG]

I imagine the request rate is an important factor...
Out of curiosity, what sort of content is this?

Otis

[Srikanth]

Each entry will most probably be a hash map like structure that will
store data such as user profile - name, mobile, etc..

> > Srikanth- Hide quoted text -
>
> - Show quoted text -

[Tom Chen]

Hi Srikanth,

How random is the lookup for your data? Is it completely random or can
you rely on a fixed number of users to come in?

In the completely random case, random disk performance will be
essential. Depending on how fast your disk storage is, you can max out
the number of entries in your BDB storage engine until performance is
unacceptable( > 300 ms in your case.)

If your data is pretty hot and you can rely on a cache, you can
probably size your cache using this utility? You will want to make
sure that your data can mostly fit in cache, and perhaps misses are
not a big deal.

http://www.oracle.com/technology/documentation/berkeley-db/je/java/com/sleepycat/je/util/DbCacheSize.html

Take a look at:

http://sna-projects.com/blog/2009/08/introducing-the-nio-socketserver-implementation/

Note as the size of your data gets bigger, the throughput decreases.

Tom

> --
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
>
>

--
Tom Chen
Software Architect
GOGII, Inc
t...@gogii.net
650-468-6318

[Srikanth]

Tom, Thanks for your inputs.