3n+1 NoSQL/Key-Value/Schema-Free/Schema-Less Database Benchmark

[K.S. Bhaskar]

At http://docs.google.com/View?id=dd5f3337_12fzjpqbc2 I have created a
proposal for a benchmark designed to allow for apples to apples
comparisons of NoSQL databases using features that should allow many
if not most NoSQL engines to be benchmarked. Comments are welcome.
I'll update the document to accommodate useful suggestions and also
create & run a GT.M benchmark.

Regards
-- Bhaskar

[K.S. Bhaskar]

I created the benchmark and ran it on my work laptop today. The
results are at http://docs.google.com/View?id=dd5f3337_13hbmxpjhj

In summary, I was able to get fairly consistently, update rates of
25-30,000 updates per second along with 250-300,000 reads per second
with eight concurrent processes working in parallel on the problem on
a dual core laptop with 4GB RAM on a jfs file system on a 200GB
7,200rpm SATA drive using GT.M V5.3-004A on Kubuntu 9.10. I ran the
database with "before image journaling" which provides for backward
recovery (i.e., in place recovery of the database from the journal
without the need to restore a backup).

I did not spend much time tuning database parameters to maximize
throughput. That's a task for another day.

Questions about how the program works are welcome as are questions and
comments on the results.

Regards
-- Bhaskar

On Nov 21, 1:15 pm, "K.S. Bhaskar" <ksbhas...@gmail.com> wrote:
> Athttp://docs.google.com/View?id=dd5f3337_12fzjpqbc2I have created a

[Cliff Moon]

This reminds me of a proposal that I had made a while back:
http://cliffmoon.tumblr.com/post/128847520/performance-followup-from-nosql

> --
>
> You received this message because you are subscribed to the Google Groups "NOSQL" group.
> To post to this group, send email to nosql-di...@googlegroups.com.
> To unsubscribe from this group, send email to nosql-discussi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nosql-discussion?hl=.
>
>
>
>

[rtweed]

Just for fun I repeated Bhaskar's benchmark on my "reference" Asus Eee
701

(900 MHz Celeron, 2Gb memory Ubuntu 9.04 UNR, with GT.M running in a
VMWare/Ubuntu 9.10 VM, using a 4Gb SD drive with 512Mb memory
allocated)

1 100000 8 351 18 (average of 3 runs)

8 parallel execution streams, total time 18 seconds

So if I've understood Bhaskar's figures, that means it was doing
100,000 database updates @ 5,555 per second and 800,000 database
reads at 44,444 per second

Not a bad demo of NoSQL performance on a piece of throwaway tat? :-)

Rob

On 23 Nov, 02:53, "K.S. Bhaskar" <ksbhas...@gmail.com> wrote:
> I created the benchmark and ran it on my work laptop today.  The

> results are athttp://docs.google.com/View?id=dd5f3337_13hbmxpjhj

>
> In summary, I was able to get fairly consistently, update rates of
> 25-30,000 updates per second along with 250-300,000 reads per second
> with eight concurrent processes working in parallel on the problem on
> a dual core laptop with 4GB RAM on a jfs file system on a 200GB
> 7,200rpm SATA drive using GT.M V5.3-004A on Kubuntu 9.10.  I ran the
> database with "before image journaling" which provides for backward
> recovery (i.e., in place recovery of the database from the journal
> without the need to restore a backup).
>
> I did not spend much time tuning database parameters to maximize
> throughput.  That's a task for another day.
>
> Questions about how the program works are welcome as are questions and
> comments on the results.
>
> Regards
> -- Bhaskar
>
> On Nov 21, 1:15 pm, "K.S.  Bhaskar" <ksbhas...@gmail.com> wrote:
>

> > Athttp://docs.google.com/View?id=dd5f3337_12fzjpqbc2Ihave created a

[rtweed]

OK now a proper repeat of the benchmark using an EC2 AMI as follows:

- Ubuntu 9.10 (ami-1515f67c) m1.small
- GT.M (M/DB installer), 5000 global buffers, Journalling on

1 100000 8 [8 execution streams, 100k updates, 800k reads] : 9 sec
1 1000000 4 [4 execution streams, 1m updates, 4m reads] : 66 sec
1 1000000 8 [8 execution streams, 1m updates, 8m reads]: 84 sec

> > > Athttp://docs.google.com/View?id=dd5f3337_12fzjpqbc2Ihavecreated a

[K.S. Bhaskar]

Rob --

I threw in some number crunching:

1 100000 8 <-- 12,500 updates/sec; 100,000 reads/sec (112,500 ops/sec)
1 1000000 4 <-- 15,152 updates/sec; 60,606 reads/sec (75,758 ops/sec)
1 1000000 8 <-- 11,905 updates/sec; 95,238 reads/sec (107,142 ops/sec)

Regards
-- Bhaskar

[rtweed]

Pushing the EC2 instance harder:

1 1000000 16: 125 seconds [8,000 updates/sec, 128,000 reads/sec]
1 1000000 32: 208 seconds [4,807 updates/sec, 153,846 reads/sec]
1 1000000 64: 365 seconds [2,740 updates/sec, 175,342 reads/sec]

[Steve Davis]

Good performance profiles of the behavior of the various "nosql"
implementations would, for sure, help greatly in making realistic
decisions about which implementation would be a good choice for a
particular application requirement. Comparison with at least one known
RDBMS would also be a great addition.

If someone stands up to do this I would definitely make a personal
contribution to any continuing profiling effort, e.g. via paypal, etc.

/s

[K.S. Bhaskar]

Results from an enhanced benchmark (where the data are stored as
strings per George James' suggestion) are at http://docs.google.com/View?id=dd5f3337_15hssw6bdh
- as expected, update rates were quite a bit slower (there was a lot
more work being done), but still respectable. Again, I did not spend
time tuning the database and just used the parameters I used the last
time.

-- Bhaskar

On Nov 22, 9:53 pm, "K.S. Bhaskar" <ksbhas...@gmail.com> wrote:
> I created the benchmark and ran it on my work laptop today.  The

> results are athttp://docs.google.com/View?id=dd5f3337_13hbmxpjhj

>
> In summary, I was able to get fairly consistently, update rates of
> 25-30,000 updates per second along with 250-300,000 reads per second
> with eight concurrent processes working in parallel on the problem on
> a dual core laptop with 4GB RAM on a jfs file system on a 200GB
> 7,200rpm SATA drive using GT.M V5.3-004A on Kubuntu 9.10.  I ran the
> database with "before image journaling" which provides for backward
> recovery (i.e., in place recovery of the database from the journal
> without the need to restore a backup).
>
> I did not spend much time tuning database parameters to maximize
> throughput.  That's a task for another day.
>
> Questions about how the program works are welcome as are questions and
> comments on the results.
>
> Regards
> -- Bhaskar
>
> On Nov 21, 1:15 pm, "K.S.  Bhaskar" <ksbhas...@gmail.com> wrote:
>

> > Athttp://docs.google.com/View?id=dd5f3337_12fzjpqbc2Ihave created a

[rtweed]

So nobody wants to re-run Bhaskar's apples v apples benchamrk against
their own favourite NoSQL database?

On 1 Dec, 20:56, "K.S. Bhaskar" <ksbhas...@gmail.com> wrote:
> Results from an enhanced benchmark (where the data are stored as

> strings per George James' suggestion) are athttp://docs.google.com/View?id=dd5f3337_15hssw6bdh

> - as expected, update rates were quite a bit slower (there was a lot
> more work being done), but still respectable.  Again, I did not spend
> time tuning the database and just used the parameters I used the last
> time.
>
> -- Bhaskar
>
> On Nov 22, 9:53 pm, "K.S.  Bhaskar" <ksbhas...@gmail.com> wrote:
>
> > I created the benchmark and ran it on my work laptop today.  The
> > results are athttp://docs.google.com/View?id=dd5f3337_13hbmxpjhj
>
> > In summary, I was able to get fairly consistently, update rates of
> > 25-30,000 updates per second along with 250-300,000 reads per second
> > with eight concurrent processes working in parallel on the problem on
> > a dual core laptop with 4GB RAM on a jfs file system on a 200GB
> > 7,200rpm SATA drive using GT.M V5.3-004A on Kubuntu 9.10.  I ran the
> > database with "before image journaling" which provides for backward
> > recovery (i.e., in place recovery of the database from the journal
> > without the need to restore a backup).
>
> > I did not spend much time tuning database parameters to maximize
> > throughput.  That's a task for another day.
>
> > Questions about how the program works are welcome as are questions and
> > comments on the results.
>
> > Regards
> > -- Bhaskar
>
> > On Nov 21, 1:15 pm, "K.S.  Bhaskar" <ksbhas...@gmail.com> wrote:
>

> > > Athttp://docs.google.com/View?id=dd5f3337_12fzjpqbc2Ihavecreated a

[mscsf...@pucit.edu.pk]

Hi Bhasker ,
I need your help regarding designing of a benchmark proposal.