VPork - Voldemort Performance Testing Framework

[Jon Travis]

Hello Voldemortarians,

I've put together a basic performance testing framework to prove out
Voldemort for use in our company (Hyperic). I am currently in the
process of running a battery of tests and will have some conclusions
in a few days, but in the spirit of Open Source wanted to put what I'd
done out there.

You can read / get it here: http://wiki.github.com/trav/vpork/vpork

The framework assumes you already have a Voldemort cluster up and are
using the configuration style layout that project-voldemort ships
with. (namely, using a config/my-cluster/config/nodes file)

It runs from a single source JVM to the cluster, so cluster based
clients are not supported -- still it does give some useful numbers.

It has 2 components, writers and readers.

A writer simply creates new records, starting at id=1, and ending at
id much > 1.
A reader attempts to read a record which has already been written. By
default, the reader will attempt to read recently written records, but
has the possibility (with decreasing probability) of reading much
older records. This conforms with our usage patterns.

Here are the results of a couple short test runs. In one case, we use
100 threads in the client, in the other we use 30. In both cases we
execute against a BDB 5 node cluster with n=3, r=2, w=2. These were
run in XLarge EC2 instances. Nothing about these 2 tests was tuned --
2GB JVM, default svn checkout of voldemort. I look forward to
tweaking it.

So far I am very interested in what I'm seeing with this community --
seems like people are excited, motivated, and interesting in making
this a very successful project. Shame about the name, though ..
searching for this project on the web has never been easier...

-- Jon

--- 30 Thread Pork ---
Mar 31, 2009 6:23:13 PM - Writes:
Mar 31, 2009 6:23:13 PM - Num Writes: 48072
Mar 31, 2009 6:23:13 PM - Write Failures: 0
Mar 31, 2009 6:23:13 PM - Write Latency: 35.79 ms
Mar 31, 2009 6:23:13 PM - Write Latency (%99): 230.00 ms
Mar 31, 2009 6:23:13 PM - Bytes Written: 1100.28 MB
Mar 31, 2009 6:23:13 PM - Thread w/Throughput: 0.65 KB / ms
Mar 31, 2009 6:23:13 PM - Total w/Throughput: 17.50 KB / ms
Mar 31, 2009 6:23:13 PM -
Mar 31, 2009 6:23:13 PM - Reads:
Mar 31, 2009 6:23:13 PM - Num Read: 5889
Mar 31, 2009 6:23:13 PM - Read Failures: 0
Mar 31, 2009 6:23:14 PM - Read Latency: 28.05 ms
Mar 31, 2009 6:23:14 PM - Read Latency (%99): 218.00 ms
Mar 31, 2009 6:23:14 PM - Read Not Found: 21 (%0.36)
Mar 31, 2009 6:23:14 PM - Bytes Read: 134.15 MB
Mar 31, 2009 6:23:14 PM - Thread r/Throughput: 0.84 KB / ms
Mar 31, 2009 6:23:14 PM - Total r/Throughput: 2.13 KB / ms

---- 100 Thread Pork ----
Mar 31, 2009 6:21:17 PM - Writes:
Mar 31, 2009 6:21:17 PM - Num Writes: 159855
Mar 31, 2009 6:21:17 PM - Write Failures: 0
Mar 31, 2009 6:21:17 PM - Write Latency: 85.94 ms
Mar 31, 2009 6:21:19 PM - Write Latency (%99): 319.00 ms
Mar 31, 2009 6:21:19 PM - Bytes Written: 3658.79 MB
Mar 31, 2009 6:21:19 PM - Thread w/Throughput: 0.27 KB / ms
Mar 31, 2009 6:21:19 PM - Total w/Throughput: 24.29 KB / ms
Mar 31, 2009 6:21:19 PM -
Mar 31, 2009 6:21:19 PM - Reads:
Mar 31, 2009 6:21:19 PM - Num Read: 19872
Mar 31, 2009 6:21:19 PM - Read Failures: 0
Mar 31, 2009 6:21:19 PM - Read Latency: 70.36 ms
Mar 31, 2009 6:21:19 PM - Read Latency (%99): 298.00 ms
Mar 31, 2009 6:21:19 PM - Read Not Found: 12 (%0.06)
Mar 31, 2009 6:21:19 PM - Bytes Read: 453.94 MB
Mar 31, 2009 6:21:19 PM - Thread r/Throughput: 0.33 KB / ms
Mar 31, 2009 6:21:19 PM - Total r/Throughput: 3.01 KB / ms

[Geir Magnusson]

I'm game... why "vpork"?

geir

[Jon Travis]

VoldemortPork was too long.

-- Jon

[Geir Magnusson]

ROTFL.

Ok. Why "Pork"? (It's prob a stupid question...)

geir

[Jay Kreps]

Hey Jon,

That is great. I would really love to get a full performance test
suite going as part of the nightly test run. Maybe this could be a
step towards that...

There is also some additional benchmarking info here:
http://groups.google.com/group/project-voldemort/browse_thread/thread/442a801ffb7f4621/66b7b2b2f16f2ee9?lnk=gst&q=Erich#66b7b2b2f16f2ee9

In general the performance should scale linearly with the number of
nodes. The per-node performance depends highly on the ratio of
data/memory so the cache size you set is very important (this is just
a fact about disks). Some more info is here:

I expect to see > 10k req/sec per node for smallish values. Not sure
if that was what you found since it looks like you have things in
bytes/sec.

-Jay

[Jon Travis]

The max I've seen on this cluster (5 node, n=3, r=2, w=2) is 3500
writes (1kb each) per second.

-- Jon