Max connected clients?

[Peter Fales]

I've been trying to get up to speed with Project Voldemort by doing
some performance tests with voldemort-performance-tool.sh. I'm
using a three node cluster with Replication Factor 3, required-
writes=3, and required-reads=3.

When I run with "--threads 25" in a single client, I get pretty
decent performance, 2400 ops/sec with an average response time under
10 milliseconds. However, if increase the number of threads to 75,
I start seeing errors like

[2010-10-04 12:41:14,508 voldemort.store.routed.RoutedStore] WARN
Timed out waiting for put # 2 of 2 to succeed.
Exception in thread "Thread-110"
voldemort.store.InsufficientOperationalNodesException: 1 writes
succeeded, but 3 are required.

And the same thing happens if I stick with 25 threads per client, but
run clients on three different machines. However, if I use three
clients and drop the number of threads to 8, I can get a successful
run.

All of this is leading me to believe that my cluster can only handle
about 30 or so simultaneous clients. Is this expected? Am I doing
something wrong?

I'm using some pretty high-powered servers for these tests (16 core,
2.3Ghz processors with the 32GB ram. I'm using the in-memory data
store.

[Alex Feinberg]

Hi Peter,

You can increase the number of threads on the servers:

max.threads in server.properties

The default setting is 100, you may want to increase it. Additionally,
we have an NIO server available ("enable.nio.connector" to true),
which removes that parameter from the tuning equation.

Here is a good comparison of both the NIO and blocking I/O servers in
terms of concurrent requests they are able to handle:

http://sna-projects.com/blog/2009/08/introducing-the-nio-socketserver-implementation/

Additionally, especially since you're using the in-memory engine, you
may want to make sure your server side garbage collector is tuned
correctly (GC pauses can also lead to timeouts).

Some tips on GC tuning are available at:

http://project-voldemort.com/configuration.php

Thanks,
- Alex

> --
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
>
>

[Peter Fales]

Alex,

Thanks for the suggestions. I had already increased my max.threads
from 100 to 300. Tried changing it to 1000, but it didn't seem to
make any difference. I still see errors when I use a single client
with 75 threads. (Perhaps I'm doing something wrong - is there
anyway to check and see if this parameters is being read and used?)

I also tried "enable.nio.connector" - I think that's being handled,
because I see messages like:

[2010-10-04 15:02:02,032 voldemort.server.niosocket.NioSocketService]
INFO Starting Voldemort NIO socket server (admin-server) on port 6667
[2010-10-04 15:02:02,036 voldemort.server.niosocket.NioSocketService]
INFO Server now listening for connections on port 6667

but that also didn't seem to have much effect (either on throoughput,
or on the number of client threads that can be started)

Any thoughts on what I should be able to expect? (How many clients
should be able connect simultaneously to a 3-node cluster) I'm very
much a newbie with Project Voldemort, but the whole thing is feeling
sort of "fragile" at the moment. I'm still suspicious that I'm
either doing something fundamentally wrong, or expecting something
unreasonable.

[Alex Feinberg]

Ah, just realized you're using the YCSB based performance tool. Have
you tried setting the GC settings for the client as well (you can pass
them as VOLD_OPTS to the shell script)?

Thanks,
- Alex

On Mon, Oct 4, 2010 at 12:07 PM, Alex Feinberg <fein...@gmail.com> wrote:
> Hi Peter,
>
> What are your garbage collection settings (in voldemort-sever.sh) set to?
>
> What other server.properties settings have you made?
>
> What are your other client settings? It's also important to tune your
> client GC and threadpool sizes (e.g., do you have 75 connections
> without 75 threads on the client).
>
> Thanks,
> - Alex

[Alex Feinberg]

Hi Peter,

What are your garbage collection settings (in voldemort-sever.sh) set to?

What other server.properties settings have you made?

What are your other client settings? It's also important to tune your
client GC and threadpool sizes (e.g., do you have 75 connections
without 75 threads on the client).

Thanks,
- Alex

[Peter Fales]

My server options are set to:


VOLD_OPTS="-Xmx2G -server -Dcom.sun.management.jmxremote -
Dcom.sun.management.jmxremote.port=9080 -
Dcom.sun.management.jmxremote.authenticate=false"
VOLD_OPTS="$VOLD_OPTS -XX:+UseConcMarkSweepGC -XX:
+CMSParallelRemarkEnabled -XX:+UseParNewGC"

I had forgotten about that, but apparently I had tried to tweak those
values.

Here are the diffs between my server.properties and the default:

4,5c4
< max.threads=1000
< enable.nio.connector=true
---
> max.threads=100
11d9
< jmx.enable=true
16c14
< bdb.cache.size=100MB
---
> bdb.cache.size=1G
25c23,26
<
storage.configs=voldemort.store.memory.InMemoryStorageConfiguration,voldemort.store.bdb.BdbStorageConfiguration
---
> #NIO connector settings.
> enable.nio.connector=false
>
> storage.configs=voldemort.store.bdb.BdbStorageConfiguration, voldemort.store.readonly.ReadOnlyStorageConfiguration


I'll look into my client settings, but I tend to think that's not the
problem. I actually started testing with a private C++ client, and
got similar results. Before posting here, I tried using the
"official" performance test client to confirm that it behaved about
the same way.

On Oct 4, 2:07 pm, Alex Feinberg <feinb...@gmail.com> wrote:
> Hi Peter,
>
> What are your garbage collection settings (in voldemort-sever.sh) set to?
>
> What other server.properties settings have you made?
>
> What are your other client settings? It's also important to tune your
> client GC and threadpool sizes (e.g., do you have 75 connections
> without 75 threads on the client).
>
> Thanks,
> - Alex
>
> On Mon, Oct 4, 2010 at 12:03 PM, Peter Fales
>

[Tom Chen]

Hi Peter,

Your Xmx2G only allocated 2 gigabytes of memory for the JVM heap, try
a much larger value. Try allocating 30G such as -Xmx30G to your
voldemort servers. I suspect that with all those client threads you
are overwhelming the JVM heap at 2G each.

--
Tom Chen
Software Architect
GOGII, Inc
t...@gogii.net
650-468-6318

[Alex Feinberg]

Try this:

max.threads=300
bdb.cache.size=10G

VOLD_OPTS for server (to take full advantage of your machine)

"-ea -server -Xms18g -Xmx18g -XX:NewSize=2048m -XX:MaxNewSize=2048 \
-XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC \
-XX:CMSInitiatingOccupancyFraction=70"

VOLD_OPTS for the client (to sustain the amount of threads you're sending with)

"-Xmx4g -Xms4g -ea -server -XX:NewSize=768m -XX:MaxNewSize=768m \
-XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseParNewGC \
-XX:CMSInitiatingOccupancyFraction=70""

I'd also suggest trying bdb storage engine along with
InMemoryStorageEngine, as BerkeleyDB should still have excellent
performance when all data fits into memory.

Thanks,
- Alex

[Peter Fales]

Does bdb.cache.size=10G matter if I'm using
voldemort.store.memory.InMemoryStorageConfiguration ?

[Alex Feinberg]

It doesn't, but it will likely mean you can use BerkeleyDB instead of
InMemoryStorageEngine without much of a performance difference if your
data mostly fits into memory. Using BerkeleyDB instead of
InMemoryStorageEngine also means you can take advantage of the
operating system page cache.

Do make sure to set -Xmx/-Xms and the GC settings correctly, however.

Thanks,
- Alex