A "fat" client

[Maarten Koopmans]

Hi,

I have a fat client ( a daemon) that accesses voldemort nodes for its metadata. Let's say I have three Voldemort nodes for now, R=W=1, replication = 3, and I handle version conflicts at the application level.

Now the daemon gets hit by e.g. 100 concurrent clients, each client spawns a thread on the daemon (for a CPU intensive task). Each thread on the daemon may access up to 10 voldemort stores. Meaning I end up with 1000 short-lived clients (I have considered pooling, but that seems to be done at the voldemort client layer with thread executors already).

This is my client config now: 

val clientconfig = new ClientConfig

clientconfig.setBootstrapUrls(bootlist)

clientconfig.setMaxBootstrapRetries(10)

clientconfig.setMaxThreads(128)

clientconfig.setMaxQueuedRequests(5000)

clientconfig.setMaxTotalConnections(25000)

clientconfig.setMaxConnectionsPerNode(9000)

So every thread on the daemon that acts as Voldemort client for its metadata starts 10 Voldemort clients using this config. It works, but I have the feeling this can be way more optimized by some smarter configuration. The profiler (YourKit) revealed nothing though.

Any thoughts? What could be changed, what kind of performance should I expect from the daemon to Voldemort. Server configs below.

Any comments, insights greatly appreciated. 

Thanks,

Maarten

.....and this is what server configs look like, on machines with 17GB memory (high mem Ec2):

node.id=2

max.threads=8192

############### DB options ######################

http.enable=false

socket.enable=true

# BDB

bdb.write.transactions=false

bdb.flush.transactions=false

bdb.cache.size=10000MB

#NIO connector settings.

#enable.nio.connector=false

storage.configs=voldemort.store.bdb.BdbStorageConfiguration

enable.verbose.logging=false

They are started with:

export VOLD_OPTS="-server -Xms16g -Xmx16g -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70"

cd voldemort-0.81

bin/voldemort-server.sh config/my-app