voldemort on EC2 and maxed file descriptor behavior

[kindageeky]

just an FYI, this was a single node stress test (meaning lots of
concurrent put / get activity), test client and server ran locally on
the same instance ... we're planning on doing some distributed scale
and failure testing in EC2 in the next couple weeks.

stress test - single instance local test agent execution time = 2 hrs
38 minutes
total operations against voldemort = 198,000,000
total inserts = 6,000,000
total updates = 6,000,000
total reads = 186,000,000
1st pass read latency = 36 ms
1st pass read throughput = 16,620 ops / sec
insert latency = 60 ms
insert throughput = 996 ops / sec
update latency = 60 ms
update throughput = 1000 ops / sec
2nd pass read latency = 32 ms
2nd pass read throughput = 18,900 ops / sec
3rd pass read latency = 25 ms
3rd pass read throughput = 23,820 ops / sec
data size of individual values = 1KB
Size of raw dataset = ~100MB (100,000 * 1024)
Size of storage on disk = ~1 GB
Size of metadata on disk = ~950 MB
Peak read byte throughput = ~24 MB / second
Floor read byte throughput = ~18 MB / second
Contention Error Rate = ~2.7% (almost all of these are expected from
the nature of the test) Server CPU Utilization = ~25%
Test Agent CPU Utilization = ~35%
Context Switching and IO = ~16%
Server RAM allocation = 4GB for voldemort, 1GB for btree cache
B-Tree fanout = 1024
Max Server Threads = 20000
Max Client Threads = 10000
Scheduler Threads = 200
Test Threads = 720
Max Configured Sockets = ~20000
Max Used Sockets = ~720 each (client and server)
Hardware Spec = 400GB single HDD, 7GB RAM, 8 cores at effectively 2.5
GHz (XL compute instance with 15 amazon processing units); Ubuntu 8.10
server

some of the errors encountered ...

Exception in thread "handler609757"
java.lang.IllegalArgumentException: No mapping code for class
voldemort.store.PersistenceFailureException
[2009-02at voldemort.store.ErrorCodeMapper.getCode
(ErrorCodeMapper.java:59)t.server.socket.SocketServer)e)
32637 ecat
voldemort.server.socket.StreamStoreRequestHandler.writeException
(StreamStoreRequestHandler.java:
132)
32612 ecat voldemort.server.socket.StreamStoreRequestHandler.handlePut
(StreamStoreRequestHandler.java:
113)
11679 ecat
voldemort.server.socket.StreamStoreRequestHandler.handleRequest
(StreamStoreRequestHandler.java:
71)
22817 ecat voldemort.server.socket.SocketServer$SocketServerSession.run
(SocketServer.java:
196)
1 roat java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:
1110)
2 roat java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:
603)
3 roat java.lang.Thread.run(Thread.java:636).0 0:00.02 ksoftirqd/
0


other exceptions encountered
- server dies when ephemeral port range is insufficient (e.g. too many
files error)
- NoSuchElementException
Exception in thread "voldemort-client-thread-1402"
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:813)
at java.util.HashMap$KeyIterator.next(HashMap.java:845)
at com.google.common.collect.StandardMultimap$WrappedCollection
$WrappedIterator.next(StandardMultimap.java:469)
at voldemort.store.routed.ReadRepairer.getRepairs(ReadRepairer.java:
108)
at voldemort.store.routed.RoutedStore$3.run(RoutedStore.java:343)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

- these happen together
Exception in thread "insert"
voldemort.store.InsufficientOperationalNodesException: No master node
succeeded!
Exception in thread "handler493151"
java.lang.IllegalArgumentException: No mapping code for class
voldemort.store.PersistenceFailureException at
voldemort.store.routed.RoutedStore.put(RoutedStore.java:425)

at voldemort.store.routed.RoutedStore.put(RoutedStore.java:60) at
voldemort.store.ErrorCodeMapper.getCode(ErrorCodeMapper.java:59)
at voldemort.store.serialized.SerializingStore.put
(SerializingStore.java:69)

at voldemort.server.socket.StreamStoreRequestHandler.writeException
(StreamStoreRequestHandler.java:132) at
voldemort.store.DelegatingStore.put(DelegatingStore.java:63)

at voldemort.server.socket.StreamStoreRequestHandler.handlePut
(StreamStoreRequestHandler.java:113) at
voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:106)

at voldemort.server.socket.StreamStoreRequestHandler.handleRequest
(StreamStoreRequestHandler.java:71) at
com.ecollege.ei.poc.asss.LoadTest$InsertThread.run(LoadTest.java:177)

Exception in thread "handler776530"
java.lang.IllegalArgumentException: No mapping code for class
voldemort.store.PersistenceFailureException RES SHR S %CPU %MEM
TIME+ COMMAND
2148 ecat voldemort.store.ErrorCodeMapper.getCode
(ErrorCodeMapper.java:59)Exception in thread "update"
voldemort.store.InsufficientOperationalNodesException: No master node
succeeded!
10339 ecat voldemort.store.routed.RoutedStore.put(RoutedStore.java:
425)
5646 ecat voldemort.store.routed.RoutedStore.put(RoutedStore.java:
60)
14143 ecat voldemort.store.serialized.SerializingStore.put
(SerializingStore.java:69)
7974 ecat voldemort.store.DelegatingStore.put(DelegatingStore.java:
63)
1 roat voldemort.client.DefaultStoreClient.put
(DefaultStoreClient.java:119)
2 roat
voldemort.server.socket.StreamStoreRequestHandler.writeException
(StreamStoreRequestHandler.java:132) at
com.ecollege.ei.poc.asss.LoadTest$UpdateThread.run(LoadTest.java:133)
0
4 root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/
0
5 roat java.lang.Thread.run(Thread.java:636).0 0:00.01 events/
0
6 roat voldemort.server.socket.StreamStoreRequestHandler.handlePut
(StreamStoreRequestHandler.java:113)
7 roat
voldemort.server.socket.StreamStoreRequestHandler.handleRequest
(StreamStoreRequestHandler.java:71)
Caused by: voldemort.VoldemortException: java.io.EOFExceptionxenat
voldemort.server.socket.SocketServer$SocketServerSession.run
(SocketServer.java:196)0 S 0 0.0 0:00.00
xenbus
17 roat java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1110)
18 roat java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

at voldemort.store.socket.SocketStore.put(SocketStore.java:139)
at voldemort.store.socket.SocketStore.put(SocketStore.java:46)
at voldemort.store.logging.LoggingStore.put(LoggingStore.java:121)
at voldemort.store.routed.RoutedStore.put(RoutedStore.java:406)
... 6 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at voldemort.store.socket.SocketStore.checkException(SocketStore.java:
150)
at voldemort.store.socket.SocketStore.put(SocketStore.java:136)
... 9 more


On balance, voldemort did great with this stress test. One idea for
enhancement would be to not have the server crash in the case of
insufficient file descriptor / sockets available for allocation; seems
like you'd rather serve some traffic rather than crash in this
scenario. On ubuntu we raised the security limit and the physical
limits on open files to 30,000 to avoid problems in this test.