ChronicleMap and memory efficiency vs CHM
Kevin Burton
I'm considering porting ActiveMQ's memory store to ChronicleMap to get tighter in-memory representation and fewer GC pauses.
The main motivator is memory. We're wasting a MASSIVE amount of memory hosting our queue. I think it's along the order of 150GB of RAM right now but only for a few million objects.
Chronicle is saying that , in their benchmarks, they're getting 18x memory savings.
In my experience this is probably realistic. In-memory data structures in Java can be horribly inefficient.
(I attached their performance numbers below).
I think the performance tradeoff here is probably realistic. While we have millions of messages our scheduling throughput is more like 1-2k per second vs the 30M per second below. (of course my mileage may vary depending on the encoding/decoding performance).
Has anyone tested this in production?
I would post to the chronicle list but I couldn't find one. If this works I could get this data < 10GB which would be pretty amazing.
We're planning on scaling a LOT larger so this could easily become 1TB of memory if I'm not careful :-P .. that would be pricey!
|
Number of entries |
Chronicle* Throughput |
Chronicle RSS |
HashMap* Throughput |
HashMap Worst GC pause |
HashMap RSS |
|
10 million |
30 Mupd/s |
½ GB |
155 Mupd/s |
2.5 secs |
9 GB |
|
50 million |
31 Mupd/s |
3⅓ GB |
120 Mupd/s |
6.9 secs |
28 GB |
|
250 million |
30 Mupd/s |
14 GB |
114 Mupd/s |
17.3 secs |
76 GB |
|
1000 million |
24 Mupd/s |
57 GB |
OOME |
43 secs |
NA |
|
2500 million |
23 Mupd/s |
126 GB |
Did not test |
NA |
|