ConcurrentHashMap on large heaps >= 15GB

[Kevin Burton]

I'm trying to debug some performance issues with our app.

We use ActiveMQ which in non-persistent mode to get better performance but it's based on ConcurrentHashMap internally.

We've been seeing poor performance when indexing more and more data and I think it primarily has do with maintaining GBs of queue data in a large heap.

My hypothesis is that splitting it out into smaller heaps <= 8GB or so would be smarter and I'm going to try this out tomorrow.

However, the problem with hashmaps is the resize latency which I think is what's biting us.  That and the over-allocation of the hashmap and the non-predictable memory usage.

Are there any modern alternatives to using ConcurrentHashMap with more predictable (and efficient) memory usage?

That and avoiding copy the world when resizing.  However, with concurrent hashmap I know this is less of an issue but it's still present.

[Vitaly Davidovich]

What are the performance issues exactly? Long STW GC pauses? Which GC are you using? Slow down in reads/writes? How many threads (and how many CPUs on the machine) are you hitting it with for reads/writes? What's the ratio of reads to writes? If you could shed more light on the exact problems/symptoms, you may get better answers.

As for CHM alternatives, I know Cliff Click's NonBlockingHashMap is used in some places; IIRC it's more frugal with memory and scales better at very high core counts (concurrency) but slightly slower at lower levels.  This is from memory, so please confirm via your own research.

sent from my phone

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Burton]

On Monday, March 16, 2015 at 7:44:55 PM UTC-7, Vitaly Davidovich wrote:

What are the performance issues exactly? Long STW GC pauses? Which GC are you using? Slow down in reads/writes? How many threads (and how many CPUs on the machine) are you hitting it with for reads/writes? What's the ratio of reads to writes? If you could shed more light on the exact problems/symptoms, you may get better answers.

I'm still debugging it but we haven't seen any huge red flags yet.  But mostly because I"m in the process of instrumenting things to figure out what's happening.

What we're seeing is continual degradation of performance over time. 

 

This directly correlates with more memory used by ActiveMQ so I suspect it's just a problem with the heap being large which makes me think CHM might be part of the problem.  

As for CHM alternatives, I know Cliff Click's NonBlockingHashMap is used in some places; IIRC it's more frugal with memory and scales better at very high core counts (concurrency) but slightly slower at lower levels.  This is from memory, so please confirm via your own research.

Nice.  I'll check into it...  

[Gil Tene]

From experience, CHM itself doesn't have much of an efficiency issue with scaling to large heaps. We have customers running CHMs with 100s of GB of objects in 1TB heaps (and yes, it's the exact same, unmodified CHM Java code that other JDKs use) without seeing any degradation in efficiency.

The resizing thing is different: I doubt that resizing it causing efficiency issues, but it will certainly cause annoying stalls, and resizing stalls will bite [linearly] harder when hash maps are larger. But that is not really "CHM's fault". Any hashmap that stalls operations during a resize will take a similar hit, and that describes pretty much any commonly available hashmaps right now. I discuss this in PauselessHashMap blog entry where I described a solution for a non-concurrent hashmap form, but to date I don't know of one that is available for a concurrent use case [someone should probably build one, right?...].

If the problem isn't the annoying stalls, but actual efficiency degradation (i.e. more and more CPU spent for the same amount of work), and assuming nothing silly is being done with the CHM (like linearly iterating over it, or using terrible hashing functions that make collisions have the same effect), you are most likely looking at the classic GC efficiency loss that occurs as the % of the heap occupancy grows (live set / heap size). As a rule of thumb, GC efficiency roughly doubles with every doubling of empty heap memory. That also means that GC efficiency drops in [roughly] half with every halving of empty heap memory. In terms of work-per-allocation-unit, GC will thrash [efficiency-wise] as you approach using the entire heap for live stuff.

So if GC times are accounting for growing % of overall CPU time, you should consider growing the heap to increase GC efficiency.

Also, keep in mind that the CHM data structure itself is typically only a small fraction of the overall space occupied by its contents. So the data structure efficiency (or not) is not a real concern there. The bulk of the live heap associated with hash maps tends to be associated with the in the values stored in it, and in some cases in the key objects as well.