Recommendations for handling KTypeOpenHashSet memory usage?

[je...@streamsets.com]

We are using a KTypeOpenHashSet typed to an int (so "IntOpenHashSet").  In our particular application, we have a large number of instances of this object, and we have observed that with heavy/sustained usage, the cumulative memory usage continues to grow, mostly in the form of int[] instances.  These are the keys in the hash set.  Periodically, we clear out the hash set by invoking the clear() method, but this does not actually deallocate any of the memory used by the keys array, it just does an Arrays.fill with the default values.

What is the recommended way to deal with this problem when using a KTypeOpenHashSet?  One option is to null out our reference to this object and recreate a new one, allowing the memory for the old instance (including keys array) to be garbage collected.  But this seems a bit messy.  I read through the Javadocs (and source) and wasn't able to find any methods to actually reduce the size of keys.  Has anyone else encountered this issue, or have recommendations?  Thanks.

[Dawid Weiss]

Hi Jeff,

Indeed, there is no gradual "deallocation" for container types. This
was intentional as most of our use cases would allocate additional
keys anyway, even if some were temporarily removed.

I see two solutions.

1. .clear() method indeed doesn't release underlying storage (for the
same reason mentioned above), but .release() method does:

http://carrotsearch.github.io/hppc/releases/0.7.1/api/com/carrotsearch/hppc/IntHashSet.html#release--

2. Other primitive collection libraries are less conservative and do
have shrinking strategies. I believe fastutil and Koloboke do, for
example.

3. A patch to HPPC to implement shrinking on remove() would be most welcome :)

Dawid

> --
> You received this message because you are subscribed to the Google Groups
> "High Performance Primitive Collections for Java" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to
> java-high-performance-primi...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.