Rocksdb Memory Usage Stats

[Samantha Steele]

Hi, 

A rocksdb cluster I'm running is causing our machines to run out of memory and get killed by the kernel. Its consuming 60GB+ memory.

I'm trying to track down the leak/issue. estimate-live-data-size is showing the 60GB+ of memory, but all the other rocksdb stats we expose (estimate-table-readers-mem, cur-size-active-mem-table, estimate-num-keys, etc) are all showing less than 5GB of memory. Based on this article https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB, I'm trying to measure block-cache-usage and pinned-usage. The rocksdb code doesn't actually seem to expose these stats. Are there any other recommended ways of tracking down memory usage? 

Setting cache_index_and_filter_blocks = true helped slightly (memory usage topped out around 50GB and then went down), but at a big performance cost. This makes it seem like its not a memory leak. Thanks for any advice. 

Sammy

[Igor Canadi]

You can get memory usage of block cache by calling GetUsage() method on block cache object (i think). Check out include/rocksdb/cache.h

If you are running with jemalloc the best way to see where memory is used is by fetching the heap profile: https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling

More information that would be useful: your database size, block size, average key/value length, block cache size.

Igor

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To post to this group, send email to roc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/a3676893-abbe-4dc9-8e03-555cc5690547%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.