Sudden memory growth during RocksDB compaction

[Jeevitha Kirubanandam]

Hi,

I have a rocksdb with ~280GB data and our process is running with memory limit set to 16G. Every few days, our process is getting killed by oom-killer. 
According to the heap profiles collected just before crash: ~75% of memory is used by TableIndexReader & Filter data created as a result of BackgroundCompaction.

Increasing the memory limit to 24G didn't help either. It eventually hits the limit and crashes.
Is there a way to limit the memory usage by rocksdb compaction?

Here is the list of DB options we use:
table_options.filter_policy.reset(NewBloomFilterPolicy(8, false /* use_block_based_builder */));

table_options.block_size = 8 * 1024;
table_options.block_cache = NewLRUCache(128 * 1024 * 1024);

options->min_write_buffer_number_to_merge = 3;
options->max_write_buffer_number = 5;

options->env->SetBackgroundThreads(3, Env::Priority::HIGH);
options->max_background_flushes = 3;

options->env->SetBackgroundThreads(4, Env::Priority::LOW);
options->max_background_compactions = 4;

options->level0_file_num_compaction_trigger = 8;
options->level0_slowdown_writes_trigger = 17;
options->level0_stop_writes_trigger = 24;
options->target_file_size_base = 64 * 1024 * 1024;
options->max_bytes_for_level_base = 8 * options->target_file_size_base;
options->max_bytes_for_level_multiplier = 8;
options->target_file_size_multiplier = 3;

Thanks!

[Jeevitha Kirubanandam]

One more data point:
When the memory usage was climbing up, I inspected the db compaction stats. It looked something like:

Level   Files   Size(MB) 

-------------------------

  L0     6/0        568  

  L1    11/0        457  

  L2    73/0       4043  

  L3   523/14     32663  

  L4   245/66    260382  

Why did it end up opening > 70 files? Is this expected? Is there a way to estimate / limit # of sstable compaction thread can bring into the memory?

[Igor Canadi]

Hi Jeevitha,

I replied to your post on Github Issues: https://github.com/facebook/rocksdb/issues/719. Let's continue the conversation there.

Best,

Igor

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To post to this group, send email to roc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/a58d785e-6389-4f4b-a148-ed8afb13517d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.