BlockTableCache ReplayLog Consumes a lot of Memory

[Pete]

Hi Rocks Team,

I'm running RocksDB from Java app that consumes data from Kafka, validates and stores them in RocksDB. I use LRU block cache, but total used memory is way beyond what I'd expected. I got jemalloc heap profile to see where the memory goes to. It seems to be consumed by appending string in ReplyLog (attached).

I could not find any documentation about this log and how to manage it or how it's used. Glad to learn more

Here are some of my configurations

prefix_extractor=rocksdb.CappedPrefix.18
compression_per_level=kNoCompression:kNoCompression:kZSTD:kZSTD:kZSTD
memtable_whole_key_filtering=true
target_file_size_multiplier=8
max_write_buffer_number=8
blob_compression_type=kNoCompression
compression=kZSTD
level0_stop_writes_trigger=32
level0_slowdown_writes_trigger=16
strict_max_successive_merges=false
level0_file_num_compaction_trigger=8
memtable_prefix_bloom_size_ratio=0.100000
max_write_buffer_number_to_maintain=8

[TableOptions/BlockBasedTable "default"]
num_file_reads_for_auto_readahead=2
initial_auto_readahead_size=8192
metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;}
enable_index_compression=true
pin_top_level_index_and_filter=true
read_amp_bytes_per_bit=0
verify_compression=false
prepopulate_block_cache=kDisable
format_version=6
use_delta_encoding=true
partition_filters=true
metadata_block_size=4096
max_auto_readahead_size=262144
index_block_restart_interval=1
block_size_deviation=10
block_size=16384
detect_filter_construct_corruption=false
no_block_cache=false
checksum=kXXH3
filter_policy=bloomfilter:10:false
data_block_hash_table_util_ratio=0.750000
block_restart_interval=16
index_type=kTwoLevelIndexSearch
pin_l0_filter_and_index_blocks_in_cache=true
data_block_index_type=kDataBlockBinarySearch
cache_index_and_filter_blocks_with_high_priority=true
whole_key_filtering=true
index_shortening=kShortenSeparators
cache_index_and_filter_blocks=true
block_align=false
optimize_filters_for_memory=true
flush_block_policy_factory=FlushBlockBySizePolicyFactory