Manifest file size is huge

[Krish Chandra]

Hi,

We are using leveldb version 1.20. Ours is a java application which will do put and delete batch operations towards leveldb via jni. in one of our servers we have seen that manifest file size has been increased to 25GB. Can we know reason for this increase. Also if we execute repair db will our application be crashed due to large Manifest size. Do our application store that manifest file in memory and process the rdb operation? Or do we have any other impacts like extra RAM consumption when we execute rdb?

Please find below the settings of our leveldb and repair db:
LevelDB:

writebuffersize=64
cachesize=3072
compressiontype=SNAPPY
totaldbinstance=1
#Bloom filter policy bit count for db
filterpolicy.bitcount=100

RepairDB:
writebuffersize=4
compressiontype=SNAPPY
cachesize=4
#Bloom filter policy bit count for repair db
bitcount=10

Also whenever we restart our JVM, we will open a close the existing db and reopen new db. During that time if manifest file size is huge, Our the RSS of our application in top command is keep on growing and at one point application is crashing due to no physical memory. Why the entire Manifest file is getting loaded? Even though we have configured the Cache size as 3GB, why our RSS increasing beyond 10GB and application is crashing

Regards,

Krishna

[Krish Chandra]

Hi All,

We have tested the same issue on 1.23 and issue still persists. We have found that below patch has been pushed recently for the similar issue:

https://github.com/google/leveldb/pull/917

We have below queries:

Does rotation Manifest frequently have any performance impacts?

During rotation Is it only the manifest file content or entire database content is read?

Regards,

Krishna

[Daniel Murphy]

Some thoughts that may or may not help (it's been a while since I've poked at leveldb):

• The manifest-reuse feature causes manifest to be reloaded on startup, which is faster, but can sometimes prevent paging that to level files because the heuristics are reset on startup (read & write count)
• The manifest file gets compacted to level files when there are enough reads or writes (I believe).
• When manifest reuse is on, and the access pattern of the database is to do little operations and then close the database, this can lead to manifest file growing quite a lot & maybe never shrinking?

These are bugs that need to be fixed - to mitigate, in general it's a good idea to keep the database open so that these compactions happen consistently. Also, I think you can change some options around max/min batch size or file size or something, which might have an effect here.