RocksDB: Hundreds of 4KB SST files

[Gurjinder Singh Brar]

We open the DB and when the operation is done, we close the DB. Is this pattern leading to too many small-size SST files? 

Options are:

 table_factory=BlockBasedTable

target_file_size_base=67108864

 block_size=4096

flush_block_policy_factory=FlushBlockBySizePolicyFactory

max_write_buffer_number=2

 write_buffer_size=67108864

[曹志超]

Is there only a small amount of write queries? Also, how many column families do you have?

zhichao

Gurjinder Singh Brar <joint...@gmail.com> 于2021年4月8日周四 上午11:58写道：

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/5ef0919c-7eaf-4d02-91b6-10462799be70n%40googlegroups.com.

[Gurjinder Singh Brar]

Yes, Write queries are fewer, and we are using only the default column family. When we need to run the Read or Write operation, we open the DB and close it after 10 seconds. 

[曹志超]

Can you share more information, like how many SST files you have seen and what's their sizes?

Zhichao

Gurjinder Singh Brar <joint...@gmail.com> 于2021年4月8日周四 下午12:10写道：

[Gurjinder Singh Brar]

Still searching for the solution,

Open rocksDB, insert a record and close it. It will create a new 4KB sst file. Is there any way to force rockDB to add files in the same sst?

[曹志超]

SST files are immutable after it is created. This is the basic design principle of LSM based KV. So, after you insert a new record and close it, a Flush is triggered and it will create a new SST file for sure.

Gurjinder Singh Brar <joint...@gmail.com> 于2021年9月13日周一 下午4:11写道：

[Tomas Kolda]

The problem is that flushed SST is not in collision with existing files so that it is just creating new ones. Normally when you have random writes it should merge together after some time.

It is happening for us as well and only solution for me was to write custom job that simply look for these small files. If there are too many small adjacent files on same level I simply compact them together with CompactFiles. My algorithm use sorted files (by user key) and count size. If there are more then X amount of adjacent files with total size less then Y then I merge there into one.

Tomas

[Eric Maynard]

Yes, that is the issue we have as well. The keys do not overlap and our application sometimes generates thousands of SST files.

We have tried a similar approach to what @Tomas proposed, but it seems that CompactFiles does not combine files that don't overlap.

Eric M.