Performance of compactions (kvrocks)

[JM C]

Hi!

I'm currently using rocksdb behind kvrocks to store 3TB of key-value data.

The main problem we face is during bulk loads (several times a day), since the auto compaction makes the database unavailable. We tried disabling auto compaction and running a compact after the load, but the memory consumption is too big.

I wonder what tuning we can try to make reads available while writing with auto compaction. I dont care if writes takes longer, as soon as the database is available. 

Tkank you!

[Siying Dong]

Can you explain why auto compaction makes the database unavailable? What error did you see?

Also, what’s the reason of memory consumption too big if running compaction after bulkloading? What consumed memory in this case?

 

Thanks,

 

Siying

 

From: roc...@googlegroups.com <roc...@googlegroups.com> On Behalf Of JM C
Sent: Wednesday, October 19, 2022 10:36 AM
To: rocksdb <roc...@googlegroups.com>
Subject: Performance of compactions (kvrocks)

 

Hi! I'm currently using rocksdb behind kvrocks to store 3TB of key-value data. The main problem we face is during bulk loads (several times a day), since the auto compaction makes the database unavailable. We tried disabling auto compaction

ZjQcmQRYFpfptBannerStart

This Message Is From an Untrusted Sender You have not previously corresponded with this sender.

ZjQcmQRYFpfptBannerEnd

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/b34639c7-5e13-43e0-a448-5018c981225cn%40googlegroups.com.

[Mark Callaghan]

If the problem is worst-case write stalls and you use leveled compaction then consider upgrading to RocksDB version 7.5.3 are significantly reduced. In my benchmarks they could be really bad prior to 7.5.3 and now are much less than one second in 7.5.3.

[Dan Carfas]

Shared your question in the Speedb hive (Discord) and this is the buzz:

"Not sure if this is exposed through the KVrocks interface, but in RocksDB you can set a write rate limiter if you don't care about writes taking longer to complete. See the rate_limiter member of DBOptions (accesible through the Options type as well)."

"also, the recommended way of loading large amounts of data is through the SstFileWriter (https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files) . incases where thats not possible, you can try to use WriteOptions.low_pri=true (https://github.com/facebook/rocksdb/wiki/Low-Priority-Write)"

Join the discussion on Discord or reply here and I'll pass the message.
If you want more assistance or got more questions, check our Discourse page.

[hulk]

> If the problem is worst-case write stalls and you use leveled compaction then consider upgrading to RocksDB version 7.5.3 are significantly reduced. In my benchmarks they could be really bad prior to 7.5.3 and now are much less than one second in 7.5.3.

That sounds good. Kvrocks now has upgraded the C++ standard to C++17 and some users also confirmed it works well with RocksDB 7.0.x: https://github.com/apache/incubator-kvrocks/discussions/1013. I think can have a try at 7.5.3 to see if it can solve the compaction issue.


>  increase where thats not possible, you can try to use WriteOptions.low_pri=true

I think it cannot work since it will slowdown the write operation in Kvrocks, this will also affect Kvrocks read since they shared the thread pool.

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/9111375f-3e61-481f-aafa-0b70a0fbbafcn%40googlegroups.com.

--

Best Regards,
- Hulk Lin

[hulk.w...@gmail.com]

Hi Mark,

Kvrocks community users tested 7.7.3 and the benchmark result was much better than 6.29.5. Many thanks for your suggestion.

By the way, I'm very curious about how rocksdb achieves this significant improvement. Could you give us the link to the issue or PR mentioned this improvement? 

Thanks a lot.

[Mark Callaghan]

Initial discussion was here -- https://github.com/facebook/rocksdb/issues/9423

First PR: https://github.com/facebook/rocksdb/pull/10057
Second PR: https://github.com/facebook/rocksdb/pull/10270

[hulk]

Hi Mark.

Thanks for your reply and input.

You received this message because you are subscribed to a topic in the Google Groups "rocksdb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rocksdb/umRPboha0as/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/f3a20bc0-4119-456d-9bf8-bd7d4a4903d1n%40googlegroups.com.