Why is the latency fluctuation of iterators so huge?
Aurora Twinkle
Hi, All:
I'm using rocksdb at work and I'm finding that the latency of my Iterator fluctuates wildly. I hope to get your help.（For more details, please refer to the screenshots. Some logs of rocksdb can be viewed in the attachments.）
My write tps is 6k/sec. The iterator operation is executed once per second, and all keys prefixed with the current timestamp (in seconds) are traversed. Usually 6k kv objects can be traversed per second. The p90 time of the iterator is basically within 500ms, but the p99 time often exceeds 500ms, which is unacceptable to me. Is there any way to improve the performance of iterators and reduce time consumption?
Machine Configuration：
8C32GB
disk: Enhanced SSD, Max throughput (MB/s): 350, Random IOPS:50000
RocksDB Configuration:
writeBufferSize: 256MB
maxwriteBufferNumber:6
maxBackgroundJobs:100
Key format:
The key consists of a second-level timestamp plus a uuid, for example:
1732573201#3070f948-b2e7-4c4a-921a-06a3df033dc2
pseudocode： (System.currentTimeMillis() / 1000 + ThreadLocalRandom.current().nextInt(10, 3600)) + ”#” + UUID.randomUUID().toString()
Value:
The value is a randomly generated string of size 2kb
Aurora Twinkle
Ted Mostly
Iterators by themselves don't use much memory, but it can prevent some resource from being released. This includes:
- memtables and SST files as of the creation time of the iterators. Even if some memtables and SST files are removed after flush or compaction, they are still preserved if an iterator pinned them.
- data blocks for the current iterating position. These blocks will be kept in memory, either pinned in block cache, or in the heap if block cache is not set. Please note that although normally blocks are small, in some extreme cases, a single block can be quite large, if the value size is very large.
So the best use of iterator is to keep it short-lived, so that these resource is freed timely.
An iterator has some creation costs. In some use cases (especially memory-only cases), people want to avoid the creation costs of iterators by reusing iterators. When you are doing it, be aware that in case an iterator getting stale, it can block resource from being released. So make sure you destroy or refresh them if they are not used after some time, e.g. one second. When you need to treat this stale iterator, before release 5.7, you'll need to destroy the iterator and recreate it if needed. Since release 5.7, you can call an API Iterator::Refresh() to refresh it. By calling this function, the iterator is refreshed to represent the recent states, and the stale resource pinned previously is released.