Significant Point Lookup Latency Regression after Upgrading to RocksDB 9.1.0

[ashwin konale]

Hello RocksDB Community,

We have encountered a significant performance regression after upgrading our Java application's rocksdbjni dependency from version 9.0.1 to 9.1.0. The dependency is pulled from Maven Central. Our service runs on AMD EPYC servers.
Our workload is heavily dominated by random point lookups (db.get()) with a read volume of approximately 3 million ops/minute.
After the upgrade, our P99 read latency increased by over 1,200% (from ~150µs to ~2ms), while our read throughput dropped significantly. These are the config overrides.

I have basically hit a wall here. Please let me know if you need more info

Options baseOptions = new Options()
.setCreateIfMissing(true)
.setMaxOpenFiles(-1)
.setMaxBytesForLevelBase(160 * SizeUnit.MB)
.setMaxBytesForLevelMultiplier(10)
.setTargetFileSizeBase(64 * SizeUnit.MB)
.setAllowMmapReads(true)
.setMemTableConfig(new HashSkipListMemTableConfig())
.useFixedLengthPrefixExtractor(8)
.setAllowConcurrentMemtableWrite(false)
.setTableFormatConfig(new BlockBasedTableConfig().setBlockCache(new LRUCache(1024 * SizeUnit.MB)))

[MARK CALLAGHAN]

I have not found problems from 9.0 to 9.1 and other 9.x releases. CPU profiles might be needed to explain this.

Some differences between my setup and yours ...
* I use db_bench and don't use Java
* I compile everything from source
* I don't use mmap reads
* I don't use HashSkipList

A performance report from me is here: https://smalldatum.blogspot.com/2025/05/rocksdb-102-benchmarks-large-small.html
* point query perf is here at low concurrency, but for 9.0 vs 9.5 (no 9.1) -> https://github.com/mdcallag/mytools/blob/master/bench/arc/may25.db_bench/ser7/byrx/sum.byrx.lru.txt#L56-L65
* point query perf is here at high concurrency, including 9.0 and 9.1 ->  https://github.com/mdcallag/mytools/blob/master/bench/arc/may25.db_bench/socket2/byrx/sum.byrx.lru.txt#L101-L119

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rocksdb/6be5498d-b38d-40d0-8e60-3f038cd70537n%40googlegroups.com.

--

Mark Callaghan
mdca...@gmail.com