RocksDB ingestExternalFile and its impact on DB.get() performance

[Kane Lee]

Hello everyone,

Looking for some clues to help debug an interesting performance issue with RocksDB. Thanks in advance!

Basically, we have an application that has two ways to ingest data:

Method 1. is doing DB.get(key), the check its values, and then do DB.put(key, newValue) conditionally.

Method 2. is to build SST files offline and ingest them via the ingestExternalFile() method, and then ingesting any remaining K/V pairs doing the DB.get()/check/DB.put() flow.

We found Method 1 was not fast enough to bootstrap the database when application restarts, so we added Method2, but found it was not always faster than Method 1, particularly it's much slower to do DB.get() when handling the remaining K/V pairs.

The ingestExternalFile() method could finish very quickly to ingest majority of the K/V pairs in few seconds vs. few minutes of Method 1. But processing the remaining K/V pairs took very long, making the total duration even longer than Method 1 sometimes. The remaining K/V pairs might be 10% or less of all the K/V pairs. 

Looking into RocksDB logs, we saw the `db.get.micros` from Method 2 was like 10X higher than those from Method 1, roughly 300us vs. 30us.

With async-profiler, we could see the two methods had very similar cpu profiles (attached two example profiles from the two methods)

The two methods used same DB/ColumnFamily configs. So the SST files exported from Method 2 used same DB/ColumnFamily configs too. Basically, using BlockBasedTable with same compression LZ4  and index HashIndex.

So I'd wonder if ingestExternalFile() has any known performance impact on following DB.get()s? Perhaps the block cache is not warmed when ingesting SST files directly. Or any other metrics I can check to debug this further. If it's cold cache, any efficient way to warm it after ingesting SST files?

Any clues to help debug this further are welcomed. Thanks!!

best,

Kane

[Kane Lee]

As more context, we put those K/V pairs in 10 files, each with about 500K+ K/Vs, to ingest them in parallel. The server had 16 cores, so could ingest all of them in parallel. 

Method 1 - would use 10 threads to ingest those files in parallel;

Method 2 - would use 1 thread to call ingestExternalFile() and then use 4 threads to ingest the remaining 4 files in parallel. And we notified the DB.get() from those 4 threads was much slower than the 10 threads in Method 1 (300us vs. 30us)

After more tests, we found that for Method 2, if we reduced the SST file size from 100MB down to 2MB, we could reduce the latency of DB.get() from those 4 threads greatly, almost down to 40us (getting very close to 30us as seen by the 10 threads in Method 1). 

Could someone help me understand why ingesting much smaller SST files would reduce DB.get() latency here? This seems a bit counter-intuitive to me as RocksDB does compaction to merge smaller files to bigger one continuously to improve read performance. Thanks!

best,

Kane