performance of MultiGet

[10141...@qq.com]

version:rocksdb 6.15.2

I open a DB and put 2000W k-v using writebatch.
The keys range from (1,20000000), and the values are normal string.

```

DB* db; Options options; options.IncreaseParallelism(); options.OptimizeLevelStyleCompaction(); options.max_open_file=-1; options.write_buffer_size=1024*1024*1024; options.create_if_missing=true; Status s=DB::Open(options,dbPath,&db);

```
 

Using MultiGet to get 500 keys costs 15 secondes.
pgsql costs only 200ms under the same conditions after build index btree.

I mean rocksDB slower than sqlDB?

Is there any wrong on options or set-up?

[Siying Dong]

Can you share perf context of your query?

Here is how you can get it: https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/930d1d27-77ed-4457-bfa3-487aa11ede69n%40googlegroups.com.

[陈庆国]

Here are some of context. 

Most data is retrieved from block.

In my scenario,cold data and hot data are equally likely to be retrieved.

user_key_comparison_count=19960

block_cache_hit_count=2

block_read_count=1058

block_read_byte=4379421

block_read_time=54048875682

block_cache_index_hit_count=0,

index_block_read_count=0

block_cache_filter_hit_count=0

filter_block_read_count=0

compression_dict_block_read_count=0,

block_checksum_time=8066710

block_decompress_time=0,

get_read_bytes=0,

multiget_read_bytes=409600

iter_read_bytes=0

internal_key_skipped_count=0

internal_delete_skipped_count=0,

internal_recent_skipped_count=0,

internal_merge_count=0,

write_wal_time=0,

get_snapshot_time=501429

get_from_memtable_time=0,

get_from_memtable_count=0

get_post_process_time=128673,

get_from_output_files_time=54143504035

seek_on_memtable_time=O

seek_on_memtable_count=0,

next_on_memtable_count=0

prev_on_memtable_count=0,

seek_child_seek_time=0

------------------ 原始邮件 ------------------

发件人: "Siying Dong" <siyi...@fb.com>;

发送时间: 2021年1月28日(星期四) 凌晨5:56

收件人: "陈庆国"<10141...@qq.com>;"rocksdb"<roc...@googlegroups.com>;

主题: RE: performance of MultiGet

[Siying Dong]

Look at these lines:

block_cache_hit_count=2

block_read_count=1058

block_read_byte=4379421

block_read_time=54048875682

 

Block cache only hit for 2 blocks, and other 1058 blocks were read from disk. By every, each one is 54 ms.

Since you read 500 keys, it’s about 2 I/Os per key. Not a crazy number.

If 54ms per I/O is what the latency your storage can provide, then increasing block cache hit rate is the only chance you have. How large is the DB and how large is your DRAM size and your block cache size? Is it possible that you tune Postgres to mostly cache in DRAM but in RocksDB cache you need to read from the disk?

[陈庆国]

Thanks for your help.

How to increase block cache hit rate. Block cache size is default in this test. 

My options is below:

DB* db; Options options; 

options.IncreaseParallelism(); 

options.OptimizeLevelStyleCompaction(); 

options.max_open_file=-1; 

options.write_buffer_size=1024*1024*1024; 

options.create_if_missing=true;

I tried to set block_cache=10gb，but doesn't work.

rocksdb::BlockBasedTableOptions table_options;
table_options.block_cache = rocksdb::NewLRUCache(10 * 1024 * 1024 * 1024LL);

The RAM I use is 128gb，the DB size is 40gb in this test and will increase to 200gb.

Is it possible that you tune Postgres to mostly cache in DRAM but in RocksDB cache you need to read from the disk?

——pg put index in DRAM by default，but how to do in RocksDB？ I definitely want RocksDB to use as much memory as possible.

------------------ 原始邮件 ------------------

发件人: "Siying Dong" <roc...@googlegroups.com>;

发送时间: 2021年1月30日(星期六) 凌晨4:07

收件人: "陈庆国"<10141...@qq.com>;"rocksdb"<roc...@googlegroups.com>;

主题: RE: 回复：RE: performance of MultiGet

You received this message because you are subscribed to a topic in the Google Groups "rocksdb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rocksdb/9WxlUHxiBdc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rocksdb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/BYAPR15MB286962CDB239743AE65C8698F8B99%40BYAPR15MB2869.namprd15.prod.outlook.com.

[Siying Dong]

Do you use direct I/O? If not, I’m surprised that the whole thing isn’t cached in OS page cache. I’m not expert on Postgres but it appears that it always uses OS page cache, so there everything should be cached too.

 

The I/O you hit is clearly the data blocks, not index or something like that. Unless you use BlobDB, RocksDB inlines data with index so there isn’t a clear difference.

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/tencent_67AE09F65A5EC436C5C4F9E2C4F6E82CDB06%40qq.com.

[陈庆国]

Okay, let's focus on rocksDB.

Is there any problem on my options?

------------------ 原始邮件 ------------------

发件人: "Siying Dong" <siyi...@fb.com>;

发送时间: 2021年2月2日(星期二) 凌晨3:46

收件人: "陈庆国"<10141...@qq.com>;"rocksdb"<roc...@googlegroups.com>;

主题: RE: 回复：RE: 回复：RE: performance of MultiGet

[Siying Dong]

I can’t see any problem, unless you didn’t apply table_options correctly to options.