Bulk Load Problem (rocksdb_bulk_load not work)

140 views

Skip to first unread message

禾众

unread,

Jan 10, 2019, 2:57:19 AM1/10/19

to MyRocks - RocksDB storage engine for MySQL

I try to use rocksdb engine, migrate from InnoDB.

After I build table, I just insert all the datas into table.
That's huge. So I used variables rocksdb_bulk_load=1.

----- filename : data2rocks.txt (around 10GB)

// SET session rocksdb_bulk_load_allow_unsorted=1;  also tried one time, not work
set session rocksdb_bulk_load=1;
set session sql_log_bin=0;
set session foreign_key_checks=0;
set session unique_checks=0;
set session rocksdb_write_disable_wal=1;
set session rocksdb_commit_in_the_middle=1;
insert ...
insert into ...
.... // something that inserts the data in PK order
insert into...

$ bin/mysql newdatabase <~/data2rocks.txt

But, after all, rocksdb_bulk_load doesn't work for me.

Still write to L0-L5.
How could I totally write all data into bottom level?
For I do have more than 10T data to insert which uses a lot of memory,
without bulk_load it killed by OOM killer.

Later, I do another test.

mysql> SET session sql_log_bin=0;

Query OK, 0 rows affected (0.00 sec)

mysql> SET session rocksdb_bulk_load=1;

Query OK, 0 rows affected (0.00 sec)

mysql> load data infile '/home/chinyajie/data.txt' into table file_meta__tablet_10_2443 fields terminated by ',';

Query OK, 488132 rows affected (20.07 sec)

But It still write to memtable and L0.

How could I make it work？