Using loader to restore a MyDumper backup is slow.

[Wilfred Chau]

Is there a faster way to restore million to billion rows of data to TiDB?

I cut a backup using MyDumper, then use the loader utility to load the data. 

The transaction execution costs reported from loader is around 4-15 seconds.

version : 5.7.10-TiDB-v2.0.0-rc.4-31-gbfb6ec2

[chen...@pingcap.com]

Yes, We do have a  faster tool which is under heavy development and not release yet. In one of our tests, the tool can restore 388 GB data in 2.5 h.

在 2018年4月11日星期三 UTC+8上午9:59:19，Wilfred Chau写道：

[Wilfred Chau]

Thanks.  When do you expect the new import tool be available?

Fyi.  We loaded 179GB (mydump uncompressed sql) in 4 days.

Can I suggest a few things to add to the new tool:
1) allow bulk upload to an existing table, currently, the loader tool requires to run the schema.sql from mydumper.  I have to uncomment the schema DDL to get around it
2) allow bulk upload to use compressed mydumper SQL file
3) allow bulk upload to use compressed and versioned mydumper SQL files.  (I used 4 threads when executing mydumper, hence it produces SQL file like <table>.00000.sql.gz, <table>.00001.sql.gz etc)