How to import faster?

[Kramer Li]

Hi All

I`m trying to import a large amount of data into openTSDB. 

Let`s say we have 288 files one day. each file contains 5 million rows of data. I`m trying to import all these 288 files. 

But the tsdb import command takes a lot time to import only one file.

Is there any way to improve the performance?

Regards

Mingwe

[Jim Scott]

Depending on how your data is organized, which metrics are in those files... the BatchedDataPoints is the fastest way to get historical data loaded into OpenTSDB

https://github.com/OpenTSDB/opentsdb/blob/d430ab057ae8163e21c12e56c422b78223bd9178/src/core/BatchedDataPoints.java 

--

Jim Scott

@kingmesal

[Kramer Li]

Thanks Scott

I did some pre-operation before the import. Like assign uid to possible values. Pre-split hbase region. 

The spead is acceptable now. Almost 10,000 per second.

Thanks very much

在 2015年9月1日星期二 UTC+8上午12:24:19，Jim Scott写道：

[Izak Marais]

Hi Mingwe

Could you explain (or post some links to references explaining) the pre-operations you did? How do you assign UID to possible values. Did you use existing scripts to pre-split the hbase regions?

Thanks

Izak