OpenTSDB dimensioning

[Mahesh Govind]

Hi,

Could you please help me  with some pointers about OpenTSDB dimensioning . 

I have a scenario where I need to write millions of data points from a plant controller to OpenTSDB  .

I might have to call 15 K  /api/put calls per second based on estimation of sampling rate of various sensors.

One  point I found was about the configuration of threads per CPU  to handle http requests .Also batching of metrics in JSON body .

But still i am not getting a clear Idea  about OpenTSDB  to scale  up/out for a large number of HTTP put requests .

Could you please provide some guidance for the same .

Thanks in advance .....

regards

Mahesh

[Thibault Godouet]

Hi Marsha,

In my experience, the settings about what metadata is recorded has a huge importance on the write perf you'll get.  You'll also want to pre-split your tables evenly, which is much easier with random uids.
Also remember that it gets busier during compaction (but for me still better than enabling appends).

As a rough guide, FYI with enable_tsuid_tracking, I can write 130K datapoints per second on 3 datanodes (with a region server and a single opentsdb per node).

Thibault.

[Izak Marais]

Also check out this blog post from Kyle Brandt at Stack-Exchange on their opentsdb architecture : http://kbrandt.com/post/bosun_arch/#reference-configs:7bc97b814cc5ce0a4508e0dd2bd2e58d