OpenTSDB Fetch Performance

[Ravindra S]

I have a process which does the following :

1. Get 'n' datapoints from OpenTSDB using Java API

2. Perform some analysis on them which generates roughly '2n' datapoints

3. Persist the output to OpenTSDB using Java API

(Initially there are 5 million datapoints. One fetch gets around 50k datapoints)

Lets call the three steps as a unit of work.

There are 100 such units of work to be done.

When I run them sequentially, fetching from OpenTSDB takes somewhere around 100-200ms for each of them.

But when I run multiple threads, say three threads, the fetch performance degrades.

Some of the fetch take 100-200ms, some take 1-5secs, some 10-30secs and some even more than a minute.

What could be the reason for this performance degradation?

Does the Java API handle concurrent requests poorly?

Later I used HTTP API instead of Java API to fetch datapoints.

(I had started two TSDs and accessed them through Varnish)

Then even multiple threads were performing decently (100-300ms)

So I am inclined towards using HTTP API to access datapoints.

I would like to know which one is better at handling concurrent requests : HTTP API or Java API.

[ManOLamancha]

They should both exhibit similar performance though the trick is that if you're using the Java API, you have to fire the queries asynchronously using the Deferred.

Also, a drawback is that if you're using a single HBase server (or a small number of them) the AsyncHBase client becomes single threaded in responding to queries. We're working on fixing that up for 3.0.