Does OpenTSDB slow down as the database gets larger?

[John Humphreys]

There is another thread with this basic question; but I'm hoping for a more on track/succinct answer.

Assuming you properly lay out your metrics / build your row-key as you should, is OpenTSDB expected to slow down over time as the database fills up?  I.e. does recording 1000 days of the same metric make the metric slower to retrieve (assume the hbase cluster isn't overloaded)?

We do about 18-billion points a day.  After the first day, query times were ~ 30ms (great).  After the 2nd day, they were ~45ms.  Now, after 5 days or so, they're up to ~160ms (okay but starting to scare me).

Assuming all of my hbase regions are limited to 4GB, should I expect the slow-down to continue, or am I probably just seeing some initial growing pains?

[ManOLamancha]

Two parts to that:

If your query is growing as well, e.g. day 1 fetch 1 day of data, day 2, fetch 2 days, etc, they'll certainly take longer since you're fetching more and more data. 

But I'll assume you're performing a sliding window kind of query, e.g. querying for maybe now to 1 day-ago. In that case, assuming you've reached a steady state with Hbase (regarding compaction runs, memstore flushes, etc), then the same query should also reach a consistent time to load.

Check the cache hits/misses in HBase. Data is written to the memstore, then flushed to cache (if cache on write is enabled) and disk. When you first fire up HBase, queries usually look great since all of the data is coming out of memory. But when it's flushed to disk, any queries that encompass data for that range will take longer, though when the flush cycle is consistent, queries should be consistent too.

Also look at the region server's compaction queue and compaction times to see if they're caught up and consistent. Also check the GCs on the RS and the TSD.