Query performance vs DB size?

[Izak Marais]

Hi all, 

We recently did an internal demo of OpenTSDB. One of the questions I could not definitively answer was whether the total size of the db will affect query performance.

To clarify the question: given an example query (a specific day's worth of data for a single metric, returning e.g. 100k data points), will the performance of that same query be slower a year later after the db has grown by many GB's of data? Assume the data added during the year does not map into the row keys for the query.

After reading and watching a presentation about the back-end hbase  schema design, as well as reading a bit about HBase's architecture, I think the answer is: the performance should not be affected at all. OpenTSDB's schema design ensures locality of data in the HBase table. I am less certain about the following: HBase's sharding should split the tsdb table across different region servers while keeping adjacent rows together, therefore the total size of the table will not affect performance?

I would really appreciate an answer form someone more knowledgeable than me. 

Thanks

Izak

[Siddartha Guthikonda]

It mainly depends on how you split your regions and their size. If regions keys have the time then newer queries will only go to a few regions instead of all the regions that has that metric. 

[John A. Tamplin]

Having the time part of the key isn't the problem since it comes after the metric uid.  Allowing metric uids sequentially is a problem, and we fixed that ourselves by reversing the bytes when the uid is allocated (which means starting from empty or rewriting the existing uids).  Then you can presplit your regions evenly and traffic will be balanced between them.  Couple that with seeing TTL on rows and it stays balanced as time goes on.

John A. Tamplin (phone)

[Izak Marais]

Thank you for for the replies. 

> Having the time part of the key isn't the problem since it comes after the metric uid.

This makes sense; it is how OpenTSDB ensures locality of data related to the same query. 

>  Allowing metric uids sequentially is a problem

This sounds interesting, could you please elaborate on what the problem is? How does modifying the UID assigment scheme facilitate evenly split HBase regions?

> Couple that with seeing TTL on rows

I'm afraid I don't know what "TTL on rows" is referring to. Do you mind elaborating?

Thanks in advance!

[Izak Marais]

> > Couple that with seeing TTL on rows

> I'm afraid I don't know what "TTL on rows" is referring to. Do you mind elaborating?

OK, I did some reading on this. It sounds excellent! I was looking for this sort of functionality form OpenTSDB itself, but I assume you do custom Hbase configuration to use this feature?

[John A. Tamplin]

On Wed, Apr 22, 2015 at 2:51 AM, 'Izak Marais' via OpenTSDB <open...@googlegroups.com> wrote:

>  Allowing metric uids sequentially is a problem

This sounds interesting, could you please elaborate on what the problem is? How does modifying the UID assigment scheme facilitate evenly split HBase regions?

So you want your keys balanced across region servers, and the best way to do that is presplit your keys so you don't have to deal with region splits on a live cluster.  Out of the box, UIDs are allocated sequentially, so \x00\x00\x00 then \x00\x00\x01 etc.  Let's say you start out with 64k metrics, so your last key is \x00\xFF\xFF.  If you presplit on that key range, you might wind up with \x00\xF0 and up for the key prefix of the last of 16 region servers.  As you add new metrics, they will be allocated \x01XXXX and be assigned to the last region server.  Metrics which are no longer used mean that earlier region servers are underutilized as well.  To fix this, you will have to split the last region (which has severe performance implications while the split is going on) or add more region servers.

If instead, you reverse the bytes of metric UIDs as they are allocated, then you get \x00\x00\x00, \x01\x00,\x00, \x02\x00\x00 etc.  Then you split your key space evenly, so if you have 16 region servers they would be \x00 - \x0F, \x10 - \x1F, etc.  The entire keyspace is covered, and traffic is equal between them (well, as equal as your metrics are, but blocks of consecutively allocated metrics will be spread out so it tends to even out).  In our experience the load difference between different region servers is at most 2x, where before we did this it was 20x.  Basically, you can decide how many region servers you want to have, presplit the key space, set the TTL to match your resources, and never have to worry about splitting or rebalancing again (though if you grow beyond expectations you will either have to reduce TTLs or add more region servers, with a headache for rebalancing).

I had mentioned it here a while back, but it seemed like the interest was in assigning random UIDs instead to solve the same problem.  If there is interest, I could package up a patch - since it only happens at UID allocation time, it is a pretty small patch.  The only real downside is that you can't have any existing data when you switch to this, or you will have to do something to rewrite all the UIDs (both in the table, and in all the rows).

--

John A. Tamplin

[John A. Tamplin]

On Wed, Apr 22, 2015 at 3:06 AM, Izak Marais <izakm...@gmail.com> wrote:

> > Couple that with seeing TTL on rows

> I'm afraid I don't know what "TTL on rows" is referring to. Do you mind elaborating?

OK, I did some reading on this. It sounds excellent! I was looking for this sort of functionality form OpenTSDB itself, but I assume you do custom Hbase configuration to use this feature?

You set it when you create the table, or you can add it later with alter.

--

John A. Tamplin

[Izak Marais]

Thank you very much for the clear and detailed description. I will try to bring this discussion (and patch possibility) under the developer's attention.

[ManOLamancha]

On Wednesday, April 22, 2015 at 11:02:05 AM UTC-7, John A. Tamplin wrote:

I had mentioned it here a while back, but it seemed like the interest was in assigning random UIDs instead to solve the same problem.  If there is interest, I could package up a patch - since it only happens at UID allocation time, it is a pretty small patch.  The only real downside is that you can't have any existing data when you switch to this, or you will have to do something to rewrite all the UIDs (both in the table, and in all the rows).

We finally upstreamed the random UID assignment for metrics along with salting support for extra distribution. It's all in 2.2/"put". 

[Izak Marais]

Good to hear, thanks for the feedback.