Combining Salting, Random UID Metric assignment and Pre-spit HBase regions

[Izak Marais]

Hi all,

I am upgrading from OpenTSDB 2.1 to 2.2 and moving from single node to distributed. I want the best performance by using the new features recommended in docs: random metric assignment,  salting and pre-split Hbase regions. Could you please advise on how to combine these features?

Specifically:

1. Should/can one use both random metric assignment and salting? (Are they orthogonal features?) It seems like they accomplish similar goals: both increase region server write distribution. From the documentation it seems that salting is better at achieving this goal. 
2. We want to pre-split HBase regions, but we don't know the metric names beforehand. Therefore we should use random metric assignment ensure we evenly divide our keyspace. Or does salting sufficiently take care of this as well?
3. Assuming we must combine a 1 byte salt prefix ( tsd.storage.salt.width=1, tsd.storage.salt.buckets=20) with the standard 3 byte metric UID using random assigment, what should the start and end keys values for the region split script be?

I will try to answer #3 above:

• Since we have 20 buckets in the first salt byte, we have possible values \x00 to \x14 for the first byte. If we use random UID assignment for metrics across the entire 3 byte metric space, we have \x00\x00\x00 to \xFF\xFF\xFF for the metrics. Concatenating the salt bytes and metric bytes we have the start key: \x00\x00\x00\x00 and end key: \x14\xFF\xFF\xFF. Does this look correct?

Thanks in advance

Izak

[Izak Marais]

> Since we have 20 buckets in the first salt byte, we have possible values \x00 to \x14 for the first byte

This assumption was incorrect. After starting a fresh tsdb table with salt.width 1 and salt.buckets unset (i.e. default 20), and letting data play in for a short while, I inspected the HBase Master web UI. It displays the request distribution for tsdb table. The final region (starting at \x14\xFF\xFF\xFF and going to the end of the table) had much more requests into it than the other regions. Running a table scan from hbase shell revealed that the values for the first byte of the row key don't stop at \x14 but continue going until \xFF even for salt.buckets=20. 

I'm not sure how the salting algorithm is implemented, confirmation from a developer would be appreciated. Based on the above opservations, I assume that the bucket IDs are not identical between metrics to allow the whole key space for the first byte to be used.

Therefore the start and end keys for region splitting should be \x00\x00\x00\x00 and end key: \xFF\xFF\xFF\xFF.

[Izak Marais]

After testing this, it appears I was mistaken. A clean install on a fresh cluster verified that the original assumption (\x00\x00\x00\x00 and end key: \x14\xFF\xFF\xFF) was correct.

[yaduvendra choubey]

Hi Ijak,

I have a small cluster with 5 region servers. Without salting most of the data is writing to one hbase region server which is causing very bad read performance when issuing multiple queries at the same time. 
Could you please advise if salting will help in this case. If yes then what should be the bucket size as i have only 5 region servers.

Im using multiple tags (5) in the metrics format. 

[ManOLamancha]

On Wednesday, September 28, 2016 at 4:10:57 AM UTC-7, Izak Marais wrote:

After testing this, it appears I was mistaken. A clean install on a fresh cluster verified that the original assumption (\x00\x00\x00\x00 and end key: \x14\xFF\xFF\xFF) was correct.

Yes, you're correct here. Likely a missconfig before :)

 

On Thursday, 18 August 2016 15:35:04 UTC+2, Izak Marais wrote:

Specifically:

1. Should/can one use both random metric assignment and salting? (Are they orthogonal features?) It seems like they accomplish similar goals: both increase region server write distribution. From the documentation it seems that salting is better at achieving this goal. 

Absolutely, using both is a good idea if you expect continual metric assignments over the life of your deployment and on average, more than #salt buckets cardinality per metric. 

1. We want to pre-split HBase regions, but we don't know the metric names beforehand. Therefore we should use random metric assignment ensure we evenly divide our keyspace. Or does salting sufficiently take care of this as well?

You definitely want to split on the salt and metric as well. 

1. Assuming we must combine a 1 byte salt prefix ( tsd.storage.salt.width=1, tsd.storage.salt.buckets=20) with the standard 3 byte metric UID using random assigment, what should the start and end keys values for the region split script be?

I will try to answer #3 above:

• Since we have 20 buckets in the first salt byte, we have possible values \x00 to \x14 for the first byte. If we use random UID assignment for metrics across the entire 3 byte metric space, we have \x00\x00\x00 to \xFF\xFF\xFF for the metrics. Concatenating the salt bytes and metric bytes we have the start key: \x00\x00\x00\x00 and end key: \x14\xFF\xFF\xFF. Does this look correct?

Correct :) We need to add these to the docs as well. 

[ManOLamancha]

On Friday, October 21, 2016 at 8:34:25 PM UTC-7, yaduvendra choubey wrote:

I have a small cluster with 5 region servers. Without salting most of the data is writing to one hbase region server which is causing very bad read performance when issuing multiple queries at the same time. 

Could you please advise if salting will help in this case. If yes then what should be the bucket size as i have only 5 region servers.

Im using multiple tags (5) in the metrics format. 

Salting will help but more that, you'll want to try and split your regions even if you don't switch to salting. Note that using salting requires starting a new data table (but you don't have to change the UID table).