metasync vs tsd.core.meta.enable_realtime_ts=true

[Izak Marais]

Hi All, 

We are load testing distributed OpenTSDB after moving from a single node. The tsd.core.meta.enable_realtime_ts=true is a problem, it limits performace to 17k dps on a beefy 4 node cluster. After disabling we handle 3-4 times as much.

This is aggravated by our inability pre-split the tsdb-meta table; the tsdb-meta key-space is an undocumented black box. So the write load is not as evenly distributed as it is for the pre-split tsdb table.

We really want to use the Grafana template variable features, which requires meta data. 

1. Does anyone have experience with this? 
   
2. Suggestions for presplitting tsdb-meta? 
   
3. Is it feasible to run commandline tsdb uid metasync every hour instead of using tsd.core.meta.enable_realtime_ts=true? Other posts seem to indicate that this can also have problems.
   

Thanks for any tips.

Izak

[Jonathan Creasy]

1) Yes, lots of us have experience with this, something is wrong with your node if your write performance is maxed at 17k dps, I have a VM here on my macbook doing nearly twice that for testing.

2) Pre-splitting your table is recommended: http://opentsdb.net/docs/build/html/user_guide/writing.html#pre-split-hbase-regions Some Examples of that: https://github.com/OpenTSDB/opentsdb/issues/538

3) Yes, I would use the metasync, I believe some folks run a separate node for this purpose

[Izak Marais]

Hi Johathan,

Thank you very much for the reply!

1) Is that performance in your test VM with enable_realtime_ts=true set? The problem is setting it true causes a high write load to tsdb-meta. With this table unsplit, the HBase regionserver node hosting it runs out of heap. With enable_realtime_ts=false we got 10x write performance increase yesterday. 

2) Yes, we did use those pre-spit guides and scripts for the tsdb table, but they don't explain how to apply them to the tsdb-meta table. The tsdb-meta key-space is undocumented.

3) OK, thanks, that gives me hope that metasync will solve our problems!

Regards

Izak

[ManOLamancha]

On Friday, October 21, 2016 at 12:19:07 AM UTC-7, Izak Marais wrote:

1) Is that performance in your test VM with enable_realtime_ts=true set? The problem is setting it true causes a high write load to tsdb-meta. With this table unsplit, the HBase regionserver node hosting it runs out of heap. With enable_realtime_ts=false we got 10x write performance increase yesterday. 

Yeah, it's crappy. One thing you can try is to enable "tsd.core.meta.enable_tsuid_tracking" and then disable "tsd.core.meta.enable_tsuid_incrementing"  That only writes puts instead of atomic increments and should help the write rate a bit. But it will require a manual meta-sync periodically to generate the meta data objects. Our HBase engineers came up with a way to improve those atomic increments a ton (Along with appends) and when that makes it open source that will help meta a ton.

2) Yes, we did use those pre-spit guides and scripts for the tsdb table, but they don't explain how to apply them to the tsdb-meta table. The tsdb-meta key-space is undocumented.

I'll try and doc that soon. In the mean time, the meta table row key is just the TSUID. That means it starts with the metric UID followed by the first tag key UID and it's paired tag value UID followed by the other tag pairs. So you can split on the metric values. If you have random metric assignments enabled, split on the full 3 bytes. If you have the default incrementing UIDs enabled then split up to your max UID.

[john....@skyscanner.net]

Are the atomic update / append performance improvements to the asynchbase library or to hbase itself ?

[ManOLamancha]

On Wednesday, December 21, 2016 at 6:17:15 AM UTC-8, john....@skyscanner.net wrote:

Are the atomic update / append performance improvements to the asynchbase library or to hbase itself ?

The improvements are a co-processor our engineers wrote that simply maintains the list of edits to HBase columns and plays them back at query time. During HBase's compactions the edits are squashed into the final result. We tested it just before the holidays and it was really impressive, helping us to store almost 10x the data in block cache compared to straight TSD writes without TSD compaction. And it didn't have the same CPU overhead as normal HBase appends. The trade off is greater storage utilization until HBase compactions have completed.

[john....@skyscanner.net]

This is really encouraging, do you have any idea of a timeline for it being open sourced 

[ManOLamancha]

On Tuesday, January 17, 2017 at 3:25:14 AM UTC-8, john....@skyscanner.net wrote:

This is really encouraging, do you have any idea of a timeline for it being open sourced 

Not quite yet, the engineers who created it are on a huge crunch right now that's due at the end of the quarter so I hope to poke them into OSSing it after that.