best way to sync datapoints between 2 opentsdb

[Huafeng Lyu]

Hi guys,

I want to set up 'high availability' for my data in opentsdb, so I want to set up 2 opentsdb clusters, one as master, another as backup. The datapoints will be written to the master, and I want to sync the data automatically from master to backup.

So what is the best approach to do this?

(An alternative is to write data both to the master and to the backup... but I guess it's not a good idea.)

Any ideas? Thanks.

--huafeng 

[IDerr]

Hi, you can use replication from hbase.

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_bdr_hbase_replication.html
https://www.ibm.com/support/knowledgecenter/en/SSPT3X_3.0.0/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_enable_disasterrecovery.html

[Huafeng Lyu]

Hi IDerr,

This is really helpful! Thanks a lot! I'll investigate. But before I dig into details, do you know 

(1) can I specify which table(s) to replicate across hbase clusters?

(2) I only want to replicate data later than a given datetime. is it possible?

(3) is it easy to switch the direction of the replication? For example, my app writes data to cluster A, and then replicated from A to B. If cluster A is down, then my app starts writing data to B, and data should be replicated in the opposite direction - from B to A. Is it easy to configure this?

Thanks!

--huafeng

[IDerr]

Hi again,
it is really easy :

1) Only tables with a replication scope set to 1 will be replicated (http://blog.cloudera.com/blog/2012/08/hbase-replication-operational-overview/)
2) The replication works only for data inserted after the replication was enabled. So you can disable the replication, bulkload your data, reactivate the replication and then insert the data you want to be replicated.
3) You can add a peer in your host A to replicate to your host B to replicate data from A to B
   and on your host B, you add the peer A to replicate the data from B to A

IDerr

Le mardi 11 juillet 2017 05:32:35 UTC+2, Huafeng Lyu a écrit :

[Huafeng Lyu]

Hi IDerr,

One more question: what's the delay/lag of the replication? It seems the replication is not in real-time so there will be a delay of data in the passive cluster. Usually how much is the delay? Seconds, minutes or even longer? I searched but didn't find authorative doc on this topic. Will you please give some hints? Thanks.

[Santosh Pingale]

You could use tsdb relay for this. You dont want to mess with tuid's if in case any random write happens to backup instance.

[Huafeng Lyu]

Do you mean bosun's tsdbrelay https://godoc.org/bosun.org/cmd/tsdbrelay?

With hbaser replication, we'll need to copy the uid table to the backup hbase cluster before we enable replication.

What kind of "random write to backup instance" can mess with tuid's? Will you please elaborate?

Thanks.