Reair - master-master replication

[Zheng Shao]

Hi,

Master-master replication allows Hive in 2 data centers to be writable at the same time.

Did anybody think about what needs to be done to allow Reair to replicate bi-directionally to support master-master replication?

--

Zheng

[Zheng Shao]

Any thoughts anybody?  See https://en.wikipedia.org/wiki/Multi-master_replication

--

Zheng

[Paul Yang]

Sorry for the delay! Many of us were out for the holidays and are just catching up on the backlog.

Since conflict resolution is difficult in incremental replication, the only viable approach for "master-master" that I can think of is to segregate the datasets so that a given table is written to from only one cluster. Then, you could setup replication processes using custom replication filters so that the replication processes don't cause conflicts as they run.

Would something like that work for the case you're thinking of?

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.
To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/CAAguJ7oNNhdtW7kUx8pgqCHRHM6%2BG9f%2Bsbz6x-yvw0CuL-O_BA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Zheng Shao]

Our use case looks like this:

1. There are two Hive clusters: A and B

2. Pipeline can run on either A or B cluster depending on which cluster has free resources.

3. The result should be replicated to the other cluster.

A simple conflict resolution like "last write wins" can work well.  The assumption (which is true for most cases) is that writes in Hive is mostly done by ETL pipelines, and there are usually a single writer for a single table any way.

The implementation can be pretty simple:  Add an option to Reair that does not overwrite "newer" table/partitions.  Then we can run 2 Reair to make the master-master replication work.

How does that sound?

--

Zheng

[Paul Yang]

Yeah, I think that will work with the given workload. Were you planning to use batch or incremental? 

For incremental, one approach might be to make changes to the ObjectConflictHandler class and how it's used so that it can skip the copy task under those conditions. I wanted to make that class plug-able as well, but it hasn't been done yet.

[Zheng Shao]

We are planning to start with the batch first.   At some point later this year, we will likely move to incremental to cut down the replication delay.

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/CALm6gcdNp3wDkWbe6AWS4UZyFMYrgB6GyEJxLMXLa41J1gd%3DAw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Zheng

[Zheng Shao]

Hi Paul/Jingwei,

The patch is ready.  Can you help take a look at it?

Zheng

--

Zheng

[Zheng Shao]

Hi Paul,

https://github.com/airbnb/reair/pull/56

I added a test and fixed the bug in the code.  Can you take a look?

Zheng

--

Zheng

[Zheng Shao]

Hi Paul,

I just made the warning messages more clear for debugging.

Can you take a look again?  https://github.com/airbnb/reair/pull/56

Zheng

--

Zheng