Incremental sync with specific set of tables

[VG]

Hi,

First of all want to say thanks for sharing this very useful project. I could able to setup batch replication in an HA environment, validated and all looks good. 

The question i have is on the incremental replication. Like batch can we specify certain set of DBs and tables for incremental sync? can someone share the incremental steps?

Thanks

Venu

[Paul Yang]

Yeah, there's a regex that can be specified as a filter for the DBs / tables / partitions to replicate. Please check the configuration file / options.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.
To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[VG]

Let me cross-check then, Thanks

On Wednesday, October 24, 2018 at 1:30:52 PM UTC-7, Paul Yang wrote:

Yeah, there's a regex that can be specified as a filter for the DBs / tables / partitions to replicate. Please check the configuration file / options.

On Wed, Oct 24, 2018 at 12:30 PM VG <venu...@gmail.com> wrote:

Hi,

First of all want to say thanks for sharing this very useful project. I could able to setup batch replication in an HA environment, validated and all looks good. 

The question i have is on the incremental replication. Like batch can we specify certain set of DBs and tables for incremental sync? can someone share the incremental steps?

Thanks

Venu

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

[VG]

Incremental setup done, but getting following error when the actual copy starts, 

2018-10-24 14:36:07 [TaskWorker-0] ERROR com.airbnb.reair.incremental.deploy.ReplicationLauncher - Exiting due to uncaught exception from thread Thread[TaskWorker-0,5,main]!

java.lang.IllegalAccessError: tried to access method org.apache.hadoop.tools.DistCp.<init>()V from class com.airbnb.reair.common.DistCpWrapper

        at com.airbnb.reair.common.DistCpWrapper.copy(DistCpWrapper.java:171)

        at com.airbnb.reair.incremental.DirectoryCopier.copy(DirectoryCopier.java:120)

        at com.airbnb.reair.incremental.primitives.CopyUnpartitionedTableTask.runTask(CopyUnpartitionedTableTask.java:159)

        at com.airbnb.reair.incremental.ReplicationJob.run(ReplicationJob.java:66)

        at com.airbnb.reair.multiprocessing.Worker.run(Worker.java:67)

appreciate if i get some clues, in the mean time investigating

Thanks

[Paul Yang]

Looks like the version of Hadoop it was built with is different from the version used to run it.

On Wed, Oct 24, 2018 at 2:46 PM VG <venu...@gmail.com> wrote:

Incremental setup done, but getting following error when the actual copy starts, 

2018-10-24 14:36:07 [TaskWorker-0] ERROR com.airbnb.reair.incremental.deploy.ReplicationLauncher - Exiting due to uncaught exception from thread Thread[TaskWorker-0,5,main]!

java.lang.IllegalAccessError: tried to access method org.apache.hadoop.tools.DistCp.<init>()V from class com.airbnb.reair.common.DistCpWrapper

        at com.airbnb.reair.common.DistCpWrapper.copy(DistCpWrapper.java:171)

        at com.airbnb.reair.incremental.DirectoryCopier.copy(DirectoryCopier.java:120)

        at com.airbnb.reair.incremental.primitives.CopyUnpartitionedTableTask.runTask(CopyUnpartitionedTableTask.java:159)

        at com.airbnb.reair.incremental.ReplicationJob.run(ReplicationJob.java:66)

        at com.airbnb.reair.multiprocessing.Worker.run(Worker.java:67)

appreciate if i get some clues, in the mean time investigating

Thanks

On Wednesday, October 24, 2018 at 2:09:21 PM UTC-7, VG wrote:

Let me cross-check then, Thanks

On Wednesday, October 24, 2018 at 1:30:52 PM UTC-7, Paul Yang wrote:

Yeah, there's a regex that can be specified as a filter for the DBs / tables / partitions to replicate. Please check the configuration file / options.

On Wed, Oct 24, 2018 at 12:30 PM VG <venu...@gmail.com> wrote:

Hi,

First of all want to say thanks for sharing this very useful project. I could able to setup batch replication in an HA environment, validated and all looks good. 

The question i have is on the incremental replication. Like batch can we specify certain set of DBs and tables for incremental sync? can someone share the incremental steps?

Thanks

Venu

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.

[VG]

Both clusters are running same HDP version, batch worked earlier

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

[Paul Yang]

What version of Hadoop is running on the cluster, and what version of Hadoop was used to build ReAir?

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/8e9a264c-14ef-4703-8280-74f8fc5715a4%40googlegroups.com.

[VG]

Both clusters are running HDP 2.6.5, HDFS 2.7.3, i used the default reair build dint change anything when building, if i need to please let me know how

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

[Paul Yang]

i used the default reair build dint change anything when building

Recommend taking a look at how the build works and what version is used there.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/8e9a264c-14ef-4703-8280-74f8fc5715a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/4c57b477-e852-4a7c-802b-4796f5f1779e%40googlegroups.com.

[VG]

Will check, thanks

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/8e9a264c-14ef-4703-8280-74f8fc5715a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.