Incremental sync with specific set of tables

43 views
Skip to first unread message

VG

unread,
Oct 24, 2018, 3:30:36 PM10/24/18
to reair
Hi,

First of all want to say thanks for sharing this very useful project. I could able to setup batch replication in an HA environment, validated and all looks good. 

The question i have is on the incremental replication. Like batch can we specify certain set of DBs and tables for incremental sync? can someone share the incremental steps?

Thanks
Venu




Paul Yang

unread,
Oct 24, 2018, 4:30:52 PM10/24/18
to venu...@gmail.com, airbnb...@googlegroups.com
Yeah, there's a regex that can be specified as a filter for the DBs / tables / partitions to replicate. Please check the configuration file / options.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.
To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

VG

unread,
Oct 24, 2018, 5:09:21 PM10/24/18
to reair
Let me cross-check then, Thanks


On Wednesday, October 24, 2018 at 1:30:52 PM UTC-7, Paul Yang wrote:
Yeah, there's a regex that can be specified as a filter for the DBs / tables / partitions to replicate. Please check the configuration file / options.

On Wed, Oct 24, 2018 at 12:30 PM VG <venu...@gmail.com> wrote:
Hi,

First of all want to say thanks for sharing this very useful project. I could able to setup batch replication in an HA environment, validated and all looks good. 

The question i have is on the incremental replication. Like batch can we specify certain set of DBs and tables for incremental sync? can someone share the incremental steps?

Thanks
Venu




--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

VG

unread,
Oct 24, 2018, 5:45:59 PM10/24/18
to reair

Incremental setup done, but getting following error when the actual copy starts, 

2018-10-24 14:36:07 [TaskWorker-0] ERROR com.airbnb.reair.incremental.deploy.ReplicationLauncher - Exiting due to uncaught exception from thread Thread[TaskWorker-0,5,main]!
java.lang.IllegalAccessError: tried to access method org.apache.hadoop.tools.DistCp.<init>()V from class com.airbnb.reair.common.DistCpWrapper
        at com.airbnb.reair.common.DistCpWrapper.copy(DistCpWrapper.java:171)
        at com.airbnb.reair.incremental.DirectoryCopier.copy(DirectoryCopier.java:120)
        at com.airbnb.reair.incremental.primitives.CopyUnpartitionedTableTask.runTask(CopyUnpartitionedTableTask.java:159)
        at com.airbnb.reair.incremental.ReplicationJob.run(ReplicationJob.java:66)
        at com.airbnb.reair.multiprocessing.Worker.run(Worker.java:67)

appreciate if i get some clues, in the mean time investigating

Thanks

Paul Yang

unread,
Oct 24, 2018, 6:06:31 PM10/24/18
to venugopal gannavarapu, airbnb...@googlegroups.com
Looks like the version of Hadoop it was built with is different from the version used to run it.

On Wed, Oct 24, 2018 at 2:46 PM VG <venu...@gmail.com> wrote:

Incremental setup done, but getting following error when the actual copy starts, 

2018-10-24 14:36:07 [TaskWorker-0] ERROR com.airbnb.reair.incremental.deploy.ReplicationLauncher - Exiting due to uncaught exception from thread Thread[TaskWorker-0,5,main]!
java.lang.IllegalAccessError: tried to access method org.apache.hadoop.tools.DistCp.<init>()V from class com.airbnb.reair.common.DistCpWrapper
        at com.airbnb.reair.common.DistCpWrapper.copy(DistCpWrapper.java:171)
        at com.airbnb.reair.incremental.DirectoryCopier.copy(DirectoryCopier.java:120)
        at com.airbnb.reair.incremental.primitives.CopyUnpartitionedTableTask.runTask(CopyUnpartitionedTableTask.java:159)
        at com.airbnb.reair.incremental.ReplicationJob.run(ReplicationJob.java:66)
        at com.airbnb.reair.multiprocessing.Worker.run(Worker.java:67)

appreciate if i get some clues, in the mean time investigating

Thanks

On Wednesday, October 24, 2018 at 2:09:21 PM UTC-7, VG wrote:
Let me cross-check then, Thanks

On Wednesday, October 24, 2018 at 1:30:52 PM UTC-7, Paul Yang wrote:
Yeah, there's a regex that can be specified as a filter for the DBs / tables / partitions to replicate. Please check the configuration file / options.

On Wed, Oct 24, 2018 at 12:30 PM VG <venu...@gmail.com> wrote:
Hi,

First of all want to say thanks for sharing this very useful project. I could able to setup batch replication in an HA environment, validated and all looks good. 

The question i have is on the incremental replication. Like batch can we specify certain set of DBs and tables for incremental sync? can someone share the incremental steps?

Thanks
Venu




--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.

VG

unread,
Oct 24, 2018, 6:19:35 PM10/24/18
to reair
Both clusters are running same HDP version, batch worked earlier
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

Paul Yang

unread,
Oct 24, 2018, 6:21:52 PM10/24/18
to venugopal gannavarapu, airbnb...@googlegroups.com
What version of Hadoop is running on the cluster, and what version of Hadoop was used to build ReAir?

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/8e9a264c-14ef-4703-8280-74f8fc5715a4%40googlegroups.com.

VG

unread,
Oct 24, 2018, 6:30:10 PM10/24/18
to reair
Both clusters are running HDP 2.6.5, HDFS 2.7.3, i used the default reair build dint change anything when building, if i need to please let me know how
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

Paul Yang

unread,
Oct 24, 2018, 6:47:41 PM10/24/18
to venugopal gannavarapu, airbnb...@googlegroups.com
i used the default reair build dint change anything when building

Recommend taking a look at how the build works and what version is used there.

To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/8e9a264c-14ef-4703-8280-74f8fc5715a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair...@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/4c57b477-e852-4a7c-802b-4796f5f1779e%40googlegroups.com.

VG

unread,
Oct 24, 2018, 6:51:32 PM10/24/18
to reair
Will check, thanks
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/b14a3cc6-667d-4e37-b580-90aac5af1ef5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/2377a202-73f9-4db2-8cf3-373dd3edada7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.

To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/8e9a264c-14ef-4703-8280-74f8fc5715a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages