Bulk replication in HDFS with HA

[Don Krapohl]

We are implementing bulk replication between two HDFS cluster with high availability enabled.  For some specific reasons we don't want to make the source cluster aware of the nameservice on the destination cluster so we're going to submit our ReAir workloads along with a custom client hdfs-site.xml on the destination cluster.  We're working it out but if you already have this addressed it would be great to see how you did it.  We're following this approach:  http://henning.kropponline.de/2015/03/15/distcp-two-ha-cluster/

Don

[Zheng Shao]

I haven't tried it out yet, but I have the same plan of following the idea from that blog post.

On Thu, Nov 3, 2016 at 8:49 AM, Don Krapohl <dkrapo...@gmail.com> wrote:

We are implementing bulk replication between two HDFS cluster with high availability enabled.  For some specific reasons we don't want to make the source cluster aware of the nameservice on the destination cluster so we're going to submit our ReAir workloads along with a custom client hdfs-site.xml on the destination cluster.  We're working it out but if you already have this addressed it would be great to see how you did it.  We're following this approach:  http://henning.kropponline.de/2015/03/15/distcp-two-ha-cluster/

Don

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.
To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/cdaab65e-49e2-4ee9-ac50-144fda944dd0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Zheng

[Paul Yang]

We currently replicate between two HDFS clusters with HA enabled on both. Since the jobs run in the destination cluster, we added the source HDFS nameservice to the hdfs-site.xml for the destination cluster and that works without issues (similar to the blog post). We've deployed the modified hdfs-site.xml to all nodes on the destination cluster to ensure that the map-reduce tasks have access to the right configuration.

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/CAAguJ7rT%3DTZL4WDj%3DBDjgxGiRctqTfXgXvg0Y7MhLffYj7qZvw%40mail.gmail.com.

[Don Krapohl]

This will probably be a distro-specific question (we're on Cloudera) but are you talking about the HDFS Service-level hdfs-site.xml or the HDFS Client-level hdfs-site.xml (Cloudera allows them to be different)?  These would be the respective safety valve sections in Cloudera Manager for us.  If you're not using that distro, no problem.

[Paul Yang]

Yeah, we're using Cloudera as well. We've put the changes in the client-level hdfs-site.xml (Input box labeled "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml").

On Thu, Nov 3, 2016 at 11:54 AM, Don Krapohl <dkrapo...@gmail.com> wrote:

This will probably be a distro-specific question (we're on Cloudera) but are you talking about the HDFS Service-level hdfs-site.xml or the HDFS Client-level hdfs-site.xml (Cloudera allows them to be different)?  These would be the respective safety valve sections in Cloudera Manager for us.  If you're not using that distro, no problem.

--

You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.
To post to this group, send email to airbnb...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/0c81200b-243c-4a9d-bce7-d6db0c80ad41%40googlegroups.com.

[Don Krapohl]

Beautiful.  We got it working about the time you  posted.  So others can shortcut it, to get replication working between two HA hdfs clusters, on the DESTINATION cluster go to Cloudera Manager, hdfs service, configuration, and filter on the HDFS client safety valve.  Change it to show in xml with the link next to the boxes.  Paste this there changing the names of the nameservices and source namenodes in the xml below with your own.  No changes need to be made on the source cluster.  NamespaceOfSource is your source cluster nameservice, NamespaceOfDestination is the destination cluster nameservice, and the two namenodes in the xml are the namenodes in your source cluster. Safety valve content:

<property>

<name>dfs.nameservices</name>

<value>NamespaceOfDestination,NamespaceOfSource</value>

</property>

<property>

<name>dfs.internal.nameservices</name>

<value>NamespaceOfDestination</value>

<description>

      IMPORTANT. This keeps the destination cluster datanodes

 from joining and becoming members of the source cluster.

</description>

</property>

<property>

<name>dfs.ha.namenodes.NamespaceOfSource</name>

<value>namenode77,namenode1</value>

</property>

<property>

<name>dfs.namenode.rpc-address.NamespaceOfSource.namenode77</name>

<value>OneSourceNamenode:8020</value>

</property>

<property>

<name>dfs.namenode.servicerpc-address.NamespaceOfSource.namenode77</name>

<value>OneSourceNamenode:8022</value>

</property>

<property>

<name>dfs.namenode.http-address.NamespaceOfSource.namenode77</name>

<value>OneSourceNamenode:50070</value>

</property>

<property>

<name>dfs.namenode.https-address.NamespaceOfSource.namenode77</name>

<value>OneSourceNamenode:50470</value>

</property>

<property>

<name>dfs.namenode.rpc-address.NamespaceOfSource.namenode1</name>

<value>AnotherSourceNamenode:8020</value>

</property>

<property>

<name>dfs.namenode.servicerpc-address.NamespaceOfSource.namenode1</name>

<value>AnotherSourceNamenode:8022</value>

</property>

<property>

<name>dfs.namenode.http-address.NamespaceOfSource.namenode1</name>

<value>AnotherSourceNamenode:50070</value>

</property>

<property>

<name>dfs.namenode.https-address.NamespaceOfSource.namenode1</name>

<value>AnotherSourceNamenode:50470</value>

</property>

<property>

<name>dfs.client.failover.proxy.provider.NamespaceOfSource</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

<name>dfs.ha.automatic-failover.enabled.NamespaceOfSource</name>

<value>true</value>

</property>