Hadoop version mismatch: Server IPC version 7 cannot communicate with client version 4

[Patrick Salami]

Hi,

I am running the latest HEAD of Spark. I have Hadoop version 2.0.0-mr1-cdh4.1.2 installed on my cluster. The following happens when I try to read a file from HDFS:

scala> val textFile = sc.textFile("hdfs://hadoop-testcluster-master:8020/user/psalami/testfile")

[...]

scala> textFile.count()

13/03/19 18:59:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

13/03/19 18:59:12 WARN snappy.LoadSnappy: Snappy native library not loaded

org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4

        at org.apache.hadoop.ipc.Client.call(Client.java:1070)

        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)

        at sun.proxy.$Proxy3.getProtocolVersion(Unknown Source)

        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)

        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)

        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)

        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)

        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)

        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)

        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)

        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)

        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)

        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)

        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)

        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)

        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)

        at spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:53)

        at spark.RDD.partitions(RDD.scala:168)

        at spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:9)

        at spark.RDD.partitions(RDD.scala:168)

        at spark.SparkContext.runJob(SparkContext.scala:627)

        at spark.RDD.count(RDD.scala:506)

[...]

How can I compile Spark with support for MRv2 / CDH 4.1.2? I see in your pom.xml that you have 2 profiles, (hadoop1 and hadoop2), hadoop2 using CDH 4.1.2. How can I switch to hadoop2 when building with sbt?

Thanks!

Patrick

[Patrick Wendell]

For SBT, if you look in:

project/SparkBuild.scala

it explains how to setup the build for Hadoop 2. You want to uncomment
the following lines (and comment the lines above them which are for
Hadoop 1):

// For Hadoop 2 versions such as "2.0.0-mr1-cdh4.1.1", set the
HADOOP_MAJOR_VERSION to "2"
//val HADOOP_VERSION = "2.0.0-mr1-cdh4.1.1"
//val HADOOP_MAJOR_VERSION = "2"

Also, you'll want to use 4.1.2 instead of 4.1.1 which is listed here.

> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Patrick Salami]

great, that worked! 

Thanks!