can spark0.7.3 work with hadoop 0.20.2

[lyk...@gmail.com]

Hi,Thanks for reading my post.

Here is my situation :

I just install spark0.7.3 on my cluster ,everything runs fine,but when I tried to reading file in HDFS,it gives error message :

java.io.IOException: Call to /10.103.244.246:6000 failed on local exception: java.io.IOException: Broken pipe

I google it,and found it may be the problem of my hadoop ,hadoop version is 0.20.2.

Because may people are using this clustering ,so I can`t change hadoop .

my question are below:

1 can spark 0.7.3 work with hadoop 0.20.2 ?

2 someone had the same trouble ,and solve by replace the hadoop-core-x.x.x.jar. with hadoop-core-0.20.2.jar in   /spark-0.7.3/lib_managed/jar

   but i want to know the detail ,did they change the  hadoop-core-0.20.2.jar name into  hadoop-core-x.x.x.jar, and copy it to the  /spark-0.7.3/lib_managed/jar,

   or just delete   hadoop-core-x.x.x.jar and copy ?

thanks again !

[lyk...@gmail.com]

And the info is this:

[WARNING]

java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)

        at java.lang.Thread.run(Thread.java:619)

Caused by: java.io.IOException: Call to /10.103.244.246:6000 failed on local exception: java.io.IOException: Broken pipe

        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)

        at org.apache.hadoop.ipc.Client.call(Client.java:1071)

        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)

        at $Proxy18.getProtocolVersion(Unknown Source)

        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)

        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)

        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)

        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)

        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)

        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)

        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)

        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)

        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)

        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)

        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)

        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)

        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)

        at spark.rdd.HadoopRDD.<init>(HadoopRDD.scala:57)

        at spark.SparkContext.hadoopFile(SparkContext.scala:238)

        at spark.SparkContext.textFile(SparkContext.scala:207)

        at spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:108)

        at SparkTest.NodeRead.main(NodeRead.java:60)

        ... 6 more

Caused by: java.io.IOException: Broken pipe

        at sun.nio.ch.FileDispatcher.write0(Native Method)

        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)

        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)

        at sun.nio.ch.IOUtil.write(IOUtil.java:75)

        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)

        at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)

        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)

        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)

        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)

        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

        at java.io.DataOutputStream.flush(DataOutputStream.java:106)

        at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:779)

        at org.apache.hadoop.ipc.Client.call(Client.java:1047)

        ... 26 more

[张包峰]

Hi, I have the same problem. I did two things and currently fixed it.

1. replace the hadoop-core-1.0.4.jar with your hadoop-client jar, mine is hadoop-2-core.jar (Although my spark is hadoop-1-prebuilt and my hadoop-client is hadoop2)

2. put hadoop-default.xml and hadoop-site.xml under SPARK_HOME/conf. Else you cant make it.

Hope this will help you!

在 2013年8月7日星期三UTC+8下午9时34分23秒，Liu Yang写道：

[Liu Yang]

Really thank you!!! It works!