problems writing to disk between steps

[Elliot Chow]

Hi,

Not sure if this is a scoobi problem, but maybe you guys will have some ideas. Within the last week, my jobs, which were running fine a week ago, all started stalling after the first step.  The first map-reduce finishes smoothly, but then it stalls.  I see messages like

[INFO] DFSClient - Could not complete file /user/ellchow/.scoobi-tmp/scoobi-20121022-140002-dec8349f-ad4e-4376-ac0b-6ed7f7e5cd80/bridges/1eb3df6c-6328-4808-8bbf-7555b069a1f3/ch0out0-r-00130 retrying...
[INFO] DFSClient - Could not complete file /user/ellchow/.scoobi-tmp/scoobi-20121022-140002-dec8349f-ad4e-4376-ac0b-6ed7f7e5cd80/bridges/1eb3df6c-6328-4808-8bbf-7555b069a1f3/ch0out0-r-00166 retrying...
[INFO] DFSClient - Could not complete file /user/ellchow/.scoobi-tmp/scoobi-20121022-140002-dec8349f-ad4e-4376-ac0b-6ed7f7e5cd80/bridges/1eb3df6c-6328-4808-8bbf-7555b069a1f3/ch0out0-r-00237 retrying...
[INFO] DFSClient - Could not complete file /user/ellchow/.scoobi-tmp/scoobi-20121022-140002-dec8349f-ad4e-4376-ac0b-6ed7f7e5cd80/bridges/1eb3df6c-6328-4808-8bbf-7555b069a1f3/ch0out0-r-00237 retrying...
[INFO] DFSClient - Could not complete file /user/ellchow/.scoobi-tmp/scoobi-20121022-140002-dec8349f-ad4e-4376-ac0b-6ed7f7e5cd80/bridges/1eb3df6c-6328-4808-8bbf-7555b069a1f3/ch0out0-r-00237 retrying...
[INFO] DFSClient - Could not complete file /user/ellchow/.scoobi-tmp/scoobi-20121022-140002-dec8349f-ad4e-4376-ac0b-6ed7f7e5cd80/bridges/1eb3df6c-6328-4808-8bbf-7555b069a1f3/ch0out0-r-00237 retrying...

and then, after a long time, I start seeing exceptions like

[WARN] DFSClient - NotReplicatedYetException sleeping /user/ellchow/.scoobi-tmp/scoobi-20121019-085306-e2bff437-aa96-47a7-8879-4a3a357460dd/bridges/2a12c7fc-c4c7-48e6-b2a2-345c1088b585/ch0out0-r-00241 retries left 7
[INFO] DFSClient - refreshBlock: block= blk_-2095192023021337706_908323730 possibleNodes= [Lorg.apache.hadoop.hdfs.protocol.DatanodeInfo;@1727d05f
[INFO] DFSClient - refreshBlock: block= blk_3284076222095078238_908324132 possibleNodes= [Lorg.apache.hadoop.hdfs.protocol.DatanodeInfo;@51bbd815
[INFO] DFSClient - org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/user/ellchow/.scoobi-tmp/scoobi-20121019-085306-e2bff437-aa96-47a7-8879-4a3a357460dd/bridges/2a12c7fc-c4c7-48e6-b2a2-345c1088b585/ch0out0-r-00241
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1401)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:596)
    at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:524)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1386)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1382)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1380)

    at org.apache.hadoop.ipc.Client.call(Client.java:1006)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy1.addBlock(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy1.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3180)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3051)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2309)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2515)

Have you guys ever encountered this before?  I may very well be a cluster problem.

[Eric Springer]

Never seen anything of the sort, but it definitely does look like a
cluster problem. You could try an older version of scoobi [0.5.0-cdh*
vs 0.6.0-cdh*-SNAPSHOT] to confirm its nothing like a scoobi
regression

[Elliot Chow]

Yea, I have used both 0.5.0 and 0.6.0 and am getting the same thing.

[Eric Springer]

I wonder if it's its this:

https://issues.apache.org/jira/browse/HDFS-611