Yet another hadoop streaming failed with error code 1 problem

[Liang Zhou]

Hi, I am running the sample code 

a <- to.dfs(seq(from=1, to=500, by=3), output="/user/cloudera/numbers")

b <- mapreduce(input=a, map=function(k,v){keyval(v,v*v)})

I got the following error, 
> b <- mapreduce(input=a, map=function(k,v){keyval(v,v*v)})
packageJobJar: [/tmp/Rtmp82ZTy0/rmr-local-env5c1b21e77298, /tmp/Rtmp82ZTy0/rmr-global-env5c1b6c395e3a, /tmp/Rtmp82ZTy0/rmr-streaming-map5c1b63407493, /tmp/hadoop-hdfs/hadoop-unjar7710151861161046761/] [] /tmp/streamjob4134270826733773111.jar tmpDir=null
14/03/26 16:33:44 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/03/26 16:33:45 INFO mapred.FileInputFormat: Total input paths to process : 1
14/03/26 16:33:46 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hdfs/mapred/local]
14/03/26 16:33:46 INFO streaming.StreamJob: Running job: job_201403261402_0011
14/03/26 16:33:46 INFO streaming.StreamJob: To kill this job, run:
14/03/26 16:33:46 INFO streaming.StreamJob: UNDEF/bin/hadoop job  -Dmapred.job.tracker=h1-3005.local:8021 -kill job_201403261402_0011
14/03/26 16:33:46 INFO streaming.StreamJob: Tracking URL: http://h1-3005.local:50030/jobdetails.jsp?jobid=job_201403261402_0011
14/03/26 16:33:47 INFO streaming.StreamJob:  map 0%  reduce 0%
14/03/26 16:33:57 INFO streaming.StreamJob:  map 50%  reduce 0%
14/03/26 16:34:28 INFO streaming.StreamJob:  map 100%  reduce 100%
14/03/26 16:34:28 INFO streaming.StreamJob: To kill this job, run:
14/03/26 16:34:28 INFO streaming.StreamJob: UNDEF/bin/hadoop job  -Dmapred.job.tracker=h1-3005.local:8021 -kill job_201403261402_0011
14/03/26 16:34:28 INFO streaming.StreamJob: Tracking URL: http://h1-3005.local:50030/jobdetails.jsp?jobid=job_201403261402_0011
14/03/26 16:34:28 ERROR streaming.StreamJob: Job not successful. Error: NA
14/03/26 16:34:28 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 
  hadoop streaming failed with error code 1

The stderr from the node is:
Loading required package: rmr2
Loading required package: Rcpp
Loading required package: RJSONIO
Loading required package: methods
Loading required package: bitops
Loading required package: digest
Loading required package: reshape2
Loading required package: stringr
Loading required package: plyr
Loading required package: caTools
Error: !is.null(template) is not TRUE
No traceback available 
Error during wrapup: 
Execution halted
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)


What could be wrong? Thank you!

[Antonio Piccolboni]

It works for me. Could you tell me what version of rmr/hadoop/os you are running on and try a from.dfs(a) to see if the problem is independent of mapreduce. Also could you, at the shell prompt, enter

hdfs dfs -ls /user/cloudera/numbers

and paste the results in your reply

Thanks

Antonio

--
post: rha...@googlegroups.com ||
unsubscribe: rhadoop+u...@googlegroups.com ||
web: https://groups.google.com/d/forum/rhadoop?hl=en-US
---
You received this message because you are subscribed to the Google Groups "RHadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhadoop+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Liang Zhou]

This is in deed the problem!! Somehow /user/cloudera/numbers is corrupted and the problem is gone once I delete the file and rerun the code. 

Thanks very much for the quick reply!

[Antonio Piccolboni]

Glad it works but a little suspicious about file corruption. If you run into other problems please let me know.