IOException Broken Pipe

60 views
Skip to first unread message

Ravi Puranik

unread,
Mar 27, 2015, 3:14:46 PM3/27/15
to rh...@googlegroups.com

I am trying to run a Rhipe M/R job in a single node CDH4.7 VM and am getting the exception below.

I have the env variables setup per the installation instructions and am using Rhipe_0.74.0 (also tried Rhipe_0.73.1)

Has anyone encountered this issue?

syslog logs

2015-03-27 12:09:31,266 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2015-03-27 12:09:33,415 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /mapred/local/taskTracker/distcache/-7015955577281540294_-1819052313_1554182977/localhost.localdomain/tmp/rhipe-temp-params-828ea5867ec9fee458144d3990ad6b80 <- /mapred/local/taskTracker/cloudera/jobcache/job_201503271205_0001/attempt_201503271205_0001_m_000000_2/work/rhipe-temp-params-828ea5867ec9fee458144d3990ad6b80
2015-03-27 12:09:33,452 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /mapred/local/taskTracker/cloudera/jobcache/job_201503271205_0001/jars/.job.jar.crc <- /mapred/local/taskTracker/cloudera/jobcache/job_201503271205_0001/attempt_201503271205_0001_m_000000_2/work/.job.jar.crc
2015-03-27 12:09:33,467 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /mapred/local/taskTracker/cloudera/jobcache/job_201503271205_0001/jars/job.jar <- /mapred/local/taskTracker/cloudera/jobcache/job_201503271205_0001/attempt_201503271205_0001_m_000000_2/work/job.jar
2015-03-27 12:09:33,824 WARN org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-03-27 12:09:33,837 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2015-03-27 12:09:35,272 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2015-03-27 12:09:35,310 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@78dc9766
2015-03-27 12:09:36,014 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://localhost.localdomain:8020/user/cloudera/access_log:0+134217728
2015-03-27 12:09:36,032 INFO org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-03-27 12:09:36,040 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 50
2015-03-27 12:09:36,216 INFO org.apache.hadoop.mapred.MapTask: data buffer = 39845888/49807360
2015-03-27 12:09:36,217 INFO org.apache.hadoop.mapred.MapTask: record buffer = 131072/163840
2015-03-27 12:09:36,446 INFO org.godhuli.rhipe.RHMRHelper: Mapper:Started external program:sh /usr/lib64/R/library/Rhipe/bin/RhipeMapReduce.sh
2015-03-27 12:09:36,448 INFO org.godhuli.rhipe.RHMRHelper: Mapper:Started Error Thread
2015-03-27 12:09:36,468 INFO org.godhuli.rhipe.RHMRHelper: Mapper:Started Output Thread
2015-03-27 12:09:37,110 INFO org.godhuli.rhipe.RHMRHelper: Mapper:MROutputThread done
2015-03-27 12:09:37,110 INFO org.godhuli.rhipe.RHMRHelper: Mapper:MRErrorThread done
2015-03-27 12:09:37,110 INFO org.godhuli.rhipe.RHMRMapper: QUIIIITING:127
2015-03-27 12:09:37,150 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2015-03-27 12:09:37,158 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.io.IOException: java.io.IOException: Broken pipe
2015-03-27 12:09:37,158 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: java.io.IOException: Broken pipe
    at org.godhuli.rhipe.RHMRMapper.map(RHMRMapper.java:113)
    at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:56)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.io.IOException: Broken pipe
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:282)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at org.godhuli.rhipe.RHBytesWritable.write(RHBytesWritable.java:121)
    at org.godhuli.rhipe.RHMRHelper.write(RHMRHelper.java:305)
    at org.godhuli.rhipe.RHMRMapper.map(RHMRMapper.java:107)
    ... 8 more
2015-03-27 12:09:37,190 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

Reply all
Reply to author
Forward
0 new messages