log4j issue

[abdelkafi ahmed]

Hello Revolution Analytics team,

I have recently successful done a Hadoop multi node cluster setup (version 2.7.1) in 4 virtualbox virtual machines running Ubuntu server 15.10.

I have installed in the master node RStudio server so that i can execute jobs remotely from my laptop, and in the 3 other nodes i installed the r-basic packages (R).

I wanted to setup a R-Hadoop envirenment, i followed your tutorial and installed the R-Hadoop packages with their dependencies :

plyrmr On every node in the cluster


ravro Only on the node that runs the R client

rhbase Only on the node that runs the R client

rhdfs Only on the node that runs the R client

rmr2 On every node in the cluster

After that, i wanted to execute the WordCount example using R this is the code :

Sys.setenv("HADOOP_PREFIX"="/usr/local/hadoop")
Sys.setenv("HADOOP_CMD"="/usr/local/hadoop/bin/hadoop")
Sys.setenv("HADOOP_STREAMING"="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.1.jar")

library(rmr2)

## map function
map <- function(k,lines) {
  words.list <- strsplit(lines, '\\s') 
  words <- unlist(words.list)
  return( keyval(words, 1) )
}

## reduce function
reduce <- function(word, counts) { 
  keyval(word, sum(counts))
}

wordcount <- function (input, output=NULL) { 
  mapreduce(input=input, output=output, input.format="text", 
            map=map, reduce=reduce)
}

## Submit job
hdfs.root <- '/user/hduser/wordcount'
hdfs.data <- file.path(hdfs.root, 'data') 
hdfs.out <- file.path(hdfs.root, 'out') 
out <- wordcount(hdfs.data, hdfs.out)

PS : There is a file named input.txt in the hdfs : /user/hduser/wordcount/data/input.txt

This is the log printed in the console :

packageJobJar: [/tmp/hadoop-unjar5542234117460705124/] [] /tmp/streamjob3511407528655074002.jar tmpDir=null
15/12/26 18:24:45 INFO client.RMProxy: Connecting to ResourceManager at hadoopmnmaster/192.168.56.11:8050
15/12/26 18:24:45 INFO client.RMProxy: Connecting to ResourceManager at hadoopmnmaster/192.168.56.11:8050
15/12/26 18:24:46 INFO mapred.FileInputFormat: Total input paths to process : 1
15/12/26 18:24:46 INFO mapreduce.JobSubmitter: number of splits:2
15/12/26 18:24:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1451147739123_0009
15/12/26 18:24:47 INFO impl.YarnClientImpl: Submitted application application_1451147739123_0009
15/12/26 18:24:47 INFO mapreduce.Job: The url to track the job: http://hadoopmnmaster:8088/proxy/application_1451147739123_0009/
15/12/26 18:24:47 INFO mapreduce.Job: Running job: job_1451147739123_0009
15/12/26 18:24:55 INFO mapreduce.Job: Job job_1451147739123_0009 running in uber mode : false
15/12/26 18:24:55 INFO mapreduce.Job:  map 0% reduce 0%
15/12/26 18:25:07 INFO mapreduce.Job: Task Id : attempt_1451147739123_0009_m_000001_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

15/12/26 18:25:07 INFO mapreduce.Job: Task Id : attempt_1451147739123_0009_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

15/12/26 18:25:16 INFO mapreduce.Job:  map 50% reduce 0%
15/12/26 18:25:16 INFO mapreduce.Job: Task Id : attempt_1451147739123_0009_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

15/12/26 18:25:16 INFO mapreduce.Job: Task Id : attempt_1451147739123_0009_m_000001_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

15/12/26 18:25:17 INFO mapreduce.Job:  map 0% reduce 0%
15/12/26 18:25:22 INFO mapreduce.Job: Task Id : attempt_1451147739123_0009_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

15/12/26 18:25:23 INFO mapreduce.Job: Task Id : attempt_1451147739123_0009_m_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

15/12/26 18:25:31 INFO mapreduce.Job:  map 100% reduce 100%
15/12/26 18:25:31 INFO mapreduce.Job: Job job_1451147739123_0009 failed with state FAILED due to: Task failed task_1451147739123_0009_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/12/26 18:25:31 INFO mapreduce.Job: Counters: 13
	Job Counters 
		Failed map tasks=7
		Killed map tasks=1
		Launched map tasks=8
		Other local map tasks=6
		Data-local map tasks=2
		Total time spent by all maps in occupied slots (ms)=56383
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=56383
		Total vcore-seconds taken by all map tasks=56383
		Total megabyte-seconds taken by all map tasks=57736192
	Map-Reduce Framework
		CPU time spent (ms)=0
		Physical memory (bytes) snapshot=0
		Virtual memory (bytes) snapshot=0
15/12/26 18:25:31 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!

And this is the stderr log :

log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

I looked in that link but didn't find any solution. 

Can you please help me ?

What should i do ?

Thank you for your help.