Dear Antonio
I am running hadoop from CDH4. I have checked up, it is running in pseudo-distributed mode.
I am unable to attach stderr file: The outputs are as below:
stderr output from one attempt
___________________________________________________________________
Loading objects:
.Last
.Random.seed
binary
count
map
mydata
myformat
mylogit
Loading objects:
backend.parameters
combine
combine.file
combine.line
debug
default.input.format
default.output.format
in.folder
in.memory.combine
input.format
keyval.length
libs
map
Warning: namespace ‘plyrmr’ is not available and has been replaced
by .GlobalEnv when processing object ‘map’
map.file
map.line
out.folder
output.format
pkg.opts
preamble
profile.nodes
reduce
reduce.file
reduce.line
rmr.global.env
rmr.local.env
save.env
vectorized.reduce
verbose
work.dir
Loading required package: plyrmr
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘plyrmr’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load plyrmr
Loading required package: hydroPSO
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘hydroPSO’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load hydroPSO
Loading required package: R.methodsS3
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘R.methodsS3’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load R.methodsS3
Loading required package: pryr
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘pryr’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load pryr
Loading required package: rhdfs
Loading required package: methods
Loading required package: rJava
Error : .onLoad failed in loadNamespace() for 'rhdfs', details:
call: fun(libname, pkgname)
error: Environment variable HADOOP_CMD must be set before loading package rhdfs
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load rhdfs
Loading required package: rmr2
Loading required package: Rcpp
Loading required package: RJSONIO
Loading required package: bitops
Loading required package: digest
Loading required package: stringr
Loading required package: plyr
Attaching package: ‘plyr’
The following object is masked _by_ ‘.GlobalEnv’:
count
Loading required package: reshape2
Error in map(keys(kv), values(kv)) : could not find function "safe.cbind"
Calls: <Anonymous> -> <Anonymous> -> as.keyval -> is.keyval -> map
Execution halted
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
________________________________________________________________________
stderr output from another attempt but same job
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Loading objects:
.Last
.Random.seed
binary
count
map
mydata
myformat
mylogit
Loading objects:
backend.parameters
combine
combine.file
combine.line
debug
default.input.format
default.output.format
in.folder
in.memory.combine
input.format
keyval.length
libs
map
Warning: namespace ‘plyrmr’ is not available and has been replaced
by .GlobalEnv when processing object ‘map’
map.file
map.line
out.folder
output.format
pkg.opts
preamble
profile.nodes
reduce
reduce.file
reduce.line
rmr.global.env
rmr.local.env
save.env
vectorized.reduce
verbose
work.dir
Loading required package: plyrmr
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘plyrmr’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load plyrmr
Loading required package: hydroPSO
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘hydroPSO’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load hydroPSO
Loading required package: R.methodsS3
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘R.methodsS3’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load R.methodsS3
Loading required package: pryr
Warning in library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘pryr’
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load pryr
Loading required package: rhdfs
Loading required package: methods
Loading required package: rJava
Error : .onLoad failed in loadNamespace() for 'rhdfs', details:
call: fun(libname, pkgname)
error: Environment variable HADOOP_CMD must be set before loading package rhdfs
Warning in FUN(c("graphics", "grDevices", "utils", "datasets", "plyrmr", :
can't load rhdfs
Loading required package: rmr2
Loading required package: Rcpp
Loading required package: RJSONIO
Loading required package: bitops
Loading required package: digest
Loading required package: stringr
Loading required package: plyr
Attaching package: ‘plyr’
The following object is masked _by_ ‘.GlobalEnv’:
count
Loading required package: reshape2
Error in map(keys(kv), values(kv)) : could not find function "safe.cbind"
Calls: <Anonymous> -> <Anonymous> -> as.keyval -> is.keyval -> map
Execution halted
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The console output is as below:
> as.data.frame(transform(input("/user/test/mtcars"), carb.per.cyl = carb/cyl))
packageJobJar: [/tmp/RtmpeD4xQ1/rmr-local-env307310da66b6, /tmp/RtmpeD4xQ1/rmr-global-env307378551d85, /tmp/RtmpeD4xQ1/rmr-streaming-map30733d28e1c2, /tmp/hadoop-ashokharnal/hadoop-unjar1040037147401611408/] [] /tmp/streamjob7463591244593531676.jar tmpDir=null
13/11/12 06:54:06 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/11/12 06:54:06 INFO mapred.FileInputFormat: Total input paths to process : 1
13/11/12 06:54:06 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-ashokharnal/mapred/local]
13/11/12 06:54:06 INFO streaming.StreamJob: Running job: job_201311120553_0002
13/11/12 06:54:06 INFO streaming.StreamJob: To kill this job, run:
13/11/12 06:54:06 INFO streaming.StreamJob: /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=master:8021 -kill job_201311120553_0002
13/11/12 06:54:06 INFO streaming.StreamJob: Tracking URL:
http://master:50030/jobdetails.jsp?jobid=job_201311120553_000213/11/12 06:54:07 INFO streaming.StreamJob: map 0% reduce 0%
13/11/12 06:54:56 INFO streaming.StreamJob: map 100% reduce 100%
13/11/12 06:54:56 INFO streaming.StreamJob: To kill this job, run:
13/11/12 06:54:56 INFO streaming.StreamJob: /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=master:8021 -kill job_201311120553_0002
13/11/12 06:54:56 INFO streaming.StreamJob: Tracking URL:
http://master:50030/jobdetails.jsp?jobid=job_201311120553_000213/11/12 06:54:56 ERROR streaming.StreamJob: Job not successful. Error: NA
13/11/12 06:54:56 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, :
hadoop streaming failed with error code 1
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
rmr: DEPRECATED: Please use 'rm -r' instead.
Moved: 'hdfs://master:8020/tmp/RtmpeD4xQ1/file30731121949b' to trash at: hdfs://master:8020/user/ashokharnal/.Trash/Current
Thanks for help.
Ashok Kumar Harnal
On Monday, November 11, 2013 1:33:01 PM UTC+5:30, Ashok Kumar Harnal wrote: