hadoop streaming jar missed

[santhi kanumuru]

Hi,
when i am running rmr job from rstudio  there is no hadoop streaming jar path it is showing an empty path [ ] in red colour

 the job output is like this

packageJobJar: [/tmp/RtmpB22WS3/rmr-local-env4a57dc1c7ab, /tmp/RtmpB22WS3/rmr-global-env4a525ecdf99, /tmp/RtmpB22WS3/rmr-streaming-map4a51545134d, /tmp/hadoop-root/hadoop-unjar4498474245388692627/] [] /tmp/streamjob4202137707137066710.jar tmpDir=null

in   PackageJobJar  description there is no hadoop streming jar information ,

i have set

Sys.setenv(HADOOP_CMD="/usr/bin/hadoop")
Sys.setenv(HADOOP_HOME="/usr/lib/hadoop")
Sys.setenv(HADOOP_CONF="/etc/hadoop/conf")
Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.3.0.jar")

let me know why hadoop streming is not located here


thanks in advance

[Antonio Piccolboni]

Can you run a streaming job independent of R? Thanks

Antonio

--
post: rha...@googlegroups.com ||
unsubscribe: rhadoop+u...@googlegroups.com ||
web: https://groups.google.com/d/forum/rhadoop?hl=en-US
---
You received this message because you are subscribed to the Google Groups "RHadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhadoop+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[santhi kanumuru]

means i run the same rmr program on single node but it is failing when i am running on cluster is there any explicit setup to run rmr jobs on hadoop cluster

[Antonio Piccolboni]

No I mean a simple streaming job

$HADOOP_CMD jar $HADOOP_STREAMING -input someinput -output someoutput -mapper cat

[santhi kanumuru]

yes i test with wordcount it was successfully run.

i installed rmr  and rhdfs on each node and i am submitting the job from namenode itself, please verify the code and output


small.ints <- to.dfs(1:1000)
out <- mapreduce(input = small.ints, map = function(k, v) keyval(v, v^2))
df <- as.data.frame(from.dfs(out))


and the output is

packageJobJar: [/tmp/RtmpNTOATZ/rmr-local-env6fd4b251818, /tmp/RtmpNTOATZ/rmr-global-env6fd42f275343, /tmp/RtmpNTOATZ/rmr-streaming-map6fd46bcd7924] [/usr/lib/hadoop/hadoop-streaming-2.0.0-mr1-cdh4.3.0.jar] /tmp/streamjob5131748567260469892.jar tmpDir=null
13/10/23 12:46:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/10/23 12:46:22 INFO mapred.FileInputFormat: Total input paths to process : 1
13/10/23 12:46:22 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-root/mapred/local]
13/10/23 12:46:22 INFO streaming.StreamJob: Running job: job_201310231229_0002
13/10/23 12:46:22 INFO streaming.StreamJob: To kill this job, run:
13/10/23 12:46:22 INFO streaming.StreamJob: /usr/lib/hadoop/bin/hadoop job  -Dmapred.job.tracker=humana1.tcshydnextgen.com:8021 -kill job_201310231229_0002
13/10/23 12:46:22 INFO streaming.StreamJob: Tracking URL: http://humana1.tcshydnextgen.com:50030/jobdetails.jsp?jobid=job_201310231229_0002
13/10/23 12:46:23 INFO streaming.StreamJob:  map 0%  reduce 0%
13/10/23 12:46:49 INFO streaming.StreamJob:  map 100%  reduce 100%
13/10/23 12:46:49 INFO streaming.StreamJob: To kill this job, run:
13/10/23 12:46:49 INFO streaming.StreamJob: /usr/lib/hadoop/bin/hadoop job  -Dmapred.job.tracker=humana1.tcshydnextgen.com:8021 -kill job_201310231229_0002
13/10/23 12:46:49 INFO streaming.StreamJob: Tracking URL: http://humana1.tcshydnextgen.com:50030/jobdetails.jsp?jobid=job_201310231229_0002
13/10/23 12:46:49 ERROR streaming.StreamJob: Job not successful. Error: NA
13/10/23 12:46:49 INFO streaming.StreamJob: killJob...
Streaming Command Failed!



and an error occured in the job tracker logs

java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja


please let me know why the job is not get initiated...

On Tuesday, October 22, 2013 6:56:26 PM UTC+5:30, santhi kanumuru wrote:

[Antonio Piccolboni]

Anything in the task attempt stderr log?

Antonio

[santhi kanumuru]

yes 
the task error log shows me,

Rscript no such file or directory then i came to know where the issue was and resolved
Thank you

On Tuesday, October 22, 2013 6:56:26 PM UTC+5:30, santhi kanumuru wrote:

[Konstantinos Mammas]

Santhi do you remember what was this issue? It happens to deal with the same issue and I dont know what to do.