I am using Rhipe 0.73, while executing a job I am getting some error. Can anyone help me?

Amal G Jose

unread,

Mar 20, 2013, 1:29:20 AM3/20/13

to rh...@googlegroups.com

My code is given below

hadoop fs -mkdir /userhipe

------------------------------R---------------------------

library(Rhipe)

rhinit()

bySpecies <- lapply(unique(iris$Species), function(x) {

list(as.character(x), subset(iris, Species==x))

})

bySpeciesSub <- lapply(1:15, function(i) {

specSub <- iris[((i - 1)*10 + 1):(i*10),]

list(c(as.character(specSub$Species[1]), (i-1) %% 5 + 1), specSub)

})

rhwrite(bySpecies, "/userhipe/bySpecies")

rhwrite(bySpeciesSub, "/userhipe/bySpeciesSub")

rhls("/userhipe/bySpecies/")

tmp <- rhread("/userhipe/bySpecies")

m.ex1.1 <- expression({

lapply(seq_along(map.values), function(r) {

v <- map.values[[r]]

k <- map.keys[[r]]

rhcollect(k, mean(v$Petal.Length))

})

z <- rhwatch(map=m.ex1.1,

input="/userhipe/bySpecies/*",

output="/userhipe/ex1.1"

)

While executing this, I am getting and error

13/03/20 10:51:29 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4ca68fd8

13/03/20 10:51:29 INFO mapred.MapTask: io.sort.mb = 100

13/03/20 10:51:30 INFO mapred.MapTask: data buffer = 79691776/99614720

13/03/20 10:51:30 INFO mapred.MapTask: record buffer = 262144/327680

mapred.input.file == file:/rhipe/bySpecies/part_1

13/03/20 10:51:30 INFO rhipe.RHMRHelper: Mapper:Started external program:R CMD /home/user1/software/amal/R-2.15.1/library/Rhipe/bin/RhipeMapReduce --slave --silent --vanilla

13/03/20 10:51:30 INFO rhipe.RHMRHelper: Mapper:Started Error Thread

13/03/20 10:51:30 INFO rhipe.RHMRHelper: Mapper:Started Output Thread

13/03/20 10:51:31 WARN rhipe.RHMRHelper: Mapper:java.lang.RuntimeException:

R ERROR BEGIN (map.setup):

=============

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

R ERROR END

===========

at org.godhuli.rhipe.RHMRHelper$MRErrorThread.run(RHMRHelper.java:391)

13/03/20 10:51:31 INFO rhipe.RHMRHelper: Mapper:MROutputThread done

13/03/20 10:51:31 WARN mapred.LocalJobRunner: job_local_0002

java.io.IOException: MROutput/MRErrThread failed:java.lang.RuntimeException:

R ERROR BEGIN (map.setup):

=============

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

R ERROR END

===========

at org.godhuli.rhipe.RHMRHelper$MRErrorThread.run(RHMRHelper.java:391)

at org.godhuli.rhipe.RHMRHelper.checkOuterrThreadsThrowable(RHMRHelper.java:232)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:68)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:263)

13/03/20 10:51:31 INFO filecache.TrackerDistributedCacheManager: Deleted path /tmp/hadoop-root/mapred/local/archive/-3027963830366467070_750245607_104772155/file/tmp/rhipe-temp-params-8dc8135d384a1a97617d95e9fa0acdb0

13/03/20 10:51:35 INFO mapred.LocalJobRunner:

Saptarshi Guha

unread,

Mar 20, 2013, 2:50:10 AM3/20/13

to rh...@googlegroups.com

Hmm,

0.73 is beta, though will be live this Friday.

This code works

hdfs.setwd("/user/sguha/tmp/f4")

bySpecies <- lapply(unique(iris$Species), function(x) {
list(as.character(x), subset(iris, Species==x))
})

rhwrite(bySpecies, "bySpecies")
m.ex1.1 <- expression({
mapply(function(k,v) {
rhcollect(k, mean(v$Petal.Length))

},map.keys,map.values,SIMPLIFY=FALSE)

})
z <- rhwatch(map=m.ex1.1, reduce=0,input="bySpecies/*", output="ex1.1")

In your case,

a) could you use paste the lines that came before

13/03/20 10:51:29 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4ca68fd8

13/03/20 10:51:29 INFO mapred.MapTask: io.sort.mb = 100

b) can you paste the job file, which can be found in the Hadoop Jobtracker e.g. and looks like
/sguha/.staging/job_201303051534_4328/job.xml

c) what value is rhoptions()$HADOOP.TMP ?

The error

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

usually occurs when a 'load' is called and the file to load is not present.

--

---
You received this message because you are subscribed to the Google Groups "rhipe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhipe+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Amal G Jose

unread,

Mar 20, 2013, 5:54:47 AM3/20/13

to rh...@googlegroups.com, saptars...@gmail.com

a)

/ex1.1

13/03/20 14:32:29 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

13/03/20 14:32:29 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

13/03/20 14:32:29 INFO input.FileInputFormat: Total input paths to process : 1

13/03/20 14:32:29 WARN conf.Configuration: dfs.df.interval is deprecated. Instead, use fs.df.interval

13/03/20 14:32:29 WARN conf.Configuration: dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects

13/03/20 14:32:29 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available

13/03/20 14:32:29 WARN conf.Configuration: dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir

13/03/20 14:32:29 WARN conf.Configuration: dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir

13/03/20 14:32:29 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS

13/03/20 14:32:29 WARN conf.Configuration: fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir

13/03/20 14:32:29 WARN conf.Configuration: dfs.block.size is deprecated. Instead, use dfs.blocksize

13/03/20 14:32:29 WARN conf.Configuration: dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision

13/03/20 14:32:29 WARN conf.Configuration: dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min

13/03/20 14:32:29 WARN conf.Configuration: dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir

13/03/20 14:32:29 WARN conf.Configuration: dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad

13/03/20 14:32:29 WARN conf.Configuration: dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec

13/03/20 14:32:29 WARN conf.Configuration: dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct

13/03/20 14:32:29 WARN conf.Configuration: dfs.http.address is deprecated. Instead, use dfs.namenode.http-address

13/03/20 14:32:29 WARN conf.Configuration: dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore

13/03/20 14:32:29 WARN conf.Configuration: dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource

13/03/20 14:32:29 WARN conf.Configuration: dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address

13/03/20 14:32:29 WARN conf.Configuration: dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address

13/03/20 14:32:29 WARN conf.Configuration: dfs.permissions is deprecated. Instead, use dfs.permissions.enabled

13/03/20 14:32:29 WARN conf.Configuration: dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension

13/03/20 14:32:29 WARN conf.Configuration: dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads

13/03/20 14:32:29 WARN conf.Configuration: dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth

13/03/20 14:32:29 WARN conf.Configuration: dfs.https.address is deprecated. Instead, use dfs.namenode.https-address

13/03/20 14:32:29 WARN conf.Configuration: dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval

13/03/20 14:32:29 WARN conf.Configuration: fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir

13/03/20 14:32:29 WARN conf.Configuration: dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size

13/03/20 14:32:29 WARN conf.Configuration: dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup

13/03/20 14:32:29 WARN conf.Configuration: topology.script.number.args is deprecated. Instead, use net.topology.script.number.args

13/03/20 14:32:29 WARN conf.Configuration: dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address

13/03/20 14:32:29 WARN conf.Configuration: fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period

13/03/20 14:32:29 WARN conf.Configuration: topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl

13/03/20 14:32:29 WARN conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae in /tmp/hadoop-root/mapred/local/archive/-193089761312830351_133345639_118032155/file/tmp-work-8333888260353126556 with rwxr-xr-x

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Cached /tmp/rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae#rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae as /tmp/hadoop-root/mapred/local/archive/-193089761312830351_133345639_118032155/file/tmp/rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae

13/03/20 14:32:30 WARN mapred.LocalJobRunner: LocalJobRunner does not support symlinking into current working dir.

13/03/20 14:32:30 INFO mapred.TaskRunner: Creating symlink: /tmp/hadoop-root/mapred/local/archive/-193089761312830351_133345639_118032155/file/tmp/rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae <- /tmp/hadoop-root/mapred/local/localRunner/rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/.job.splitmetainfo.crc <- /tmp/hadoop-root/mapred/local/localRunner/.job.splitmetainfo.crc

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/.job.xml.crc <- /tmp/hadoop-root/mapred/local/localRunner/.job.xml.crc

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/job.xml <- /tmp/hadoop-root/mapred/local/localRunner/job.xml

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/job.split <- /tmp/hadoop-root/mapred/local/localRunner/job.split

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/.job.split.crc <- /tmp/hadoop-root/mapred/local/localRunner/.job.split.crc

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/.job.jar.crc <- /tmp/hadoop-root/mapred/local/localRunner/.job.jar.crc

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/job.jar <- /tmp/hadoop-root/mapred/local/localRunner/job.jar

13/03/20 14:32:30 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/staging/root1083697648/.staging/job_local_0001/job.splitmetainfo <- /tmp/hadoop-root/mapred/local/localRunner/job.splitmetainfo

13/03/20 14:32:30 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS

13/03/20 14:32:30 WARN conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

Error in .jcall("RJavaTools", "Ljava/lang/Object;", "invokeMethod", cl, :

java.lang.ArrayIndexOutOfBoundsException: 1

> 13/03/20 14:32:30 INFO mapred.LocalJobRunner: OutputCommitter set in config null

13/03/20 14:32:30 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

13/03/20 14:32:30 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead

13/03/20 14:32:31 INFO util.ProcessTree: setsid exited with exit code 0

13/03/20 14:32:31 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4987b287

mapred.input.file == file:/bySpecies/part_1

13/03/20 14:32:31 INFO rhipe.RHMRHelper: Mapper:Started external program:R CMD /home/user1/software/amal/R-2.15.1/library/Rhipe/bin/RhipeMapReduce --slave --silent --vanilla

13/03/20 14:32:31 INFO rhipe.RHMRHelper: Mapper:Started Error Thread

13/03/20 14:32:31 INFO rhipe.RHMRHelper: Mapper:Started Output Thread

13/03/20 14:32:32 WARN rhipe.RHMRHelper: Mapper:java.lang.RuntimeException:

R ERROR BEGIN (map.setup):

=============

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

R ERROR END

===========

at org.godhuli.rhipe.RHMRHelper$MRErrorThread.run(RHMRHelper.java:391)

13/03/20 14:32:32 INFO rhipe.RHMRHelper: Mapper:MROutputThread done

13/03/20 14:32:32 WARN mapred.LocalJobRunner: job_local_0001

java.io.IOException: MROutput/MRErrThread failed:java.lang.RuntimeException:

R ERROR BEGIN (map.setup):

=============

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

R ERROR END

===========

at org.godhuli.rhipe.RHMRHelper$MRErrorThread.run(RHMRHelper.java:391)

at org.godhuli.rhipe.RHMRHelper.checkOuterrThreadsThrowable(RHMRHelper.java:232)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:68)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:263)

13/03/20 14:32:32 INFO filecache.TrackerDistributedCacheManager: Deleted path /tmp/hadoop-root/mapred/local/archive/-193089761312830351_133345639_118032155/file/tmp/rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae

13/03/20 14:32:37 INFO mapred.LocalJobRunner:

b) I will attach the job.xml soon

c)

> rhoptions()$HADOOP.TMP

[1] "/tmp"

Amal G Jose

unread,

Mar 20, 2013, 6:39:48 AM3/20/13

to rh...@googlegroups.com

This the the error that I am getting while running the code that you have given, please help me in resolving this error.

> z <- rhwatch(map=m.ex1.1, reduce=0,input="/user/root/ip1/*", output="/user/root/op1")

Loading required package: codetools

/user/root/op1

13/03/20 16:04:03 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

13/03/20 16:04:04 INFO input.FileInputFormat: Total input paths to process : 1

13/03/20 16:04:05 WARN conf.Configuration: dfs.df.interval is deprecated. Instead, use fs.df.interval

13/03/20 16:04:05 WARN conf.Configuration: dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects

13/03/20 16:04:05 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available

13/03/20 16:04:05 WARN conf.Configuration: dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir

13/03/20 16:04:05 WARN conf.Configuration: dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir

13/03/20 16:04:05 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS

13/03/20 16:04:05 WARN conf.Configuration: fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir

13/03/20 16:04:05 WARN conf.Configuration: dfs.block.size is deprecated. Instead, use dfs.blocksize

13/03/20 16:04:05 WARN conf.Configuration: dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision

13/03/20 16:04:05 WARN conf.Configuration: dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min

13/03/20 16:04:05 WARN conf.Configuration: dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir

13/03/20 16:04:05 WARN conf.Configuration: dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad

13/03/20 16:04:05 WARN conf.Configuration: dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec

13/03/20 16:04:05 WARN conf.Configuration: dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct

13/03/20 16:04:05 WARN conf.Configuration: dfs.http.address is deprecated. Instead, use dfs.namenode.http-address

13/03/20 16:04:05 WARN conf.Configuration: dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore

13/03/20 16:04:05 WARN conf.Configuration: dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource

13/03/20 16:04:05 WARN conf.Configuration: dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address

13/03/20 16:04:05 WARN conf.Configuration: dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address

13/03/20 16:04:05 WARN conf.Configuration: dfs.permissions is deprecated. Instead, use dfs.permissions.enabled

13/03/20 16:04:05 WARN conf.Configuration: dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension

13/03/20 16:04:05 WARN conf.Configuration: dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads

13/03/20 16:04:05 WARN conf.Configuration: dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth

13/03/20 16:04:05 WARN conf.Configuration: dfs.https.address is deprecated. Instead, use dfs.namenode.https-address

13/03/20 16:04:05 WARN conf.Configuration: dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval

13/03/20 16:04:05 WARN conf.Configuration: fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir

13/03/20 16:04:05 WARN conf.Configuration: dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size

13/03/20 16:04:05 WARN conf.Configuration: dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup

13/03/20 16:04:05 WARN conf.Configuration: topology.script.number.args is deprecated. Instead, use net.topology.script.number.args

13/03/20 16:04:05 WARN conf.Configuration: dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address

13/03/20 16:04:05 WARN conf.Configuration: fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period

13/03/20 16:04:05 WARN conf.Configuration: topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl

13/03/20 16:04:05 WARN conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

[Wed Mar 20 16:04:06 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: PREP Duration: 341.838

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 0 0 0 0 0 0 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

[Wed Mar 20 16:04:11 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: PREP Duration: 347.204

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 1 1 0 0 0 0 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

[Wed Mar 20 16:04:17 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: RUNNING Duration: 352.372

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 1 0 1 0 0 0 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

There were Hadoop specific errors (autokill will not kill job), showing at most 30:

java.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

at java.lang.UNI[Wed Mar 20 16:04:22 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: RUNNING Duration: 357.511

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 1 0 1 0 0 1 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

There were Hadoop specific errors (autokill will not kill job), showing at most 30:

java.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

at java.lang.UNI[Wed Mar 20 16:04:27 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: RUNNING Duration: 362.653

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 1 0 1 0 0 2 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

There were Hadoop specific errors (autokill will not kill job), showing at most 30:

java.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

at java.lang.UNI[Wed Mar 20 16:04:32 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: RUNNING Duration: 367.782

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 1 0 1 0 0 3 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

There were Hadoop specific errors (autokill will not kill job), showing at most 30:

java.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

at java.lang.UNI[Wed Mar 20 16:04:37 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: RUNNING Duration: 372.889

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 1 0 1 0 0 3 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

There were Hadoop specific errors (autokill will not kill job), showing at most 30:

java.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

at java.lang.UNIjava.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=13, Permission denied

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied

at java.lang.UNIXProcess.<init[Wed Mar 20 16:04:42 2013] Name:2013-03-20 16:04:03 Job: job_201302142117_4147 State: RUNNING Duration: 377.995

URL: http://BHUKRK105508D:50030/jobdetails.jsp?jobid=job_201302142117_4147

pct numtasks pending running complete killed failed_attempts killed_attempts

map 0 1 0 0 0 1 4 0

reduce 0 0 0 0 0 0 0 0

Waiting 5 seconds

There were Hadoop specific errors (autokill will not kill job), showing at most 30:

java.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

at java.lang.UNIjava.lang.RuntimeException: configuration exception

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)

at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:86)

at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:53)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.io.IOException: Cannot run program "R": java.io.IOException: error=13, Permission denied

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)

... 9 more

Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied

at java.lang.UNIXProcess.<initDeleted hdfs://bhukrk105508d:51400/user/root/op1

Warning message:

In Rhipe:::rhwatch.runner(job = job, mon.sec = mon.sec, readback = readback, :

Job failure, deleting output: /user/root/op1:

Saptarshi Guha

unread,

Mar 20, 2013, 1:09:17 PM3/20/13

to rh...@googlegroups.com

Did you call rhinit? I wont be copying code verbatim. I was simplifying your code
to make it easier to debug.

Instead of your rhwatch, use the rhwatch call I provided.

Amal G Jose

unread,

Mar 20, 2013, 1:31:39 PM3/20/13

to rh...@googlegroups.com

The No such file or directory error is resolved now.
Rhipe was not present in all the tasktracker nodes. So while running the job it goes to some tasktracker where R is not present.
So by installing Rhipe everywhere I resoved this issue.
Now I got another error.
Error 13 permission denied Java IO exception.

What may be the issue.?

Actually while running the job some files are getting created in /tmp folder which is having 700 permission. Will this create any issue. I think these are in intermediate output generated by the job.

The job is getting failed at map phase.

--

---
You received this message because you are subscribed to a topic in the Google Groups "rhipe" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rhipe/9lKL6ioSHX0/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to rhipe+un...@googlegroups.com.

Saptarshi Guha

unread,

Mar 20, 2013, 1:40:01 PM3/20/13

to rh...@googlegroups.com

On Wed, Mar 20, 2013 at 10:31 AM, Amal G Jose <amal...@gmail.com> wrote:

Actually while running the job some files are getting created in /tmp folder which is having 700 permission.

This is an issue and probably causing your initial error. So once you resolve your installation issues, make a directory in your home directory

which i guess is "/user/root/" , called /user/root/tmp and then set

rhoptions(HADOOP.TMP.FOLDER = "/user/root/tmp" )

and then re-run.

In the first query you ran, i saw this

rhipe-temp-params-96dffe81e6cd9d4d3a8532b1307641ae

which means rhipe is trying to create a file with temporary parameters in the HDFS '/tmp' folder (and is likely failing).

Hence creatting user-specific tmp folders is a good approach.

Also RHIPE will ceate many temp files in rhoptions()$HADOOP.TMP.FOLDE. So every now and then,

(when no jobs are running), run rhclean.

Amal G Jose

unread,

Mar 22, 2013, 1:50:25 PM3/22/13

to rh...@googlegroups.com

Thank you very much... issues are solved..:)

Please let us know the release of Rhipe 0.73 stable version.

Reply all

Reply to author

Forward