Rhipe issue

9 views

Skip to first unread message

ymzh...@gmail.com

unread,

Jul 9, 2015, 12:17:11 PM7/9/15

to tesser...@googlegroups.com

Installed RHIPE and try to run the example. Here is the script and output. How can I found out why it is not working?

R script file: test1.r

library(Rhipe)
rhinit()
rhoptions(zips = "/user/mike/bin/R.Pkg.tar.gz")
rhoptions(runner = "sh ./R.Pkg/library/Rhipe/bin/RhipeMapReduce.sh")

map1 <- expression({
lapply(seq_along(map.keys), function(r) {
line = strsplit(map.values[[r]], ",")[[1]]
outputkey <- line[1:3]
outputvalue <- data.frame(
date = as.numeric(line[4]),
units = as.numeric(line[5]),
listing = as.numeric(line[6]),
selling = as.numeric(line[7]),
stringsAsFactors = FALSE
)
rhcollect(outputkey, outputvalue)
})
})

reduce1 <- expression(
pre = {
reduceoutputvalue <- data.frame()
},
reduce = {
reduceoutputvalue <- rbind(reduceoutputvalue, do.call(rbind, reduce.values))
},
post = {
reduceoutputkey <- reduce.key[1]
attr(reduceoutputvalue, "location") <- reduce.key[1:3]
names(attr(reduceoutputvalue, "location")) <- c("FIPS","county","state")
rhcollect(reduceoutputkey, reduceoutputvalue)
}
)

mr1 <- rhwatch(
map = map1,
reduce = reduce1,
input = rhfmt("/user/mike/housing/housing.txt", type = "text"),
output = rhfmt("/user/mike/housing/byCounty", type = "sequence"),
readback = FALSE
)

--------------------------------------------------

> source("test1.r")
Loading required package: codetools
Loading required package: rJava
------------------------------------------------
| Please call rhinit() else RHIPE will not run |
------------------------------------------------
Rhipe: Using Rhipe.jar file
Initializing Rhipe v0.75.0
Initializing mapfile caches
Saving 1 parameter to /user/mike/tmp/rhipe-temp-params-edee821ec95ad18faeab334739de0b02 (use rhclean to delete all temp files)
[Thu Jul 9 10:00:09 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: PREP Duration: 0.216
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 0 0 0 0 0 0
reduce 0 0 0 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:14 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: PREP Duration: 5.468
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 0 0 0 0 0 0
reduce 0 0 0 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:20 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 11.249
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 1 0 0 0 0
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:25 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 16.742
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 0
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:30 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 21.902
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 0
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:35 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 27.03
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 0
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:40 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 32.163
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 1
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:46 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 37.315
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 1
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:51 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 42.475
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 1
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:00:56 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 47.606
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 2
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:01:01 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 52.746
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 2
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:01:06 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 57.895
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 2
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:01:11 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 63.019
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 3
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:01:16 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 68.146
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 3
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
[Thu Jul 9 10:01:22 2015] Name:2015-07-09 10:00:06 Job: job_1436452252461_0005 State: RUNNING Duration: 73.259
URL: http://node001:8088/proxy/application_1436452252461_0005/
pct numtasks pending running complete killed failed_attempts
map 0 1 0 1 0 0 3
reduce 0 1 1 0 0 0 0
killed_attempts
map 0
reduce 0
Waiting 5 seconds
Warning message:
In Rhipe:::rhwatch.runner(job = job, mon.sec = mon.sec, readback = readback, :
Job failure, deleting output: /user/mike/housing/byCounty:
>

Saptarshi Guha

unread,

Jul 9, 2015, 1:09:46 PM7/9/15

to ymzh...@gmail.com, tesser...@googlegroups.com

Sometimes (unfortunately) the errors dont'c ome back to the console. If possible go to the job tracker and see the stdout/stderr logs - you will likely see the error there.

--
You received this message because you are subscribed to the Google Groups "Tessera-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tessera-user...@googlegroups.com.
To post to this group, send email to tesser...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tessera-users/92ae736e-e9bd-4f64-b2e5-426ac1450193%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward

0 new messages