compatibility link on RHadoop wiki is broken

[Vincent Carey]

https://github.com/RevolutionAnalytics/rmr2/blob/master/docs/compatibility.md

I get 404.

That seems to be a place to go for information on which Hadoop distros can be used.

It would be very nice to have a set of commands that will verify that the environment is

suitable for rmr2 to work.  With the code in the tutorial (modified to use rmr2 ... yes?)

library(rmr2)

small.ints = 1:1000

sapply(small.ints, function(x) x^2)

 small.ints = to.dfs(1:1000)

  mapreduce(

    input = small.ints, 

    map = function(k, v) cbind(v, v^2))

I get (for the second part)

Not a valid JAR: /home/stvjc/DEMOS/-D

Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 

  hadoop streaming failed with error code 255

Can there be a little more bulletproofing?

R Under development (unstable) (2014-12-02 r67094)

Platform: x86_64-unknown-linux-gnu (64-bit)

locale:

[1] C

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:

[1] rmr2_3.2.0           BiocInstaller_1.17.1

loaded via a namespace (and not attached):

[1] RJSONIO_1.3-0  Rcpp_0.11.3    bitops_1.0-6   caTools_1.17.1 functional_0.6

[6] plyr_1.8.1     reshape2_1.4   stringr_0.6.2 

[Antonio Piccolboni]

On Tue, Dec 2, 2014 at 1:07 PM, Vincent Carey <st...@channing.harvard.edu> wrote:

https://github.com/RevolutionAnalytics/rmr2/blob/master/docs/compatibility.md

I get 404.

Would you mind sharing which page contains the offending link?

That seems to be a place to go for information on which Hadoop distros can be used.

It used to. Recently we realized that it gave the wrong impression that whatever distro was not included there was not compatible. Our ability to test many distros is limited, and the community did not take upon itself to add to the list. We believe it should work with all recent distros. The days when you had to make sure that a list of critical patches were included are behind us.

 

It would be very nice to have a set of commands that will verify that the environment is

suitable for rmr2 to work.  With the code in the tutorial (modified to use rmr2 ... yes?)

R CMD check path-to-pkg is how we do it. It runs the tutorial among many other things.

 

library(rmr2)

small.ints = 1:1000

sapply(small.ints, function(x) x^2)

 small.ints = to.dfs(1:1000)

  mapreduce(

    input = small.ints, 

    map = function(k, v) cbind(v, v^2))

I get (for the second part)

Not a valid JAR: /home/stvjc/DEMOS/-D

Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 

  hadoop streaming failed with error code 255

Can there be a little more bulletproofing?

I am sure there can be, pull requests are welcome.

I think if you assumed a configuration problem as a working hypothesis you might be able to get to the bottom of this. What are you environment variables HADOOP_STREAMING and HADOOP_CMD?

R Under development (unstable) (2014-12-02 r67094)

Platform: x86_64-unknown-linux-gnu (64-bit)

locale:

[1] C

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:

[1] rmr2_3.2.0           BiocInstaller_1.17.1

loaded via a namespace (and not attached):

[1] RJSONIO_1.3-0  Rcpp_0.11.3    bitops_1.0-6   caTools_1.17.1 functional_0.6

[6] plyr_1.8.1     reshape2_1.4   stringr_0.6.2 

--
post: rha...@googlegroups.com ||
unsubscribe: rhadoop+u...@googlegroups.com ||
web: https://groups.google.com/d/forum/rhadoop?hl=en-US
---
You received this message because you are subscribed to the Google Groups "RHadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhadoop+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vincent Carey]

On Tuesday, December 2, 2014 4:24:21 PM UTC-5, Antonio Piccolboni wrote:

On Tue, Dec 2, 2014 at 1:07 PM, Vincent Carey <st...@channing.harvard.edu> wrote:

https://github.com/RevolutionAnalytics/rmr2/blob/master/docs/compatibility.md

I get 404.

Would you mind sharing which page contains the offending link?

https://github.com/RevolutionAnalytics/RHadoop/wiki

second paragraph in "About"

 

That seems to be a place to go for information on which Hadoop distros can be used.

It used to. Recently we realized that it gave the wrong impression that whatever distro was not included there was not compatible. Our ability to test many distros is limited, and the community did not take upon itself to add to the list. We believe it should work with all recent distros. The days when you had to make sure that a list of critical patches were included are behind us.

 

It would be very nice to have a set of commands that will verify that the environment is

suitable for rmr2 to work.  With the code in the tutorial (modified to use rmr2 ... yes?)

R CMD check path-to-pkg is how we do it. It runs the tutorial among many other things.

I tried this.  quickcheck distribution from github has the current source in a folder called "pkg".  I renamed that folder

to get quickcheck installed. 

* checking whether package 'rmr2' can be installed ... OK

* checking installed package size ... OK

* checking package directory ... OK

* checking DESCRIPTION meta-information ... ERROR

During startup - Warning message:

Setting LC_CTYPE failed, using "C"

Nothing informative in the 00check.log

 

 

library(rmr2)

small.ints = 1:1000

sapply(small.ints, function(x) x^2)

 small.ints = to.dfs(1:1000)

  mapreduce(

    input = small.ints, 

    map = function(k, v) cbind(v, v^2))

I get (for the second part)

Not a valid JAR: /home/stvjc/DEMOS/-D

Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 

  hadoop streaming failed with error code 255

Can there be a little more bulletproofing?

I am sure there can be, pull requests are welcome.

I think if you assumed a configuration problem as a working hypothesis you might be able to get to the bottom of this. What are you environment variables HADOOP_STREAMING and HADOOP_CMD?

empty.  what should they be?

thank you for your advice.  i have  the following output for 

 hadoop classpath

/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar

[Antonio Piccolboni]

On Tue, Dec 2, 2014 at 7:44 PM, Vincent Carey <st...@channing.harvard.edu> wrote:

On Tuesday, December 2, 2014 4:24:21 PM UTC-5, Antonio Piccolboni wrote:

On Tue, Dec 2, 2014 at 1:07 PM, Vincent Carey <st...@channing.harvard.edu> wrote:

https://github.com/RevolutionAnalytics/rmr2/blob/master/docs/compatibility.md

I get 404.

Would you mind sharing which page contains the offending link?

https://github.com/RevolutionAnalytics/RHadoop/wiki

second paragraph in "About"

 

Thanks

That is a problem. See https://github.com/RevolutionAnalytics/RHadoop/wiki/user%3Ermr%3EHome somewhere at the end of the second section "Prerequisites and Installation". 

[Vincent Carey]

My bad, I should have seen that page.   I also had to learn that hadoop must be run by a special

user on this system.

Additionally, there is no coordination between the R used to invoke CMD check

and the Rscript used by rmr2 ... I had to ensure that the Rscript I needed was first

in PATH.

Otherwise, CMD check is moving along, with various warnings against R-devel that

I assume you have knowledge of.

Thanks for your assistance.

You received this message because you are subscribed to a topic in the Google Groups "RHadoop" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rhadoop/9O6rNwasWmo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rhadoop+u...@googlegroups.com.