Problem of installing rhipe

[cls3415]

Hi there, I've tried to install rhipe but got the following error
message:
-------
cls3415@ubuntu:~/Downloads$ sudo R CMD INSTALL Rhipe_0.60.tar.gz *
installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘Rhipe’ ...
checking for gcc... gcc -std=gnu99
checking for C compiler default output file name... rm: cannot remove
`a.out.dSYM': Is a directory
a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc -std=gnu99 accepts -g... yes
checking for gcc -std=gnu99 option to accept ISO C89... none needed
configure: creating ./config.status
config.status: creating src/Makevars
** libs
g++ -I/usr/share/R/include -fpic -g -O2 -I. -g -
DHAVE_UINTPTR_T `pkg-config --cflags protobuf` -Wall -c display.cc -o
display.o
In file included from ream.h:9,
from display.cc:1:
./rexp.pb.h:12:2: error: #error This file was generated by a newer
version of protoc which is
./rexp.pb.h:13:2: error: #error incompatible with your Protocol Buffer
headers. Please update
./rexp.pb.h:14:2: error: #error your headers.
display.cc: In function ‘SEXPREC* collect_buffer(SEXPREC*, SEXPREC*)’:
display.cc:247: warning: deprecated conversion from string constant to
‘char*’
display.cc:247: warning: deprecated conversion from string constant to
‘char*’
display.cc: In function ‘int32_t readJavaInt(FILE*)’:
display.cc:349: warning: ignoring return value of ‘size_t fread(void*,
size_t, size_t, FILE*)’, declared with attribute warn_unused_result
make: *** [display.o] Error 1
ERROR: compilation failed for package ‘Rhipe’
* removing ‘/usr/local/lib/R/site-library/Rhipe’
---------------------
Could anyone tell me how to install rhipe correctly??

Thanks.
S.H.

[Saptarshi Guha]

Hello,

This is the offending message

./rexp.pb.h:12:2: error: #error This file was generated by a newer 

version of protoc which is 

I have built this using protobuf 2.3

> pkg-config --modversion protobuf

2.3.0

You will need to upgrade your version of protouf to 2.3.0

http://code.google.com/p/protobuf/

Hope it helps

Saptarshi

[SH.Chou]

Thanks Saptarshi. I've downloaded the latest version of protocol buffers (protobuf 2.3.0) and tried to install in my ubuntu 10.04 (lucid). But I'm not sure if I have installed or not. I unzipped the file I downloaded from http://code.google.com/p/protobuf/downloads/detail?name=protobuf-2.3.0.tar.bz2&can=2&q=, and then type "sh configure". Then I install rhipe, but still got the same error message:

-------

* installing to library ‘/usr/local/lib/R/site-library’

* installing *source* package ‘Rhipe’ ...

checking for gcc... gcc -std=gnu99

checking for C compiler default output file name... rm: cannot remove `a.out.dSYM': Is a directory

a.out

checking whether the C compiler works... yes

checking whether we are cross compiling... no

checking for suffix of executables... 

checking for suffix of object files... o

checking whether we are using the GNU C compiler... yes

checking whether gcc -std=gnu99 accepts -g... yes

checking for gcc -std=gnu99 option to accept ISO C89... none needed

configure: creating ./config.status

config.status: creating src/Makevars

** libs

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c display.cc -o display.o

In file included from ream.h:9,

                 from display.cc:1:

./rexp.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is

./rexp.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update

./rexp.pb.h:14:2: error: #error your headers.

display.cc: In function ‘SEXPREC* collect_buffer(SEXPREC*, SEXPREC*)’:

display.cc:247: warning: deprecated conversion from string constant to ‘char*’

display.cc:247: warning: deprecated conversion from string constant to ‘char*’

display.cc: In function ‘int32_t readJavaInt(FILE*)’:

display.cc:349: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result

make: *** [display.o] Error 1

ERROR: compilation failed for package ‘Rhipe’

* removing ‘/usr/local/lib/R/site-library/Rhipe’

--------------------

Did I really upgrade protobuf??

Thanks 

S.H.

--
=====================================
Shih-Hsiung, Chou
System Administrator / PH.D Student at
Department of Industrial Manufacturing
and Systems Engineering
Kansas State University

[Saptarshi Guha]

> I unzipped the file I downloaded from http://code.google.com/p/protobuf/downloads/detail?name=protobuf-2.3.0.tar.bz2&can=2&q=, and then type "sh configure". 

You did type:(?)

 make && make install 

# you might need to run sudo make install (assuming you have root permissions)

If you have already done so, try running this

pkg-config --modversion protobuf

and report the output.

It is possible that the protobuf 2.3 was not installed in the same place(as the old one) and okg-config picks up the old version.

Regards

Saptarshi

[SH.Chou]

Hi Saptarshi, thanks again!!

After using make && make install, I think I've successfully installed 2.3.0 version of protobuf. I typed pkg-config --modversion protobuf, and it shows 2.3.0. However, when I reinstall rhipe again, it shows the following error message:

--------------------------

* installing to library ‘/usr/local/lib/R/site-library’

* installing *source* package ‘Rhipe’ ...

checking for gcc... gcc -std=gnu99

checking for C compiler default output file name... rm: cannot remove `a.out.dSYM': Is a directory

a.out

checking whether the C compiler works... yes

checking whether we are cross compiling... no

checking for suffix of executables... 

checking for suffix of object files... o

checking whether we are using the GNU C compiler... yes

checking whether gcc -std=gnu99 accepts -g... yes

checking for gcc -std=gnu99 option to accept ISO C89... none needed

configure: creating ./config.status

config.status: creating src/Makevars

** libs

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c display.cc -o display.o

display.cc: In function ‘SEXPREC* collect_buffer(SEXPREC*, SEXPREC*)’:

display.cc:247: warning: deprecated conversion from string constant to ‘char*’

display.cc:247: warning: deprecated conversion from string constant to ‘char*’

display.cc: In function ‘int32_t readJavaInt(FILE*)’:

display.cc:349: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c main.cc -o main.o

main.cc: In function ‘int main(int, char**)’:

main.cc:219: warning: deprecated conversion from string constant to ‘char*’

main.cc:219: warning: deprecated conversion from string constant to ‘char*’

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c mapper.cc -o mapper.o

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c message.cc -o message.o

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c reducer.cc -o reducer.o

reducer.cc:8: warning: deprecated conversion from string constant to ‘char*’

reducer.cc:9: warning: deprecated conversion from string constant to ‘char*’

reducer.cc:10: warning: deprecated conversion from string constant to ‘char*’

reducer.cc:11: warning: deprecated conversion from string constant to ‘char*’

reducer.cc:12: warning: deprecated conversion from string constant to ‘char*’

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c rexp.pb.cc -o rexp.pb.o

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c rhooks.cc -o rhooks.o

rhooks.cc: In function ‘void CaptureLogInLibrary(google::protobuf::LogLevel, const char*, int, const std::string&)’:

rhooks.cc:18: warning: deprecated conversion from string constant to ‘char*’

rhooks.cc:18: warning: deprecated conversion from string constant to ‘char*’

rhooks.cc:18: warning: deprecated conversion from string constant to ‘char*’

rhooks.cc:18: warning: deprecated conversion from string constant to ‘char*’

rhooks.cc:18: warning: deprecated conversion from string constant to ‘char*’

rhooks.cc: In function ‘SEXPREC* readBinaryFile(SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*)’:

rhooks.cc:309: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result

rhooks.cc:330: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result

rhooks.cc: In function ‘SEXPREC* readSQFromPipe(SEXPREC*, SEXPREC*, SEXPREC*)’:

rhooks.cc:455: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result

rhooks.cc:469: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result

g++ -I/usr/share/R/include      -fpic  -g -O2 -c tests.cc -o tests.o

g++ -I/usr/share/R/include      -fpic  -g -O2 -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall -c utility.cc -o utility.o

utility.cc: In function ‘int64_t readVInt64FromFileDescriptor(FILE*)’:

utility.cc:186: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result

g++ -shared -o Rhipe.so display.o main.o mapper.o message.o reducer.o rexp.pb.o rhooks.o tests.o utility.o -pthread -L/usr/local/lib -lprotobuf -lz -L/usr/lib/R/lib -lR

g++   -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall `pkg-config --libs protobuf`  `R CMD config --cppflags` -fpic `R CMD config --ldflags` rexp.pb.o message.o utility.o display.o reducer.o mapper.o main.o -o imperious.so

installing to /usr/local/lib/R/site-library/Rhipe/libs

** R

** inst

** preparing package for lazy loading

** help

*** installing help indices

** building package indices ...

** testing if installed package can be loaded

Error in dyn.load(file, DLLpath = DLLpath, ...) : 

  unable to load shared library '/usr/local/lib/R/site-library/Rhipe/libs/Rhipe.so':

  libprotobuf.so.6: cannot open shared object file: No such file or directory

ERROR: loading failed

* removing ‘/usr/local/lib/R/site-library/Rhipe’

[Saptarshi Guha]

g++   -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall `pkg-config --libs protobuf`  `R CMD config --cppflags` -fpic `R CMD config --ldflags` rexp.pb.o message.o utility.o display.o reducer.o mapper.o main.o -o imperious.so

installing to /usr/local/lib/R/site-library/Rhipe/libs

** R

** inst

** preparing package for lazy loading

** help

*** installing help indices

** building package indices ...

** testing if installed package can be loaded

Error in dyn.load(file, DLLpath = DLLpath, ...) : 

  unable to load shared library '/usr/local/lib/R/site-library/Rhipe/libs/Rhipe.so':

  libprotobuf.so.6: cannot open shared object file: No such file or directory

Hi,

Hmm,

1) what does 

pkg-config --libs protobuf 

say 

2) Also, run this

(a) tar zxvf Rhipe_0.60.tar.gz

(b) R CMD INSTALL Rhipe

(this should fail, as you have observed)

(c)  ldd Rhipe/src/Rhipe.so 

libprotobuf.so.6 => /ln/meraki/custom/lib/libprotobuf.so.6 (0x00002ab9506e2000)

libz.so.1 => /usr/lib64/libz.so.1 (0x00002ab9509fc000)

libR.so => /ln/meraki/custom/lib64/R/lib/libR.so (0x00002ab950c10000)

        .....

See that Rhipe.so is linked to the correct libprotobuf.so.6 (usually libprotobuf.so.6 is soft linked to the probotuf lib with the version).

In your case, the protobuf library might not be in the library load path.

Consider exporting LD_LIBRARY_PATH to include the path to where your protobuf was installed

[SH.Chou]

Hi Saptarshi:

    Thanks.

You are totally correct. 

When I type pkg-config --libs protobuf , it shows:

-pthread -L/usr/local/lib -lprotobuf -lz 

and ldd Rhipe/src/Rhipe.so, shows:

--------------------------

linux-gate.so.1 =>  (0x006de000)

libprotobuf.so.6 => not found

libz.so.1 => /lib/libz.so.1 (0x00c97000)

libR.so => /usr/lib/libR.so (0x0037a000)

libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00d75000)

libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x00852000)

libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00110000)

libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0x0012f000)

libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x0018e000)

libblas.so.3gf => /usr/lib/libblas.so.3gf (0x0073f000)

libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00944000)

libreadline.so.6 => /lib/libreadline.so.6 (0x002e8000)

libpcre.so.3 => /lib/libpcre.so.3 (0x0087c000)

libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x00148000)

libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x00b13000)

/lib/ld-linux.so.2 (0x00927000)

libncurses.so.5 => /lib/libncurses.so.5 (0x0031c000)

----------------------------

May I know where I can modify LD_LIBRARY_PATH??

(sorry, I'm new in using Linux) 

S.H.

=============================================================================

[Saptarshi Guha]

Hi,

Sure. There are 3 approaches

1) Install protobuf into /usr/lib and /usr/include i.e. run ./configure --prefix=/usr && make && make install

    I do not prefer this since protobuf is not system specific, rather an add on(like an application) and really should go to /usr/local

2) In the bash profile oyu need 

EXPORT LD_LIBRARY_PATH=/to/path1:/to/path2:/where/to/path3

but this needs to be done for every user or (assuming you're sysadmin) you can just modify /etc/profile (again this works if everyone uses bash)

See http://www.network-theory.co.uk/docs/gccintro/gccintro_25.html , specifically

"It is possible for the system administrator to set the LD_LIBRARY_PATH variable for all users, by adding it to a default login script, such as ‘/etc/profile’. "

3) Modify or place an entry in /etc/ld.so.conf.d/ or /etc/ld.so.conf (to include /usr/local/lib  - i.e the place where protobuf installs its libs)

And then update the cache, by ldconfig

(see man ld.so and man ldconfig)

Finally, to confirm all is indeed okay, do the ldd once again.

Hope it helps

Saptarshi

[SH.Chou]

Hi Saptarshi:

     I've add "include /usr/local/lib" in /etc/ld.so.conf. And I tried ldd Rhipe/src/Rhipe.so again. It shows:

-----

hadoop@cls_master:~/Downloads$ ldd Rhipe/src/Rhipe.so 

linux-gate.so.1 =>  (0x00c4e000)

libprotobuf.so.6 => /usr/local/lib/libprotobuf.so.6 (0x00d76000)

libz.so.1 => /lib/libz.so.1 (0x00229000)

libR.so => /usr/lib/libR.so (0x003b5000)

libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x009d6000)

libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x00d2b000)

libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00ec5000)

libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0x00110000)

libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x0023e000)

libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00129000)

libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x007c9000)

libreadline.so.6 => /lib/libreadline.so.6 (0x001a6000)

libpcre.so.3 => /lib/libpcre.so.3 (0x001da000)

libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x0020b000)

libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x0021d000)

/lib/ld-linux.so.2 (0x00398000)

libncurses.so.5 => /lib/libncurses.so.5 (0x0071d000)

--------

But when I install Rhipe again, it still shows some errors as below:

--------

hadoop@cls_master:~/Downloads$ sudo R CMD INSTALL Rhipe_0.60.tar.gz 

* installing to library ‘/usr/local/lib/R/site-library’

* installing *source* package ‘Rhipe’ ...

checking for gcc... gcc -std=gnu99

checking for C compiler default output file name... rm: cannot remove `a.out.dSYM': Is a directory

a.out

checking whether the C compiler works... yes

checking whether we are cross compiling... no

checking for suffix of executables... 

checking for suffix of object files... o

checking whether we are using the GNU C compiler... yes

checking whether gcc -std=gnu99 accepts -g... yes

checking for gcc -std=gnu99 option to accept ISO C89... none needed

configure: creating ./config.status

config.status: creating src/Makevars

** libs

g++   -I. -g  -DHAVE_UINTPTR_T  `pkg-config --cflags protobuf` -Wall `pkg-config --libs protobuf`  `R CMD config --cppflags` -fpic `R CMD config --ldflags` rexp.pb.o message.o utility.o display.o reducer.o mapper.o main.o -o imperious.so

installing to /usr/local/lib/R/site-library/Rhipe/libs

** R

** inst

** preparing package for lazy loading

** help

*** installing help indices

** building package indices ...

** testing if installed package can be loaded

Rhipe lnitializing

Error : .onLoad failed in loadNamespace() for 'Rhipe', details:

  call: fun(...)

  error: Rhipe requires the HADOOP environment variable to be present

ERROR: loading failed

* removing ‘/usr/local/lib/R/site-library/Rhipe’

------------

It shows I have to setup HADOOP environment variable. May I know where I can setup?? I'm using Clouldera's hadoop.

thanks so much for your help!!!!!

S.H.

[Saptarshi Guha]

You're almost there. You need to set the environment variable HADOOP to the folder containing the bin folder that contains the hadoop shell file. 

For example if hadoop is installed in /a/hadoop-2/ ( so that /a/hadoop-2/bin contains the hadoop shell file) then set HADOOP to /a/hadoop-2/ . On cloudera I /think/you can set HADOOP to / . 

Do this in your bash profile or .bashrc .

You can also do

HADOOP=path R CMD INSTALL ....

To install. But library(Rhipe) will fail, for that set the environment variables ( which you could do with Sys.setenv)

Saptarshi

[SH.Chou]

Thanks. 

I still can not get it right. 

I added HADOOP=/usr/lib/hadoop into ~/.bashrc, and then install Rhipe again. But it still argues Rhipe requires the HADOOP environment variable to be present.

S.H.

[Saptarshi Guha]

But did you reload .bashrc ? Confirm that indeed HADOOP is present.

Before R CMD INSTALL

$HADOOP

should be /usr/lib/hadoop

and you need export HADOOP=/usr/lib/hadoop , without the export it won't be visible to other processes.

Saptarshi

[SH.Chou]

Hi Saptarshi:

    I still have problem with that. So I record you a mp4 to demonstrate the step I have done and error message I got. 

please see http://www.youtube.com/watch?v=zphpk59aX4E

Thanks.

S.H. 

[Saptarshi Guha]

Pretty neat  to provide a youtube video (what software did you use?)

Not sure why this is happening

1) In .bashrc , change HADOOP=/usr ... with export HADOOP=/usr...

    Then, in terminal, type 

[Saptarshi Guha]

Sorry, got sent by accident.

Pretty neat  to provide a youtube video (what software did you use?)

Not sure why this is happening

1) In .bashrc , change HADOOP=/usr ... with export HADOOP=/usr...

    Then, in terminal, type 

    . ~/.bashrc   #note the leading .

Also, now check

2) in terminal, type echo $HADOOP

3) start R, type Sys.getenv("HADOOP")

If all this works, then type R CMD INSTALL Rhipe....

[SH.Chou]

I recorded you another one. Still it is not working. 

p.s. I'm using Camtasia

http://www.youtube.com/watch?v=KL8gmOgX02Y

[Saptarshi Guha]

You setting the environment variable as yourself, but installing RHIPE as root.

The new variable is not transferred over.

E.g.

echo 'print(Sys.getenv("HADOOP"))' > a.r

R CMD BATCH a.r

cat a.r.Rout

sudo R CMD BATCH a.r

cat a.r.Rout

note the difference.

So, what you can do

a) modify the /etc/bashrc and see that everyones bashrc sources this.

or

(b) su root and do the rest

Not sure which is better.

Regards

Saptarshi

[SH.Chou]

It works!!!!!

I changed the owner and group of /usr/local/lib/R to hadoop:hadoop. 

Then everything works!!!

Thank-You very much!!!!!!!!!

S.H.

[Saptarshi Guha]

Good to hear

Saptarshi