Trouble with libopen-rte.so.4

[Jana]

Dear all, 

I have managed to compile cp2k-2.3 at a supercomputer using their mklroot, mpi and my fftw and libint but there is a problem with run while using different nodes probably due to the old version of mpi available...so i was sent another version compiled by another user at the same supercomputer but the user didn't use anything default, all libs his own.

I got his binary and dynamically linked libs via cp -a so with links, but I keep getting

$ ldd ./bin/mpiexec
        linux-vdso.so.1 =>  (0x00007fff12a74000)
        libopen-rte.so.4 => not found

or 

$ ldd /storage/home/his-username/bin/mpiexec
        linux-vdso.so.1 =>  (0x00007fff2cdff000)
        libopen-rte.so.4 => not found

while he has at the same time

$ ldd /home/his-username/storage/bin/mpiexec
        linux-vdso.so.1 =>  (0x00007fff57d33000)
        libopen-rte.so.4 => /home/his-username/CP2K/lib/libopen-rte.so.4 (0x00007fdb28ffd000)

I changed my script for PBS so that I added the red line

...
echo 'export LD_LIBRARY_PATH=/storage/home/$USER/lib' >> $go
echo 'export PATH=/storage/home/$USER/bin:'$PATH >> $go
echo 'echo "env:"; env' >> $go
echo 'echo "nodes:"; cat $PBS_NODEFILE; echo libs:; ls /storage/home/$USER/lib; echo bin:; ls /storage/home/$USER/bin' >> $go
echo 'export LD_LIBRARY_PATH=/storage/home/$USER/lib/libopen-rte.so.4' >> $go
echo /storage/home/$USER/bin/'mpiexec --hostfile $PBS_NODEFILE -x LD_LIBRARY_PATH='$LD_LIBRARY_PATH' -x PATH='$PATH' cp2k.popt '$base'.in > '$base'.log 2>&1' >> $go
...

but in the logfile (using input for h2o from regtests) I keep getting

/storage/home/mathausj/bin/mpiexec: error while loading shared libraries: libopen-rte.so.4: cannot open shared object file: No such file or directory

though

$ ls lib

total 15936

-rwxr-xr-x 1 mathausj meta    20323 2013-03-14 09:59 libmpi_f90.so.3

-rwxr-xr-x 1 mathausj meta  2002819 2013-03-14 09:59 libmpi_f77.so.1

-rwxr-xr-x 1 mathausj meta 10282011 2013-03-14 09:59 libmpi.so.1

-rwxr-xr-x 1 mathausj meta  4006084 2013-03-14 16:52 libopen-rte.so.4

I am sorry to bother you with this problem, but I am really clueless... Have emailed helpdesk and the user who compiled it but none of them has solution to this problem, if executed from that user's account, cp2k works perfectly...

[Ari Paavo Seitsonen]

Dear Jana(?),

  I don't know whether this helps, but a few notes:

 - With the second "export LD_LIBRARY_PATH..." you overwrite the first assigment; is this what you wanted? Usually one uses

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:<new_directory> (or prepending the <new_directory>)

 - The value of the second assignment points to a dynamic library directly, but it should point to a _directory_

 - This problem comes directly from the 'mpiexec', not CP2K itself. It seems that the user has compiled the MPI oneself; hopefully this is useful on your machine: In general it is the system administrators who set up a consistent and working MPI environment

    Greetings,

       apsi

2013/3/17 Jana <jahodov...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
To post to this group, send email to cp...@googlegroups.com.
Visit this group at http://groups.google.com/group/cp2k?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-
  Ari P Seitsonen / Ari.P.S...@iki.fi / http://www.iki.fi/~apsi/
  Physikalisch-Chemisches Institut der Universität Zürich
  Tel: +41 44 63 55 44 97  /  Mobile: +41 79 71 90 935

[Frank Uhlig]

Hello Jana,

I agree with Ari (who to my disappointment was faster than me ;-),

export LD_LIBRARY_PATH=/storage/home/$USER/lib:$LD_LIBRARY_PATH

should be the way to go.

Some other things:

Can you tell me what is done with the variable "go" later in the PBS script? In the lines you provided, you never actually start cp2k.

Furthermore, be careful with your apostrophes. According to the apostrophes you used in this line:

echo /storage/home/$USER/bin/'mpiexec --hostfile $PBS_NODEFILE -x LD_LIBRARY_PATH='$LD_LIBRARY_PATH' -x PATH='$PATH' cp2k.popt '$base'.in > '$base'.log 2>&1' >> $go

its content will be interpreted in different ways. In this sense the line is "split" into the following parts:

echo
/storage/home/$USER/bin/
'mpiexec --hostfile $PBS_NODEFILE -x LD_LIBRARY_PATH='
$LD_LIBRARY_PATH
' -x PATH='
$PATH
' cp2k.popt '
$base
'.in > '
$base
'.log 2>&1'
 >> $go

It will still be interpreted as one line by echo, but any part in apostrophes will be written to $go "as is" (e.g. $LD_LIBRARY_PATH). The variables will not be substituted by their content. On the other hand, everything that is not in apostrophes will be substituted in terms of the contents of the associated variables, e.g., $PATH in the above line will be substituted by its contents. In general, it is advisable to use quotation marks for accessing contents of these variables. Depending on how you treat "$go" that might not matter at all.

It is sufficient to specify -x VARIABLE. You do not need to specify -x VARIABLE=.... that only makes thing messier.

Btw, it looks like the supercomputer you are referring to is the Czech metacentrum. If that is correct, then please acknowledge the following. The version of OpenMPI they use is a reasonable choice (v1.4.3). It is the prior stable version of OpenMPI and should give no trouble at all.
I am a user at the metacentrum as well and have a working cp2k version installed with their OpenMPI version and MKL. FFTW3 and libint built separately and linked statically to the cp2k executable.
Feel free to check out the contents (arch file used was Linux-x86-64-gfortran.popt) here:

/storage/brno1/home/fuhlig/prog/cp2k-2.3

and a simple, but working test under:

/storage/brno1/home/fuhlig/test_cp2k

Best,

Frank

[Jana]

Dear Ari,

thanks a lot for your response 

this is what I have actually tried before but I got a strange output in my logfile (while stderr was empty)

[minos32-1.zcu.cz:06096] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file runtime/orte_init.c at line 79

[minos32-1.zcu.cz:06096] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file orterun.c at line 694

so I searched thru the documentation for another version of openmpi in /software (as I don't have permissions to any other directories of that user than bin and lib and yes, he used his own openmpi) and I got 

[opal_init:startup:internal-failure]
It looks like opal_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure.

So I am confused because from that user's account everything works fine so I didn't understand how could this be mpi configuration error.

Dne neděle, 17. března 2013 14:17:07 UTC+1 Ari Paavo Seitsonen napsal(a):

[Jana]

Hi,

I am really ruined by hearing that someone at metacentrum experienced no difficulties while compiling and running cp2k. 

I spent 4 full weeks in January trying to manage it, finally I (thought I) succeeded using intel compiler and advices from this forum

INTEL_MKL = $MKLROOT
INTEL_INC = $(INTEL_MKL)/include/fftw
INTEL_LIB = $(INTEL_MKL)/lib/intel64
CC       = cc
CPP      =
FC       = mpiifort
LD       = mpiifort
AR       = ar -r
DFLAGS   = -D__INTEL -D__FFTSG -D__parallel -D__BLACS -D__SCALAPACK -D__FFTW3 -D__LIBINT
CPPFLAGS =
FCFLAGS  = $(DFLAGS) -I$(INTEL_INC) -O3 -xW -heap-arrays 64 -funroll-loops -fpp -free
FCFLAGS2 = $(DFLAGS) -I$(INTEL_INC) -O1 -xW -heap-arrays 64 -fpp -free
LDFLAGS  = $(FCFLAGS) -I$(INTEL_INC)
LIBS     = -L$(INTEL_LIB) -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -lpthread -lm -L/home/mathausj/fftw3xf -lfftw3xf_intel -L/home/mathausj/libint/lib -lderiv -lint -lstdc++




OBJECTS_ARCHITECTURE = machine_intel.o

graphcon.o: graphcon.F
        $(FC) -c $(FCFLAGS2) $<

 but during regtests and later I discovered occasional errors in run and I haven't really managed to get cp2k to run properly. 

So that's why I asked the other user for help but what he did was compiling it himself and letting me cp bin and lib folders so here I am not really having any idea about the version of openmpi he used or anything else regarding his procedure (he just said the openmpi in /software is old and didn't work)

Thanks a lot for your advices and folder access, tho you used gfortran which I tried and didn't succeed to compile cp2k, but I guess trying to compile the programme all over again is the only way now, even if I still don't get how is that possible that from that user's account it just works.

Regarding $go - I use it to redirect there PBS commandst and in the end I do 

qsub -q short -l walltime=1:00:00 -l nodes=2:ppn=2 -l mem=2gb ./$go

splitting of the last echo actually gives me a bunch of errors

[Ari Paavo Seitsonen]

Dear Jana,

  I'm sorry this depresses you, but at the beginning of this week I was in Nove Hrady, we managed to compile CP2K in about 15 minutes at the metacentre and it looked as if people at the nano-CP2K-tutorial were also able to run the code in the batch system. Probably some one who was present can help you - maybe Babak Minofar (working in Nove Hrady) or so can forward you to the people who started running the code in the batch system. It could be that some regtests will fail, but at least the first test runs they made were successful.

    Greetings,

       apsi

2013/3/17 Jana <jahodov...@gmail.com>

Hi,

--

You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
To post to this group, send email to cp...@googlegroups.com.
Visit this group at http://groups.google.com/group/cp2k?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jana]

Dear Frank, 

could I just simply ask what modules you had loaded while compiling cp2k with that gfortran.popt arch file?

[Frank Uhlig]

Hello Jana,

openmpi and mkl-11.0 which was the recommendation from the support staff at metacentrum.

Cheers

On Sun, Mar 17, 2013 at 4:41 PM, Jana <jahodov...@gmail.com> wrote:

Dear Frank, 

could I just simply ask what modules you had loaded while compiling cp2k with that gfortran.popt arch file?

--

[Frank Uhlig]

Hi again,

one thing that just came to mind, try using /storage/NEARCITY/home instead of /storage/home.  I was told /storage/home is deprecated.

Good luck,

Frank

Dear Frank, 

could I just simply ask what modules you had loaded while compiling cp2k with that gfortran.popt arch file?

--

[Jana]

It's seriously amazing how two people do the same thing and the result is totally different.

[tarkil.cesnet.cz:24339] *** An error occurred in MPI_Comm_set_errhandler

[tarkil.cesnet.cz:24339] *** on communicator MPI_COMM_WORLD

[tarkil.cesnet.cz:24339] *** MPI_ERR_COMM: invalid communicator

[tarkil.cesnet.cz:24339] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)

(got the exe, not working)

I actually e-mailed the helpdesk too, before my first try, and they recommended mpich2-intel + intelcdk-11 + intelcdk-12, so that's why I was asking.

Thanks for your help anyway, I guess I am hopeless then so now I am even more clueless than ever before.