[Rocks-Discuss] Running OpenMPI jobs
Kennie Cruz
/usr/bin/mpirun -np 2 -machinefile mynodes /opt/hpl/openmpi-hpl/bin/xhpl
/opt/hpl/openmpi-hpl/bin/xhpl: error while loading shared libraries:
liborte.so.0: cannot open shared object file: No such file or directory
/opt/hpl/openmpi-hpl/bin/xhpl: error while loading shared libraries:
liborte.so.0: cannot open shared object file: No such file or directory
--
Kennie J. Cruz Gutierrez
kjc...@gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20080206/6a4884ba/attachment.html
Tim Carlson
I'll just add a "me too" for Rocks 4.2.1 and 4.3 x86_64 based on RHEL 4.6
Tim
Greg Bruno
> On Wed, 6 Feb 2008, Kennie Cruz wrote:
>
> I'll just add a "me too" for Rocks 4.2.1 and 4.3 x86_64 based on RHEL 4.6
tim and kennie,
on your frontend and one compute node, what is the output of:
# rpm -qi openmpi
and, on the frontend, please send us the output of:
# rocks list roll
i just ran xhpl on an x86_64 rocks 4.3 system built with the OS rolls
and it ran successfully.
- gb
Scott Beardsley
> On Wed, 6 Feb 2008, Kennie Cruz wrote:
>
> I'll just add a "me too" for Rocks 4.2.1 and 4.3 x86_64 based on RHEL 4.6
Ahhh this looks like a problem with moving to 4.6. I have a Rocks 4.3
system based on CentOS 4.5 where it works:
$ ldd /opt/hpl/openmpi-hpl/bin/xhpl
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00000035d7200000)
libmpi.so.0 => /usr/lib64/openmpi/libmpi.so.0 (0x00000035d7600000)
liborte.so.0 => /usr/lib64/openmpi/liborte.so.0
(0x00000035d7400000)
libopal.so.0 => /usr/lib64/openmpi/libopal.so.0
(0x00000035d6900000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000035d6e00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00000035dac00000)
libutil.so.1 => /lib64/libutil.so.1 (0x00000035d7a00000)
libm.so.6 => /lib64/tls/libm.so.6 (0x00000035d7000000)
libc.so.6 => /lib64/tls/libc.so.6 (0x00000035d6b00000)
/lib64/ld-linux-x86-64.so.2 (0x00000035d6700000)
But on a Rocks 4.3 system based on CentOS 4.6 it is broken:
$ ldd /opt/hpl/openmpi-hpl/bin/xhpl
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00000035cb800000)
libmpi.so.0 => /usr/lib64/openmpi/1.2.3-gcc/libmpi.so.0
(0x00000035cbc00000)
liborte.so.0 => not found
libopal.so.0 => not found
libdl.so.2 => /lib64/libdl.so.2 (0x00000035cb600000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00000035cce00000)
libutil.so.1 => /lib64/libutil.so.1 (0x00000035cf800000)
libm.so.6 => /lib64/tls/libm.so.6 (0x00000035cb400000)
libc.so.6 => /lib64/tls/libc.so.6 (0x00000035cb100000)
/lib64/ld-linux-x86-64.so.2 (0x00000035cad00000)
libopen-rte.so.0 =>
/usr/lib64/openmpi/1.2.3-gcc/libopen-rte.so.0 (0x00000035cba00000)
libopen-pal.so.0 =>
/usr/lib64/openmpi/1.2.3-gcc/libopen-pal.so.0 (0x00000035caf00000)
Even though openmpi-libs is installed. It looks like openmpi changed
their interface. liborte is no longer in openmpi-libs (v 1.2.3):
rpm -ql openmpi-libs|grep /usr/lib64/openmpi
/usr/lib64/openmpi/1.2.3-gcc
/usr/lib64/openmpi/1.2.3-gcc/libmca_common_sm.so.0
/usr/lib64/openmpi/1.2.3-gcc/libmca_common_sm.so.0.0.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi.so.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi.so.0.0.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi_cxx.so.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi_cxx.so.0.0.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi_f77.so.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi_f77.so.0.0.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi_f90.so.0
/usr/lib64/openmpi/1.2.3-gcc/libmpi_f90.so.0.0.0
/usr/lib64/openmpi/1.2.3-gcc/libopen-pal.so.0
/usr/lib64/openmpi/1.2.3-gcc/libopen-pal.so.0.0.0
/usr/lib64/openmpi/1.2.3-gcc/libopen-rte.so.0
/usr/lib64/openmpi/1.2.3-gcc/libopen-rte.so.0.0.0
/usr/lib64/openmpi/1.2.3-gcc/openmpi
/usr/lib64/openmpi/1.2.3-gcc/openmpi.ld.conf
/usr/lib64/openmpi/1.2.3-gcc/openmpi/libompitv.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_allocator_basic.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_allocator_bucket.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_bml_r2.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_btl_openib.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_btl_self.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_btl_sm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_btl_tcp.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_btl_udapl.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_coll_basic.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_coll_self.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_coll_sm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_coll_tuned.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_errmgr_hnp.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_errmgr_orted.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_errmgr_proxy.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_gpr_null.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_gpr_proxy.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_gpr_replica.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_io_romio.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_iof_proxy.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_iof_svc.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_maffinity_first_use.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_maffinity_libnuma.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_mpool_rdma.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_mpool_sm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_ns_proxy.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_ns_replica.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_odls_default.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_oob_tcp.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_osc_pt2pt.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_paffinity_linux.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_pls_gridengine.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_pls_proxy.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_pls_rsh.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_pls_slurm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_pml_cm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_pml_ob1.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_ras_dash_host.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_ras_gridengine.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_ras_localhost.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_ras_slurm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rcache_vma.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rds_hostfile.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rds_proxy.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rds_resfile.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rmaps_round_robin.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rmgr_proxy.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rmgr_urm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_rml_oob.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_sds_env.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_sds_pipe.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_sds_seed.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_sds_singleton.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_sds_slurm.so
/usr/lib64/openmpi/1.2.3-gcc/openmpi/mca_topo_unity.so
It look slike hpl needs to be rebuilt to work with CentOS 4.6.
Scott
Tim Carlson
As was pointed out this appears to be an issue with using RHEL/Centos 4.6
as the base. That kind of makes sense to me now because when I have been
patching these systems based on 4.x where x<6 I have had to remove the
xhpl rpm due to incompatibilites. Looks like this time the rpm was just
forced into place and the install went merrily along :)
# rpm -qi openmpi
Name : openmpi Relocations: (not relocatable)
Version : 1.2.3 Vendor: Red Hat, Inc.
Release : 1 Build Date: Tue 07 Aug 2007
02:21:06 PM PDT
Install Date: Mon 21 Jan 2008 01:43:44 AM PST Build Host:
hs20-bc2-2.build.redhat.com
Group : Development/Libraries Source RPM:
openmpi-1.2.3-1.src.rpm
Size : 2206331 License: BSD
Signature : DSA/SHA1, Wed 17 Oct 2007 10:56:45 PM PDT, Key ID
219180cddb42a60e
Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL : http://www.open-mpi.org/
Summary : Open Message Passing Interface
Description :
Open MPI is an open source, freely available implementation of both the
MPI-1 and MPI-2 standards, combining technologies and resources from
several other projects (FT-MPI, LA-MPI, LAM/MPI, and PACX-MPI) in
order to build the best MPI library available. A completely new MPI-2
compliant implementation, Open MPI offers advantages for system and
software vendors, application developers, and computer science
researchers. For more information, see http://www.open-mpi.org/ .
# dbreport rolls
Red_Hat_Enterprise_Linux_4 4.3 x86_64 enabled
base 4.3 x86_64 enabled
ganglia 4.3 x86_64 enabled
hpc 4.3 x86_64 enabled
kernel 4.3 x86_64 enabled
web-server 4.3 x86_64 enabled
Tim Carlson
Voice: (509) 371 6435 (new 1/12/08)
Email: Tim.C...@pnl.gov
Pacific Northwest National Laboratory
HPCaNS: High Performance Computing and Networking Services