[Rocks-Discuss] HPCC 1.4.1 Benchmark on Rocks 5.4.2
Gowtham
Here are our notes:
http://sgowtham.net/blog/2012/02/28/hpcc-1-4-1-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/
As always, any thoughts to make it better
will be greatly appreciated.
Best,
g
--
Gowtham
Information Technology Services
Michigan Technological University
Mario Benitez
Hi Gowtham,
I'm trying to compile hpcc and I get the following error:
----
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(init.o): In function `MPI_Init':
init.c:(.text+0x34): undefined reference to `MPL_env2str'
init.c:(.text+0x54): undefined reference to `MPL_env2bool'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(initthread.o): In function `MPI_Init_thread':
initthread.c:(.text+0x4a6): undefined reference to `MPL_env2bool'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params':
param_vals.c:(.text+0x16): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x33): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x50): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x6d): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x8a): undefined reference to `MPL_env2int'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o):param_vals.c:(.text+0xa7): more undefined references to `MPL_env2int' follow
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params':
param_vals.c:(.text+0x3d3): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x3f0): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x40d): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x42a): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x447): undefined reference to `MPL_env2bool'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o):param_vals.c:(.text+0x464): more undefined references to `MPL_env2bool' follow
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params':
param_vals.c:(.text+0x52f): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x54c): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x569): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x586): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x5a3): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x5c0): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x5dd): undefined reference to `MPL_env2bool'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o):param_vals.c:(.text+0x5fa): more undefined references to `MPL_env2bool' follow
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params':
param_vals.c:(.text+0x617): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x634): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x651): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x66e): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x68b): undefined reference to `MPL_env2str'
param_vals.c:(.text+0x6a8): undefined reference to `MPL_env2str'
param_vals.c:(.text+0x6c5): undefined reference to `MPL_env2str'
param_vals.c:(.text+0x6e2): undefined reference to `MPL_env2str'
param_vals.c:(.text+0x6ff): undefined reference to `MPL_env2str'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o):param_vals.c:(.text+0x71c): more undefined references to `MPL_env2str' follow
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params':
param_vals.c:(.text+0x739): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x756): undefined reference to `MPL_env2int'
param_vals.c:(.text+0x773): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x790): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x7ad): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x7ca): undefined reference to `MPL_env2bool'
param_vals.c:(.text+0x7e7): undefined reference to `MPL_env2bool'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o):param_vals.c:(.text+0x804): more undefined references to `MPL_env2bool' follow
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params':
param_vals.c:(.text+0x863): undefined reference to `MPL_env2range'
param_vals.c:(.text+0x888): undefined reference to `MPL_env2range'
param_vals.c:(.text+0x8ad): undefined reference to `MPL_env2range'
param_vals.c:(.text+0x8d2): undefined reference to `MPL_env2range'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(mpid_init.o): In function `MPID_Init':
mpid_init.c:(.text+0x162): undefined reference to `MPL_env2bool'
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(mpid_vc.o): In function `MPIDI_Populate_vc_node_ids':
mpid_vc.c:(.text+0x2b9): undefined reference to `MPL_env2int'
mpid_vc.c:(.text+0x2cc): undefined reference to `MPL_env2int'
collect2: ld returned 1 exit status
make[1]: *** [../../../../hpcc] Error 1
make[1]: Leaving directory `/share/apps/hpcc/1.4.1/hpl/lib/arch/build'
make: *** [all] Error 2
Step #1: Update /root/.bashrc
Add the following lines to /root/.bashrc and remember to source it
# HPCC (1.4.1) settings
export HPCC="/share/apps/hpcc/1.4.1/mpich2/1.4.1p1/gcc/4.1.2"
export PATH="${PATH}:${HPCC}/bin/MPICH2141p1_GCC412"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HPCC}/MPICH2141p1_GCC412"
----
Any hint to overcome this issue?. Thanx in advance.
Marinho.-
> Date: Tue, 28 Feb 2012 14:12:54 -0500
> From: g...@mtu.edu
> To: npaci-rocks...@sdsc.edu
> CC: ccw...@mtu.edu
> Subject: [Rocks-Discuss] HPCC 1.4.1 Benchmark on Rocks 5.4.2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120307/6e387d64/attachment.html
Gowtham
Hello Mario,
Can you please
1. send your Makefile?
2. let us know the output of
which mpif77
which mpif90
which mpicc
command?
Best,
g
--
Gowtham
Information Technology Services
Michigan Technological University
(906) 487/3593
http://www.it.mtu.edu/
Mario Benitez
Make.MPICH2141p1_GCC412
----
... (comments)
# Shell Details
SHELL = /bin/sh
CD = cd
CP = cp
LN_S = ln -s
MKDIR = mkdir
RM = /bin/rm -f
TOUCH = touch
#
# Platform Identifier
ARCH = MPICH2141p1_GCC412
#
# HPL Directory Structure / HPL library
TOPdir = ../../..
INCdir = $(TOPdir)/include
BINdir = $(TOPdir)/bin/$(ARCH)
LIBdir = $(TOPdir)/lib/$(ARCH)
HPLlib = $(LIBdir)/libhpl.a
#
# Message Passing library (MPI)
MPdir = /share/apps/mpich2/1.4.1p1/gcc/4.1.2
MPinc = -I$(MPdir)/include
MPlib = $(MPdir)/lib/libmpich.a
#
# Linear Algebra library (BLAS)
LAlib = /share/apps/gotoblas2/1.13/gcc/4.1.2/libgoto2.a
... (comments)
F2CDEFS = -DAdd_
#
# HPL Includes / Libraries / Specifics
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib)
#
# HPL Compile Time Options
# -DHPL_COPY_L force the copy of the panel L before bcast
# -DHPL_CALL_CBLAS call the cblas interface
# -DHPL_CALL_VSIPL call the vsip library
# -DHPL_DETAILED_TIMING enable detailed timers
#
# By default HPL will:
# *) not copy L before broadcast
# *) call the BLAS Fortran 77 interface
# *) not display detailed timing information
HPL_OPTS = -DHPL_COPY_L -DHPL_CALL_CBLAS -DHPL_DETAILED_TIMING
#
# HPL Definitions
HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# Compilers / linkers - Optimization Flags
CC = mpicc
CCNOOPT = $(HPL_DEFS)
CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall
#
# On some platforms, it is necessary to use the Fortran linker
# to find the Fortran internals used in the BLAS library
LINKER = mpif77
LINKFLAGS = $(CCFLAGS)
ARCHIVER = ar
ARFLAGS = r
RANLIB = echo
#
----
'which' output
----
[root@myhpccluster hpl]# which mpif77
/opt/openmpi/bin/mpif77
[root@myhpccluster hpl]# which mpif90
/opt/openmpi/bin/mpif90
[root@myhpccluster hpl]# which mpicc
/opt/openmpi/bin/mpicc
----
Everything seems normal. Thanx in advance.
Marinho.-
> Date: Wed, 7 Mar 2012 15:45:19 -0500
> From: g...@mtu.edu
> To: npaci-rocks...@sdsc.edu
> CC: mariob...@hotmail.com
> Subject: Re: [Rocks-Discuss] HPCC 1.4.1 Benchmark on Rocks 5.4.2
Gowtham
Mario,
The makefile looks ok; however, the output of 'which' commands
is not what is expected. Please read the steps posted in
article carefully.
'which mpicc' should yield
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/bin/mpicc
and so on.
Mario Benitez
I tried everything from scratch and I got the same error. My question now is, how or where the path por mpi* binaries should be set?
Is there any other way to compile HPL for Rocks?. Thanx in advance.
Marinho.-
> Date: Wed, 7 Mar 2012 16:39:59 -0500
> From: g...@mtu.edu
> To: npaci-rocks...@sdsc.edu
Gowtham
Mario,
The instructions for HPL 2.0 are here:
http://sgowtham.net/blog/2012/02/27/hpl-2-0-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/
This article was linked in HPCC 1.4.1 instructions post
http://sgowtham.net/blog/2012/02/28/hpcc-1-4-1-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/
Please read carefully and follow every step as indicated.
May be you can make a checklist and check things off as
you do them, so that you don't get errors during final
stages (of compilation, etc.)
To make sure that 'which mpif77', etc. return the
appropriate ones, please look into PATH variable -
this can be set at the command line OR in your
.bashrc file
Best,
g
--
Gowtham
Information Technology Services
Michigan Technological University
(906) 487/3593
http://www.it.mtu.edu/
Mario Benitez
Thanx a lot Gowtham, I'll take a look in a few hours.
Marinho.-
> Date: Sat, 10 Mar 2012 14:32:33 -0500
Mario Benitez
Hi there, following the Gowtham article to get HPCC running, while trying to compile GotoBLAS2 I got the following error:
----
../kernel/x86_64/gemm_ncopy_4.S: Assembler messages:
../kernel/x86_64/gemm_ncopy_4.S:175: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:176: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:177: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:178: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:180: Error: undefined symbol `WPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:328: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:329: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:331: Error: undefined symbol `WPREFETCHSIZE' in operation
make[1]: *** [sgemm_oncopy.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/share/apps/gotoblas2/1.13/gcc/4.1.2/kernel'
make: *** [libs] Error 1
----
I made some research and it seems is a problem related with newer processors or when processor is not identified (http://saintaardvarkthecarpeted.com/blog/archive/2011/05/Trouble_compiling_GotoBLAS2_on_newer_CPU.html). So looking at the content of cpuid.h and config.h from GotoBLAS2 source code basically I found defined values depending on the processor, now my question is if I got i7/8core processor and /proc/cpuinfo says that:
----
[root@hunab-ku apps]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
stepping : 7
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc arat pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips : 6820.42
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management: [8]
...
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
stepping : 7
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc arat pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips : 6820.08
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management: [8]
----
What I should add/change on these .h files?. Thanx in advance.
Marinho.-
> Date: Sat, 10 Mar 2012 14:32:33 -0500
James Rudd
I'm not sure if it will fix your problem but have you tried OpenBLAS
https://github.com/xianyi/OpenBLAS
GotoBLAS2 is no longer maintained and has been forked to OpenBLAS. I
believe they have added some support for newer processors in OpenBLAS.
>From Changelog:
* Auto-detect Intel Core i6/i7 (Sandy Bridge) CPU with Nehalem assembly
kernels
Regards,
James
James Rudd
http://jrudd.org/
---------------------
HPC Cluster Administrator
Centre of Excellence for Silicon Photovoltaics and Photonics
University of New South Wales
Sydney NSW 2052
AUSTRALIA
-------------- next part --------------
An HTML attachment was scrubbed...
Mario Benitez
Thanx a lot James,
I'll give it a try. I'll post later to write how it was.
Best regards,
Marinho.-
> From: james...@gmail.com
> Date: Wed, 14 Mar 2012 13:10:36 +1100
> To: npaci-rocks...@sdsc.edu
Mario Benitez
This was fixed using:
gmake clean
gmake TARGET=NEHALEM
So, now trying to get up HPL-2.0, I got this error:
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/lib/libmpich.a(mpid_vc.o): In function `MPIDI_Populate_vc_node_ids':
mpid_vc.c:(.text+0x2be): undefined reference to `MPL_env2int'
mpid_vc.c:(.text+0x2d0): undefined reference to `MPL_env2int'
collect2: ld returned 1 exit status
make[2]: *** [dexe.grd] Error 1
make[2]: Leaving directory `/share/apps/hpl/2.0/mpich2/1.4.1p1/gcc/4.1.2/testing/ptest/MPICH2141p1_GCC412'
make[1]: *** [build_tst] Error 2
make[1]: Leaving directory `/share/apps/hpl/2.0/mpich2/1.4.1p1/gcc/4.1.2'
make: *** [build] Error 2
Any to hint to get around it?. Thanx in advance.
Marinho.-
Mario Benitez
Ok, MPICH2+GotoBLAS2+HPCC installed.
I had to rename mpif77, mpif90 and mpicc at
/opt/openmpi/bin/
So 'which' command now finds the compiled files with the same name at
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/bin/
So, now how I run test to se my cluster performance???
Best regards,
Marinho.-
Gowtham
Hello Marinho,
I'm glad it worked but ...
renaming the executables in /opt/openmpi/bin/
is not a good idea
One can control the output of 'which' command
as follows. For e.g., when you typed
which mpif77
it looks in all the folders listed by the
PATH variable, in the order specified (from
left to right). You can check the output of
PATH variable by typing
echo $PATH
in a terminal.
All you have to do is redefine this PATH
variable so that
/share/apps/mpich2/1.4.1p1/gcc/4.1.2/bin
is listed before
/opt/openmpi/bin
Hope this helps.
To run a HPCC bechmark, you can still follow the
guidelines in the 'Running HPL Benchmark' section of
http://sgowtham.net/blog/2012/02/27/hpl-2-0-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/
to set up the hpccinf.txt (it's same as HPL.dat used
in HPL benchmark run).
Best,
g
--
Gowtham
Information Technology Services
Michigan Technological University
(906) 487/3593
http://www.it.mtu.edu/
Gustavo Correa
On Mar 14, 2012, at 5:14 PM, Mario Benitez wrote:
>
> Ok, MPICH2+GotoBLAS2+HPCC installed.
>
> I had to rename mpif77, mpif90 and mpicc at
>
> /opt/openmpi/bin/
>
> So 'which' command now finds the compiled files with the same name at
>
> /share/apps/mpich2/1.4.1p1/gcc/4.1.2/bin/
>
You could have set your PATH so that the latter takes precedence over /opt/openmpi/bin.
If you use csh/tcsh, at the bottom of your ${HOME}/.[t]cshrc file add this line:
setenv PATH /share/apps/mpich2/1.4.1p1/gcc/4.1.2/bin/:${PATH}
If you use sh/bash, at the bottom of your ${HOME}/.profile file add this line:
export PATH=/share/apps/mpich2/1.4.1p1/gcc/4.1.2/bin/:${PATH}
> So, now how I run test to se my cluster performance???
>
You could follow the next steps in Gowtham's recipe
to compile and run HPL.
Adjust your makefile according to your cluster.
http://sgowtham.net/blog/
Also of help are the HPL instructions, specially about HPL tuning:
http://www.netlib.org/benchmark/hpl/
Note also the recent message James Rudd sent to you about OpenBLAS,
as an alternative to GotoBLAS.
Although for starters any BLAS+Lapack should work.
I would suggest that you try first on a single node, then move to the cluster.
There is a calculator for memory use here:
http://hpl-calculator.sourceforge.net/
I hope this helps,
Gus Correa
Hung-Sheng Tsao (LaoTsao) Ph.D
there are two different mpi:
openmpi
mpich2
please donot mix two or rename two
-lt
Sent from my iPad
Mario Benitez
The fastest way was to rename /opt/openmpi :-)
Now, trying to figure it out how to run a consistent test in my cluster.
Marinho.-
> CC: npaci-rocks...@sdsc.edu
> From: lao...@gmail.com
> Date: Wed, 14 Mar 2012 19:12:29 -0400
> To: npaci-rocks...@sdsc.edu
> Subject: Re: [Rocks-Discuss] HPCC 1.4.1 Benchmark ... finally installed
Hung-Sheng Tsao (LaoTsao) Ph.D
Sent from my iPad
On Mar 14, 2012, at 19:56, Mario Benitez <mariob...@hotmail.com> wrote:
>
> The fastest way was to rename /opt/openmpi :-)
wrong way