Installing ADDA without sudo
60 views
Skip to first unread message
Liberman, Vladimir - 1006 - MITLL
Nov 23, 2021, 9:21:43 AM11/23/21
to adda-d...@googlegroups.com
I am trying to install ADDA on our large Ubuntu cluster at work. My sysadmin advises me the following
1. Install the software just in my own directory with a –user flag without using sudo apt.
2. “how new are these dev versions – gfortran libfftw3-dev and libopenmpi-dev – might they be in the versions of gcc and openmpi that we have in our module files on the system already?? – we have the following modules
“openmpi-4.0”
“Gnu Compiler
beta-gcc-5.2.0
beta-gcc-6.4.0
beta-openmpi-1.8.8-gcc520”
Any advice on how I should approach the installation process in this case?
Vlad Liberman
Maxim Yurkin
Nov 23, 2021, 11:29:54 AM11/23/21
to adda-d...@googlegroups.com
Dear Vlad,
The most robust option is to try to use
existing modules at the cluster. Ideally, you should enable the
modules (gcc, fftw3, and openmpi) and ADDA may compile out of the
box. The module versions seems new enough, however, you may have
problems with paths to headers and libraries. In that case, see
advices in:
Alternatively, you may try to install
Ubuntu packages with -user flag. This can be the fastest option,
but there can be some problems with adda_mpi due to conflicting
versions of openmpi (installed in user and system - you may need
to disable/hide the latter). Also, I am not sure if openmpi
package (that should interact between nodes) can be installed
without sudo.
Please let us know, which option (if
any) works for you,
Maxim.
23.11.2021 21:21, Liberman, Vladimir -
1006 - MITLL пишет:
--
You received this message because you are subscribed to the Google Groups "ADDA questions and answers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adda-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adda-discuss/SN5P110MB0896AD3ADD10B2B5CA2BC626E1609%40SN5P110MB0896.NAMP110.PROD.OUTLOOK.COM.
Liberman, Vladimir - 1006 - MITLL
Nov 23, 2021, 11:35:23 AM11/23/21
to adda-d...@googlegroups.com
So, to be clear when invoking the –user option, what should the install command look like?
Would I still be using “apt install” without “sudo” in front of it?
Vlad Liberman
To view this discussion on the web visit https://groups.google.com/d/msgid/adda-discuss/10a8b43d-2fcd-81bb-8423-efaa097cef62%40gmail.com.
Maxim Yurkin
Nov 23, 2021, 12:01:17 PM11/23/21
to adda-d...@googlegroups.com
Actually, I assumed that such flag is
available since your sysadmin mentioned it (and I haven't tried
anything like that on a cluster). But then I started googling and
it seems that there is no such option to either apt or apt-get.
Although some workarounds are mentioned to install packages for a
particular user, they are not that convenient. It's best to ask
your sysadmin, what exactly he meant by this flag.
Maxim.
23.11.2021 23:35, Liberman, Vladimir -
1006 - MITLL пишет:
To view this discussion on the web visit https://groups.google.com/d/msgid/adda-discuss/SN5P110MB08966A8D9388094C97AED646E1609%40SN5P110MB0896.NAMP110.PROD.OUTLOOK.COM.
Liberman, Vladimir - 1006 - MITLL
Nov 23, 2021, 4:02:24 PM11/23/21
to adda-d...@googlegroups.com
Our sysadmin suggested the following command for a local install
export DESTDIR=$HOME/adda_install && make all
When I did that, I ran into errors
_________________________________
Compiling OpenCL version of ADDA
make -C ocl
make[1]: Entering directory '/home/gridsan/VL16177/adda/src/ocl'
gcc -c -Ofast -w -std=c99 -DOPENCL -MD ../calculator.c
In file included from ../calculator.c:26:0:
../oclcore.h:30:11: fatal error: CL/cl.h: No such file or directory
# include <CL/cl.h>
^~~~~~~~~
compilation terminated.
../common.mk:75: recipe for target 'calculator.o' failed
make[1]: *** [calculator.o] Error 1
make[1]: Leaving directory '/home/gridsan/VL16177/adda/src/ocl'
Makefile:516: recipe for target 'ocl' failed
make: *** [ocl] Error 2
___________________________________
To view this discussion on the web visit https://groups.google.com/d/msgid/adda-discuss/cc0c2f2a-2a58-7132-0e7f-dffb65ddfe7b%40gmail.com.
Maxim Yurkin
Nov 23, 2021, 9:02:09 PM11/23/21
to adda-d...@googlegroups.com
If that is the first error, then it seems that your module environment is working fine. Basically, "make all" compiles
three versions of ADDA, and the last one (OpenCL one) you probably do not need. It is designed for running on a single
GPU - read https://github.com/adda-team/adda/wiki/OpenCL for details.
To compile two other ADDA versions (sequential and MPI) run "make seq mpi" instead. Actually, it was effectively done by
"make all" before producing an error. Note, however, that Makefile of ADDA ignores DESTDIR, so it performs only the
compilation, not the installation. You need to copy/move the produced executable yourself, if you need.
Maxim.
three versions of ADDA, and the last one (OpenCL one) you probably do not need. It is designed for running on a single
GPU - read https://github.com/adda-team/adda/wiki/OpenCL for details.
To compile two other ADDA versions (sequential and MPI) run "make seq mpi" instead. Actually, it was effectively done by
"make all" before producing an error. Note, however, that Makefile of ADDA ignores DESTDIR, so it performs only the
compilation, not the installation. You need to copy/move the produced executable yourself, if you need.
Maxim.
Liberman, Vladimir - 1006 - MITLL
Dec 4, 2021, 5:24:54 PM12/4/21
to adda-d...@googlegroups.com
OK, seem to be running adda_mpi on our cluster using openmpi4.0, but still a
few bugs.
First of all, why do I get warning messages from the job output when I start
the job such as
INFO: (ringID=275) No real dipoles are assigned
INFO: (ringID=277) No real dipoles are assigned
INFO: (ringID=279) No real dipoles are assigned
....
...
The number of these warning messages varies depending on how "robustly" the
job launches, does not seem to depend directly on the syntax of adda_mpi
command.
-----Original Message-----
From: adda-d...@googlegroups.com <adda-d...@googlegroups.com> On Behalf
Of Maxim Yurkin
Sent: Tuesday, November 23, 2021 21:02
To: adda-d...@googlegroups.com
Subject: Re: Installing ADDA without sudo
few bugs.
First of all, why do I get warning messages from the job output when I start
the job such as
INFO: (ringID=275) No real dipoles are assigned
INFO: (ringID=277) No real dipoles are assigned
INFO: (ringID=279) No real dipoles are assigned
....
...
The number of these warning messages varies depending on how "robustly" the
job launches, does not seem to depend directly on the syntax of adda_mpi
command.
-----Original Message-----
From: adda-d...@googlegroups.com <adda-d...@googlegroups.com> On Behalf
Of Maxim Yurkin
Sent: Tuesday, November 23, 2021 21:02
To: adda-d...@googlegroups.com
Subject: Re: Installing ADDA without sudo
--
You received this message because you are subscribed to the Google Groups
"ADDA questions and answers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to adda-discuss...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/adda-discuss/35a786c7-73dc-b451-75a4-88e07cbd99a4%40gmail.com.
You received this message because you are subscribed to the Google Groups
"ADDA questions and answers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to adda-discuss...@googlegroups.com.
To view this discussion on the web visit
Maxim Yurkin
Dec 4, 2021, 10:10:51 PM12/4/21
to adda-d...@googlegroups.com
Dear Vladimir,
This is legitimate messages (that's why they are labeled info) meaning that some MPI threads have no dipoles to deal
with. This is OK, but may indicate that you may decrease the number of MPI threads to use the hardware more efficiently.
The partitioning scheme used by ADDA is robust, but not the most optimal one. See Section 6.6 "Partition over processors
in parallel mode" of the manual.
This partition is deterministic, i.e. fully determined by the command line (unless you use some shape with randomly
placed voids - but I believe that is not possible with command line only). However, the messages may appear in different
places in the output (in some range) due to the parallel nature of computations. Synchronization points are placed only
when they are strictly necessary.
Let me know, if that explains your observations,
Maxim.
05.12.2021 5:24, Liberman, Vladimir - 1006 - MITLL пишет:
This is legitimate messages (that's why they are labeled info) meaning that some MPI threads have no dipoles to deal
with. This is OK, but may indicate that you may decrease the number of MPI threads to use the hardware more efficiently.
The partitioning scheme used by ADDA is robust, but not the most optimal one. See Section 6.6 "Partition over processors
in parallel mode" of the manual.
This partition is deterministic, i.e. fully determined by the command line (unless you use some shape with randomly
placed voids - but I believe that is not possible with command line only). However, the messages may appear in different
places in the output (in some range) due to the parallel nature of computations. Synchronization points are placed only
when they are strictly necessary.
Let me know, if that explains your observations,
Maxim.
05.12.2021 5:24, Liberman, Vladimir - 1006 - MITLL пишет:
Liberman, Vladimir - 1006 - MITLL
Dec 4, 2021, 10:28:32 PM12/4/21
to adda-d...@googlegroups.com
That explains it, perhaps partially.
For instance, if I have a fixed geometry and keep sending jobs to an increasingly higher number of processors, the number of warning messages increases, because more and more processors, presumably, are idle. However, in this case, the output of the code still looks good and the job completes successfully because in all cases, there is enough memory for the job.
However, here is what does not make sense. If I have a job that is very large and clearly requires more processor cores to fit into memory, I am not able to launch the adda_mpi job successfully on more cores
I need to study the situation more closely as per section 6.6, I guess, but just stumbling around, I have not been able to run a job successfully on more than ~300 cores.
What happens if I try to use more cores is that the program produces a very large number of such warnings and then crashes with various mpi cluster communication errors. I am guessing that it crashes because a lot of processors are not able to be utilized and so I no longer have enough memory for the simulation??
Staff Member, Active Optical Systems Group
MIT Lincoln Laboratory E-mail: vl...@ll.mit.edu
Room S3-357 Phone: 781 981 3706
Cell: 339 368 9811
244 Wood St.
Lexington, MA 02421-6426
To view this discussion on the web visit https://groups.google.com/d/msgid/adda-discuss/422f9d95-b5b7-8a20-7f95-d4c155880a1c%40gmail.com.
For instance, if I have a fixed geometry and keep sending jobs to an increasingly higher number of processors, the number of warning messages increases, because more and more processors, presumably, are idle. However, in this case, the output of the code still looks good and the job completes successfully because in all cases, there is enough memory for the job.
However, here is what does not make sense. If I have a job that is very large and clearly requires more processor cores to fit into memory, I am not able to launch the adda_mpi job successfully on more cores
I need to study the situation more closely as per section 6.6, I guess, but just stumbling around, I have not been able to run a job successfully on more than ~300 cores.
What happens if I try to use more cores is that the program produces a very large number of such warnings and then crashes with various mpi cluster communication errors. I am guessing that it crashes because a lot of processors are not able to be utilized and so I no longer have enough memory for the simulation??
Staff Member, Active Optical Systems Group
MIT Lincoln Laboratory E-mail: vl...@ll.mit.edu
Room S3-357 Phone: 781 981 3706
Cell: 339 368 9811
244 Wood St.
Lexington, MA 02421-6426
Maxim Yurkin
Dec 4, 2021, 11:27:11 PM12/4/21
to adda-d...@googlegroups.com
Yes, your explanation is almost correct. You might find some memory allocation errors in the output of larger runs, but
sometimes it is indeed only some MPI errors, which are hard to diagnose specifically. In the log (and stdout) there are
some values of "Maximum memory usage of single processor" that you may compare with that available on your hardware.
The main limitation of parallelization strategy is that it uses at least one slice (yz-plane with one dipole along
z-axis) per core (or zero, if no slices are left). Thus, using more cores than z-slices is a bad idea. However, if your
particle is oblate (z-dimension significantly smaller than along another axis), you may rotate the particle (in your
shape generation routine) to make the largest dimension along z. This way, you will be able to use more cores efficiently.
Next, concerning the memory limitation. Memory per core is largely determined by the size of a single slice (cannot be
smaller). However, available memory per core is memory per node / number of cores at this node. By default, one uses all
available cores, but if memory is a limitation, you may use lesser number (down to 1 core per node). Thus, you may
handle larger problems, but only slowly.
Maxim.
sometimes it is indeed only some MPI errors, which are hard to diagnose specifically. In the log (and stdout) there are
some values of "Maximum memory usage of single processor" that you may compare with that available on your hardware.
The main limitation of parallelization strategy is that it uses at least one slice (yz-plane with one dipole along
z-axis) per core (or zero, if no slices are left). Thus, using more cores than z-slices is a bad idea. However, if your
particle is oblate (z-dimension significantly smaller than along another axis), you may rotate the particle (in your
shape generation routine) to make the largest dimension along z. This way, you will be able to use more cores efficiently.
Next, concerning the memory limitation. Memory per core is largely determined by the size of a single slice (cannot be
smaller). However, available memory per core is memory per node / number of cores at this node. By default, one uses all
available cores, but if memory is a limitation, you may use lesser number (down to 1 core per node). Thus, you may
handle larger problems, but only slowly.
Maxim.
Reply all
Reply to author
Forward
0 new messages