ADDA Parallelization Scaling

[Claudia M]

Hello,

I am attempting a fairly large ADDA simulation (orientationally averaged spheroid with m=1.33+1.45E-87i, dpl=15, lambda=0.651, eq_rad=10 ). I tried it on a 20-core cluster and it took about 3 weeks and eventually crashed. I tried smaller particles (same index and wavelength but only eq_rad=4) and those worked fine and took a couple days. I want to try it on a much large computer (300 nodes, 7500 cores), but have to request how many resources I need. Do you know how ADDA scales with more computational power or have any guesses on how much I may need to request?

Thanks in advance,

Claudia Morello

[Maxim Yurkin]

Dear Claudia,

For your problem the computational performance will probably be determined by number of dipoles along the z-axis - nz
(you can find it in the log file). ADDA efficiently parallelizes computations when number of cores is not larger than
nz, since the particle is divided in slices over the z direction (minimum one slice per core), and similar partition is
used along the x-axis as well. If you use oblate spheroids with moderate eq_rad and dpl, nz can be small implying poor
parallelization potential for ADDA. Using larger number of cores can even lead to crashes, but ADDA should produce
warnings during such runs. Unfortunately, ADDA parallelization focus is on large particles rather than a large number of
independent runs (as required for orientation averaging). The latter is actually much easier, it is just not yet
implemented in ADDA. See Section 6.6 "Partition over processors in parallel mode" of the manual for more details - it is
the users responsibility to choose the optimal number of cores.

Speaking of possible workarounds, there are a few options:
1) Rotate the particle so that its largest dimension will become along the z-axis, and smallest - along the y-axis. You
can not use -orient command line option for that (since the latter is somewhat virtual), but for spheroids it boils down
to changing the parameters of the built-in shapes. For more complicated built-in shapes (e.g., those that are
necessarily symmetric over the z-axis) the transformation of the shape file can be done outside of ADDA.

2) Do not use the orientation averaging inside ADDA, but rather by internal script. Then each fixed-orientation
simulation (for such moderate sizes) can be done on a single core, and any number of them can be run in parallel. This
can be done by a python script and combined with some advanced cubatures - this has been discussed previously in this
discussion group. In principle, an intermediate regime is also possible - each ADDA run averages over a small set of
orientations, and many such independent (parallel) runs are further combined into the final result.

Bye,
Maxim.

[Claudia M]

Hello,

Thanks so much for your quick response! That is all good to know and would explain a couple problems I ran into. I will try orientation averaging outside of ADDA!

Thanks,

Claudia Morello