tetmesh tallies in MCNP6-MPI

[Miguel Magán Romero]

I have been able to get DAGMC working with MCNP6.2 using OpenMPI in my local machine. However, I have noticed that, when using tet mesh tallies, the KD tree seems to be built for each node. See this output.

----------------------------------

     _/      _/        _/_/_/       _/      _/       _/_/_/         _/_/_/
    _/_/  _/_/      _/             _/_/    _/       _/    _/     _/
   _/  _/  _/      _/             _/  _/  _/       _/_/_/       _/_/_/
  _/      _/      _/             _/    _/_/       _/           _/    _/
 _/      _/        _/_/_/       _/      _/       _/             _/_/

  comment.  Physics models enabled.
 mode n H D T S A / Z P
  comment.  photonuclear physics may be needed (phys:p).
 comment. using random number generator  1, initial seed = 19073486328125
Set overlap thickness = 0

  comment.  total nubar used if fissionable isotopes are present.
  comment.  photon   importances have been set equal to 1.
  comment.  proton   importances have been set equal to 1.
  comment.  pi_plus  importances have been set equal to 1.
  comment.  pi_zero  importances have been set equal to 1.
  comment.  deuteron importances have been set equal to 1.
  comment.  triton   importances have been set equal to 1.
  comment.  helion   importances have been set equal to 1.
  comment.  alpha    importances have been set equal to 1.
  warning.  there are only neutron tallies in this problem.
  warning.   FM card uses material         2 cross sections over all of mesh tally        14
           0           0           0           1           1           2
DAGMC tally 14 has these 2 energy bin boundaries:
     0
     30
tot bin: no
Creating dagmc mesh tally14, input: mesh.h5m, output: mesh_out_n.h5m
  There are 180768 tetrahedrons in this tally mesh.
    Tally range has psize: 1
  Tally mesh has 386499 triangles.  Building KD tree of size 567267... done.

  warning.  use models for the following missing data tables:

 (Bunch of materials cut for briefty)

  warning.    1 materials had unnormalized fractions. print table 40.
 imcn   is done

 runtpe already exists.  runtpf is created instead.
 runtpf already exists.  runtpg is created instead.
 runtpg already exists.  runtph is created instead.
  warning.    8017.70c lacks gamma-ray production cross sections.
  warning.   30000.70c lacks gamma-ray production cross sections.
  warning.   38088.70c lacks gamma-ray production cross sections.
  warning.   44102.70c lacks gamma-ray production cross sections.
  warning.  material        2 has been set to a conductor.
  comment.  setting up hash-based fast table search for xsec tables

 ctm =        0.00   nrn =                 0
 dump    1 on file runtph   nps =           0   coll =                0
 xact   is done

 cp0 =   1.32
 master starting       3 MPI slave tasks with       1 threads each  06/22/22 11:37:31
 master broadcasting static commons...
 master broadcasting dynamic commons...
           0           0           0           1           1           2
DAGMC tally 14 has these 2 energy bin boundaries:
     0
     30
tot bin: no
Creating dagmc mesh tally14, input: mesh.h5m, output: mesh_out_n.h5m
           0           0           0           1           1           2
           0           0           0           1           1           2

DAGMC tally 14 has these 2 energy bin boundaries:
DAGMC tally 14 has these 2 energy bin boundaries:
     0
     30
tot bin: no
     0
     30
tot bin: no
Creating dagmc mesh tally14, input: mesh.h5m, output: mesh_out_n.h5m
Creating dagmc mesh tally14, input: mesh.h5m, output: mesh_out_n.h5m
  There are 180768 tetrahedrons in this tally mesh.
  There are 180768 tetrahedrons in this tally mesh.
    Tally range has psize: 1
    Tally range has psize: 1
  There are 180768 tetrahedrons in this tally mesh.
    Tally range has psize: 1
  Tally mesh has 386499 triangles.  Building KD tree of size 567267...   Tally mesh has 386499 triangles.  Building KD tree of size 567267...   Tally mesh has 386499 triangles.  Building KD tree of size 567267... done.

done.

done.

 master broadcasting cross section data...
 master sending DAGMC information...
Set overlap thickness = 0
Set numerical precision = 0.001
Loading file geom.h5m
Set overlap thickness = 0
Set numerical precision = 0.001
Set overlap thickness = 0
Set numerical precision = 0.001
Set overlap thickness = 0
Set numerical precision = 0.001
Loading file geom.h5m
Loading file geom.h5m
Loading file geom.h5m
Initializing the GeomQueryTool...
Using faceting tolerance: 0.0001
Building acceleration data structures...
Initializing the GeomQueryTool...
Initializing the GeomQueryTool...
Initializing the GeomQueryTool...
Using faceting tolerance: 0.0001
Using faceting tolerance: 0.0001
Building acceleration data structures...
Using faceting tolerance: 0.0001
Building acceleration data structures...
Building acceleration data structures...
Implicit Complement assumed to be Vacuum
Implicit Complement assumed to be Vacuum
Set overlap thickness = 0
 master completed initialization broadcasts.
 master set rendezvous nps =      500000,  work chunks =     3    06/22/22 11:38:41

======================================

Is this intended behavour? for 4 nodes, it is not terrible, but I am concerned about what happens if I run this in a 200 core cluster.

One thing I noticed is that I have not built MOAB with MPI.  I did not see anything of that in the instructions, but perhaps I need to?

Thanks

[Paul Wilson]

Hello Miguel,

 

It is true that the KD tree is generated on each node.  It must exist on each node anyway, so it’s a choice between generating it once while other nodes sit idle and sharing it vs generating it on each node.  Compiling MOAB with MPI won’t make a difference here.

 

Your follow on question gets at a slightly different issue, perhaps!  Since DAG-MCNP does not yet support OpenMP/shared memory parallel, some users rely on MPI for multi-core parallelization on a single node, or across multiple cores on each node.   A known issue with this is that it duplicates most of the memory structures, with one copy for each core, which is inefficient and may limit the number of cores that can be used on each node.

 

We have a long standing desire to validate the OpenMP/shared memory operation with reduced memory use, but haven’t managed to do that yet.

 

Paul

 

-- 
-- ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ --
Paul P.H. Wilson (he/him/his)
Grainger Professor of Nuclear Engineering
Chair, Department of Engineering Physics
o: 608-263-0807, c: 608-469-9615
paul....@wisc.edu
153 Engineering Research Bldg
1500 Engineering Dr, Madison, WI 53706
Zoom Meeting Room: https://uwmadison.zoom.us/j/6082630807
Zoom Phone Access: +1-929-205-6099, Access code: 6082630807

Computational Nuclear Engineering Research Group

 

--
You received this message because you are subscribed to the Google Groups "DAGMC Users and Collaborators" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dagmc-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dagmc-users/412e3d94-fce5-419b-8736-ad045187f3c5n%40googlegroups.com.

[Miguel Magán Romero]

Hello Paul.

Thanks for the very informative answer. I see now that it will not be a significant delay. The log may look a bit "spammy", but that's nothing to lose sleep over.

I guess that the duplicated memory issues will be similar to what happens with "standard" MCNP using MPI. But even that was tricky to get working with OMP, and, IIRC there were some limitations regarding what could be parallelized  (such as charged particles). 

Regards

Miguel

[Miguel Magán Romero]

Well, I am finding an issue when using these tallies in a production machine. The run starts apparently fine, but at the first rendezvous/dump, I get a memory allocation error. Here is the final output.

=====================================================

....

Implicit Complement assumed to be Vacuum
Set overlap thickness = 0
Implicit Complement assumed to be Vacuum
Implicit Complement assumed to be Vacuum
Set overlap thickness = 0
 master completed initialization broadcasts.

 master set rendezvous nps =     1000000,  work chunks =    15    06/28/22 12:47:32

Set overlap thickness = 0

Implicit Complement assumed to be Vacuum
Set overlap thickness = 0
Implicit Complement assumed to be Vacuum
Set overlap thickness = 0

Operating system error: Cannot allocate memory
Memory allocation failed in xmallocarray

Error termination. Backtrace:
#0  0x8c7855 in ???
#1  0x7c5424 in ???
#2  0x8e008f in ???
#3  0x8b2b3a in ???
#4  0x87ae67 in ???
#5  0x87b3a6 in ???
#6  0x7fc71289a6a2 in ???
#7  0x409c9d in ???
#8  0xffffffffffffffff in ???

===============================================  
This seems related to the writing of tally data, I suppose in the runtpe file. If I comment out the dag tally, it runs fine. Likewise, serial runs have no problems.

Has anybody found a similar issue, or can anyone help me tackle this? 

[Andrew Davis]

Hi Miguel

'Work Chunks' looks like you're using threading to me? Could you confirm if you're using threads/mpi, what size mesh, etc?

Thanks

Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/dagmc-users/00ff5643-ff70-40a2-9eb5-c76112ad359bn%40googlegroups.com.

[Miguel Magán Romero]

Hi Andrew:

Not unless I am seriously misunderstanding something...I am using MPI with no threads, the master simply divides the work into chunks and distributes them to the workers.

Regarding the mesh size, I have this in the output: 

  There are 180768 tetrahedrons in this tally mesh.
    Tally range has psize: 1
  Tally mesh has 386499 triangles.  Building KD tree of size 567267

This same problem runs in my local machine, so I do not expect size to be a problem... My guess is something related to the toolchain, but that is a bit beyond my ken.