Number of DOFs after refinement depends on number of MPI processes (p:d:Triangulation)

[Stephen DeWitt]

Hello,

I'm running matrix free calculations using a parallel::distributed::Triangulation (loosely based on step-48). One thing that I've noticed is the the final number of DOFs at the end of a simulation vary slightly depending on my number of MPI processes. For example, one run on 16 cores finished with 2,988,700 DOF and nominally the same run on 512 cores finished with 2,988,864. Is this to be expected? My working assumption has been that there are a number of heuristics that go into "smoothing" the refinement of the mesh and that some/all of these only operate on the local part of the mesh. Is that the case? Or should I be concerned that I have a subtle issue in my code? Nothing changes qualitatively between simulations with different numbers of cores, but of course there is a very small difference in the solutions due to the different meshes.

Thanks,

Steve

[Wolfgang Bangerth]

On 6/20/19 9:19 AM, Stephen DeWitt wrote:
> I'm running matrix free calculations using a
> parallel::distributed::Triangulation (loosely based on step-48). One thing
> that I've noticed is the the final number of DOFs at the end of a simulation
> vary slightly depending on my number of MPI processes. For example, one run on
> 16 cores finished with 2,988,700 DOF and nominally the same run on 512 cores
> finished with 2,988,864. Is this to be expected?

If the number of cells is also different, then yes. For the same number of
cells, you need to get the same number of DoFs, so the number of cells is the
normative question you need to investigate.

If you do use adaptive mesh refinement, then just as you mention there are
heuristics that sometimes do or do not lead to certain cells to be refined.
For example, computing the threshold for the local error indicators above
which a cell should be refined is one algorithm that might slightly depend on
the number of processes you use.

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/