Distributing objects on cluster nodes according to distributed triangulation (MPI)

[Konrad]

Hi deal.ii community,

I have a little conceptual question. Maybe it is dumb but I am new to MPI...

Say I have a triangulation of type parallel::distributed::Triangulation<dim> that gets distributed over N nodes on a cluster. Each active cell should own (or point to) an object that makes up a (modified non-polynomial) finite element basis. That means the object is not so simple in that it contains objects itself (e.g., a triangulation, several vectors etc).

How can I distribute a set of these basis objects (after creating the distributed triangulation) so that each active cell and its corresponding basis object are on the same cluster node? 

Is there something like a distributed STL-like container whose elements I can initialize on different cluster nodes (for example by querying cell->is_locally_owned()) when looping over the triangulation?

Best,
Konrad

[Daniel Arndt]

Konrad,

You could create an std::map using CellId (https://www.dealii.org/developer/doxygen/deal.II/classCellId.html) as key if you need to identify cells uniquely globally.

 

Best,

Daniel

[Konrad]

Hi Daniel,

 

You could create an std::map using CellId (https://www.dealii.org/developer/doxygen/deal.II/classCellId.html) as key if you need to identify cells uniquely globally.

 

Best,

Daniel

Thanks, I see this class makes it possible to identify cells globally in a distributed triangulation. I don't see though how the objects in the std::map are distributed among cluster nodes as the cells are. Am I missing something here?

Best,
Konrad

[Wolfgang Bangerth]

On 8/23/19 7:33 AM, Konrad wrote:
>
> Thanks, I see this class makes it possible to identify cells globally in a
> distributed triangulation. I don't see though how the objects in the std::map
> are distributed among cluster nodes as the cells are. Am I missing something here?

Yes, the data transfer is difficult because you need to figure out who owns a
cell for which you have an entry in your local map after
refinement/rebalancing. Then you need to pack the data, send it to the new
owner, and possibly receive information about other cells you now own after
refinement/rebalancing.

You might want to look at how other classes do this that store non-trivial
data on each cell. An example is ContinuousQuadratureDataTransfer.

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Konrad]

Thank you, Wolfgang, problem solved. But in a much simpler way. My basis object is not easily serializable so I have to recreate it and destroy the old one anyway (for now) once cells are moved to other nodes. So I do not send send them but rather create them and work with them on each node (if the corresponding cell is locally owned) and just collect some serializable information on a master process with mpi_gather.

I have a little technical question though: Can I attach the mpi_communicators that you use in the deal.ii tutorials also somehow to a boost::mpi::communicator?

https://www.boost.org/doc/libs/1_48_0/doc/html/boost/mpi/communicator.html

Never tried it but according to the documentation it seems possible.

Best,
Konrad

[Wolfgang Bangerth]

On 9/3/19 2:40 AM, Konrad wrote:
>
> I have a little technical question though: Can I attach the mpi_communicators
> that you use in the deal.ii tutorials also somehow to a boost::mpi::communicator?
>
> https://www.boost.org/doc/libs/1_48_0/doc/html/boost/mpi/communicator.html
>
> Never tried it but according to the documentation it seems possible.

Yes, boost::mpi is a wrapper around MPI, and so the communicators can be
converted to each other.

boost::mpi is a nice idea in principle, but it's not particularly well
designed and few people use it. Some reasons are listed here:

https://scicomp.stackexchange.com/questions/3019/boostmpi-or-c-mpi-for-high-performance-scientific-applications

[Konrad]

boost::mpi is a nice idea in principle, but it's not particularly well
designed and few people use it. Some reasons are listed here:
 
https://scicomp.stackexchange.com/questions/3019/boostmpi-or-c-mpi-for-high-performance-scientific-applications

Best
  W.

I think it is a nice interface to send not so complex objects but this functionality is already wrapped nicely by deal.ii. The problem starts when an object is not easily serializable. And, honestly, I do not have a clue what boost::mpi (or the boost serialization) is doing exactly when an object contains pointers to whatever distributed data somewhere in memory etc.

At any rate, Thanks for the hints.

Konrad