Communicating constraints in a parallel environment

[Markus Klinsmann]

Dear all,

I have been looking for a way to set constraints, as for example in step-11 a mean-value constraint or as in step-45 a periodic boundary constraint, in a parallel computation. It appears to me that it might be a really non-trivial task to to set up the necessary communication between the diffferent parts of a partitioned mesh, i.e the different processors. And furthermore, it could probably lead to a unacceptable costly communication time. But anyway, are there already some thoughts about that topic or a tutorial for me to start with?

Thank you for all kind of responses,
Markus

[Wolfgang Bangerth]

The trick is to only set constraints that correspond to degrees of freedom you
may write to on any given processor -- i.e., exactly the locally_active or
locally_relevant ones. This way it is at least possible to store them all.

Of course, it may at times be difficult to compute constraints in parallel
(e.g. for periodic boundary conditions) but since you don't say what you want
to do with your constraints there is little I can beyond speculating.

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@math.tamu.edu
www: http://www.math.tamu.edu/~bangerth/

[Markus Klinsmann]

Hi Wolfgang,
thank you for the fast response. To be more precise, I have something like a thermomechanical problem with free boundaries and I want to get rid of the (physical,) undetermined degrees of freedom, i.e. translations and rotations. I planned to do it in a similar fashion as in step-11, i.e. by constraining one dof by an algebraic relation to, for example, all other dofs located at the boundary of the geometry. However, in this case, the chosen dof is then partially constrained by dofs living on other parts of the partitioned mesh. If I now select as constrained dof a locally active or relevant one, is the problem really solved? What I mean is, that also the locally active or relevant dofs are only accesible on adjacent parts of the mesh. Or am I wrong? Just to be sure that in the end, I am not restricted to comunication between neighbouring parts.
Concerning the periodic b.c. question, I am also thinking about a model where I would come across this problem in the future snd I thought that most of the critical points of the parallel implementation would be the same.
Since I am still pretty new to deal.Ii, I hope there weren't too many obvious points in my concerns.
Thanks a lot,
Markus

[Martin Kronbichler]

Dear Markus,

On Monday, April 1, 2013 1:48:34 AM UTC+2, Markus Klinsmann wrote:

To be more precise, I have something like a thermomechanical problem with free boundaries and I want to get rid of the (physical,) undetermined degrees of freedom, i.e. translations and rotations. I planned to do it in a similar fashion as in step-11, i.e. by constraining one dof by an algebraic relation to, for example, all other dofs located at the boundary of the geometry. However, in this case, the chosen dof is then partially constrained by dofs living on other parts of the partitioned mesh.

You are right that the procedure in step-11 will not work in a parallel setting. Have you tried to fix the degree of freedom to zero instead of the average? This is strategy 1 in the discussion of step-11 and works in many applications (but not all, therefore it is not recommended). However, in parallel this is the only sensible strategy among the ones presented in step-11. And also it avoids that the matrices get many coupling entries (with your strategy, all boundary dofs would couple to each other, giving a quite dense matrix).

Another alternative would be to set the condition \int_dOmega u = 0 as an additional expression for the constrained dof manually, which has the same effect but avoids denser matrices and is perfectly parallel (you only evaluate it for locally owned dofs) - it is really just an inner product on a (sparse) vector.

Best,

Martin

[Markus Klinsmann]

Hi Martin,
thank you for your answer. I think I will go for the solution of constraining certain dofs directly to zero. Or to get rid of the rotations, by only a small number of other dofs living on, let's say, only the first processor used. Constraining via the average seemed to me just as a somehow cleaner way, but the direct approach is probably more pragmatic.

Thanks again,
Markus

[Daniel Arndt]

Dear all,

Of course, it may at times be difficult to compute constraints in parallel
(e.g. for periodic boundary conditions) but since you don't say what you want
to do with your constraints there is little I can beyond speculating.

I am really interested in using periodic boundary conditions on distributed meshes.
Are there any ideas or suggestions on how to accomplish this?

Since communicating the needed dofs after the mesh has been distributed is quite difficult, may modifying the grid and somehow hardcode the periodicity be an option?

Thanks for all responses,
Daniel

[Timo Heister]

Hi Daniel,

this is a non-trivial task to do and the difficulty kind of depends on
your problem:
1. If you use adaptive refinement, you need to make sure the
refinement is compatible across the periodic faces. This is possible
in p4est and it just requires you to hand the periodicity information
to p4est when the coarse mesh is initialized.
2. The advantage of doing 1) is also that the cells on the other side
are part of the ghost layer automatically, so you will have the
necessary information (DoF numbers) at hand.
2. Setting the constraints. This should now work in the same way as it
is done right now.

Take a look at https://groups.google.com/d/topic/dealii-developers/aInLZX8sE6Q/discussion
and let us know if you need more information. As a first step I would
try to hack in a periodicity in the p4est create_connectivity call and
see what happens. We can work on a nicer interface afterwards.

> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dealii+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Timo Heister
http://www.math.tamu.edu/~heister/

[Wolfgang Bangerth]

> 1. If you use adaptive refinement, you need to make sure the
> refinement is compatible across the periodic faces. This is possible
> in p4est and it just requires you to hand the periodicity information
> to p4est when the coarse mesh is initialized.

I've been communicating with Craig Michoski (CC:) about exactly this option.
Daniel -- you may want to communicate with Craig about where he currently is
and what needs to be done.

[Zhicheng Wang]

Hi, Daniel,

       I met same problem several monthes ago. Thanks to Timo, he  gave me some suggestion. however, my way is :

    

    1 , looking for all the dofs supposed to be constrainted,

        2,  copying these dofs to all the processes,

        3. add these dofs to locally relevant IndexSet,

        4, initialize the ConstraintMatrix by using  IndexSet,

        5, add periodic dofs to  ConstraintMatrix on every process.

       

        I'm not sure if there are bugs on these steps, but it works for me if using a mesh created by using GridGenerator 

         

        

       zhicheng

[Daniel Arndt]

Thanks for the fast responses. I will have a look at the suggestions and communicate with Craig Michoski.