Query regarding DoFTools::dof_indices_with_subdomain_association()

[vachan potluri]

Hello all,

I am writing an MPI parallel DG code for linear advection equation, just for understanding the parallel programming paradigm of deal.II.

To evaluate numerical flux at a face connecting two subdomains, I only require the solution (from a different mpi process) at dofs which lie on the face; the dofs which lie inside the neighboring cell (of the neighbor subdomain) are not required. Consequently, I think the ghost indices for my solution vector would be those provided by the function DoFTools::dof_indices_with_subdomain_association(), rather than DoFTools::extract_locally_relevant_dofs(). However, the documentation of the former function states

Note that this function is of questionable use for DoFHandler objects built on parallel::distributed::Triangulation since in that case ownership of individual degrees of freedom by MPI processes is controlled by the DoF handler object, not based on some geometric algorithm in conjunction with subdomain id. In particular, the degrees of freedom identified by the functions in this namespace as associated with a subdomain are not the same the DoFHandler class identifies as those it owns.

What is the meaning of the second line? I suppose this has something to do with the FiniteElement type of the dof handler. Does this mean to say that irrespective of whether the FiniteElement has dofs on cell faces or not, this function approves a dof if it is geometrically residing on a face? This behaviour is actually what I want because although FE_DGQ element doesn't attach any dofs to cell faces, I want to get dof indices of the dofs (geometrically) lying on the face.

Thanks!

Vachan

[Daniel Arndt]

Vachan,

[...]

However, the documentation of the former function states

Note that this function is of questionable use for DoFHandler objects built on parallel::distributed::Triangulation since in that case ownership of individual degrees of freedom by MPI processes is controlled by the DoF handler object, not based on some geometric algorithm in conjunction with subdomain id. In particular, the degrees of freedom identified by the functions in this namespace as associated with a subdomain are not the same the DoFHandler class identifies as those it owns.

What is the meaning of the second line? I suppose this has something to do with the FiniteElement type of the dof handler. Does this mean to say that irrespective of whether the FiniteElement has dofs on cell faces or not, this function approves a dof if it is geometrically residing on a face? This behaviour is actually what I want because although FE_DGQ element doesn't attach any dofs to cell faces, I want to get dof indices of the dofs (geometrically) lying on the face.

DoFTools::dof_indices_with_subdomain_association returns the degrees of freedom of all the cells that have the given subdomain id. For parallel::distributed::Triangulation objects the subdomain id is the the MPI rank and this is the only valid input.

In this case, the function simply returns all the degrees of freedom on locally owned cells. If a finite element defines degrees of freedoms that are associated with a face, these degrees of freedom are part of all the IndexSet objects in case the face is located on the interface between two subdomains. For a parallel::distributed::Triangulation object, these are the degrees of freedom on the faces to ghosted cells.

FE_DGQ doesn't define any face degrees of freedom. All its degrees of freedom are defined to be inside of a cell. You could still ask for the unit support points and figure out which of them are geometrically on a given face.

However, note that as soon as gradients are involved all the degrees of freedom contribute to values on faces.

Best,

Daniel

[vachan potluri]

Daniel,

DoFTools::dof_indices_with_subdomain_association returns the degrees of freedom of all the cells that have the given subdomain id. For parallel::distributed::Triangulation objects the subdomain id is the the MPI rank and this is the only valid input.
In this case, the function simply returns all the degrees of freedom on locally owned cells. If a finite element defines degrees of freedoms that are associated with a face, these degrees of freedom are part of all the IndexSet objects in case the face is located on the interface between two subdomains. For a parallel::distributed::Triangulation object, these are the degrees of freedom on the faces to ghosted cells

Thanks for the reply. Indeed the indices given by DoFHandler::locally_owned_dofs() and DoFTools::dof_indices_with_subdomain_association() are identical for FE_DGQ element. I have checked this in my code.

You could still ask for the unit support points and figure out which of them are geometrically on a given face.

This is definitely possible, but would probably be inefficient if I code for it. Isn't there a function in DoFTools which does this? Because not marking all dofs of ghost cells as relevant will give much savings in communication, I was wondering if DoFTools already doesn't have an implementation for this.

However, note that as soon as gradients are involved all the degrees of freedom contribute to values on faces.

I don't have much experience in parallel programming, but I think we can circumvent this by computing gradients at all dof locations in a subdomain and again only communicating data of dofs lying on subdomain interfaces. I might need some correction on this :).

[Doug Shi-Dong]

Hello Vachan,

Sounds like you're implementing nodal DG, hence why you only need values and gradients at the quadrature points from the neighbor. Something you might want to consider rather than communicating them at solution points, in case you ever decide to overintegrate.

You could still ask for the unit support points and figure out which of them are geometrically on a given face.

This is definitely possible, but would probably be inefficient if I code for it. Isn't there a function in DoFTools which does this? Because not marking all dofs of ghost cells as relevant will give much savings in communication, I was wondering if DoFTools already doesn't have an implementation for this.

I don't think there is currently an implementation. You should be able to mark your degrees of freedom on the subdomain interface using FE_DGQ::has_support_on_face(...). Basically, if you loop over the ghost elements, you can loop over the dof indices of the cell, and use that function to figure out if its on a face. Efficiency shouldn't be an issue, especially since this is pre-processing.

 

However, note that as soon as gradients are involved all the degrees of freedom contribute to values on faces.

I don't have much experience in parallel programming, but I think we can circumvent this by computing gradients at all dof locations in a subdomain and again only communicating data of dofs lying on subdomain interfaces. I might need some correction on this :).

Yes, that works if you decide to first compute all your gradients and store them. Note that in 3D, you're doing 3x the communication for the face gradients, so your efficiency only shows up when using elements of order 4 or more. 

Depending on where you want to take your code in the future, I'd be careful with early code optimization like this. I would just stick with communicating all the neighbor's dofs and optimize the code if you really see a slowdown. 

Doug