Parallel distributed triangulation: informations on part of the mesh which are not owned

[giovann...@hotmail.it]

Hi everyone!
  I'm GivAlz, I've recently started to develop some additions to the deal.II library as a project for my master and PhD; I'm currently working on some pull request but I guess this is a better place for discussion (https://github.com/dealii/dealii/pull/5117).

Currently I'm working on a project for FSI simulations. The idea is having a "space mesh" and another one interacting inside. The interacting mesh, while moving, could go to ghost cells but also end up in an artificial cell of the space mesh. In this case or, if this "interacting mesh" is initialized in parallel, I need the ability to guess who's the owner of a portion of space inside an artificial cell. Another thing I'm working on is trying to modify find_point_locations routine of deal.II so that it can find point and returns map for arbitrary points even in distributed parallel meshes.

Recently I've stumbled on this post which I found quite interesting: https://groups.google.com/forum/#!topic/dealii/5UjaE3Y1WEk

So I was wondering:

- I see there are some promising variables, such as forest::global_first_position , but I can't really find the documentation: how much can they really tell? The discussion doesn't seem to reach a clear solution; if Martin Steigemann is still reading could you tell me, please, if you managed to solve this problem?
- Does Cellid contain some information about the owners? Could it be useful for a process to pass Cellid to another process CellId? I mean: modifying the
find_point_location could the Cellid give information about where to start looking for that point?

I'm wondering this because p4est naturally splits the space but I'm trying to achieve another description of the space using boundingboxes: every process describes it's "locally owned" space with few of them
which are then sent to all the others. In this way one can "guess" with relative accuracy the owner of a point and pass it around. Do you think it would be better/easier/more effective or efficient to try implementing
some functions/properties from the p4est description?

I tried to look for documentation but I couldn't find much so, please, if someone has links to articles or ideas that would be very welcome.
Thank you very much!
Giovanni

[Wolfgang Bangerth]

> - Does Cellid contain some information about the owners? Could it be
> useful for a process to pass Cellid to another process CellId? I mean:
> modifying the
> find_point_location could the Cellid give information about where to
> start looking for that point?

CellIds are globally unique, so you can pass them around to other
processors. A CellId of an *active* cell on the global mesh can also be
used to query which processor owns the cell.

The problem is that you don't know the CellId of the cell in which a
point is located, because chances are that on your processor, the cell
the point is located in is artificial, and so all you know that one of
its children in the global mesh is the active cell that has the point --
but this child cell could be owned by any number of processors.

> I'm wondering this because p4est naturally splits the space but I'm
> trying to achieve another description of the space using boundingboxes:
> every process describes it's "locally owned" space with few of them
> which are then sent to all the others. In this way one can "guess" with
> relative accuracy the owner of a point and pass it around. Do you think
> it would be better/easier/more effective or efficient to try implementing
> some functions/properties from the p4est description?

No. I think the bounding box approach is the way to go. It allows you to
shrink the set of processors you have to ask whether they own the point
to a manageable set.

Have you done a literature search on finding what processor owns a point
in a parallel mesh? This doesn't seem like a problem you're the first
one to encounter. You may also want to search for literature on building
collections of bounding boxes -- that, too, ought to be a problem for
which there is a good solution available.

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Bruno Turcksin]

Giovanni,

It looks like you are doing a lot of things similar to what we do in this project https://github.com/ORNL-CEES/DataTransferKit In particular, one of our subroutine allows the user to ask for the value of the field at arbitrary points. We need our code to use GPUs, so we based our algorithms on Karras' work https://devblogs.nvidia.com/parallelforall/thinking-parallel-part-iii-tree-construction-gpu/ We use two search trees to speed up the computation: one across node to know on which processor(s) the point can be found and then a second tree on the node, i.e., for each MPI rank. I'll try to take a look at your PR.

Best,

Bruno