Exception when using FETools::extrapolate with ghost hanging nodes
Guilhem Poy
I am trying to use the FETools::extrapolate method on a distributed
triangulation that is adaptively refined. However, I get an exception
inside this function every time my triangulation contains hanging nodes
at ghost interfaces. I have attached a minimal working example that
illustrates this on deal.II version 9.6.1. It should be compiled in
debug mode as for usual dealii example codes, and then run with "mpirun
-np 2 main" with two mpi ranks to generate the exception. The macro
NO_GHOST_HANGING_NODES can be commented out to check that
FETools::extrapolate works correctly when there are no hanging nodes at
ghost interfaces.
Before opening a bug, I would like to check with you that I am not doing
something wrong in this test program. I was careful to set up my
distributed vectors as large as possible (i.e. with all relevant dofs)
to see if the problem was coming from there, and of course I updated the
ghost values of the "coarse" vector before calling FETools::extrapolate.
If I am not mistaken, the exception is thrown when an internal vector
with all relevant dofs is compressed line 1455 of
fe_tools_extrapolate.templates.h, but of course this internal vector
depends on the input vector I give to extrapolate so the problem could
very much come from my code. I am ready to provide any further
information that may be useful! Thanks in advance.
Best regards,
Guilhem
Guilhem Poy
As a complement (I realized I should have put this in the first email), here is (one of) the exception that is thrown with the attached MWE from my first mail:
An error occurred in line <738> of file
</home/gpoy/.local/share/deal-ii-candi/tmp/unpack/deal.II-v9.6.1/include/deal.II/base/partitioner.templates.h>
in function
void
dealii::Utilities::MPI::Partitioner::import_from_ghosted_array_finish(dealii::VectorOperation::values,
const dealii::ArrayView<const ElementType,
MemorySpaceType>&, const
dealii::ArrayView<ElementType, MemorySpace>&, const
dealii::ArrayView<ElementType, MemorySpace>&,
std::vector<ompi_request_t*>&) const [with Number =
double; MemorySpaceType = dealii::MemorySpace::Host]
The violated condition was:
*read_position == Number() ||
internal::get_abs(locally_owned_array[j] - *read_position) <=
internal::get_abs(locally_owned_array[j] + *read_position) *
100000. * std::numeric_limits<typename
numbers::NumberTraits< Number>::real_type>::epsilon()
Additional information:
Called compress(VectorOperation::insert), but the element
received
from a remote processor, value 0.7071067811865476, does not
match with
the value 0 on the owner processor 0
I also realized I was wrong below for the actual vector that
fails to compress when there are ghost hanging nodes: I think it
corresponds to the output vector "u2" in
ExtrapolateImplementation<dim, spacedim,
OutVector>::extrapolate_parallel.
Best
Guilhem
Guilhem Poy
Hi everyone,
I have continued trying to fix the issue below on my own. First, I discovered that the extrapolate function is thoroughly tested inside the testsuite, so I went there to see how it is used to fix my MWE, but I still get the exception. This led me to conclude that some triangulations with certain refinement pattern triggers the compress exception, while other do not, including the triangulation that is created in the make_tria function of the file "tests/mpi/fe_tools_extrapolate_common.h" of the deal.II repository. To convince you of this I have attached a patch that changes the refinement pattern in the aforementioned make_tria function. With the patch applied on the v9.6.1 repository, running again the tests "mpi/fe_tools_extrapolate_03***" will throw the same exception mentioned in my previous mails for three mpi ranks.
To summarize, I think there is a real bug involved since one can trigger the exception just by changing the triangulation involved in the "mpi/fe_tools_extrapolate*" tests, touching nothing about the interpolation and extrapolation code. Now, the problem is that I still don't understand why some triangulations works and others don't. Clearly, my affirmation on the triggering condition from my previous emails is imperfect, ghost hanging nodes seems to be a sufficient but not necessary condition to throw the exception. For example, the below distributed triangulation on the left triggers the exception, while the one on the right don't:
I will continue to investigate by diving into the internal
functions of extrapolate, but I would welcome some help on this
since I am not really familiar with the p4est code.
Best regards,
Guilhem
Guilhem Poy
Dear all,
I think I have pinpointed the problem, which comes from l.
724-733 in deal.II/fe/fe_tools_extrapolate.templates.h: when a
cell which has a more refined ghost neighbour owns a DoF that
lives at the interface between this cell and this refined
neighbour, the associated DoF value will never be set and
therefore stays at 0. However, on the neighbouring process, the
corresponding ghost DoF value will be set, which explains the
compress("insert") exception later thrown due to the inconsistency
between a nonzero ghost value and zero "owned" value. So
really, the triggering condition for my bug is not that there
are ghost hanging nodes, but that one of the entry in
dofs_on_refined_neighbors is associated with a locally owned DoF
on a given process.
I guess the idea behind these lines is that we should favour extrapolation on the side that has more information, i.e. with the refined cell. But for this to work, I think one need to communicate data between process if the relevant DoFs on the refined cell are owned by a different process than the one that owns this refined cell. Or maybe I completely misunderstood the thinking behind this dofs_on_refined_neighbors map?
Best regards,
Guilhem
Guilhem Poy
Luca Heltai
this is indeed the right channel to discuss these topics. Lack of answers usually indicates that what you ask is genuinely difficult, not that people don’t want to help you. Also, people tend to respond on areas where they are most familiar with, and what you are asking is a very specific topic, in a very specific case.
For example, I am not very familiar with functions in fe_tools_extrapolate.h, therefore I read with interest your questions, but could not give an answer right away, and, like the others, waited to see if someone else indeed had ideas.
If you think you have found a bug, and it looks like this is indeed the case, writing a minimal test that triggers the bug (like you did) is the perfect way to proceed. This also allowed you to find a solution.
Also, it is very useful what you did, to follow up on your own questions on the mailing list, as this indeeds may help other people.
Can I suggest that you open a Pull Request on the github repository containing both the failing test and your proposed patch?
At the stage in which you are, this is probably going to work better, as you have already done all of the work, and we can include your fix in the library.
If it’s your first time contributing to open source projects, here we have a short summary of how to do it:
https://github.com/dealii/dealii/wiki/Contributing
and there’s even a video lecture by Wolfgang:
https://www.math.colostate.edu/~bangerth/videos.676.32.8.html
Thanks, and I’m sorry you got the impression no-one cared. It’s just that no-one knew, this time, the answer to your question…
L.
> On 20 Apr 2025, at 09:54, Guilhem Poy <guilhe...@gmail.com> wrote:
>
> I have designed a simple patch that fixes the bug for simple situations that were previously failing with deal.II 9.6.1's FETools::extrapolate. I will continue the testing and propose a PR together with a bug report on github once I have time. Given the lack of answers here, I guess this is not the right channel to discuss all this?
> Best
> Guilhem
>
> On Wednesday, 16 April 2025 at 18:27:15 UTC+2 Guilhem Poy wrote:
> Dear all,
> I think I have pinpointed the problem, which comes from l. 724-733 in deal.II/fe/fe_tools_extrapolate.templates.h: when a cell which has a more refined ghost neighbour owns a DoF that lives at the interface between this cell and this refined neighbour, the associated DoF value will never be set and therefore stays at 0. However, on the neighbouring process, the corresponding ghost DoF value will be set, which explains the compress("insert") exception later thrown due to the inconsistency between a nonzero ghost value and zero "owned" value. So really, the triggering condition for my bug is not that there are ghost hanging nodes, but that one of the entry in dofs_on_refined_neighbors is associated with a locally owned DoF on a given process.
> I guess the idea behind these lines is that we should favour extrapolation on the side that has more information, i.e. with the refined cell. But for this to work, I think one need to communicate data between process if the relevant DoFs on the refined cell are owned by a different process than the one that owns this refined cell. Or maybe I completely misunderstood the thinking behind this dofs_on_refined_neighbors map?
> Best regards,
> Guilhem
>
> Le 15/04/2025 à 09:04, Guilhem Poy a écrit :
>> Hi everyone,
>> I have continued trying to fix the issue below on my own. First, I discovered that the extrapolate function is thoroughly tested inside the testsuite, so I went there to see how it is used to fix my MWE, but I still get the exception. This led me to conclude that some triangulations with certain refinement pattern triggers the compress exception, while other do not, including the triangulation that is created in the make_tria function of the file "tests/mpi/fe_tools_extrapolate_common.h" of the deal.II repository. To convince you of this I have attached a patch that changes the refinement pattern in the aforementioned make_tria function. With the patch applied on the v9.6.1 repository, running again the tests "mpi/fe_tools_extrapolate_03***" will throw the same exception mentioned in my previous mails for three mpi ranks.
>> Guilhem
>>
>> On 10/04/2025 14:31, Guilhem Poy wrote:
>>> As a complement (I realized I should have put this in the first email), here is (one of) the exception that is thrown with the attached MWE from my first mail:
>>> An error occurred in line <738> of file </home/gpoy/.local/share/deal-ii-candi/tmp/unpack/deal.II-v9.6.1/include/deal.II/base/partitioner.templates.h> in function
>>> void dealii::Utilities::MPI::Partitioner::import_from_ghosted_array_finish(dealii::VectorOperation::values, const dealii::ArrayView<const ElementType, MemorySpaceType>&, const dealii::ArrayView<ElementType, MemorySpace>&, const dealii::ArrayView<ElementType, MemorySpace>&, std::vector<ompi_request_t*>&) const [with Number = double; MemorySpaceType = dealii::MemorySpace::Host]
>>> The violated condition was:
>>> *read_position == Number() || internal::get_abs(locally_owned_array[j] - *read_position) <= internal::get_abs(locally_owned_array[j] + *read_position) * 100000. * std::numeric_limits<typename numbers::NumberTraits< Number>::real_type>::epsilon()
>>> Additional information:
>>> Called compress(VectorOperation::insert), but the element received
>>> from a remote processor, value 0.7071067811865476, does not match with
>>> the value 0 on the owner processor 0
>>> I also realized I was wrong below for the actual vector that fails to compress when there are ghost hanging nodes: I think it corresponds to the output vector "u2" in ExtrapolateImplementation<dim, spacedim, OutVector>::extrapolate_parallel.
>>> Best
>>> Guilhem
>>> On 10/04/2025 10:13, Guilhem Poy wrote:
>>>> Dear deal-ii users and developers,
>>>>
>>>> I am trying to use the FETools::extrapolate method on a distributed triangulation that is adaptively refined. However, I get an exception inside this function every time my triangulation contains hanging nodes at ghost interfaces. I have attached a minimal working example that illustrates this on deal.II version 9.6.1. It should be compiled in debug mode as for usual dealii example codes, and then run with "mpirun -np 2 main" with two mpi ranks to generate the exception. The macro NO_GHOST_HANGING_NODES can be commented out to check that FETools::extrapolate works correctly when there are no hanging nodes at ghost interfaces.
>>>>
>>>> Before opening a bug, I would like to check with you that I am not doing something wrong in this test program. I was careful to set up my distributed vectors as large as possible (i.e. with all relevant dofs) to see if the problem was coming from there, and of course I updated the ghost values of the "coarse" vector before calling FETools::extrapolate. If I am not mistaken, the exception is thrown when an internal vector with all relevant dofs is compressed line 1455 of fe_tools_extrapolate.templates.h, but of course this internal vector depends on the input vector I give to extrapolate so the problem could very much come from my code. I am ready to provide any further information that may be useful! Thanks in advance.
>>>>
>>>> Best regards,
>>>>
>>>> Guilhem
>
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/dealii/85e0c48b-60e4-4f5f-87d7-cce0a92bfbcdn%40googlegroups.com.
Guilhem POY
Thanks for the helpful links and explanations. You guessed right, this
is my first public contribution to a large-scale collaborative
open-source project (up until now I only developed my physics simulation
software internally or with single collaborator) so I was not so sure
about the right channels and procedures :) I can also perfectly
understand that this is a very specific topic with probably a low
priority, so I am very happy to contribute however modestly.
I will start testing my patch on a fresh master build and if everything
goes smoothly submit my bug report and PR.
Best regards,
Guilhem