Mean value of component of TrilinosWrappers::MPI::BlockVector

[Konrad Simon]

Dear all,

I am trying to compute the mean value of the pressure component of a Trilinos block vector with three blocks and 7 components (vorticity-velocity-pressure).

Using one MPI rank is fine but if I use more I get the error

"An error occurred in line <666> of file </home/ksimon/lib/dealii-9.2.0-src/source/lac/trilinos_vector.cc> in function 

   dealii::TrilinosScalar dealii::TrilinosWrappers::MPI::Vector::operator()(dealii::TrilinosWrappers::MPI::Vector::size_type) const
The violated condition was:  
   false
Additional information:  
   You tried to access element 3023 of a distributed vector, but this element is not stored on the current processor. Note: There are 4456 elements stored on the current processor from within the range 5544 through 9999 but Trilinos vectors need not store contiguous ranges on each processor, and not every element in this range may in fact be stored locally."

Now, I know what that means. Is there anything I can do? A workaround?

Thanks in advance and best regards,
Konrad

[Konrad Simon]

I am pretty sure that this is related to the following problem:

When I try to run AMR with this FESystem (Nedelec-RaviartThomas-DGQ) with more than one MPI rank I get this error

"An error occurred in line <1626> of file </home/ksimon/lib/dealii-9.2.0-src/include/deal.II/lac/affine_constraints.h> in function 

   void dealii::AffineConstraints<number>::add_line(dealii::AffineConstraints<number>::size_type) [with number = double; dealii::AffineConstraints<number>::size_type
= unsigned int]
The violated condition was:  
   line_n != numbers::invalid_size_type
Additional information:  
   This exception -- which is used in many places in the library -- usually indicates that some condition which the author of the code thought must be satisfied at a
certain point in an algorithm, is not fulfilled. An example would be that the first part of an algorithm sorts elements of an array in ascending order, and a second
part of the algorithm later encounters an element that is not larger than the previous one.

There is usually not very much you can do if you encounter such an exception since it indicates an error in deal.II, not in your own program. Try to come up with the
smallest possible program that still demonstrates the error and contact the deal.II mailing lists with it to obtain help.

"

which is thrown in AffineConstraints::add_line (see the second assert in line 1826) after being called by make_hanging_node_constraints. It seems that deal.ii is trying to setup constraints on dofs that are not owned by the MPI process.

I read a little bit in the discussions referring to Nedelec and refinement - seems a difficult issue. Did anyone run into this, too?

Best,
Konrad

[Wolfgang Bangerth]

On 1/10/21 12:43 PM, Konrad Simon wrote:
> ).

>
> Using one MPI rank is fine but if I use more I get the error
>
> "An error occurred in line <666> of file
> </home/ksimon/lib/dealii-9.2.0-src/source/lac/trilinos_vector.cc> in function
>    dealii::TrilinosScalar
> dealii::TrilinosWrappers::MPI::Vector::operator()(dealii::TrilinosWrappers::MPI::Vector::size_type)
> const
> The violated condition was:
>    false
> Additional information:
>    You tried to access element 3023 of a distributed vector, but this element
> is not stored on the current processor. Note: There are 4456 elements stored
> on the current processor from within the range 5544 through 9999 but Trilinos
> vectors need not store contiguous ranges on each processor, and not every
> element in this range may in fact be stored locally."
>
> Now, I know what that means. Is there anything I can do? A workaround?

I suspect you are passing a fully distributed vector to that function, but it
needs to read ghost elements of the vector. Have you tried copying the vector
into a locally_relevant vector, and passing that to the function in question?

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Wolfgang Bangerth]

On 1/11/21 5:18 AM, Konrad Simon wrote:
>
> which is thrown in AffineConstraints::add_line

> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdealii.org%2Fcurrent%2Fdoxygen%2Fdeal.II%2Faffine__constraints_8h_source.html%23l01618&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7C022627123a914a008c9408d8b62b01ba%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637459643116494271%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oHJI2cLWMP6kAQhW5f71Lq2p2MGdAGFbpE7F9BqUPro%3D&reserved=0>

> (see the second assert in line 1826) after being called by
> make_hanging_node_constraints. It seems that deal.ii is trying to setup
> constraints on dofs that are not owned by the MPI process.

That looks like an unrelated error. Can you create a small testcase for this
issue here?

[Konrad Simon]

I suspect you are passing a fully distributed vector to that function, but it
needs to read ghost elements of the vector. Have you tried copying the vector
into a locally_relevant vector, and passing that to the function in question?

Thank you, Wolfgang, that was the issue.  I am using the mean value function at several different places in my code. One of them is inside a preconditioner vmult that accepts only a fully distributed vector. Is there also a simple workaround? I have a patch since I only need the pressure which is the last component of the distributed vector in each locally owned cell but the patch is ugly...

[Konrad Simon]

That looks like an unrelated error. Can you create a small testcase for this
issue here?

I will try to come up with an example. 

Thank you again and best regards,
Konrad

[Wolfgang Bangerth]

On 1/12/21 9:18 AM, Konrad Simon wrote:
>
> Thank you, Wolfgang, that was the issue.  I am using the mean value function
> at several different places in my code. One of them is inside a preconditioner
> vmult that accepts only a fully distributed vector. Is there also a simple
> workaround?

Then you need to create a vector with locally relevant entries as ghosts and
copy your fully distributed vector to it. There is no other way if you want to
use that function.

But if your goal is to fix the pressure in a Stokes problem, it doesn't have
to be the integral mean of the pressure that is zero. It could just be the
arithmetic mean, which you can compute without access to ghost elements.

[Konrad Simon]

Then you need to create a vector with locally relevant entries as ghosts and
copy your fully distributed vector to it. There is no other way if you want to
use that function.

But if your goal is to fix the pressure in a Stokes problem, it doesn't have
to be the integral mean of the pressure that is zero. It could just be the
arithmetic mean, which you can compute without access to ghost elements.  

Many  thanks, Wolfgang, that works! :-)

Konrad