Using KINSOL with distributed vectors

[Norihiro Watanabe]

Hi,

I got the below error when calling KINSOL::solve() with Trilinos MPI vectors.
It looks KINSOL tried to call scale() for a ghosted vector.
The vector has ghost elemets probably because I passed
a ghost vector as "initial_guess_and_solution" argument to solve().

I can pass a non-ghosted vector as "initial_guess_and_solution", but this will cause another problem in the residual assembly function which requires ghosted elements.

Any advice on this issue would be appreciated.

--------------------------------------------------------
An error occurred in line <2146> of file </usr/src/dealii-v9.5.0/include/deal.II/lac/trilinos_vector.h> in function
void dealii::TrilinosWrappers::MPI::Vector::scale(const dealii::TrilinosWrappers::MPI::Vector&)
The violated condition was:
!has_ghost_elements()
Additional information:
You are trying an operation on a vector that is only allowed if the
vector has no ghost elements, but the vector you are operating on does
have ghost elements. Specifically, vectors with ghost elements are
read-only and cannot appear in operations that write into these
vectors.

See the glossary entry on 'Ghosted vectors' for more information.

Stacktrace:
-----------
#0 /usr/local/lib/libdeal_II.g.so.9.5.0: dealii::TrilinosWrappers::MPI::Vector::scale(dealii::TrilinosWrappers::MPI::Vector const&)
#1 /usr/local/lib/libdeal_II.g.so.9.5.0: double dealii::SUNDIALS::internal::NVectorOperations::weighted_l2_norm<dealii::TrilinosWrappers::MPI::Vector>(_generic_N_Vector*, _generic_N_Vector*)
#2 /lib/x86_64-linux-gnu/libsundials_kinsol.so.5: N_VWL2Norm
#3 /lib/x86_64-linux-gnu/libsundials_kinsol.so.5:
#4 /lib/x86_64-linux-gnu/libsundials_kinsol.so.5: KINSol
#5 /usr/local/lib/libdeal_II.g.so.9.5.0: dealii::SUNDIALS::KINSOL<dealii::TrilinosWrappers::MPI::Vector>::solve(dealii::TrilinosWrappers::MPI::Vector&)

Best regards,
Norihiro

[Luca Heltai]

The vector you pass to kinsol must be non-ghosted. In the residual assembly you must create a local ghosted copy. Just as you would if you were doing things manually. (I.e., After a solve, you copy to a ghosted vector to output or to compute error estimators. Here it is the same. )

Luca

> Il giorno 16 ago 2023, alle ore 09:25, 'Norihiro Watanabe' via deal.II User Group <dea...@googlegroups.com> ha scritto:
>
> Hi,

> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/TYWPR01MB7185E469D5E4148829CA3D4BC415A%40TYWPR01MB7185.jpnprd01.prod.outlook.com.

[Norihiro Watanabe]

Hi Luca, 

Many thanks for the reply! You are right that I should create a ghosted copy in the assembly.

Best,

Nori

2023年8月16日水曜日 20:19:15 UTC+9 Luca Heltai:

[Tiny Y]

HI Norihiro Watanabe,

Hello, I am a novice in Dealii. I have encountered a similar problem to yours. Have you solved this problem? How did you solve it?Can you explain what do you mean by ' create a ghosted copy in the assemble '.Thank you very much!

Best,

Yu

[Norihiro Watanabe]

Hi Yu,

Though I'm not sure this is optimum, how I solved the problem was
1) prepare two MPI vectors for stroring initial solutions. One is with ghost elements and another one is without ghost (I guess you have alredy one of them). Values of the vectors should be same.
2) assemble residual and/or jacobian using the solution vector with ghost
3) pass the non-ghosted solution vector to KINSOL and solve the nonlinear
4) copy the KINSOL's returned non-ghosed solution vector to the ghosted solution vector  
5) go back to step 2 to evalulate new residual

Details about ghosted vectors are explained in, for example,
https://www.dealii.org/current/doxygen/deal.II/DEALGlossary.html#GlossGhostedVector
https://www.dealii.org/current/doxygen/deal.II/group__distributed.html

Best,
Nori

2023年10月27日金曜日 11:10:44 UTC+9 Tiny Y:

[Norihiro Watanabe]

Hi Yu,

Sorry,  what I wrote above is not correct at some part.

Whe using KINSOL, one needs to prepare call back functions to assemble residual and jacobian. At the beginning of these functions, I made a copy of the non-ghosted solution to a ghosted vector because KINSOL passes a non-ghosted solution vector to these functions. The following assembly part is done with the ghosted solution vector.

In addition, when calling KINSOL::solve(), I changed my code to pass a non-ghosted solution vector instead of a ghosted one.

Best,

Nori

2023年10月27日金曜日 11:22:28 UTC+9 Norihiro Watanabe:

[Tiny Y]

Hi Nori,

Thank you for your reply. Your method is very enlightening to me.

Best,
Yu

[Tiny Y]

Hi Nori,

I'm sorry to bother you again. I would like to know which function you used to obtain vectors without ghost element from vectors with ghost element. If convenient, could you please share the code in this area or your ideas for implementing this process.  Thank you very much for your kind help.

Best,

Yu

[Norihiro Watanabe]

Hi Yu,

The code I wrote are something like this:

TrilinosWrappers::MPI::Vector vector_without_ghost;
vector_without_ghost.reinit(jacobian.locally_owned_range_indices(), jacobian.get_mpi_communicator());
vector_without_ghost = vector_with_ghost;

In the above, I used a jacobian matrix object to get local index set and MPI communicator. 

However you should also be able to get them from a ghosted vector object.

Best,

Nori

2023年10月31日火曜日 15:55:52 UTC+9 Tiny Y: