small issues with assembly

[Ashkan Dorostkar]

Hi all,

I am writing a piece of parallel code and it seems that I don't understand a few issues.

My code is close to step-31 I construct the matrices this way

system_matrix.clear();
system_precon.clear();
BlockDynamicSparsityPattern dsp(2, 2);
dsp.block(0,0).reinit (n_u, n_u);
dsp.block(0,1).reinit (n_u, n_p);
dsp.block(1,0).reinit (n_p, n_u);
dsp.block(1,1).reinit (n_p, n_p);
dsp.collect_sizes();

DoFTools::make_sparsity_pattern(dof_handler, dsp,
                                constraints, true);
SparsityTools::distribute_sparsity_pattern (dsp,
                                            dof_handler.locally_owned_dofs_per_processor(),
                                            mpi_communicator,
                                            locally_relevant_dofs);

system_matrix.reinit(dsp);
system_precon.reinit(dsp);
locally_relevant_solution.reinit (owned_partitioning,
                                  relevant_partitioning,
                                  mpi_communicator);
system_rhs.reinit (owned_partitioning,
                   mpi_communicator);

Next I assemble the matrices and use "constraint object" to distribute local to global as follows
cell->get_dof_indices (local_dof_indices);
constraints.distribute_local_to_global(cell_matrix, cell_rhs,
                                       local_dof_indices,
                                       system_matrix,
                                       system_rhs);

constraints.distribute_local_to_global(cell_precond,
                                       local_dof_indices,
                                       system_precon);

However at these two last lines the excution crashes with the message 

You are trying an operation on a vector that is only allowed if the vector has no ghost elements, but the vector you are operating on does have ghost elements. Specifically, vectors with ghost elements are read-only and cannot appear in operations that write into these vectors.

why is this happening. I am not creating the system_rhs with ghost elements.
Any help is appriciated. Thank you.

Best,
Ashkan

[Wolfgang Bangerth]

On 11/09/2016 02:40 AM, Ashkan Dorostkar wrote:
> constraints.distribute_local_to_global(cell_matrix,cell_rhs,

>
> local_dof_indices,
>
> system_matrix,
>
> system_rhs);
>
>
> constraints.distribute_local_to_global(cell_precond,
>
> local_dof_indices,
>
> system_precon);
>
>
> However at these two last lines the excution crashes with the message
>
>

> *You are trying an operation on a vector that is only allowed if the

> vector has no ghost elements, but the vector you are operating on does
> have ghost elements. Specifically, vectors with ghost elements are

> read-only and cannot appear in operations that write into these vectors.*

>
>
> why is this happening. I am not creating the system_rhs with ghost elements.

It's hard to tell without a complete example that shows the issue. Are
you sure the exception is triggered in the function you mention? Are you
sure that you don't modify system_rhs somewhere in between?

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Ashkan Dorostkar]

Here is a part of the stacktrace that shows where the issue arises

An error occurred in line <1357> of file </private/var/folders/0q/n7_qq9dd723bhdps0nvnd9jw0000gn/T/alidorostkar/spack-stage/spack-stage-3tAB3e/dealii-8.4.2/include/deal.II/lac/trilinos_vector_base.h> in function

    void dealii::TrilinosWrappers::VectorBase::add(dealii::TrilinosWrappers::VectorBase::size_type, const size_type*, const double*)

The violated condition was: 

    !has_ghost_elements()

The name and call sequence of the exception was:

    ExcGhostsPresent()

Additional Information: 

You are trying an operation on a vector that is only allowed if the vector has no ghost elements, but the vector you are operating on does have ghost elements. Specifically, vectors with ghost elements are read-only and cannot appear in operations that write into these vectors.

See the glossary entry on 'Ghosted vectors' for more information.

Stacktrace:

-----------

#0  1   libdeal_II.g.8.4.2.dylib            0x0000000110560188 _ZN6dealii16TrilinosWrappers10VectorBase3addEjPKjPKd + 236: 1   libdeal_II.g.8.4.2.dylib            0x0000000110560188 _ZN6dealii16TrilinosWrappers10VectorBase3addEjPKjPKd 

#1  2   libdeal_II.g.8.4.2.dylib            0x00000001121830f0 _ZNK6dealii16ConstraintMatrix26distribute_local_to_globalINS_16TrilinosWrappers17BlockSparseMatrixENS2_3MPI11BlockVectorEEEvRKNS_10FullMatrixINT_10value_typeEEERKNS_6VectorINT0_10value_typeEEERKSt6vectorIjSaIjEERS7_RSD_bNS_8internal9bool2typeILb1EEE + 3124: 2   libdeal_II.g.8.4.2.dylib            0x00000001121830f0 _ZNK6dealii16ConstraintMatrix26distribute_local_to_globalINS_16TrilinosWrappers17BlockSparseMatrixENS2_3MPI11BlockVectorEEEvRKNS_10FullMatrixINT_10value_typeEEERKNS_6VectorINT0_10value_typeEEERKSt6vectorIjSaIjEERS7_RSD_bNS_8internal9bool2typeILb1EEE 

#2  3   gia                                 0x000000010cb1a548 _ZN10GeoPhysics3gia8assembleEv + 13366: 3   gia                                 0x000000010cb1a548 _ZN10GeoPhysics3gia8assembleEv 

#3  4   gia                                 0x000000010cb1bcfd _ZN10GeoPhysics3gia3runEv + 421: 4   gia                                 0x000000010cb1bcfd _ZN10GeoPhysics3gia3runEv 

system_rhs is not touched anywhere between.

[Wolfgang Bangerth]

On 11/09/2016 08:24 AM, Ashkan Dorostkar wrote:
> void
> dealii::TrilinosWrappers::VectorBase::add(dealii::TrilinosWrappers::VectorBase::size_type,
> const size_type*, const double*)
> The violated condition was:
> !has_ghost_elements()
> The name and call sequence of the exception was:
> ExcGhostsPresent()
> Additional Information:
> You are trying an operation on a vector that is only allowed if the
> vector has no ghost elements, but the vector you are operating on does
> have ghost elements. Specifically, vectors with ghost elements are
> read-only and cannot appear in operations that write into these vectors.
>
> See the glossary entry on 'Ghosted vectors' for more information.
>
> Stacktrace:
> -----------
> #0 1 libdeal_II.g.8.4.2.dylib 0x0000000110560188
> _ZN6dealii16TrilinosWrappers10VectorBase3addEjPKjPKd + 236: 1
> libdeal_II.g.8.4.2.dylib 0x0000000110560188
> _ZN6dealii16TrilinosWrappers10VectorBase3addEjPKjPKd
> #1 2 libdeal_II.g.8.4.2.dylib 0x00000001121830f0
> _ZNK6dealii16ConstraintMatrix26distribute_local_to_globalINS_16TrilinosWrappers17BlockSparseMatrixENS2_3MPI11BlockVectorEEEvRKNS_10FullMatrixINT_10value_typeEEERKNS_6VectorINT0_10value_typeEEERKSt6vectorIjSaIjEERS7_RSD_bNS_8internal9bool2typeILb1EEE
> + 3124: 2 libdeal_II.g.8.4.2.dylib 0x00000001121830f0

> _ZNK6*dealii16ConstraintMatrix26distribute_local_to_global*INS_16TrilinosWrappers17BlockSparseMatrixENS2_3MPI11BlockVectorEEEvRKNS_10FullMatrixINT_10value_typeEEERKNS_6VectorINT0_10value_typeEEERKSt6vectorIjSaIjEERS7_RSD_bNS_8internal9bool2typeILb1EEE

> #2 3 gia 0x000000010cb1a548
> _ZN10GeoPhysics3gia8assembleEv + 13366: 3 gia
> 0x000000010cb1a548 _ZN10GeoPhysics3gia8assembleEv
> #3 4 gia 0x000000010cb1bcfd
> _ZN10GeoPhysics3gia3runEv + 421: 4 gia
> 0x000000010cb1bcfd _ZN10GeoPhysics3gia3runEv
>
> system_rhs is not touched anywhere between.

So if you output immediately before the call to
distribute_local_to_global what
system_rhs.has_ghost_elements()
says, what do you find?