CDR solver: exception happens when using MPI

[Tingchang Yin]

Hi everyone,

I downloaded the convection diffusion reaction solver from https://github.com/dealii/code-gallery  

The cdr solver is really what I need.  I changed the user interface code a little bit regarding loading mesh, setting boundary conditions and initial conditions. Now the code can load a mesh file (*.inp).  

When I use one processor to run it with the command `mpirun -np 1  ./cdr  SmallPorous.inp`, the simulation is conducted without any error. 

When I use multiple processor with command `mpirun -np 14  ./cdr  SmallPorous.inp`, I get the following exceptions, which seems to be unmatched dof between processors:

```

...

...

cycle 128 and number of degrees of freedom: 20796
cycle 129 and number of degrees of freedom: 20800
cycle 130 and number of degrees of freedom: 20803

--------------------------------------------------------
An error occurred in line <3561> of file </home/tingchang/dealii-9.7.0/source/dofs/dof_handler_policy.cc> in function
    dealii::internal::DoFHandlerImplementation::Policy::{anonymous}::communicate_dof_indices_on_marked_cells<2, 2>(const dealii::DoFHandler<2, 2>&, std::vector<bool>&)::<lambda(const auto:243&, const auto:244&)>::<lambda(auto:245&, auto:246)> [with auto:245 = unsigned int; auto:246 = unsigned int*]
The violated condition was:
    (stored_index == invalid_dof_index) || (stored_index == *received_index)
Additional information:
    This exception -- which is used in many places in the library --
    usually indicates that some condition which the author of the code
    thought must be satisfied at a certain point in an algorithm, is not
    fulfilled. An example would be that the first part of an algorithm
    sorts elements of an array in ascending order, and a second part of
    the algorithm later encounters an element that is not larger than the
    previous one.
   
    There is usually not very much you can do if you encounter such an
    exception since it indicates an error in deal.II, not in your own
    program. Try to come up with the smallest possible program that still
    demonstrates the error and contact the deal.II mailing lists with it
    to obtain help.

```

I am thinking that some parts of my code might be improved to aviod this exception, although i have seen `There is usually not very much you can do if you encounter such an
    exception`.

[Tingchang Yin]

I realize that maybe this is because the mesh is too irregular. I have a lot of the following features as shown in the picture below (enclosed by red circles):

  

I think these shared nodes make the dof problemtic? But why one single processor is find?

[Wolfgang Bangerth]

Tingchang:
yes, there is probably little you can do about this -- some code
encounters a situation it does not expect, and unless the code is fixed,
there is not much you can do. Like you, I suspect that it is a situation
that has to do with points where two cells meet at a vertex without
sharing a whole face. This is uncommon enough in real meshes that I'm
not surprised that there are bugs.

The code you attached has 470 lines of code and your example shows that
it runs for 130 cycles. Do you think you can come up with a "minimal
working example", or at least a much reduced but self-contained code
that shows the problem? It does not actually have to do anything useful,
just (i) be as short as possible to make it easy to understand what it
is doing, and (ii) trigger the error.

Best
W.