3D hp dof handler with distributed triangulation generates error

[Xiukun Hu]

Hi everybody,

I've been trying to parallelize step-46 for a while (mainly by combining with step-40). Now I can make the code run in release mode and get the correct output, but still get (different kind of) errors in debug mode. After some efforts I finally reproduced one in a quite short code.

Code To Reproduce the Error:

#include <deal.II/base/utilities.h>
#include <deal.II/grid/tria.h>
#include <deal.II/grid/grid_generator.h>
#include <deal.II/distributed/tria.h>
#include <deal.II/dofs/dof_accessor.h>
#include <deal.II/hp/dof_handler.h>
#include <deal.II/hp/fe_collection.h>
#include <deal.II/hp/fe_values.h>
#include <iostream>


int main(int argc, char *argv[]) {
 using namespace dealii;

 Utilities::MPI::MPI_InitFinalize mpi_init_fin(argc, argv, 1);
 MPI_Comm mpi_comm(MPI_COMM_WORLD);

 constexpr int dim = 3;

 parallel::distributed::Triangulation<dim> mesh(mpi_comm);
 // Triangulation<dim> mesh;
 GridGenerator::subdivided_hyper_cube(mesh, 8, -1, 1);
 hp::DoFHandler dof_handler(mesh);

 hp::FECollection<dim> fe_collection(FE_Q<dim>(2), FE_Q<dim>(2));
 for (auto && cell:dof_handler.active_cell_iterators()){
   if (cell->is_locally_owned()) {
     cell->set_active_fe_index(0);
   }
 }

 dof_handler.distribute_dofs(fe_collection);

 std::cout << "Done.\n";

 return 0;
}

Error Message:

--------------------------------------------------------

An error occurred in line <3654> of file </Applications/deal.II-9.0.0.app/Contents/Resources/spack/src/deal.II-9.0.0/include/deal.II/dofs/dof_accessor.templates.h> in function

    unsigned int dealii::DoFCellAccessor<dealii::hp::DoFHandler<3, 3>, false>::active_fe_index() const [DoFHandlerType = dealii::hp::DoFHandler<3, 3>, lda = false]

The violated condition was:

    (dynamic_cast<const dealii::DoFHandler<DoFHandlerType::dimension,DoFHandlerType::space_dimension>*> (this->dof_handler) != nullptr) || (this->is_locally_owned() || this->is_ghost())

Additional information:

    You can only query active_fe_index information on cells that are either locally owned or (after distributing degrees of freedom) are ghost cells.

Stacktrace:

-----------

#0  2   libdeal_II.g.9.0.0.dylib            0x000000011842d31c _ZNK6dealii15DoFCellAccessorINS_2hp10DoFHandlerILi3ELi3EEELb0EE15active_fe_indexEv + 380: 2   libdeal_II.g.9.0.0.dylib            0x000000011842d31c _ZNK6dealii15DoFCellAccessorINS_2hp10DoFHandlerILi3ELi3EEELb0EE15active_fe_indexEv

#1  3   libdeal_II.g.9.0.0.dylib            0x00000001195af850 _ZN6dealii8internal2hp24DoFHandlerImplementation14Implementation13reserve_spaceILi3EEEvRNS_2hp10DoFHandlerILi3EXT_EEE + 304: 3   libdeal_II.g.9.0.0.dylib            0x00000001195af850 _ZN6dealii8internal2hp24DoFHandlerImplementation14Implementation13reserve_spaceILi3EEEvRNS_2hp10DoFHandlerILi3EXT_EEE

#2  4   libdeal_II.g.9.0.0.dylib            0x00000001195aeb06 _ZN6dealii2hp10DoFHandlerILi3ELi3EE15distribute_dofsERKNS0_12FECollectionILi3ELi3EEE + 630: 4   libdeal_II.g.9.0.0.dylib            0x00000001195aeb06 _ZN6dealii2hp10DoFHandlerILi3ELi3EE15distribute_dofsERKNS0_12FECollectionILi3ELi3EEE

#3  5   rep                                 0x000000010bb5e2fd main + 413: 5   rep                                 0x000000010bb5e2fd main

#4  6   libdyld.dylib                       0x00007fff6004f015 start + 1: 6   libdyld.dylib                       0x00007fff6004f015 start

--------------------------------------------------------

Calling MPI_Abort now.

...

Environment:

macOS High Sierra + llvm 9.1.0 + cmake 3.12.1 + deal.II 9.0.0

Observations:

Won't throw error if in any following situation:

• build type is release,
• dim = 2,
• use (non-distributed) Triangulation class, or
• run with only one processor.

The error happens inside distribute_dofs(). I'm not sure if I used the wrong classes/functions for parallel or there is a bug in dealii. I'd be very glad to see some examples/tutorials of using hp::* in distributed system, but couldn't find any yet.

Thanks ahead for any help.

Xiukun

[Wolfgang Bangerth]

On 08/23/2018 05:51 PM, Xiukun Hu wrote:
>
> I've been trying to parallelize step-46 for a while (mainly by combining with
> step-40). Now I can make the code run in release mode and get the correct
> output, but still get (different kind of) errors in debug mode. After some
> efforts I finally reproduced one in a quite short code.

There will likely be many more errors to come, and I would doubt that your
results will be correct even in release mode. The issue, fundamentally, is
that in deal.II 9.0, hp::DoFHandler does not yet work with
parallel::distributed::Triangulation. With some luck, this combination will be
available in deal.II 9.1, though.

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Xiukun Hu]

Thank you so much for your explanation, professor.