Interaction with the boundary description for a parallel distributed triangulation

[Seyed Ali Mohseni]

Hi,

I assume there is a mistake within the documentation of parallel::distributed::Triangulation<dim, spacedim> class. Below the "detailed description" part within the second box there is written &MyGeometry<dim>. Doesn't it have to be &MyClass<dim>?

With regard to the same topic: I would like to know, if it is possible to read from a .msh input file and refine afterwards for a parallel distributed mesh. Hence, using the procedure from the documentation by means of the set_boundary_ids() function there should be a way to reuse the predefined boundary ID structure from the input file, am I right? How can we reuse the initial boundary ID setting efficiently without much effort?  

In my case mesh refinement works for 1 core, but for several cores it fails with the following error:

Assertion failed in file ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 560: !vc_ch->recv_active

internal ABORT - process 0

Assertion failed in file ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 560: !vc_ch->recv_active

internal ABORT - process 4

I assume this problem rises from the boundary description since when I refine adjacent cells to other nodes not directly related to the boundary, it also works in parallel mode.

Kind regards,

S. A. Mohseni

[Jean-Paul Pelteret]

Dear Seyed,

I assume there is a mistake within the documentation of parallel::distributed::Triangulation<dim, spacedim> class. Below the "detailed description" part within the second box there is written &MyGeometry<dim>. Doesn't it have to be &MyClass<dim>?

Yes, this looks like a copy-paste error in the documentation. Would you like to submit a patch to correct it? 

 

With regard to the same topic: I would like to know, if it is possible to read from a .msh input file and refine afterwards for a parallel distributed mesh. Hence, using the procedure from the documentation by means of the set_boundary_ids() function there should be a way to reuse the predefined boundary ID structure from the input file, am I right? How can we reuse the initial boundary ID setting efficiently without much effort?  

When you refine cells at a boundary, the child cell's faces will inherit the boundary and manifold ID's of the parent cells regardless of whether the mesh was read in or created using the GridGenerator functions. 

 

In my case mesh refinement works for 1 core, but for several cores it fails with the following error:

Assertion failed in file ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 560: !vc_ch->recv_active

internal ABORT - process 0

Assertion failed in file ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 560: !vc_ch->recv_active

internal ABORT - process 4

 

This file is not a part of the deal.II project, so unless you can provide more details (i.e. a proper stacktrace) then we can only assume that the problem does not relate to deal.II itself.

Kind regards,

Jean-Paul

[Seyed Ali Mohseni]

Dear Jean-Paul,

 

Yes, this looks like a copy-paste error in the documentation. Would you like to submit a patch to correct it? 

Sure. How and where can I submit a patch?  

When you refine cells at a boundary, the child cell's faces will inherit the boundary and manifold ID's of the parent cells regardless of whether the mesh was read in or created using the GridGenerator functions. 

But according to the documentation this is only true for sequential triangulations and not for distributed triangulations. It is more safe to update boundary IDs while the distributed::parallel::Triangulation::execute_coarsening_and_refinement function is running.

I cite: 

This is not necessary for sequential triangulations because, there, these flags are inherited from mother to child cell and remain with a cell even if it is refined and the children are later coarsened again, but this does not hold for distributed triangulations.

 See: https://www.dealii.org/developer/doxygen/deal.II/classparallel_1_1distributed_1_1Triangulation.html

There is a slight problem using the approach written there. I am trying to compile the following:

triangulation.signals.post_refinement.connect(std_cxx11::bind(&SolidMechanics<dim>::set_boundary_ids, std_cxx11::cref(*this), std_cxx11::ref(triangulation)));

Then I receive the following compilation error:

/usr/include/c++/5/functional(1426): error: static assertion failed with "Wrong number of arguments for pointer-to-member"

        static_assert(_Varargs::value

        ^

          detected during:

            instantiation of class "std::_Bind_check_arity<_Tp _Class::*, _BoundArgs...> [with _Tp=void (), _Class=SolidMechanics<2>, _BoundArgs=<std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>>]" at line 1440

            instantiation of class "std::_Bind_helper<_SocketLike, _Func, _BoundArgs...> [with _SocketLike=false, _Func=void (SolidMechanics<2>::*)(), _BoundArgs=<std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>>]" at line 787 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

            instantiation of "void SolidMechanics<dim>::create_geometry() [with dim=2]" at line 544 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

            instantiation of "void SolidMechanics<dim>::do_initial_timestep() [with dim=2]" at line 1440 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

            instantiation of "void SolidMechanics<dim>::run() [with dim=2]" at line 1472 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

/home/seyedali/programming/c++/libraries/boost-1.62.0/include/boost/function/function_template.hpp(159): error: no instance of overloaded function "std::_Bind<_Functor (_Bound_args...)>::operator() [with _Functor=std::_Mem_fn<void (SolidMechanics<2>::*)()>, _Bound_args=<std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>>]" matches the argument list

            object type is: std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>

            BOOST_FUNCTION_RETURN((*f)(BOOST_FUNCTION_ARGS));

            ^

/usr/include/c++/5/functional(1171): note: this candidate was rejected because at least one template argument could not be deduced

  operator()(_Args&&... __args) const volatile

  ^

/usr/include/c++/5/functional(1157): note: this candidate was rejected because at least one template argument could not be deduced

  operator()(_Args&&... __args) volatile

  ^

/usr/include/c++/5/functional(1143): note: this candidate was rejected because at least one template argument could not be deduced

  operator()(_Args&&... __args) const

  ^

/usr/include/c++/5/functional(1129): note: this candidate was rejected because at least one template argument could not be deduced

  operator()(_Args&&... __args)

  ^

          detected during:

            instantiation of "void boost::detail::function::void_function_obj_invoker0<FunctionObj, R>::invoke(boost::detail::function::function_buffer &) [with FunctionObj=std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>, R=void]" at line 936

            instantiation of "void boost::function0<R>::assign_to(Functor) [with R=void, Functor=std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>]" at line 727

            instantiation of "boost::function0<R>::function0(Functor, boost::enable_if_c<<expression>, int>::type) [with R=void, Functor=std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>]" at line 1073

            instantiation of "boost::function<R ()>::function(Functor, boost::enable_if_c<<expression>, int>::type) [with R=void, Functor=std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>]" at line 1125

            instantiation of "boost::enable_if_c<<expression>, boost::function<R ()>::self_type &>::type boost::function<R ()>::operator=(Functor) [with R=void, Functor=std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>]" at line 160 of "/home/seyedali/programming/c++/libraries/boost-1.62.0/include/boost/signals2/detail/slot_template.hpp"

            instantiation of "void boost::signals2::slot<R (Args...), SlotFunction>::init_slot_function(const F &) [with SlotFunction=boost::function<void ()>, R=void, Args=<>, F=std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>]" at line 85 of "/home/seyedali/programming/c++/libraries/boost-1.62.0/include/boost/signals2/detail/slot_template.hpp"

            instantiation of "boost::signals2::slot<R (Args...), SlotFunction>::slot(const F &) [with SlotFunction=boost::function<void ()>, R=void, Args=<>, F=std::_Bind<std::_Mem_fn<void (SolidMechanics<2>::*)()> (std::reference_wrapper<const SolidMechanics<2>>, std::reference_wrapper<dealii::parallel::distributed::Triangulation<2, 2>>)>]" at line 787 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

            instantiation of "void SolidMechanics<dim>::create_geometry() [with dim=2]" at line 544 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

            instantiation of "void SolidMechanics<dim>::do_initial_timestep() [with dim=2]" at line 1440 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

            instantiation of "void SolidMechanics<dim>::run() [with dim=2]" at line 1472 of "/home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc"

compilation aborted for /home/seyedali/fe_models/deal.II/solid_mechanics/solid_mechanics.cc (code 2)

make[2]: *** [CMakeFiles/solid_mechanics.dir/solid_mechanics.cc.o] Fehler 2

make[1]: *** [CMakeFiles/solid_mechanics.dir/all] Fehler 2

CMakeFiles/solid_mechanics.dir/build.make:62: die Regel für Ziel „CMakeFiles/solid_mechanics.dir/solid_mechanics.cc.o“ scheiterte

make: *** [all] Fehler 2

CMakeFiles/Makefile2:195: die Regel für Ziel „CMakeFiles/solid_mechanics.dir/all“ scheiterte

Makefile:83: die Regel für Ziel „all“ scheiterte

I already tried using std::bind by including the <functional> library instead of #include <deal.II/base/std_cxx11/bind.h> , but it also won't compile correctly.

Maybe there is something missing in the connect function?

Kind regards,

Seyed Ali

[Seyed Ali Mohseni]

I think I found the problem regarding the mesh refinement issue in parallel mode. Probably, it is not even related to the boundary indicators. From what I understood, there is a problem with the MPI_Allreduce function I mentioned in the previous post: https://groups.google.com/forum/#!topic/dealii/aDEP1MSS0JA. Hence, Daniel was totally right. I cite:

- n_vertices() only gives you the number of vertices for one process, not the global one. In particular, you can't rely on the fact that this is the same for all processes. Furthermore, you (probably) don't initialize all the values of your struct. This might be the reason for the large numbers you are observing.

I observed that my total number of nodes ( triangulation.n_vertices() ) and elements ( triangulation.n_active_cells() ) remain the same even after refinement, but the total number of degrees of freedom ( dof_handler.n_dofs() ) are updated. Is there a possibility to obtain the total number of nodes after refinement/coarsening or do I have to use the DoF information somehow?

Sorry for the confusion.

Best,

Seyed Ali 

[Seyed Ali Mohseni]

I solved the problem. The following line had to be fixed in order to prevent the error due to the changing size of the vectors which were passed to MPI_Allreduce: 

int disp_size = dof_handler.n_dofs()/dim;

Nevertheless, I wonder, if there are any commands like dof_handler.n_dofs() which also update information in the case of mesh refinement.

Furthermore, the boundary ID issue remains. Although my refinement procedure works, yet I would like to understand why and when do I have to use the approach written in the documentation. The connect or bind functions are not compiled correctly. Maybe there is also some mistake in the documentation somewhere?

Best regards,

Seyed Ali

[Jean-Paul Pelteret]

Dear Seyed,

But according to the documentation this is only true for sequential triangulations and not for distributed triangulations.

Ah yes, of course, my mistake.

Sure. How and where can I submit a patch?  

You can clone the development version of deal.II, make a new branch, make the change and submit a pull request. Simple :-)

Kind regards,

Jean-Paul