Problems in solving two PDE systems using MPI+Thread parallel computing

Jack

unread,

May 19, 2017, 3:52:39 AM5/19/17

to deal.II User Group

Dear all,

I’m trying to solve the thermal diffusion and Stokes flow problem simultaneously, similar to the step-32. I opened two threads to solve thermal diffusion and Stokes equations during solving by linear solvers (the former is solved by CG solver and the latter is solved by GMRES solver) after assembling the matrices and RHS. I encountered a problem when I using parallel computing in combination with distributed and shared memory.

The code is like following:

assemble_thermal_diffusion_system();
assemble_stokes_flow_system ();
Threads::Task<void> task =
Threads::new_task (&Problem<dim>::solve_thermal_diffsuion, *this);
solve_stokes_flow ();
task.join();

The program ran successfully with single process and showed a high performance of thread parallel computing, unfortunately, when I ran the program using several processes I got following error. I have googled this error and can not find any solution. Could someone help me?

Thanks in advance!

[warn] opal_libevent2021_event_base_loop: reentrant invocation.  Only one event_base_loop can run on each event_base at once.

[warn] opal_libevent2021_event_base_loop: reentrant invocation. Only one event_base_loop can run on each event_base at once.

Thank you very much.

Best,

Jack

Bruno Turcksin

unread,

May 19, 2017, 8:25:55 AM5/19/17

to deal.II User Group

Jack,

are your solvers using MPI? This looks similar to this problem https://github.com/open-mpi/ompi/issues/1081 Which version of MPI are you using? How do you initialize MPI? MPI_InitFinalize set MPI_THREAD_SERIALIZED which "tells MPI that

we might use several threads but never call two MPI functions at the same time." This is not want you want if your solvers use MPI. Then you need to initialize MPI and TBB yourself and use MPI THREAD MULTIPLE.

Best,

Bruno

Jack

unread,

May 19, 2017, 10:06:44 PM5/19/17

to deal.II User Group

Hi Bruno,

I appreciated it very much for your responses.

The Matrices and Vectors are of TrilinosWrappers, the solvers should use MPI, too. I used the OpenMPI 1.8.1. The release date is Apr 22 2014, later than the post on the Github. I initialized the MPI as following

try

    {
      using namespace dealii;
      using namespace problem;
      Utilities::MPI::MPI_InitFinalize mpi_initialization (argc, argv,
numbers::invalid_unsigned_int);
      Problem<2> app;
      app.infofile = argv[1];
      app.run ();
}

So, It seems this error is because I called two functions which use MPI during solving thermal diffusion and stokes flow equations.

According to this post https://github.com/open-mpi/ompi/issues/1081

“It turns out that if one tries to use Open MPI with MPI THREAD MULTIPLE (at least master) and actually has multiple threads making concurrent MPI calls, one gets a flood of messages of the following ilk:

opal_libevent2022_event_base_loop: reentrant invocation. Only one event_base_loop can run on each event_base at once.

showing up in stderr.

This needs to be fixed before release of v2.x”

I should update my OpenMPI to the latest version and such problem will be avoided and I do not need to change my code for initialization of MPI and TBB myself?

Thanks for your kind help!

Regards,

Jack

在 2017年5月19日星期五 UTC+8下午8:25:55，Bruno Turcksin写道：

Bruno Turcksin

unread,

May 20, 2017, 1:33:51 PM5/20/17

to dea...@googlegroups.com

Jack,

2017-05-19 22:06 GMT-04:00 Jack <yche...@gmail.com>:
> I should update my OpenMPI to the latest version and such problem will be
> avoided and I do not need to change my code for initialization of MPI and
> TBB myself?

Unfortunately, you wil need to initialize MPI yourself or you can
change this line
https://github.com/dealii/dealii/blob/master/source/base/mpi.cc#L321
in
dealii/source/base/mpi.cc (replace MPI_THREAD_SERIALIZED by
MPI_THREAD_MULTIPLE) and recompile deal. Here is some explanation on
the different options available
https://www.open-mpi.org/doc/current/man3/MPI_Init_thread.3.php If you
still have the problem then you can try to update you version of
OpenMPI.

Best,

Bruno

Jack

unread,

May 21, 2017, 1:46:38 AM5/21/17

to deal.II User Group

Hi Bruno,

I changed the corresponding code and remake the library again, unfortunately, the same warn came out as before.

I changed this in another workstation which uses MPICH2, it also failed during solving the two systems.

I'm trying to update my OpenMPI to v2.1, but in the latest version, it does not has the libs of libmpi_usempi.so which is ignored since v2.0.

Thank you for your help

Regards,

Jack

在 2017年5月21日星期日 UTC+8上午1:33:51，Bruno Turcksin写道：

Wolfgang Bangerth

unread,

May 22, 2017, 9:50:32 AM5/22/17

to dea...@googlegroups.com

Jack,
unrelated to the question about versions of MPI libraries, when you solve two
linear systems on separate threads, you need to pay attention that the two
solvers (and all associated objects such as matrices, vectors, etc) use
separate MPI communicators. Otherwise you will have the two solvers send
messages on the same communicator at the same time, and this is certain to
lead to trouble.

The way to do this is to clone the MPI communicator you use for your overall
problem once for each linear system. This way, each linear system uses its own
communication channel, and one solver can not get in the way of the other.

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

Jack

unread,

May 23, 2017, 5:08:05 AM5/23/17

to deal.II User Group, bang...@colostate.edu

Hi Prof. Wolfgang，

Thanks so much！

“The way to do this is to clone the MPI communicator you use for your overall
problem once for each linear system. ”

That means for my problem I have to copy the Vector and Matrix of one linear system（either the thermal diffusion or stokes flow） to another Vector and Matrix which are built with a clone of present communicator before starting a thread to solve this linear system.

And then I have to copy the solution of this linear system to the targeted Vector which is built using the present MPI communicator for assembling systems.

By the way is it possible to copy Vector and Matrix of TrilinosWrappers to another ones those are initialized with another MPI communicator?

Thanks,

All best，

Jack

在 2017年5月22日星期一 UTC+8下午9:50:32，Wolfgang Bangerth写道：

Wolfgang Bangerth

unread,

May 23, 2017, 1:59:17 PM5/23/17

to Jack, deal.II User Group

Jack,

> “The way to do this is to clone the MPI communicator you use for your
> overall
> problem once for each linear system. ”
>
> That means for my problem I have to copy the Vector and Matrix of one
> linear system（either the thermal diffusion or stokes flow） to another
> Vector and Matrix which are built with a clone of present communicator
> before starting a thread to solve this linear system.
>
> And then I have to copy the solution of this linear system to the
> targeted Vector which is built using the present MPI communicator for
> assembling systems.

That seems too complicated. Just set up the linear system (=matrix,
vectors) with a different communicator from the beginning.

> By the way is it possible to copy Vector and Matrix of TrilinosWrappers
> to another ones those are initialized with another MPI communicator?

I don't remember. It is certainly nor a usual operation. You'll have to
investigate whether copying also copies the communicator or not by
looking at the actual source.

It is not common to solve linear systems on different threads in
parallel when using MPI. I don't want to say that that is not a good
idea -- it seems like a good idea for sure because it can hide some of
the latency associated with MPI communication, but it is not usually
done. I suspect that you will not save much time compared to just
solving the two linear systems one after the other.

Jack

unread,

May 23, 2017, 9:24:41 PM5/23/17

to deal.II User Group, yche...@gmail.com, bang...@colostate.edu

Hi Prof. Wolfgang,

Many thanks!

Originally, I suspected that when solving two linear systems simultaneously by two threads would reduce time. But now it seems that this idea increases the complexity of communication between MPI communicators and coding, and also not surely to decrease computing time. I’m not going to try this idea at present. When I’m free, I will test its feasibility and post here the result.

Thank you very much!

All my best wishes,

Jack

在 2017年5月24日星期三 UTC+8上午1:59:17，Wolfgang Bangerth写道：

Wolfgang Bangerth

unread,

May 24, 2017, 10:38:57 AM5/24/17

to Jack, deal.II User Group

On 05/23/2017 07:24 PM, Jack wrote:
> Originally, I suspected that when solving two linear systems simultaneously by
> two threads would reduce time. But now it seems that this idea increases the
> complexity of communication between MPI communicators and coding, and also not
> surely to decrease computing time. I’m not going to try this idea at present.
> When I’m free, I will test its feasibility and post here the result.

I think we'd all be curious about the result. As I mentioned, it's not a
stupid idea, but it requires significant experience in both thread and MPI
programming to deal with the errors you're likely going to get if you get
anything wrong.

Jack

unread,

May 25, 2017, 10:31:18 PM5/25/17

to deal.II User Group, yche...@gmail.com, bang...@colostate.edu

Many thanks for your encouragement.

I’m neither an expert on MPI nor on thread. I underestimated the difficulty.

This may take a very long time for me to figure out.

Hopefully some experts on these parallel computing regimes would like to address such a curious problem soon.