Spurious Current Introduced by Parallelism (MPI)?

[Jeffery Y]

Hello all,

I encountered a problem for VOF simulation. Long story short, the expected steady-uniform free-surface flow down an inclined channel cannot maintain the unchanged fluid thickness when parallel running (MPI) is used. On the contrary, this problem does NOT exist when the serial running is used. The steady-uniform flow is very similar with the sample codes in the sandbox M1EMN's example, Vatsal's example, but for me I have the free-surface captured by VOF scheme.

Here are the details for my problem. I am interested in the hydrodynamic instability in the open-channel flow and I would like to investigate the evolution of an initial disturbance imposed on the steady-uniform flow. Therefore, the initial condition is a localized half sinusoidal disturbance imposed on the steady-uniform flow of non-Newtonian fluids. The maintenance for steady-uniform flow in front of the disturbance is of great importance to this problem (this is also expected from the theory, after all the flow follows the analytical solution of the steady-uniform flow). I run the simulation in parallel and in serial. The mesh adaptation uses two methods: (1) Basilisk's built-in wavelet function. (2) Oystein's adaptation function uniformly refining the free-surface interface. However, I found that the parallelism somehow introduced spurious current on the steady-uniform flow in front of the hump disturbance. The spurious current cannot be tolerated here, since it can grow with time and finally significantly pollute the simulation. The free-surface profiles at t=0.25s for parallel runs and serial run are plotted below. This weird phenomenon is unexpected and I think this spurious current is ONLY related to parallel running instead of the Poisson iterator, viscosity term or VOF scheme. Because everything works well in serial and the spurious current does not appear in serial. I guess the potential reason is the communication between processors is not reasonably good when the partition between two processors cuts the free-surface.

I have tried to make my original code as simple as possible. Attached are my source code (roll-wave.c) and the bash script for running the code in parallel (runCase.sh). 

I greatly appreciate your patience for reading my question. Any solution and comment is welcome.

Boyuan Yu

[aniszewski]

Hello,

two questions straight off the bat.

1. Did you try with OpenMp? That would rule out MPI communicaiton errors.

2. Did you try with (any kind of) parallelism, but without adaptation? Just comment out your adapt event?

cheers

voitek

[Jeffery Y]

Hello  Voitek  ,

Thanks for your inspiring question. 

First of all, I tried to use OpenMP for parallel simulation, but I did not succeed with parallelism and the code still run in serial for some reason. The major reason for this failure is due to the lack of examples in Basilisk website on how to use OpenMP. So, may I ask if you can kindly provide some sample codes on how to use OpenMP to parallelize Basilisk? And another question: have you improved the parallel performance when you switched to OpenMP from MPI?

Secondly, I have not tried to run the simulation without adaptation. But this is a very good piece of advice. I will try the static mesh for the simulation.

Boyuan Yu

[aniszewski]

Concerning OpenMP.

yes, please compile with something like this:

qcc  -std=c99 -fopenmp -Wall -O2 roll-wave.c -o rollwave -lm -grid=octree -L$BASILISK/gl -lglutils -lfb_tiny

(the last 3 argumens are, of course, optional for runtime visualization).

Once compiled, you just do

./rollwave

in the terminal to start it. This however will not let you control how many threads run, such control is obtained via an environmental variable:

 export OMP_NUM_THREADS=N

to get N threads.

The reason I asked is that OpenMP is shared memory parallelism, that means it rules out the errors that can take place when the typical MPI comands (mpi_sendrecv()) are used in the code. That said, I don't believe there actually are any MPI "errors" in basilisk, it's just a way to debug your issue...

The suggestion with adaptation has a similar goal.

good luck

v

[Jeffery Y]

Hello Voitek ,

Thank you for your comment and code sharing. I have tried OpenMP using your method. Surprisingly, the spurious current disappears. Maybe that is due to different parallel mechanisms of OpenMP and MPI. And again, I appreciate your useful and valuable solution. But I still think there are some small "flaws" in the MPI code in Basilisk. Perhaps, my setup is the evidence for it. I also duplicated the same case in Gerris. Using Gerris's MPI parallelism, the spurious current does not appear. As is known to us all, Gerris uses the domain decomposition strategy, which is different from the parallel strategy in Basilisk.

Boyuan Yu