Problem in running in parallel

[Suhas Jain]

Hi all,

 I modified the atomization example and changed the boundary conditions such that the jet is entering the domain from -z. But I am not able to run it in parallel, it runs in serial. I have attached the script. Am I missing something?

I use this to compile:

qcc -source -grid=octree -D_MPI=1 jet.c 

mpicc -Wall -std=c99 -O2 _jet.c -lm -o jet

Suhas

[Wojciech Aniszewski]

The commands you cite will parse (first line) and compile (second) the jet.c code,
creating a 'jet' executable which then has to be launched with mpirun. How do you
run the code? example launch could look like:

mpirun -np 8 jet

(note: mpirun has many aliases on various systems)

Since you just used atomization.c, it is possible you just tried to replicate recipes
found there. They (recipes) are however written for "Mesu" and "Occigen" supercomputers
where MPI (distributed memory) parallelism is used. If however you're running locally,
e.g. on a laptop or a small (less than 32 cores) machine, the shared memory parallelism
(OpenMP) makes much more sense. (If you don't know what I'm talking about, then probably
you also should use OpenMP).

Citing (http://basilisk.fr/sandbox/bugs/FloatingPoint), compilation with -fopenmp switch is done typically like:

qcc -O2 -fopenmp -g -Wall -o jet jet.c -lm

(other linking commands and switches might be necessary e.g. when using bview) which is then launched by simply typing

./jet .

good luck
w

[Antoon van Hooft]

Hallo Suhas,

> Am i missing something?

Have you read the frequently asked questions entry:

http://basilisk.fr/sandbox/jmf/faq#minimal-example

A minimal error-reproducing example could look like,

scalar s[];
int main() {
 init_grid (32);
 mask (y > 0.5 ? top : none);
 boundary (all);
}

Then it would be easy to identify that even tough it compiles, mask does

not work with MPI (and does never compile using a multi/Cartesian grid). see,

http://basilisk.fr/src/COMPATIBILITY

under "Solids"

Note the importance of this effort as currently your question hints at other causes.

Antoon

[Suhas Jain]

Hi Wojciech,

Thank you. I run the code using: mpirun -np 8 ./jet. 

I would prefer not to use open MP because, I am only setting up the case for now on my workstation. But I have installed Basilisk on Mira Supercomputer at Argonne National Lab and would want to eventually run large scale simulations over there.

Hi Antoon,

 Thanks a lot for those links! They are really helpful. So the only way to use a rectangular/cuboid domain and run a parallel simulation in Basilisk now is to use multigrid3D grid (http://basilisk.fr/src/Tips#non-cubic-domains)? But would this work with two-phase flows? I haven't been able to call "grid/multigrid3D.h" and "two-phase.h" headers at the same time. Does it mean I cannot use rectangular domain for two-phase flows? 

For example, a code like this doesn't compile:

#include "grid/multigrid3D.h"

#include "two-phase.h"

 int main() {

}

Regards,

Suhas

[Antoon van Hooft]

Hallo Suhas,

With or without the multigrid3d, the code does not compile. See the first sentence in "two-phase.h" or any the many examples.

But I doubt that this is the original error you aim to reproduce.

Antoon

[Suhas Jain]

Hi Antoon,

 Yeah, "two-phase.h" requires "navier-stokes/centered.h". But multigrid3D doesn't compile with "navier-stokes/centered.h". 

For example I am not able to compile this following code either:

#include "grid/multigrid.h"

#include "navier-stokes/centered.h"

int main() { 

}

I am trying to setup a jet atomization case on a non-cubic domain. What headers should I be using? My guess is "multigrid3D.h", "two-phase.h" and "navier-stokes/centered.h". But I am not able to get these compile together!

Thanks,

Suhas

[Antoon van Hooft]

Could you indicate how you compile and what compilations-error message(s) you are getting?  Your code should not raise any errors.

You could consider having a look at the website to familiarize yourself with setting up and running cases etc. with,

http://basilisk.fr/src/examples/atomisation.c

you hit the ground running....

Apart from the possibility to change the domain's aspect ratio with the MPI multigrid dimensions (nx, ny nz), you could have a look at this brief inter-comparison:

http://basilisk.fr/sandbox/Antoonvh/test_straight_boundaries.c

Where currently(!) the only strategy able to run with MPI in 3D is dubbed "Stephane's trick".  

FYI, a similar test with a curved internal boundary is presented here:

http://basilisk.fr/sandbox/Antoonvh/test_curved_boundaries.c

Antoon

[Suhas Jain]

Hi Antoon,

 Thanks for those links, but I would prefer to use multigrid, since I am planning on using a uniform mesh. I assume multigrid implementation is faster for uniform grids compared to octree along with "Stephane's trick".

 I use this command to compile:

 qcc -Wall -O2 -std=c99 -lm test.c

"test.c" code is the following:

#include "navier-stokes/centered.h"

#include "grid/multigrid.h"

int main() {

}

Compilation errors that I get are in the file attached.

Thank you,

Suhas 

[Antoon van Hooft]

Hallo Suhas,

Per cell, a multigrid if typically faster and requires less memory.     

For your error: The sequence is of the essence. Either you can catch up on the C-programming language or stick more closely to the examples and tutorial. 

this code:

#include "grid/multigrid.h"
#include "navier-stokes/centered.h"
int main(){
}

compiled like so

$ qcc -Wall -O2 test.c -lm

Should not raise any error and the executable `works' as intended. 


Antoon

[Suhas Jain]

Hi Antoon,

 Thank you, I can compile and run my jet now without any errors. I was not aware of the sequence!

 

 I have another question regarding writing output to either vtk/vtu/vts. I have a non-cubic domain with multigrid solver and MPI. How do I save the data for this configuration? 

output_vtu_foreach doesn't seem to work with multigrid (I suppose this is only for Octree).

Thanks,

Suhas