Gravitational wave module

[mgur...@andrew.cmu.edu]

Hi everyone,

I'm encountering an issue with the /sample/GravitationalWaves commands. When I run pc_start, it takes about 70 seconds even for 16^3 or 8^3 simulations. If I then run pc_run immediately after pc_start completes, it executes in about a second. However, if I wait 10 seconds or more before running pc_run, it again takes around 70 seconds to complete.

I haven't experienced this issue with other samples I've tested.

Any suggestions on how to resolve this behavior would be greatly appreciated.

all the best,

Murman

[Philippe-A. Bourdin]

Hello Murman,
what you describe seems very unusual and I do not know of any such behaviour. Did your try the same on a different system, in partucular on a Linux-based computing cluster?
Thanks and best greetings,

Philippe.

[mgur...@andrew.cmu.edu]

Hi Philippe,

Thanks for the advice. I am running Pencil locally on a Linux-based machine.

I think I found the cause of the problem. After changing the value of `MPICOMM` to `nompicomm` in the file `src/Makefile.local`, both the `pc_start` and `pc_run` commands execute in a fraction of a second, even for the larger resolution of 64^3.

However, when I run the code with `mpicomm`, `pc_build` gives a bunch of warnings similar to:

```
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/INTEGER(8)).
mpicomm.f90:366:45:

  366 |         call MPI_COMM_SPLIT(MPI_COMM_PENCIL, int(iproc/ncpus), mod(iproc,ncpus), MPI_COMM_GRID, mpierr)
      |                                             1
......
11166 |       call MPI_COMM_SPLIT(MPI_COMM_PENCIL, lwrite_slice_r, iproc, MPI_COMM_RSLICE, mpierr)
      |                                           2  
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/LOGICAL(4)).
mpicomm.f90:246:38:

  246 |       call MPI_TYPE_SIZE(MPI_INTEGER, size_of_int, mpierr)
      |                                      1
......
 4417 |       call MPI_TYPE_SIZE(mpi_precision, sizeofreal, mpierr)
```

Despite these warnings, the code eventually executes, and `pc_start` and `pc_run` produce data, but they are slow.

Any further suggestions would be appreciated.

---

[Philippe-A. Bourdin]

Hello Murman and Matthias,
thanks for the hint. Even though, these are just warnings, they should get fixed. Matthias, this is likely your code, can you have a look?

Honestly, I do not think these warnings would lead to such a delay at code startup. Very likely this is caused from something else.
How many CPU cores do you try to use - and how many do you have? Which MPI library are you using?

I would recommend OpenMPI v3 or v4 for your private machine, and not use more CPUs than you have phyiscally.

And I would not suggest using a Linux virtual machine on Windows, because you loose performance.

Thanks and best greetings,

Philippe.