Dump and restore crash with HTG output and MPI

[Joshua Christopher]

Hello,

I am experiencing crashes when I try to restore from a dump file. I am using the output_htg file from here: http://basilisk.dalembert.upmc.fr/sandbox/sander/output_htg.h

As a MWE, I took the Tangaroa example case and added output to the HTG format. Please see the attached file for my exact changes (I changed a few other small things that don't cause problems). I've also attached a back trace of where the crash is happening.

I'm afraid I don't really know enough about Basilisk to properly track this down on my own. Is there a known incompatibility between HTG output and dump/restore with MPI?

Some more information:

1. I am following the steps described in the section "Running in parallel on Irene" in the Tangaroa case. Specifically, I am running in serial until the dump at time step zero occurs, then I do a restore with MPI using the dump-0 file.

2. To cause the crash, I run with MPI for 60 steps, then dump again. I try to restore using the dump-60 file but the program crashes during the restore function.

3. The crash happens both with and without mesh adaptivity.

4. The crash happens no matter the number of processors used with MPI.

Thank you for any help,

Joshua

[Linkai]

Hello Joshua,

 

I have also met this problem. 

 

From my observation, there may be some conflicts between output_htg(..., (vector *) {u}, ...) and dump().  With the presence of output_htg(..., (vector *) {u}, ...),  the dump file is generally wrong.  

 

I tried to use output_htg(..., (vector *) {uf}, ...)  to avoid this problem. It seems to work.

 In my code, I use something like: 

- output_htg((scalar *) {f, T, l2, pid},(vector *){uf}, path, name, i, t); 

- dump (name, list = (scalar *){u.x, u.y, u.z, g.x, g.y, g.z, f, T, pid});

Pls let me know if you/anyone can have a real fix.

Best regards

Linkai