Potential heap memory error after cluster update

[陈金石]

Hi everyone,

I am running two-phase (air & water) VOF DNS simulation on Basilisk. I have run these kinds of simulations for about a year without issues. However, after our university update our cluster two weeks ago, the simulations start to report heap memory type errors, causing task failures. I have done some tests and so far I found the following patterns:

1. The identical code has been run successfully before the slurm update.

2. The errors does not occur in the first time step, but rather "random" in the later time steps, where I have run hours into the simulation.

3. Recompiling the code does not fix the issue.

4. There are more than one type of errors reported when I run the same code with one different physical parameter values, but all seem to point to heap memory errors. 

5. I am trying to reproduce the errors with valgrind checker but it hasn't given me meaningful message yet, since it occurs a bit randomly and not at the first time stamp.

The errors that I have encountered are listed below:

double free or corruption (out) (last log: properties src/two-phase-generic.h:83)

free(): invalid pointer (last log: properties src/two-phase-generic.h:83)

malloc(): unaligned tcache chunk detected (last log: outputting customized files)

munmap_chunk(): invalid pointer (last log: acceleration src/iforce.h:45

double free or corruption (!prev)   (last log:  properties src/two-phase-generic.h:83)

I have asked our cluster staff and they said that it is likely because there is a bug in the code that previous versions of clusters didn't catch, but with the new update (especially security updates) the cluster is able to catch these errors. 

I am wondering whether anyone has experienced this kind of errors before and if so, what could be the cause of these and where should I look for. Any tips and suggestions are greatly appreciated! 

Best,

Jinshi

[Stephane Popinet]

Hi Jinshi,

Could you try to change the lines at src/grid/tree.h:26 to

#include "memindex/range.h"
//#include "memindex/virtual.h"

recompile, rerun etc. and see if this changes anything?

cheers,

Stephane

[kevin.a....@gmail.com]

I'm running Basilisk on Windows using Cygwin.  After an updated Basilisk installation in the past few months, my runs were terminating with segmentation faults.  I determined they were caused by the introduction of virtual.h, and I reverted to range.h as suggested by Stephane.  I had been meaning to post this info earlier. 

Kevin McTaggart 

[Stephane Popinet]

Hi Kevin,

Thanks for the feedback. However I doubt that this is the same issue. I
am not surpised that using Unix Virtual memory with cygwin and Windows
virtual memory does not work. But (I hope) Jinshi's cluster is not
running on windows.

Of course a bug in the way Basilisk uses virtual memory is not excluded.

It would be useful to also have other people reporting if they have had
issues (or not) when using the recent version of Basilisk on other clusters.

cheers,

Stephane

[陈金石]

Hi Stephane and Kevin,

Thank you so much for the suggestions! Unfortunately my tree.h is already using range.h rather than virtual.h, probably because I haven't updated the source code since last year. 

I have tested some revisions over the weekend and it seems that the errors disappeared once I disable all the file outputs (including writing movies to .ppm, data files and figures). I am working on revising output codes and rerun (unfortunately our clusters are pretty congested recently...) I will update with you if I have any progress. It would be greatly appreciated if you have any suggestions on this!

Best,

Jinshi