[Gfs-devel] bug restarting simulations with mpi

[Daniel Fuster]

I would like to report I bug I find often (not always) restarting
simulations in parallel. The error I get looks like

[bazant:15919] *** Process received signal ***
[bazant:15919] Signal: Floating point exception (8)
[bazant:15919] Signal code: (7)
[bazant:15919] Failing at address: 0x7ff56bccd7fc
[bazant:15919] [ 0] /lib/libpthread.so.0(+0xeff0) [0x7ff569f44ff0]
[bazant:15919] [ 1]
/home/fuster/lib/libgfs3D-1.3.so.2(gfs_face_velocity_advection_flux+0x28c)
[0x7ff56bccd7fc]

As the message points out, the problem comes from the subroutine
gfs_face_velocity_advection_flux (called from
gfs_centered_velocity_advection_diffusion -> variable_sources).
Indeed, I have verified that the error is induced because
face->neighbor gives Floating point exception. Just for fun, if you
comment out the last lines of this subroutine, where face->neighbor is
used,

switch (ftt_face_type (face)) {
case FTT_FINE_FINE:
GFS_VALUE (face->neighbor, par->fv) += flux;
break;
case FTT_FINE_COARSE:
GFS_VALUE (face->neighbor, par->fv) += flux/FTT_CELLS;
break;
default:
g_assert_not_reached ();
}

the error dissapears (although obviously the result is no longer
correct). If then you insert a simple evaluation of face->neighbor as
this

if (face->neighbor) {
/* do nothing*/
}

the error appears again.....

I think that you need relatively heavy simulations to observe this
problem. I wonder if anybody has an idea about from where the error
comes from. I would say that before reaching this subroutine
face->neighbor values have been used in mac_projection

--
www.danielfuster.com

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gfs-devel mailing list
Gfs-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/gfs-devel

[Daniel Fuster]

ok, I found the bug. Although I need help for a proper correction

The problem was indeed that the values of par->fv were not properly
reset. In variable sources we find the funcion

gfs_domain_cell_traverse (domain, FTT_PRE_ORDER, FTT_TRAVERSE_LEAFS, -1,
(FttCellTraverseFunc) gfs_cell_reset, par->fv);

but it seems like some of the values used later in par->flux are not
reset. If I add an extra function to be sure that all the fluxes are
reset

gfs_domain_face_traverse (domain, FTT_XYZ,
FTT_PRE_ORDER, FTT_TRAVERSE_LEAFS, -1,
(FttFaceTraverseFunc) resetfaces, par);

where

static void resetfaces (const FttCellFace * face, const
GfsAdvectionParams * par)
{
GFS_VALUE (face->neighbor, par->fv) = 0.;
}

then the problem is solved.

Could this bug also cause problems at the boundaries during the
simulation? and most important, what is the proper way of reset the
variable in this case?

Thanks for your advice

Daniel



2012/10/2 Daniel Fuster <dfu...@gmail.com>:

[Daniel Fuster]

Just as an aside note, it is not clear to me if there is any similar
problem in gfs_tracer_vof_advection, where the flux is reset here:

gfs_domain_traverse_leaves (domain, (FttCellTraverseFunc) reset_fluxes, &p);

but I am not sure if the values at the gosh layer at the mpi
boundaries are reset (and if this really matters). As long as I reset
the fluxes explained in my previous email there is no crash here, but
the fluxes may be wrong.

best

[Stephane Popinet]

Hi Daniel,

> As the message points out, the problem comes from the subroutine
> gfs_face_velocity_advection_flux (called from
> gfs_centered_velocity_advection_diffusion -> variable_sources).

As you suggest, what seems to happen is that GFS_VALUE
(face->neighbor, par->fv) is somehow not initialised. If this is the
case, then it can contain any garbage (depending on what the memory
location has been used for previously) including invalid
floating-point numbers which then causes the Floating Point Exception
(FPE) error. This would also explain why the error is "random".

This is strange anyway since memory locations used for variables are
always allocated and initialised at the same time. This can be checked
by using valgrind which should not complain about unitialised memory.

Also why does this seem to happen only when restarting parallel simulations?

Another explanation may be that the boundary values (i.e. the
corresponding memory location) is initialised to e.g. G_MAXDOUBLE in
another part of the code (e.g. when the same memory location is used
for another temporary variable). Using this G_MAXDOUBLE value then
causes the FPE. One way to check this is to display the value of
GFS_VALUE (face->neighbor, par->fv) before it is used and/or add a
check like:

g_assert (GFS_VALUE (face->neighbor, par->fv) != G_MAXDOUBLE);

A problem with this explanation is that it does not account for the
randomness of the error...

As you see more digging is required.

cheers

Stephane

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov