Metal mass not conserved?

[John Forbes]

Hi everyone,

I've been attempting to run some isolated galaxy simulations (using Nathan's AgoraRestart problem type) in something closely resembling the repository version of enzo 2.3. My parameters include:

RadiativeCooling = 1

FluxCorrection = 1

ConservativeInterpolation = 1

CorrectParentBoundaryFlux = 1

MultiSpecies = 1

DualEnergyFormalism = 1

HydroMethod = 0

RiemannSolver = 4 //HLLC

I've noticed that when I initialize all the gas in the simulation to be the same metallicity, after a short time during which no stars form (so there should be no change in metallicity anywhere), there are ~10% fluctuations in the metallicity field, possibly associated with grid boundaries. I also looked at the total metal mass in the simulation box, and over this short period of time, the total mass in the box changes by a factor of ~2e-11, whereas the total metal mass changes by ~5e-7. 

Any ideas where something might be going wrong?

Thank you,

John

[John Forbes]

A little more information for completeness:

Build information and parameter file:

http://pastebin.com/DiXsF7Xr

A slice of metallicity 4 Myr after the beginning of the simulation (color bar goes from .09 to .11 solar metallicity).

https://www.dropbox.com/s/d30e57gfebc1sot/metallicity.png

[John Forbes]

One more update, possibly related to the metallicity issue: this same simulation crashes with very high temperature gas after stars begin to form. The worst-offending cell is on a grid boundary but not a level boundary, at least by the time I got a snapshot. Here's a slice of the temperature centered on the highest-temperature cell (the other orientations look similar, i.e. everything in the vicinity is on the highest refinement level). Grid boundaries are shown in black, the white points are newly-formed star particles within 10 pc along the line of sight.

https://www.dropbox.com/s/vw1n1wayuh8hry8/temperature.png

As far as I know, this is a new issue for the PPM solver - I know Nathan has run into similar-looking problems at level boundaries, but in this instance everything appears to be at the same level.

Apologies for all the messages, especially if it turns out to be a simple mistake on my part!

Thank you,

John

[John Wise]

Hi John F,

Sorry that you're having problems with Enzo, but hopefully we can figure
out what's happening!

You might be encountering a bug in the flux correction across AMR grids.
Could you try re-running the simulation with

ConservativeInterpolation = 0

If this does not fix the problem with metal mass conservation, please try

FluxCorrection = 0

This will help us pinpoint what's causing the deviations in metallicity
when there is no star formation occurs.

Also, a few of us (Sam Skillman, Nathan Goldbaum, myself, and probably a
couple of others) have seen the unnaturally high temperatures at grid
boundaries. Here are a couple of threads describing these problems and
possible solutions.

https://groups.google.com/forum/#!topic/enzo-dev/K2Lls0FpsOA
https://groups.google.com/forum/#!topic/enzo-dev/vbf5Okn6_Jg

Let us know what you find. We're pretty eager to fix this outstanding
bug...

Cheers,
John

> --
> You received this message because you are subscribed to the Google
> Groups "enzo-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to enzo-dev+u...@googlegroups.com.
> To post to this group, send email to enzo...@googlegroups.com.
> Visit this group at http://groups.google.com/group/enzo-dev.
> For more options, visit https://groups.google.com/groups/opt_out.

--
John Wise
Assistant Professor of Physics
Center for Relativistic Astrophysics, Georgia Tech
http://cosmo.gatech.edu

[John Forbes]

Hi John,

Thanks for your quick response. I've re-run with every combination of ConservativeInterpolation and FluxCorrection. It looks like the metallicity issue is associated with FluxCorrection; here are the results:

Both on (the original sim)

-- Total mass changes by 2e-11

-- Metal mass changes by 5e-7

-- 10% features in metallicity field

FluxCorrection=1, ConservativeInterpolation=0

-- Total mass changes by 1e-11

-- Metal mass changes by 4e-7

-- 10% features in metallicity field

FluxCorrection=0, ConservativeInterpolation=1

-- Total mass and metal mass both change by 5.3e-7

-- No visible features at the 10% level

Both off

-- Total mass and metal mass change by 4.3e-7

-- No visible features at the 10% level

Thanks also for the pointers to the previous discussions -- it looks like I can try Sam's fix, Nathan's adjustment of the FluxCorrection logic, and/or a couple other suggestions from Nathan (adjusting subgrid sizes so grids have axis ratios closer to 1, and adjusting DualEnergyFormalismEta2). My first attempt to turn off SubgridSizeAutoAdjust resulted in the simulation hanging on a communication step very soon after the second output, but I can try messing around to get that working.

Thanks for your help,

John

an email to enzo-dev+unsubscribe@googlegroups.com.

To post to this group, send email to enzo...@googlegroups.com.
Visit this group at http://groups.google.com/group/enzo-dev.
For more options, visit https://groups.google.com/groups/opt_out.

--
John Wise
Assistant Professor of Physics
Center for Relativistic Astrophysics, Georgia Tech
http://cosmo.gatech.edu

--
You received this message because you are subscribed to the Google Groups "enzo-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+unsubscribe@googlegroups.com.

[John Forbes]

A few updates:

-- High temperature crashes: the simulations with ConservativeInterpolation have made it past where my standard simulations crashed, and have not (yet) exhibited the forever-rising temperature problem. Thus it appears that FluxCorrection causes the metallicity features, while ConservativeInterpolation somehow is causing the high-temperature crashes in this case.

-- The problem I've been having with SubgridSizeAutoAdjust=0 seems to be related to the timers. Enzo hangs immediately after the second output (i.e. DD0001 in this case), with the root processor stuck in ReduceTimes(), called from line ~330 of EnzoTiming.h. This appears to be because of a mismatch in the number of timers between processors. I find that for some rank r, all processors with higher ranks have N timers, while all processes <=r have N+1 timers, so those processes end up waiting forever to gather information about a timer which doesn't exist on the other processors. The timer these processes are missing is called SolveHydroEquations, which I believe means those processes somehow never called SolveHydroEquations, which makes me think I've set up my simulation in an odd way - maybe I can get around the issue by messing around with load balancing parameters, larger root grid timesteps, or starting simulations with SubgridSizeAutoAdjust from a more evolved snapshot.

[Sam Skillman]

Hi John,

It's possible that if one of your processors don't call SolveHydroEquations that it will get stuck.  You can get around this by a few ways:

1) Run without the timing information: make enzo-performance-no; make clean; make

2) In enzo.C line 288, put the following line:

TIMER_REGISTER("SolveHydroEquations");

Then you can just run "make" again and it will recompile just that file. 

The issue must be that your simulation when using SubgridSizeAutoAdjust=0 is that on the root-grid level (and I suppose all other levels as well), there are only r grids where r < the number of processors you are using. I think you could get around this by manually setting MaximumSubgridSize to something smaller (try 5000 or so, the default is 32768).  32768 will attempt to make grids that are 64^3 == (32768 * 8) in size, so choose a number that makes grids small enough to give you at least Ngrids >= Nprocessors.

Sam

--
You received this message because you are subscribed to the Google Groups "enzo-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+u...@googlegroups.com.

[Sam Skillman]

On Mon, Dec 16, 2013 at 8:07 AM, Sam Skillman <samsk...@gmail.com> wrote:

Hi John,

It's possible that if one of your processors don't call SolveHydroEquations that it will get stuck.  You can get around this by a few ways:

1) Run without the timing information: make enzo-performance-no; make clean; make

2) In enzo.C line 288, put the following line:

Sorry, just after line 288, so that area looks like:

288   enzo_timer = new enzo_timing::enzo_timer();

289   TIMER_REGISTER("SolveHydroEquations");

[Nathan Goldbaum]

For what it's worth, I don't see crashes or hangs when I run with SubgridSizeAutoAdjust disabled and usually only have one subgrid on the first few levels.

Also John, you might want to emphasize that the high temperature crashes you're seeing only happen when feedback is turned on.

Nathan

[John Forbes]

Hi everyone,

I wanted to resurrect this thread because I still haven't been able to track down the issue in the subject. (Whereas I've been able to avoid the high-temperature crash issue by altering my refinement criteria, and the timer issue following Sam's advice).

To summarize the problem I've been having with metals: when I have FluxCorrection=1, I see substantial (~10%) variations in the Metallicity field (as visualized by yt slices) near grid boundaries, even with a completely uniform initial metallicity field and no metals added via stellar feedback or the like. FluxCorrection=1 also seemed to conserve metal mass about 1000 times worse than total mass. With FluxCorrection=0, the features went away, and both total and metal mass were conserved to ~5e-7 (as opposed to ~1e-11 for total mass with FluxCorrection=1). The problem persists regardless of whether I set ConservativeInterpolation={0,1}, or InterpolationMethod={1,4}.

I also found the following sentences in the docs probably relevant, although I'm still somewhat lost trying to figure this out in the source code:
"If you need to advect a species as a color field, you will have to investigate how that works. Specifically, the means of conservation – by fraction of by density – as well as the inputs into the hydro solver."
from http://enzo.readthedocs.org/en/latest/developer_guide/HowToAddNewBaryonField.html

and
"Species quantities are not flux corrected directly but are modified to keep the fraction constant based on the density change." under FluxCorrection here http://enzo.readthedocs.org/en/latest/parameters/hydro.html

If I understand all of this correctly, it means that density is flux corrected but metal density is not, or maybe the metal density is adjusted before flux correction but not after? Any tips on how to fix the issue or interpreting that first quote from the docs would be greatly appreciated.

Thank you very much,

John