AMR FLD solver -- friendly user release
7 views
Skip to first unread message
Daniel R. Reynolds
Jan 13, 2012, 4:38:50 PM1/13/12
to enzo...@googlegroups.com, Daniel R. Reynolds
Hi all,
I'm writing to announce the alpha release of the AMR-FLD radiation transport module for Enzo. By "alpha" I mean that it's at the friendly user stage, and that there are a number of things that need to be improved before it's fully operational. I wanted to get this out to the group since I know that a number of you have been waiting on it for some time, but also because I feel that progress will be much more rapid with more people hammering on it. Hopefully the saying "many hands make light work" will apply here.
I've included a file README_AMRFLD in the top level of the repository that outlines how to build and use the new solver module. So here I will list issues that currently DO NOT work. As people find more, please let me know about them, and hopefully I/we can fix them pretty quickly.
1. Time stepping currently occurs through the radiation module taking time steps over the full AMR hierarchy at the root grid step size. At every subcycled child-grid iteration, the radiation solver is not called since it has already evolved further in time. There are a number of alternate time-stepping strategies that could be explored, so I'd appreciate input into new approaches, as well as assistance in setting them up and trying them out. I'll mention that the AMRsolve framework that AMRFLD is built on can solve any subset of levels in a given hierarchy (e.g. levels 2-6), so the current levels 0-max_level is only one possibility.
2. AMRsolve (which AMRFLD is built on) requires a strict nesting of the AMR hierarchy, i.e. there cannot exist subgrid boundaries that jump multiple levels of refinement to neighbors. Instead, all refined levels require at least one layer of surrounding zones at the next-coarser grid. Unfortunately, in all of my (very simple) dynamic AMR test problems Enzo violates this requirement. However, even though AMRsolve has this technical restriction, it still runs and seems to give reasonable results, as long as these level jumps don't occur at the domain boundary, or maybe from grids to their great-grandchildren. Some work on Enzo's adaptivity algorithms to guarantee strict nesting would be greatly appreciated.
However, these refinement problems should be avoidable when using statically-nested grids. I've tested AMRFLD in serial using a static hierarchy, but I don't know how to set up similar tests in parallel, and I'd appreciate assistance on that front.
3. Selection of good time steps for the hierarchical radiation problem is still an open question. I believe that what I have implemented provides a reasonable approach, in which the requisite number of time steps is roughly independent of the mesh refinement, but suggestions/improvements are welcome. The relevant files are src/amr_solve/AMRsolve_hypre_fld.cpp (rdiff_norm) and src/enzo/AMRFLDSplit_ComputeTimeStep.C
4. After each AMRFLD implicit time step, I must exchange boundary values with all neighbors (and parents), at around line 217 of the file src/enzo/AMRFLDSplit_Evolve.C. Currently, Enzo has one large routine to do this exchange (SetBoundaryConditions), but it unnecessarily exchanges boundary conditions in ALL FIELDS, not just the one I need. While this is not exactly a bug, the AMRFLD module would be more efficient for parallel runs if Enzo had a routine to set boundary conditions on only a single field.
5. The FAC solver option within AMRFLD is currently unusable, even for a perfectly nested hierarchy. This is a solver from HYPRE, called by AMRsolve, which is in turn called by AMRFLD. Hence the default AMRFLD solver is now BiCGStab. While BiCGStab works fine, it is not scalable to large problem sizes whereas in theory FAC has optimal scalability). This is therefore a higher priority on my end than other AMRFLD enhancements.
There are three major areas where I can use your help:
First, it would be helpful if you could set up your problem of choice to use the AMRFLD solver, and just try it out. Let me know if the interface is difficult to use and/or how it could be improved.
Second, I'd like your help in making it better, and I'd welcome any coding assistance on the aforementioned issues that you can provide.
Third, try out your current favorite problems and ensure that I didn't accidentally break anything else in Enzo. I think this is fine, but I haven't yet set up or run our new "gold standard" with this version of the code.
The repository is here: https://bitbucket.org/drreynolds/enzo-dev-amrfld
It was forked off of the main enzo-dev Bitbucket repository (week-of-code branch) earlier this week, so it should be pretty up to date with whatever version you're used to.
Thanks!
Dan
I'm writing to announce the alpha release of the AMR-FLD radiation transport module for Enzo. By "alpha" I mean that it's at the friendly user stage, and that there are a number of things that need to be improved before it's fully operational. I wanted to get this out to the group since I know that a number of you have been waiting on it for some time, but also because I feel that progress will be much more rapid with more people hammering on it. Hopefully the saying "many hands make light work" will apply here.
I've included a file README_AMRFLD in the top level of the repository that outlines how to build and use the new solver module. So here I will list issues that currently DO NOT work. As people find more, please let me know about them, and hopefully I/we can fix them pretty quickly.
1. Time stepping currently occurs through the radiation module taking time steps over the full AMR hierarchy at the root grid step size. At every subcycled child-grid iteration, the radiation solver is not called since it has already evolved further in time. There are a number of alternate time-stepping strategies that could be explored, so I'd appreciate input into new approaches, as well as assistance in setting them up and trying them out. I'll mention that the AMRsolve framework that AMRFLD is built on can solve any subset of levels in a given hierarchy (e.g. levels 2-6), so the current levels 0-max_level is only one possibility.
2. AMRsolve (which AMRFLD is built on) requires a strict nesting of the AMR hierarchy, i.e. there cannot exist subgrid boundaries that jump multiple levels of refinement to neighbors. Instead, all refined levels require at least one layer of surrounding zones at the next-coarser grid. Unfortunately, in all of my (very simple) dynamic AMR test problems Enzo violates this requirement. However, even though AMRsolve has this technical restriction, it still runs and seems to give reasonable results, as long as these level jumps don't occur at the domain boundary, or maybe from grids to their great-grandchildren. Some work on Enzo's adaptivity algorithms to guarantee strict nesting would be greatly appreciated.
However, these refinement problems should be avoidable when using statically-nested grids. I've tested AMRFLD in serial using a static hierarchy, but I don't know how to set up similar tests in parallel, and I'd appreciate assistance on that front.
3. Selection of good time steps for the hierarchical radiation problem is still an open question. I believe that what I have implemented provides a reasonable approach, in which the requisite number of time steps is roughly independent of the mesh refinement, but suggestions/improvements are welcome. The relevant files are src/amr_solve/AMRsolve_hypre_fld.cpp (rdiff_norm) and src/enzo/AMRFLDSplit_ComputeTimeStep.C
4. After each AMRFLD implicit time step, I must exchange boundary values with all neighbors (and parents), at around line 217 of the file src/enzo/AMRFLDSplit_Evolve.C. Currently, Enzo has one large routine to do this exchange (SetBoundaryConditions), but it unnecessarily exchanges boundary conditions in ALL FIELDS, not just the one I need. While this is not exactly a bug, the AMRFLD module would be more efficient for parallel runs if Enzo had a routine to set boundary conditions on only a single field.
5. The FAC solver option within AMRFLD is currently unusable, even for a perfectly nested hierarchy. This is a solver from HYPRE, called by AMRsolve, which is in turn called by AMRFLD. Hence the default AMRFLD solver is now BiCGStab. While BiCGStab works fine, it is not scalable to large problem sizes whereas in theory FAC has optimal scalability). This is therefore a higher priority on my end than other AMRFLD enhancements.
There are three major areas where I can use your help:
First, it would be helpful if you could set up your problem of choice to use the AMRFLD solver, and just try it out. Let me know if the interface is difficult to use and/or how it could be improved.
Second, I'd like your help in making it better, and I'd welcome any coding assistance on the aforementioned issues that you can provide.
Third, try out your current favorite problems and ensure that I didn't accidentally break anything else in Enzo. I think this is fine, but I haven't yet set up or run our new "gold standard" with this version of the code.
The repository is here: https://bitbucket.org/drreynolds/enzo-dev-amrfld
It was forked off of the main enzo-dev Bitbucket repository (week-of-code branch) earlier this week, so it should be pretty up to date with whatever version you're used to.
Thanks!
Dan
-- ------------------------------------ Dr. Daniel R. Reynolds Assistant Professor of Mathematics Southern Methodist University 214-768-4339 http://faculty.smu.edu/reynolds/ ------------------------------------
David Collins
Jan 13, 2012, 6:54:33 PM1/13/12
to enzo...@googlegroups.com
That's super exciting, Congratulations, Dan.
d.
> --
> You received this message because you are subscribed to the Google Groups
> "enzo-dev" group.
> To post to this group, send email to enzo...@googlegroups.com.
> To unsubscribe from this group, send email to
> enzo-dev+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/enzo-dev?hl=en.
--
Sent from my computer.
Junhwan Choi
Jan 13, 2012, 7:52:39 PM1/13/12
to enzo...@googlegroups.com, <enzo-dev@googlegroups.com>, Daniel R. Reynolds
It is great news. Thank you very much Daniel.
Junhwan
--
Brian O'Shea
Jan 15, 2012, 10:31:55 AM1/15/12
to enzo...@googlegroups.com
Dan,
This is amazing. Thank you SO much! I'll try setting up a test problem using it very soon. Also, regarding the strict nesting issue, I think that Dave Collins may have had to deal with that for one of his MHD implementations. Dave, am I remember that correctly?
--Brian
This is amazing. Thank you SO much! I'll try setting up a test problem using it very soon. Also, regarding the strict nesting issue, I think that Dave Collins may have had to deal with that for one of his MHD implementations. Dave, am I remember that correctly?
--Brian
--
David Collins
Jan 17, 2012, 11:07:10 AM1/17/12
to enzo...@googlegroups.com
> using it very soon. Also, regarding the strict nesting issue, I think that
> Dave Collins may have had to deal with that for one of his MHD
> implementations. Dave, am I remember that correctly?
> Dave Collins may have had to deal with that for one of his MHD
> implementations. Dave, am I remember that correctly?
I have a work around so it doesn't cause DivB errors, but I never made
the problem go away.
We had some discussion about this at the very first week of code, and
if I recall, it sounded like a whole new RebuildHierarchy would be
necessary if we really want to force it to be strictly true in all
cases. Which actually sounds like a fun project, I'd be for
discussion of it.
--
Sent from my computer.
Reply all
Reply to author
Forward
0 new messages