auto_inits branch of enzo

[Cameron Hummels]

Hello enzo devs (specifically Greg and Christine),

I have recently been trying to resurrect some old initial conditions I once used with enzo a couple of years ago.  These were generated with gregzo (the Greg branch of enzo prior to the remerge with week-of-code) and inits.  Back then, there was the ability of inits and enzo to cooperate when creating and running multi-level nested initial conditions, at least in gregzo, which was later lost in the remerge to week-of-code.

At some point there was an additional head that was created in the hg repository to deal with this called auto_inits.  I had thought that this got merged in long ago, as I can now successfully use my inits file from long ago to generate initial conditions with multiple levels of nested refinement.  Unfortunately, enzo fails when trying to read these initial conditions with the following error:

---

./enzo.exe sim.par

MPI_Init: NumberOfProcessors = 1

Global Dir set to .

magnetic field: dim 0, 0.006982 0.000000e+00

magnetic field: dim 1, 0.006982 0.000000e+00

magnetic field: dim 2, 0.006982 0.000000e+00

CosmologySimulation: Initializing grid 0

Standard Serial I/O

CosmologySimulation: Initializing grid 1

HDF5-DIAG: Error detected in HDF5 (1.8.11) thread 0:

  #000: H5Dio.c line 161 in H5Dread(): selection+offset not within extent

    major: Dataspace

    minor: Out of range

Caught fatal exception:

   'IO Problem'

at ReadFile.C:379

Backtrace:

BT symbol: ./enzo.exe() [0x40b166]

BT symbol: ./enzo.exe() [0x706699]

BT symbol: ./enzo.exe() [0x5d9bad]

BT symbol: ./enzo.exe() [0x482eed]

BT symbol: ./enzo.exe() [0x697b8f]

BT symbol: ./enzo.exe() [0x40a068]

BT symbol: /lib64/libc.so.6(__libc_start_main+0xfd) [0x390881ecdd]

BT symbol: ./enzo.exe() [0x409a99]

terminate called after throwing an instance of 'EnzoFatalException'

[login3.stampede.tacc.utexas.edu:mpi_rank_0][error_sighandler] Caught error: Aborted (signal 6)

Abort (core dumped)

---

I had thought it was simply some HDF5 error that I was encountering, but thanks to the sleuthing of Sam Skillman and Nathan Goldbaum, we now believe that it is due to the fact that the auto_inits head never actually made it into the mainline of enzo development.  I see the last commit to this head being from Christine 3 years ago, but it never merged with week-of-code.  Is there a reason for this?  Was it not ready?  It seems to make sense that if it were not in the main repo that I'd get an error like this if I tried to give enzo too much information in the initial conditions.  Or perhaps this error is totally unrelated?

I was wondering if anyone has used inits in the last year or two to generate multi-level nested initial conditions successfully (and then run them successfully), or perhaps someone has encountered this problem before?  Or is there even a need for having inits in the code anymore or is it deprecated?  I'd really like to make this work if possible with the old ICs, but if it is going to take a huge amount of work, then I guess I can move to using MUSIC and start over with new ICs.

Thanks for the help, everyone!

Cameron

--
Cameron Hummels

Postdoctoral Researcher

Steward Observatory

University of Arizona

http://chummels.org

[Christine Simpson]

Hi Cameron,

My recollection of this is that the changes associated with the automatic generation of nested ic's were merged into week-of-code, and that at the time of that incorporation, the inits in week-of-code worked with the enzo in that branch. 

I do recall, however, encountering a problem at some point with ReadFile.C in the way that the slabs are read in.  Well, it was a bug for the ic's I was using (generated by inits), but it looks like John had made a change that I guess was needed for his ic's (I'm not sure).  The change John made is changeset 5798970.  I think this was after the auto_inits changes were incorporated.  In my fork, I found that the slab dimensions needed to be assigned like this:

  for ( dim = 1; dim < Slab_Rank; dim++ )

    Slab_Dims[dim] = field_dims_attr[dim-1];

not like this:

  for ( dim = 1; dim < Slab_Rank; dim++ )
    Slab_Dims[dim] = field_dims_attr[Rank-dim];

so the field_dims_attr index changes from Rank-dim to dim-1.  I'm not sure if this is the source of your problem.

The auto_inits branch was opened in the old repository and was a catch all for the auto_inits stuff, but also for unrelated code changes I was making.  We moved to bitbucket around this time and part of the utility of branches was replaced by forks.  So I made a fork of enzo_dev and continued my work in auto_inits in my fork of enzo-dev.  I think all the outstanding changes in this fork that have not been incorporated into enzo-dev are only my stuff, which should not affect this issue.  One thing to keep in mind is that I think the number of users that use the automatic subgrid generation is reatively small, and so changes that are made to the nested grid infrastructure tend to be tested with ic's generated with other methods.

Greg, is this consistent with your memory on this?

Christine

--
You received this message because you are subscribed to the Google Groups "enzo-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+u...@googlegroups.com.
To post to this group, send email to enzo...@googlegroups.com.
Visit this group at http://groups.google.com/group/enzo-dev.
For more options, visit https://groups.google.com/groups/opt_out.

[Greg Bryan]

Christine is exactly correct: auto_inits has been merged in (and can be closed).  The problem is the assumption of the ordering of the input data.  I guess various IC generators make different assumptions about this.  I guess possible solutions are:

1. Change the ordering in inits.  The advantage is that going forward, there is no confusion.  The disadvantage is that previous IC's generated by inits will become unreadable without the change that Christine mentions.

2. Include a flag specifying the ordering.  This is unappealing as it ads yet another parameter, but is maybe the least disruptive path.  This can be set to the current default ordering and inits can explicitly spit out a warning that this flag needs to be set.  We will have to update inits parameter files in the documentation.

3. Eject inits from the current codebase and officially adopt another IC generator. 

Thoughts?

Greg

[Christine Simpson]

I feel rather torn on this issue.  On the one hand, I find the nested intitializer nasty in large part because it is trying to accommodate so many different use cases.  On the other hand, I like that there is an ic generator that ships with the enzo source code.  On the other, other hand, I like the tremendous amount of work done in recent years on improving external ic generators (i.e. MUSIC), so I think less people will be using inits going forward.  Then there is the issue of reading old ic's as Greg mentions, which is not trivial.  So I don't know.  I would be interested in hearing more opinions before I can formulate one.  I do think taking a fresh look at simplifying the initializer and building in some more redundancies and better error reporting would be good.  I remember that slab issue as being really tough to track down.

Christine

[Britton Smith]

I don't have too much to add to this discussion, but I did want to mention that I recently (about 6 months ago) used the autoinits functionality successfully and was very glad to have it there.

One nice thing about continuing to ship an IC generator with the code is that if, one day, we get around to doing answering testing of cosmology simulations, this will make it easier to automate that process.  This is something that has been on my list for a while now and I would very much like to get it done.

Finally, I had a thought about another possible way to support the ability to use old IC parameter files.  Since this functionality did exist in the past, it should be somewhere in the repository history.  Perhaps we can simply find the revision where inits last worked in the desired way, put a mercurial tag on it, and then put a note somewhere about this.  Then, you could just update to that old revision and build inits.  Such as option wouldn't work as well with Enzo, since you would lose a lot of functionality by going back to an old version, but inits has stayed relatively static over time.

Britton

[Cameron Hummels]

Hi everyone,

Thank you everyone for your information on this topic.  I have an update on this end, which may alleviate this dilemma.

Christine, you were indeed correct that it was John's changeset #5798970 that caused the breakage that I was encountering as described in my original email.  When I use the changeset directly preceding this, I am able to read in the initial conditions without problems in serial mode (i.e. ParallelRootGridIO = 0).  Interestingly, I cannot read in these ICs with ParallelRootGridIO *prior* to John's changeset, but his changes enable me to read them in OK.  I suppose this means that slab order isn't as important when reading things in a parallel fashion, or at least it's done differently so that this works for reading in old ICs.

That said, if you try to read in old ICs using ParallelRootGridIO in the current tip of enzo, you get another unrelated error:

 cloudsize > cellsize in cic_deposit!

==================

=== ENZO ERROR ===   cic_deposit.F: 81   node 0

==================

I was able to track this down to changeset # 44d406c7053c from last year, where Greg introduced the new method for depositing particles in subgrids into the main grid.  By simply adding the parameter below to my parameter file I was able to get it to work with ParallelRootGridIO turned on, but not when it is turned off because of the slab-ordering issue discussed above.

ParticleSubgridDepositMode = 0       // default = 1

(see http://enzo.readthedocs.org/en/latest/parameters/gravity.html?highlight=particlesubgriddepositmode )

I know this may give me some gravity fluctuations since this is a nested-grid simulation, but frankly, I'm just happy to have something that *reads in* the ICs.

I'm not going to say this solves the problems for running enzo with all ICs generated with inits.exe, but it may alleviate the bulk of the problem if people just add this parameter and set ParallelRootGridIO to be on.  This could mean postponing any hard decisions about whether or not to kick out inits.exe from the source distribution.  If we are faced with this dilemma, I am in favor of voting for Greg's option #2: Include a flag specifying the ordering.  That retains backwards compatibility but allows us to deal with MUSIC-generated ICs.

Thanks again for the help from everyone, including Matt, Nathan, and Sam on the irc channel.

Cameron

[Greg Bryan]

Hi Cameron -- The bug you ran into (the cloud size > cell size error) has to do with the precision that inits generates the positions of the sub grids.  The problem is that they were lower than they should be (due to my use of the incorrect precision OUTSYM) and that causes the cell sizes to vary slightly with level.  I just issued a pull request to fix this, although of course, it won't solve the problem for old output files.  Your change of going back to the old subgrid deposition mode should be fine -- that only tweaks halo positions.

I haven't looked at the other point you made about the ordering being ok if you turn ParallelRootGridIO on.

Greg

[Cameron Hummels]

Hi Greg,

Thanks for looking into this!  I had been baffled by this error, since there are numerous points in the code that explicitly set the cloud size = to the cell size directly before this check.  But I guess if the precision is wrong, I can see how that might cause problems.  Thanks!

Cameron

[Pengfei Chen]

Hi Greg,

I also run into this problem. I print out both sizes and get 

cloudsize=   8.6805555555555551E-004  cellsize=   8.6805555555555540E-004

...

cloudsize=   8.6805555555555594E-004  cellsize=   8.6805555555555540E-004

...

I think it's also related with the precision, but I have double checked that your fix in src/inits/AutomaticSubgridGeneration.C is already in my current code.  
Do you have any idea about this problem?

Thank you.

Pengfei

在 2014年1月18日星期六 UTC-8上午6:04:38，Greg Bryan写道：

[Pengfei Chen]

Hi Greg,

It could run with ParticleSubgridDepositMode = 0, but I still prefer to run with ParticleSubgridDepositMode = 1, which is the default value.

I tried to modify the type of cellsize and cloudsize from float to FLOAT in src/enzo/Grid_DepositParticlePositions.C and Grid_DepositPositions.C, and from R_PREC to P_PREC in src/enzo/cic_deposit.F, according to http://enzo.readthedocs.org/en/latest/developer_guide/FloatIsDouble.html#floatisdouble. But now the same problem still exists.

I also tried to make CONFIG_IO-64 when compiling enzo, but it also doesn't help.

Could you please give me some suggestions?

Thanks,

Pengfei

在 2015年5月4日星期一 UTC-7下午1:25:07，Pengfei Chen写道：

[Greg Bryan]

Hi Pengfei — Sorry you’re having this problem.  A couple of thoughts:

- After incorporating the fix did you make sure to remake the code and regenerate the initial conditions with inits?  The problem was the precision of the grid positions generated by inits.

- Another option is to switch to using MUSIC to generate initial conditions (see paper by Hahn and Abel 2012?).  MUSIC is more flexible and faster than inits, and I would encourage you to move in that direction — Enzo supports reading MUSIC ICs.

Cheers,

Greg

For more options, visit https://groups.google.com/d/optout.

[Pengfei Chen]

Hi Greg,

Thank you for your reply! Yes, I did remake the code and regenerate the initial conditions with inits. But I still got that error. Now we make a temporary change in cic_deposit.F, changing

      if (cloudsize .gt. cellsize) then

         write(6,*) "cloudsize > cellsize in cic_deposit!"

         ERROR_MESSAGE

      endif

to 

      if ((cloudsize-cellsize) .gt. 1E-7) then

         write(6,*) "cloudsize > cellsize in cic_deposit!"

         ERROR_MESSAGE

      endif

to avoid the failure of our unigrid simulation. I also agree that one day we could move to use MUSIC, since more colleagues are using it.

Thank you and have a nice day!

Pengfei

--
You received this message because you are subscribed to a topic in the Google Groups "enzo-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/enzo-dev/kNMr6hJJ2BY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to enzo-dev+u...@googlegroups.com.