reproducibilty of Enzo

[Samson Cheung]

Hello !

I am working on setting up an Enzo test case. I am not a domain expert. I was given a set of Particle and Grid input data and a parameter file by a user. It is a Cosmology run which produces RedShift hdf output files.

I noticed that the generated hdf files differ from run to run, not only in numerical values, but also in hierarchy. An h5diff comparison between the generated files indicates non-comparable data sets. The runs are executed with the same executable, same input.
Is it expected that the RedShiftfiles differ?

The people gave me the data do not care about numerical reproducibility, they use visualization tools.
I would like to set up automated testing of run to run reproducibilty. Are there any values in the generated ASCII outputs (coolrates, rates, Evtime) which could be used for correctness check between run and the next?

Thanks in advance and best regards!!

~Samson

[David Collins]

Hi, Samson--

The results from Enzo are reproducible, with some caveats.  

Hydrodynamics and gravitational collapse are chaotic systems, so small differences in the evolution can lead to noticeable differences in the final outcome.  Small differences in communication times between processors can lead to slight differences in, for example, gravitational accelerations from one run to the next.  Nothing of physical significance, but enough to shift a galaxy halo by a zone, which will then lead to the hierarchy files being different.  The two hierarchies will represent the same structures, but the layout will be different.

So comparing two runs by just looking at the grid layout is not a good way to measure the reproducibility of physics results.  

We have a suite of tests that gets run regularly to ensure the results in the main repository don't change in any un-documented manner.   You can find documentation here:

http://enzo.readthedocs.io/en/latest/user_guide/EnzoTestSuite.html

This also requires yt,

yt-project.org

Note that over the years, some changes have been made that do change the results in order to improve the code. So some changes you may see between your current dataset and a new run may be due to know changes to the code.  

I hope that helps!

d.

--
You received this message because you are subscribed to the Google Groups "enzo-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+unsubscribe@googlegroups.com.
To post to this group, send email to enzo...@googlegroups.com.
Visit this group at https://groups.google.com/group/enzo-dev.
For more options, visit https://groups.google.com/d/optout.

--

-- Sent from a computer.

[Wise, John H]

Hi Samson,

I echo everything that Dave has said in his email, but I wanted to add one thing.

You should turn on the bitwise-identicality flag when compiling. You can do this with the following commands

make bitwise-yes
make

Thanks,
John

> To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+u...@googlegroups.com.

> To post to this group, send email to enzo...@googlegroups.com.
> Visit this group at https://groups.google.com/group/enzo-dev.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> -- Sent from a computer.
>

> --
> You received this message because you are subscribed to the Google Groups "enzo-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+u...@googlegroups.com.

> To post to this group, send email to enzo...@googlegroups.com.
> Visit this group at https://groups.google.com/group/enzo-dev.
> For more options, visit https://groups.google.com/d/optout.

--

John Wise
Associate Professor of Physics
Center for Relativistic Astrophysics, Georgia Tech
http://cosmo.gatech.edu

[Samson Cheung]

Thanks you so much for you quick reply!  The problem that we are faced with it is:

* We have a given data set that we need to work with. We would like to make sure that 2 subsequent runs, on this particular data set produce reasonable results, not necessarily bit-wise reproducible.

* I think using yt means using visual inspection for the result.   That is not good to us because we need to do checking automatically. Also, we are no domain experts, and would not know if the resulting images are good or not.

Well, bit-wise reproducible may be good, would it make the performance speed reduced dramatically?
Is there any value in Enzo ascii output, eg rates, coll_rates, Evtime, which we can use as basic verification?

Many thanks for your kind help!!
~Samson

[Nathan Goldbaum]

On Saturday, October 29, 2016, Samson Cheung <cheung...@gmail.com> wrote:

Thanks you so much for you quick reply!  The problem that we are faced with it is:

* We have a given data set that we need to work with. We would like to make sure that 2 subsequent runs, on this particular data set produce reasonable results, not necessarily bit-wise reproducible.

* I think using yt means using visual inspection for the result.   That is not good to us because we need to do checking automatically. Also, we are no domain experts, and would not know if the resulting images are good or not.

yt is not only visualization software, it's really an analysis API for processing volumetric data, and Enzo data are fully supported. You can use it to access the raw data stored in the Enzo outputs:

http://yt-project.org/doc/examining/low_level_inspection.html

The Enzo test suite uses yt to verify that the data are bitwise identical from run to run.

To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+unsubscribe@googlegroups.com.

[David Collins]

* We have a given data set that we need to work with. We would like to make sure that 2 subsequent runs, on this particular data set produce reasonable results, not necessarily bit-wise reproducible.

Can you elaborate on what your project is and why you want to ensure the answer is repeatable?  I can probably point you to tests in the test suite that do exactly what you want.  I imagine you want to be changing something within Enzo or your machine, having a test that targets the bits your changing is the most efficient way to go.

Is there any value in Enzo ascii output, eg rates, coll_rates, Evtime, which we can use as basic verification?

 

The test runner says "Failed" if things aren't good enough by our standards.  The test runner is pretty easy to run, it does all the analysis of the data results so you don't need to be an expert on any individual module or yt.  The easiest thing to do is:

1.) run the suite to generate a standard

2.) Every time you change (your machine or enzo or whatever) run the suite again and compare to your standard.  

It takes about 10 minutes, unless you want a really comprehensive test.

Enzo itself doesn't do any answer test outside of catching NaNs periodically.  Getting useful information from the output is really problem dependent, and is generally limited to  total failure (e.g. if the timestep goes below 1e-300, you have a problem)

d.

[Samson Cheung]

Many thanks to you all!  OK, now I have the test runner and the yt to look at first!
With gratitude!

~Samson

[Samson Cheung]

"Thanks again for pointing us to the yt software. The test runner is not going to help us, because we are not modifying the source code.

We are faced with the following situation:
* We are not modifying the Enzo code.
* We are using the same Enzo executable in an automated test suite, when system software, hardware, kernel, etc have changed.
* We have a given parameter input file and a set of Grid and Particle files, the run produced a set of Redshift hdf files.
* We want to make sure that when changing the system or compiler, nothing bad happens to Enzo execution; such as, Perfromance degradation, incorrect results, etc.

If you could point us to an example how "yt" can be used to extract some meaningful data from Redshift file, that would be great :-)

Thanks and best regards,

~Samson

[David Collins]

Hi--

I still think the test runner is your best bet.  It doesn't need Enzo to be the thing that changes.  If your changing hardware or kernels in a way that you think will change the answers, this will pick up those changes.  It takes care of running the tests, extracting the results using yt, and comparing to your 'gold standard' that you made before you started changing things.  It saves you the hassle of learning yt, and deciding what properties of Enzo to measure, what appropriate tolerances are.  These are non-trivial decisions to make, Enzo is a pretty complicated piece of code, and different components will tax hardware in different ways; having more than one test will be better for your verification.  We've already taken care of those decisions for you. From the enzo-dev/run directory, do the following:

Set up your standard, like this:

test_runner.py --run-suffix=FiducialTest  --suite=quick --local --local-dir=/scratch/wherever   --answer-name=FiducialGoldStandard --answer-store

then after you change your hardware and kernel, run

test_runner.py --run-suffix=Test1  --suite=quick --local --local-dir=/scratch/wherever   --answer-name=FiducialGoldStandard

The major difference is that for the first, you give it '--answer-store.'  That stores the answers to the tests.

Then the output is in /scratch/wherever/<hash>_Test1, where <hash> is the mercurial hash.  Every time you change your system, re-run with Test1 replaced by something that indicates your change.  

That being said, if you're really interested in learning yt, go here:

http://yt-project.org/doc/quickstart/index.html

You also may be interested in examining the results of the performance with the performance timers, which are described here:

http://enzo.readthedocs.io/en/latest/reference/PerformanceMeasurement.html

d.

--
You received this message because you are subscribed to the Google Groups "enzo-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+unsubscribe@googlegroups.com.

To post to this group, send email to enzo...@googlegroups.com.
Visit this group at https://groups.google.com/group/enzo-dev.
For more options, visit https://groups.google.com/d/optout.

[Samson Cheung]

Hello,

This is a follow to up to our question regarding the comparson of results of different Enzo runs on the same input data. We have installed "yt" and we are using the command:
   yt plot DDD000X/data000x
To generate the default density plots in x, y, z. From inspecting them visually, they look very similar.

Is it possible to use "yt" to compare the data that is used to generate these plots and check if the images are "similar" within a certain acceptable range?

We found a couple of public domain software tools, like "pdiff" (perceptual image diff), but think that "yt" might be a better choice.

Thanks in advance and greetings,
Samson and Gabriele

To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+u...@googlegroups.com.

To post to this group, send email to enzo...@googlegroups.com.
Visit this group at https://groups.google.com/group/enzo-dev.
For more options, visit https://groups.google.com/d/optout.

[Nathan Goldbaum]

On Tue, Nov 8, 2016 at 1:22 PM, Samson Cheung <cheung...@gmail.com> wrote:

Hello,

This is a follow to up to our question regarding the comparson of results of different Enzo runs on the same input data. We have installed "yt" and we are using the command:
   yt plot DDD000X/data000x
To generate the default density plots in x, y, z. From inspecting them visually, they look very similar.

It may be easier to get data for comparison using e.g. the following script:

    import yt

    ds = yt.load('DD000x/data000x')

    ad = ds.all_data()

    field_data = {}

    for field in ds.field_list:

         field_data[field] = np.array(ad[field])

At the end of this script, field_data will be a dictionary mapping on-disk field names to 1D ndarrays containing the corresponding field data for that field. The data for all grids are concatenated together into a 1D data structure.

You could then save these arrays to disk in your preferred format (e.g. i'd use hdf5 via h5py) and directly compare these arrays from run to run.

If you're dead set on comparing images, matplotlib has some capabilities for diffing images in their testing routines:

https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/testing/compare.py#L265

 

To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+unsubscribe@googlegroups.com.

[David Collins]

Hi--

To add to Nathan's ideas, you can also compare the images in yt using fixed resolution buffers:

import yt

ds1 = yt.load('sim1/DDxxxx/dataxxxx')

proj1 = ds1.proj('density',0)

width = 1.0

nzones = <whatever your top grid dimension is>

frb1 = proj1.to_frb(width, nzones)

<repeat for your second simulation>

Then you can use matplotlib to make images of the density fields from the frb:

>>> pyplot.imshow( frb1['density'] )

and take differences

>>> diff = frb1['density']-frb2['density']

d.