engine v3.15 and current master produce different ground motion fields?

[Cecilia I. Nievas]

Dear OpenQuake developers,

I'm still in the process of moving tests from OpenQuake v3.15 to the current master. A damage scenario test that I have produces different ground motion results (LanzanoEtAl2019_RJB_OMO, AvgSA) with each of those versions of the engine, even with the same seed (see note below, please). The resulting "sitemesh_XXX.csv" exports are different. My suspicion is that (i) the way the seeds are used in a scenario damage calculation has changed between OQ v3.15 and current master, and/or (ii) the behaviour of numpy's random number generator has changed from v1.20.0 (used by OQ v3.15) to v2.2.6 (used by OQ v3.25+) in a way that may lead to different results between the two versions of the engine. Do you know if any of these are true or, alternatively, if there is any known cause of discrepancy between ground motion fields calculated with OQ v3.15 vs current master? I'd just like to confirm if this is what's likely happening.

I am sending you the associated input files. The job.ini can be found under input_test_RLA_01/current/job.ini.

Regarding the seeds:

• The job.ini sets ses_seed = 159. This guaranteed reproducibility when using OQ v3.15.
• I have tried fixing random_seed instead, and still got different results with the two engine versions.
• The report_XXX.rst file generated by v3.15 tells me the values of random_seed, master_seed and ses_seed. I ran a case in which I just set all of these to the values from the report file, and still got different results with the two engine versions. According to the current OQ docs, "master_seed" should not do anything in this calculation. 
• When I compare the current description of the way in which OpenQuake uses random_seed, master_seed and ses_seed against my notes from April 2022, I find as a potential difference that, back then, I had written that, from v3.11 onward, ses_seed controls the sampling of ground motion values from the GMPE when truncation_level is different from zero. In the current docs, I see it described as only controlling the sampling of ruptures, but not ground motions. However, ses_seed = 159 does produce stable results within current master itself (i.e. different runs of the same input with current master), despite not explicitly setting the random_seed.
• I'd be happy to know if I am missing something regarding how I should be setting the seeds.

Thanks in advance!

Best regards,

Cecilia

[Michele Simionato]

The seed algorithm changes very frequently across versions, it changed even last week in master (https://github.com/gem/oq-engine/pull/11267).

Even if you do not specify a seed there is a default value so *within the same version of the engine there is always reproducibility*.

Across different versions this is not the case, one must focus on statistics quantities which have a small dependency on the seed.

The various seeds are explained here:

https://docs.openquake.org/oq-engine/master/manual/user-guide/extras/faq.html#what-implications-do-random-seed-ses-seed-and-master-seed-have-on-my-calculation

From ses_seed individual rupture-specific seeds are generated and they are using in the GMF generation, so ses_seed is relevant both for sampling the ruptures

and sampling the GMFs.

HTH,

                Michele