issues with non-parametric sources + source model logic tree

[era]

Hello!

We're running a logic tree of source models with non-parametric sources and coming across an error that seems a bit inconsistent and we're having trouble identifying if it is our fault or lies somewhere in OQ's interpretation of our source models and/or logic tree.  We're running OQ nightly build version 2.3.0-0~dev1485813444-1c0d463 from 31 January 2017 (New Zealand).

We have tested the attached example and received the error pasted at the bottom of this message. When we tested smaller subsets of that data, we had (what seems to us to be) inconsistent results - for example, running source models (NOTE: when the combination failed, the error output was identical (with the exception of the weight after Key Error - logic tree dependent) to the example below):

1. (calc 167) #1-100 in a logic tree fails and produces error
2.  #1-10 in a logic tree fails and produces error
3. #1-4 calculation fails and produces error
4. (calc 170) #1-3 +#5 completes without error
5. (calc 171) #4 alone completes without error
6. #1 and #7 together fails and produces error
7. (calc 173) #4 and #7 together completes without error
8. (calc 174) #1 alone completes without error
9. (calc 175) #1 and #2 together completes without error

We have also noticed that the outputs vary between the ones that do work (ie. sometimes the calculation completes, but no hazard curve files are produced; sometimes no mean hazard curve file is produced but the quantiles are - in this case, is the mean represented by quantile 0.5?, etc.).

The most bizarre thing about all of this, is that Nick H ran the same suite of models using OQ v2.0 (I think) and it came out successfully. Ultimately, we do need to run this with about 2000 source model branches (Nick has had success with this in the past).

In all, we're unsure if we just don't understand how the non-parametric sources are implemented in OQ or if we're not understanding the logic tree combinations at the end.  Any thoughts would be greatly appreciated.

Cheers,

Elizabeth and Matt

Error message (copied from run 1 (above), calc 167):

[2017-01-31 04:21:26,835 #167 INFO] classical  90%

[2017-01-31 04:21:26,886 #167 INFO] classical  91%

[2017-01-31 04:21:26,941 #167 INFO] classical  92%

[2017-01-31 04:21:26,998 #167 INFO] classical  93%

[2017-01-31 04:21:27,050 #167 INFO] classical  94%

[2017-01-31 04:21:27,105 #167 INFO] classical  95%

[2017-01-31 04:21:27,163 #167 INFO] classical  96%

[2017-01-31 04:21:27,219 #167 INFO] classical  97%

[2017-01-31 04:21:27,277 #167 INFO] classical  98%

[2017-01-31 04:21:27,337 #167 INFO] classical  99%

[2017-01-31 04:21:27,394 #167 INFO] classical 100%

[2017-01-31 04:21:27,449 #167 INFO] Received 195.17 KB of data, maximum per task 2.3 KB

[2017-01-31 04:21:27,777 #167 INFO] exported realizations: [u'./output/realizations_167.csv']

[2017-01-31 04:21:28,133 #167 INFO] Building hazard curves

[2017-01-31 04:21:28,196 #167 INFO] Executing a single task in process

[2017-01-31 04:21:28,253 #167 CRITICAL] 

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/base.py", line 195, in run

    self.result = self.execute()

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/classical.py", line 487, in execute

    nbytes = reduce(self.save_hcurves, res, AccumDict())

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/parallel.py", line 392, in __iter__

    raise RuntimeError(val)

RuntimeError: 

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/parallel.py", line 237, in safely_call

    got = func(*args)

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/classical.py", line 429, in build_hcurves_and_stats

    pstats.compute(sids, [pmap_by_rlz[rlz] for rlz in rlzs]))

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/general.py", line 583, in __missing__

    raise KeyError(key)

KeyError: <0,b1~b1_@_@_@,w=0.01>

[2017-01-31 04:21:28,304 #167 CRITICAL] Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/openquake/engine/engine.py", line 184, in run_calc

    _do_run_calc(calc, exports, hazard_calculation_id)

  File "/usr/lib/python2.7/dist-packages/openquake/engine/engine.py", line 217, in _do_run_calc

    close=False)  # don't close the datastore too soon

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/base.py", line 195, in run

    self.result = self.execute()

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/classical.py", line 487, in execute

    nbytes = reduce(self.save_hcurves, res, AccumDict())

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/parallel.py", line 392, in __iter__

    raise RuntimeError(val)

RuntimeError: 

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/parallel.py", line 237, in safely_call

    got = func(*args)

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/classical.py", line 429, in build_hcurves_and_stats

    pstats.compute(sids, [pmap_by_rlz[rlz] for rlz in rlzs]))

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/general.py", line 583, in __missing__

    raise KeyError(key)

KeyError: <0,b1~b1_@_@_@,w=0.01>

Traceback (most recent call last):

  File "/usr/bin/oq", line 30, in <module>

    main.oq()

  File "/usr/lib/python2.7/dist-packages/openquake/commands/__main__.py", line 49, in oq

    parser.callfunc()

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/sap.py", line 186, in callfunc

    return self.func(**vars(namespace))

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/sap.py", line 245, in main

    return func(**kw)

  File "/usr/lib/python2.7/dist-packages/openquake/commands/engine.py", line 166, in engine

    exports, hazard_calculation_id=hc_id)

  File "/usr/lib/python2.7/dist-packages/openquake/commands/engine.py", line 64, in run_job

    hazard_calculation_id=hazard_calculation_id)

  File "/usr/lib/python2.7/dist-packages/openquake/engine/engine.py", line 184, in run_calc

    _do_run_calc(calc, exports, hazard_calculation_id)

  File "/usr/lib/python2.7/dist-packages/openquake/engine/engine.py", line 217, in _do_run_calc

    close=False)  # don't close the datastore too soon

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/base.py", line 195, in run

    self.result = self.execute()

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/classical.py", line 487, in execute

    nbytes = reduce(self.save_hcurves, res, AccumDict())

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/parallel.py", line 392, in __iter__

    raise RuntimeError(val)

RuntimeError: 

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/parallel.py", line 237, in safely_call

    got = func(*args)

  File "/usr/lib/python2.7/dist-packages/openquake/calculators/classical.py", line 429, in build_hcurves_and_stats

    pstats.compute(sids, [pmap_by_rlz[rlz] for rlz in rlzs]))

  File "/usr/lib/python2.7/dist-packages/openquake/baselib/general.py", line 583, in __missing__

    raise KeyError(key)

KeyError: <0,b1~b1_@_@_@,w=0.01>

[Michele Simionato]

Hi Elisabeth,

thank you for the bug report, as usual clear and detailed. I have fixed the bug here: https://github.com/gem/oq-engine/pull/2503

The fix will enter soon in master and then in the nightly builds. We also plan to make an official release of engine 2.3 soon, with this and other fixes.

The problem had to do with branches with zero probability that were discarded.

    Michele

[era]

Hi Michele,

Thank you very much for your reply - the information on what went wrong was very interesting - and getting the bug fixed so quickly! It appears to be working now (though I have only done one run with all 100 source models).  I will let you know if we run into anything else and will keep an eye out for engine 2.3 in the near future.

Cheers,

Elizabeth