Deaggragtion error on Cluster

[Navya Vishnu]

Hi,

I have some issues regrading running deaggregation using the USGS source models for Central US on the cluster. I am able to successfully run the classical mode. Only deaggreagtion seems to throw up issues. The same code works just fine on my computer (Windows). I am pasting the error I get on the cluster below:

[2017-10-17 14:10:14,812 #35 WARNING] Reducing the logic tree of Conterminous_U_S_USA08_v1.xml from 10584 to 392 realizations

[2017-10-17 14:10:17,934 #35 INFO] Submitting  "build_hcurves_and_stats" tasks

[2017-10-17 14:10:17,938 #35 INFO] Sent 97.19 KB of data in 1 task(s)

[2017-10-17 14:10:17,967 #35 INFO] build_hcurves_and_stats 100%

[2017-10-17 14:10:17,976 #35 INFO] Received 1.26 KB of data, maximum per task 1.26 KB

[2017-10-17 14:10:17,984 #35 INFO] 3 epsilon bins from -3.0 to 3.0

[2017-10-17 14:10:20,539 #35 INFO] 5 mag bins from 5.05 to 9.2

[2017-10-17 14:10:20,544 #35 INFO] [sid=0] 20 dist bins from 0.0 to 200.0

[2017-10-17 14:10:20,547 #35 INFO] [sid=0] 23 lon bins from -92.247187217 to -87.8510240133

[2017-10-17 14:10:20,550 #35 INFO] [sid=0] 23 lat bins from 33.3512514572 to 36.9468014213

[2017-10-17 14:10:20,553 #35 CRITICAL] 

Traceback (most recent call last):

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/base.py", line 210, in run

    self.post_execute(self.result)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/disaggregation.py", line 120, in post_execute

    self.full_disaggregation()

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/disaggregation.py", line 233, in full_disaggregation

    for srcs in split_in_blocks(split_sources, nblocks):

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/baselib/general.py", line 264, in split_in_blocks

AssertionError: 0

[2017-10-17 14:10:20,559 #35 CRITICAL] Traceback (most recent call last):

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/engine/engine.py", line 212, in run_calc

    _do_run_calc(calc, exports, hazard_calculation_id, **kw)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/engine/engine.py", line 247, in _do_run_calc

    close=False, **kw)  # don't close the datastore too soon

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/base.py", line 210, in run

    self.post_execute(self.result)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/disaggregation.py", line 120, in post_execute

    self.full_disaggregation()

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/disaggregation.py", line 233, in full_disaggregation

    for srcs in split_in_blocks(split_sources, nblocks):

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/baselib/general.py", line 264, in split_in_blocks

AssertionError: 0

Traceback (most recent call last):

  File "/home/nv7/openquake/bin/oq", line 35, in <module>

    main.oq()

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/commands/__main__.py", line 49, in oq

    parser.callfunc()

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/baselib/sap.py", line 186, in callfunc

    return self.func(**vars(namespace))

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/baselib/sap.py", line 245, in main

    return func(**kw)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/commands/engine.py", line 178, in engine

    exports, hazard_calculation_id=hc_id)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/commands/engine.py", line 65, in run_job

    hazard_calculation_id=hazard_calculation_id, **kw)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/engine/engine.py", line 212, in run_calc

    _do_run_calc(calc, exports, hazard_calculation_id, **kw)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/engine/engine.py", line 247, in _do_run_calc

    close=False, **kw)  # don't close the datastore too soon

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/base.py", line 210, in run

    self.post_execute(self.result)

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/disaggregation.py", line 120, in post_execute

    self.full_disaggregation()

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/calculators/disaggregation.py", line 233, in full_disaggregation

    for srcs in split_in_blocks(split_sources, nblocks):

  File "/home/nv7/openquake/lib/python2.7/site-packages/openquake/baselib/general.py", line 264, in split_in_blocks

AssertionError: 0

Any help would be greatly appreciated.

Thanks,

Navya

[Michele Simionato]

When running on a cluster the first line of your log should give you something like this:

2017-10-17T12:43:52.33,INFO,MainProcess/38872,Using celery@cobain, celery@dylan, celery@marley, celery@mercury, 256 cores

(this is our cluster with 4 machines and 256 cores). By any chance, do you get 0 cores? That would explain why nblocks is 0 in the error message you get.

In that case it means that there is something wrong with you cluster (like celery was not started). Otherwise I am at loss at how you can get nblocks=0. Or

perhaps you have an old version of the engine with a bug.