Error in Disaggregation calculation with Source Model containing an Area Source and a Simple Fault Source

[Sujan Raj Adhikari]

Dear OpenQuake user

will you please give your thoughts in "The disaggregation matrix is too large (72900000 elements): fix the binning!"

the console is given below along with the job file.

regards

Sujan

2020-12-09T15:42:31.55,INFO,MainProcess/25508,sadhika6 running C:\Users\sadhika6\AppData\Local\Temp\tmpwwei6wof\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard\OQDisaggregation.mps.ini [--hc=None]

2020-12-09T15:42:31.56,INFO,MainProcess/25508,Using engine version 3.10.1
2020-12-09T15:42:32.82,WARNING,MainProcess/25508,Using 12 cores on molnar-2
2020-12-09T15:42:32.84,INFO,MainProcess/25508,Validated CanadaSHM6trial_W_CANADA_simplified_collapsedRates_source_model_logic_tree.xml in 0.01 seconds
2020-12-09T15:42:32.92,INFO,MainProcess/25508,Checksum of the input files: 853253271
2020-12-09T15:42:32.93,INFO,MainProcess/25508,Reading the risk model if present
2020-12-09T15:42:32.94,INFO,MainProcess/25508,Read N=1 hazard sites and L=40 hazard levels
2020-12-09T15:42:33.03,INFO,MainProcess/25508,Potential number of logic tree paths = 4_096
2020-12-09T15:42:33.04,INFO,MainProcess/25508,1 effective smlt realization(s)
2020-12-09T15:42:33.05,INFO,MainProcess/25508,Reading the source model(s) in parallel
2020-12-09T15:42:33.29,INFO,MainProcess/25508,read_source_model 100% [1 submitted, 0 queued]
2020-12-09T15:42:33.30,INFO,MainProcess/25508,Received {'C:\\Users\\sadhika6\\AppData\\Local\\Temp\\tmpwwei6wof\\Users\\sadhika6\\Desktop\\oqPSHA\\Metro\\Hazard\\ssms\\simplifiedCollapsed\\sourceModels\\CanadaSHM6trial_W_CANADA_simplified_collapsedRates.xml': '92.31 KB'} in 0 seconds
2020-12-09T15:42:33.37,INFO,MainProcess/25508,There are 81 sources
2020-12-09T15:42:33.39,INFO,MainProcess/25508,Checking the sources bounding box
2020-12-09T15:42:33.40,ERROR,MainProcess/25508,source ACMWA: The buffer of 600 km is too large, the bounding box is larger than half the globe: 219 degrees
2020-12-09T15:42:33.52,INFO,MainProcess/25508,Rupture floating factor = 2.92
2020-12-09T15:42:33.53,INFO,MainProcess/25508,Rupture spinning factor = 6.12
2020-12-09T15:42:33.87,INFO,MainProcess/25508,gzipping the input files
2020-12-09T15:42:33.99,INFO,MainProcess/25508,Weighting the sources
2020-12-09T15:42:50.04,INFO,MainProcess/25508,tot_weight=14_408_660, max_weight=300_000
2020-12-09T15:42:50.05,INFO,MainProcess/25508,TRT = Subduction IntraSlab30
2020-12-09T15:42:50.06,INFO,MainProcess/25508,max_dist=4.85->400 ... 7.25->400, gsims=4, weight=15_900, blocks=1
2020-12-09T15:42:50.08,INFO,MainProcess/25508,TRT = Subduction IntraSlab55
2020-12-09T15:42:50.08,INFO,MainProcess/25508,max_dist=4.85->400 ... 7.65->400, gsims=4, weight=303_852, blocks=1
2020-12-09T15:42:50.11,INFO,MainProcess/25508,TRT = Stable Shallow Crust
2020-12-09T15:42:50.12,INFO,MainProcess/25508,max_dist=4.85->600 ... 7.95->600, gsims=16, weight=10_364_864, blocks=5
2020-12-09T15:42:50.13,INFO,MainProcess/25508,TRT = Subduction Interface
2020-12-09T15:42:50.14,INFO,MainProcess/25508,max_dist=7.05->1000 ... 9.45->1000, gsims=4, weight=267_972, blocks=1
2020-12-09T15:42:50.16,INFO,MainProcess/25508,TRT = Active Shallow Crust
2020-12-09T15:42:50.17,INFO,MainProcess/25508,max_dist=4.85->400 ... 8.45->400, gsims=4, weight=3_456_072, blocks=7
2020-12-09T15:42:50.34,INFO,MainProcess/25508,Estimated upper memory limit per core:
1 sites x 40 levels x 16 gsims x 1 src_multiplicity * 8 bytes = 5 KB
2020-12-09T15:42:50.37,INFO,MainProcess/25508,classical 6% [15 submitted, 0 queued]
2020-12-09T15:42:51.41,DEBUG,classical_split_filter#1,produced 1 subtask(s) with mean weight 299160
2020-12-09T15:42:51.52,INFO,MainProcess/25508,classical 12% [16 submitted, 0 queued]
2020-12-09T15:42:51.97,INFO,MainProcess/25508,classical 18% [16 submitted, 0 queued]
2020-12-09T15:42:52.00,INFO,MainProcess/25508,classical 25% [16 submitted, 0 queued]
2020-12-09T15:42:52.32,DEBUG,classical_split_filter#5,produced 9 subtask(s) with mean weight 298080
2020-12-09T15:42:52.48,DEBUG,classical_split_filter#11,produced 5 subtask(s) with mean weight 299160
2020-12-09T15:42:52.68,DEBUG,classical_split_filter#12,produced 4 subtask(s) with mean weight 299280
2020-12-09T15:42:52.85,DEBUG,classical_split_filter#9,produced 3 subtask(s) with mean weight 299160
2020-12-09T15:42:53.38,DEBUG,classical_split_filter#14,produced 1 subtask(s) with mean weight 299480
2020-12-09T15:42:54.28,DEBUG,classical_split_filter#13,produced 4 subtask(s) with mean weight 299410
2020-12-09T15:43:33.37,DEBUG,classical_split_filter#7,produced 7 subtask(s) with mean weight 300000
2020-12-09T15:43:52.59,INFO,MainProcess/25508,classical 26% [49 submitted, 0 queued]
2020-12-09T15:44:07.10,INFO,MainProcess/25508,classical 28% [49 submitted, 0 queued]
2020-12-09T15:44:11.42,INFO,MainProcess/25508,classical 30% [49 submitted, 0 queued]
2020-12-09T15:44:20.14,INFO,MainProcess/25508,classical 32% [49 submitted, 0 queued]
2020-12-09T15:44:29.00,INFO,MainProcess/25508,classical 34% [49 submitted, 0 queued]
2020-12-09T15:44:43.41,INFO,MainProcess/25508,classical 36% [49 submitted, 0 queued]
2020-12-09T15:45:02.65,INFO,MainProcess/25508,classical 38% [49 submitted, 0 queued]
2020-12-09T15:45:32.38,INFO,MainProcess/25508,classical 40% [49 submitted, 0 queued]
2020-12-09T15:45:50.45,INFO,MainProcess/25508,classical 42% [49 submitted, 0 queued]
2020-12-09T15:46:00.28,INFO,MainProcess/25508,classical 44% [49 submitted, 0 queued]
2020-12-09T15:46:12.98,INFO,MainProcess/25508,classical 46% [49 submitted, 0 queued]
2020-12-09T15:46:16.84,INFO,MainProcess/25508,classical 48% [49 submitted, 0 queued]
2020-12-09T15:46:41.88,INFO,MainProcess/25508,classical 51% [49 submitted, 0 queued]
2020-12-09T15:48:24.64,INFO,MainProcess/25508,classical 53% [49 submitted, 0 queued]
2020-12-09T15:48:41.76,INFO,MainProcess/25508,classical 55% [49 submitted, 0 queued]
2020-12-09T15:49:08.46,INFO,MainProcess/25508,classical 57% [49 submitted, 0 queued]
2020-12-09T15:51:03.89,INFO,MainProcess/25508,classical 59% [49 submitted, 0 queued]
2020-12-09T15:51:22.88,INFO,MainProcess/25508,classical 61% [49 submitted, 0 queued]
2020-12-09T15:51:48.15,INFO,MainProcess/25508,classical 63% [49 submitted, 0 queued]
2020-12-09T15:51:48.16,INFO,MainProcess/25508,classical 65% [49 submitted, 0 queued]
2020-12-09T15:51:48.18,INFO,MainProcess/25508,classical 67% [49 submitted, 0 queued]
2020-12-09T15:52:03.39,INFO,MainProcess/25508,classical 69% [49 submitted, 0 queued]
2020-12-09T15:52:27.93,INFO,MainProcess/25508,classical 71% [49 submitted, 0 queued]
2020-12-09T15:54:11.95,INFO,MainProcess/25508,classical 73% [49 submitted, 0 queued]
2020-12-09T15:54:22.94,INFO,MainProcess/25508,classical 75% [49 submitted, 0 queued]
2020-12-09T15:55:15.02,INFO,MainProcess/25508,classical 77% [49 submitted, 0 queued]
2020-12-09T15:55:29.94,INFO,MainProcess/25508,classical 79% [49 submitted, 0 queued]
2020-12-09T15:55:50.22,INFO,MainProcess/25508,classical 81% [49 submitted, 0 queued]
2020-12-09T15:56:05.73,INFO,MainProcess/25508,classical 83% [49 submitted, 0 queued]
2020-12-09T15:56:20.63,INFO,MainProcess/25508,classical 85% [49 submitted, 0 queued]
2020-12-09T15:56:33.81,INFO,MainProcess/25508,classical 87% [49 submitted, 0 queued]
2020-12-09T15:56:38.40,INFO,MainProcess/25508,classical 89% [49 submitted, 0 queued]
2020-12-09T15:56:53.95,INFO,MainProcess/25508,classical 91% [49 submitted, 0 queued]
2020-12-09T15:57:15.98,INFO,MainProcess/25508,classical 93% [49 submitted, 0 queued]
2020-12-09T15:57:17.50,INFO,MainProcess/25508,classical 95% [49 submitted, 0 queued]
2020-12-09T15:57:19.49,INFO,MainProcess/25508,classical 97% [49 submitted, 0 queued]
2020-12-09T15:57:56.87,INFO,MainProcess/25508,classical 100% [49 submitted, 0 queued]
2020-12-09T15:57:56.88,INFO,MainProcess/25508,Received {'rup_data': '212.08 MB', 'pmap': '55.08 KB', 'calc_times': '13.04 KB', 'extra': '4.94 KB'} in 906 seconds
2020-12-09T15:57:57.07,INFO,MainProcess/25508,There are 4096 realization(s)
2020-12-09T15:57:57.07,WARNING,MainProcess/25508,No sources for some TRTs: you should set
discard_trts = Stable Shallow Crust
in C:\Users\sadhika6\AppData\Local\Temp\tmpwwei6wof\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard\OQDisaggregation.mps.ini
2020-12-09T15:57:57.08,INFO,MainProcess/25508,Storing by_task information
2020-12-09T15:57:57.11,INFO,MainProcess/25508,Effective number of ruptures: 2_136_816/4_598_567
2020-12-09T15:57:57.12,INFO,MainProcess/25508,Effective number of sites per rupture: 1
2020-12-09T15:57:57.44,INFO,MainProcess/25508,Building hazard statistics
2020-12-09T15:57:58.86,INFO,MainProcess/25508,build_hazard 100% [1 submitted, 0 queued]
2020-12-09T15:57:58.87,INFO,MainProcess/25508,Received {'hcurves-rlzs': '1.51 MB', 'hmaps-rlzs': '352.33 KB', 'hcurves-stats': '667 B', 'hmaps-stats': '368 B'} in 1 seconds
2020-12-09T15:57:58.88,INFO,MainProcess/25508,The maximum hazard map values are {'PGA': 0.52354276}
2020-12-09T15:57:59.60,CRITICAL,MainProcess/25508,
2020-12-09T15:57:59.61,CRITICAL,MainProcess/25508,Traceback (most recent call last):
File "c:\program files\openquake engine\python3.6\lib\site-packages\openquake\engine\engine.py", line 346, in run_calc
calc.run(exports=exports, **kw)
File "c:\program files\openquake engine\python3.6\lib\site-packages\openquake\calculators\base.py", line 252, in run
self.result = self.execute()
File "c:\program files\openquake engine\python3.6\lib\site-packages\openquake\calculators\disaggregation.py", line 209, in execute
return self.full_disaggregation()
File "c:\program files\openquake engine\python3.6\lib\site-packages\openquake\calculators\disaggregation.py", line 293, in full_disaggregation
self.save_bin_edges()
File "c:\program files\openquake engine\python3.6\lib\site-packages\openquake\calculators\disaggregation.py", line 402, in save_bin_edges
'(%d elements): fix the binning!' % matrix_size)
ValueError: The disaggregation matrix is too large (72900000 elements): fix the binning!


JOBfile

[General]

description = MetroVancouverDisaggregation450
calculation_mode = disaggregation
random_seed = 23

[Geometry]
sites = -123.077607 49.007151

[erf]
rupture_mesh_spacing = 5.0
width_of_mfd_bin = 0.1
area_source_discretization = 10.0

[site_params]
reference_vs30_type = measured
reference_vs30_value = 450.0
reference_depth_to_2pt5km_per_sec = 2.0
reference_depth_to_1pt0km_per_sec = 300.0

[Logic_tree]
source_model_logic_tree_file = ssms/simplifiedCollapsed/logicTree/CanadaSHM6trial_W_CANADA_simplified_collapsedRates_source_model_logic_tree.xml
gsim_logic_tree_file = gmms/logicTree/Vs30_450.xml
number_of_logic_tree_samples = 0

[Hazard_calculation]
intensity_measure_types_and_levels = {"PGA": logscale(0.001,6.0, 40)}
truncation_level = 5
investigation_time = 50.0
maximum_distance = {"Active Shallow Crust": 400.0, "Stable Shallow Crust": 600.0, "Subduction IntraSlab30": 400, "Subduction IntraSlab55": 400, "Subduction Interface": 1000.0}

[disaggregation]
poes_disagg = 0.02, 0.05, 0.1, 0.2, 0.4, 0.5
mag_bin_width = 1.0
distance_bin_width = 10.0
coordinate_bin_width = 0.2
num_epsilon_bins = 3

[output]
individual_curves = true
export_dir = /tmp

[EarthKuei]

Hi Sujan, 

Not sure if you figured out your problem already, but though I would chime in with some input in case this might be helpful to others in the future. For reference, I'm also running the Canadian 6th gen trial model and was running into similar issues with deaggregation calculations initially. In short, I was able to successfully complete the calculation by adjusting/coarsening-up some of the deaggregation binning parameters. The issue I believe for you, is as the ValueError indicates, that the size of your disaggregation matrix is too large, and thus the bins need to be increased in width (to reduce the matrix size).

If you trace your error message to the source code, you'll find this code snippet below, which indicates that the disaggregation matrix size cannot exceed 1e6. As to why a limit of 1e6 has been set I do not know (some context from the developers would be appreciated) but I assume it's performance related.

If you dig a little further into the .bin_edges and .trts fields above that are used for calculating matrix_size, you'll find find the code snippets below. 

So clearly, the deaggregation matrix size is computed as a product of six components as indicated above. You should reduce the size of the matrix by adjusting the configuration parameters such as:

• mag_bin_width
• distance_bin_width
• coordinate_bin_width
• num_epsilon_bins, etc.

I would only remark that adjusting these by trial-and-error can be a little time-consuming since you don't actually get to this error message about the disaggregation matrix size being too large until the stage for classical PSHA calculations are completed, and you begin the stage for deaggregation calculations (which seem fast in comparison). If I could recommend a change to the developers, it would be to perform this check in advance, or better yet, to provide some explicit guidance on how to calculate these things in the OQ manual (although I suppose one could educe it from the source code after starting at it long enough). 

I also tried commenting out the ValueError 1e6 limit and re-running the deaggregation but it had no effect, I assume because it is running off a compiled version.

Cheers,

Kevin Kuei

[EarthKuei]

Just wanted to add that the output deaggregation file contains some useful info for double checking your inputs. For example the following bins were returned from the deaggregation calculation I ran:

1. mag_bin_edges=[4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 9.5]
2. dist_bin_edges=[0.0, 25.0, 50.0, 75.0, 100.0, 125.0, 150.0, 175.0, 200.0, 225.0, 250.0, 275.0, 300.0, 325.0, 350.0, 375.0, 400.0]
3. lon_bin_edges=[-128.79021723289458, -126.08010861644729, -123.37, -120.65989138355272, -117.94978276710542]
4. lat_bin_edges=[44.82172, 46.62036, 48.419, 50.217639999999996, 52.016279999999995]
5. eps_bin_edges=[-5.0, -3.0, -1.0, 1.0, 3.0, 5.0]
6. tectonic_region_types=['Active Shallow Crust', 'Stable Shallow Crust', 'Subduction Interface', 'Subduction IntraSlab30', 'Subduction IntraSlab55']

These are the 6 components that are used for checking the deaggregation matrix size (tectonic_region_types corresponds to trts). The product of len()-1 of items 1-5 and len() of item 6 is less than 1e6.

[Michele Simionato]

On Monday, December 21, 2020 at 12:50:16 AM UTC+1 EarthKuei wrote:

If you trace your error message to the source code, you'll find this code snippet below, which indicates that the disaggregation matrix size cannot exceed 1e6. As to why a limit of 1e6 has been set I do not know (some context from the developers would be appreciated) but I assume it's performance related.

I arbitrarily decided that a disaggregation matrix of 1 million elements was big enough. Without a limit, users would set wrong binning parameters and then the engine would collapse with out of memory errors or worse (it happened times and again).

I would only remark that adjusting these by trial-and-error can be a little time-consuming since you don't actually get to this error message about the disaggregation matrix size being too large until the stage for classical PSHA calculations are completed, and you begin the stage for deaggregation calculations (which seem fast in comparison). If I could recommend a change to the developers, it would be to perform this check in advance, or better yet, to provide some explicit guidance on how to calculate these things in the OQ manual (although I suppose one could educe it from the source code after starting at it long enough). 

Done: the check is now done before starting the classical calculation, see https://github.com/gem/oq-engine/pull/6389. This is actually something that I wanted to do but got lost in the pile of things to do.

Thank you for reminding me of that and thanks for your contribution,

               Michele

[Sujan Raj Adhikari]

thanks, Kevin and Michel.

   Best Regards

Sujan Raj Adhikari

a: University of Western Ontario | London On, Canada e: asu...@gmail.com | w: www.asujan.net.np m: 4033978011 | p: 4033978011

Don’t print everything.....please do protect trees...GO GREEN...save energy

--
You received this message because you are subscribed to the Google Groups "OpenQuake Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openquake-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openquake-users/c5f5facb-8d07-4a2d-9cbd-b0ef25d9ef3dn%40googlegroups.com.