Running classical PSHA risk

164 views
Skip to first unread message

Sujan Raj Adhikari

unread,
Dec 5, 2020, 2:57:40 PM12/5/20
to OpenQuake Users

Hello Openquake user

after running for 4 hours console log is showing this. why I am not receiving any result.


regards

Sujan



2020-12-05T16:37:21.82,INFO,MainProcess/21324,sadhika6 running C:\Users\sadhika6\AppData\Local\Temp\tmpjx9xami_\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard_risk\SHM6_450.mps.ini [--hc=None]

2020-12-05T16:37:21.84,INFO,MainProcess/21324,Using engine version 3.10.1

2020-12-05T16:37:23.12,WARNING,MainProcess/21324,Using 12 cores on molnar-2

2020-12-05T16:37:23.14,INFO,MainProcess/21324,Validated CanadaSHM6trial_W_CANADA_simplified_collapsedRates_source_model_logic_tree.xml in 0.01 seconds

2020-12-05T16:37:23.22,INFO,MainProcess/21324,Checksum of the input files: 4200604625

2020-12-05T16:37:23.24,INFO,MainProcess/21324,Reading the risk model if present

2020-12-05T16:37:56.18,INFO,MainProcess/21324,Reading C:\Users\sadhika6\AppData\Local\Temp\tmpjx9xami_\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard_risk\Exposure_MV.xml

2020-12-05T16:37:59.09,INFO,MainProcess/21324,Associated 61056 assets to 123 sites

2020-12-05T16:38:00.36,INFO,MainProcess/21324,Reducing risk model from 7260 to 592 taxonomies

2020-12-05T16:38:00.45,INFO,MainProcess/21324,Potential number of logic tree paths = 4_096

2020-12-05T16:38:00.46,INFO,MainProcess/21324,1 effective smlt realization(s)

2020-12-05T16:38:00.48,INFO,MainProcess/21324,Reading the source model(s) in parallel

2020-12-05T16:38:00.68,INFO,MainProcess/21324,read_source_model 100% [1 submitted, 0 queued]

2020-12-05T16:38:00.68,INFO,MainProcess/21324,Received {'C:\\Users\\sadhika6\\AppData\\Local\\Temp\\tmpjx9xami_\\Users\\sadhika6\\Desktop\\oqPSHA\\Metro\\Hazard_risk\\ssms\\simplifiedCollapsed\\sourceModels\\CanadaSHM6trial_W_CANADA_simplified_collapsedRates.xml': '92.32 KB'} in 0 seconds

2020-12-05T16:38:00.77,INFO,MainProcess/21324,There are 81 sources

2020-12-05T16:38:00.79,INFO,MainProcess/21324,Checking the sources bounding box

2020-12-05T16:38:00.80,ERROR,MainProcess/21324,source ACMWA: The buffer of 600 km is too large, the bounding box is larger than half the globe: 219 degrees

2020-12-05T16:38:00.95,INFO,MainProcess/21324,Rupture floating factor = 2.92

2020-12-05T16:38:00.96,INFO,MainProcess/21324,Rupture spinning factor = 6.12

2020-12-05T16:38:00.97,INFO,MainProcess/21324,You are not using the pointsource_distance approximation:

https://docs.openquake.org/oq-engine/advanced/common-mistakes.html#pointsource-distance

2020-12-05T16:38:01.31,INFO,MainProcess/21324,gzipping the input files

2020-12-05T16:38:04.73,INFO,MainProcess/21324,Storing risk model

2020-12-05T16:38:06.96,INFO,MainProcess/21324,Weighting the sources

2020-12-05T16:38:22.00,INFO,MainProcess/21324,tot_weight=14_408_660, max_weight=300_000

2020-12-05T16:38:22.01,INFO,MainProcess/21324,TRT = Subduction IntraSlab30

2020-12-05T16:38:22.02,INFO,MainProcess/21324,max_dist=4.85->400 ... 7.25->400, gsims=4, weight=15_900, blocks=1

2020-12-05T16:38:22.03,INFO,MainProcess/21324,TRT = Subduction IntraSlab55

2020-12-05T16:38:22.03,INFO,MainProcess/21324,max_dist=4.85->400 ... 7.65->400, gsims=4, weight=303_852, blocks=1

2020-12-05T16:38:22.06,INFO,MainProcess/21324,TRT = Stable Shallow Crust

2020-12-05T16:38:22.07,INFO,MainProcess/21324,max_dist=4.85->600 ... 7.95->600, gsims=16, weight=10_364_864, blocks=5

2020-12-05T16:38:22.08,INFO,MainProcess/21324,TRT = Subduction Interface

2020-12-05T16:38:22.08,INFO,MainProcess/21324,max_dist=7.05->1000 ... 9.45->1000, gsims=4, weight=267_972, blocks=1

2020-12-05T16:38:22.10,INFO,MainProcess/21324,TRT = Active Shallow Crust

2020-12-05T16:38:22.11,INFO,MainProcess/21324,max_dist=4.85->400 ... 8.45->400, gsims=4, weight=3_456_072, blocks=7

2020-12-05T16:38:22.12,INFO,MainProcess/21324,Estimated upper memory limit per core:

123 sites x 400 levels x 16 gsims x 1 src_multiplicity * 8 bytes = 6.01 MB

2020-12-05T16:38:22.25,INFO,MainProcess/21324,classical 6% [15 submitted, 0 queued]

2020-12-05T16:38:23.41,DEBUG,classical_split_filter#1,produced 1 subtask(s) with mean weight 299160

2020-12-05T16:38:23.47,INFO,MainProcess/21324,classical 12% [16 submitted, 0 queued]

2020-12-05T16:38:23.97,INFO,MainProcess/21324,classical 18% [16 submitted, 0 queued]

2020-12-05T16:38:24.30,DEBUG,classical_split_filter#5,produced 11 subtask(s) with mean weight 297774

2020-12-05T16:38:24.34,DEBUG,classical_split_filter#11,produced 5 subtask(s) with mean weight 299264

2020-12-05T16:38:24.61,DEBUG,classical_split_filter#12,produced 4 subtask(s) with mean weight 299316

2020-12-05T16:38:24.85,DEBUG,classical_split_filter#9,produced 4 subtask(s) with mean weight 299445

2020-12-05T16:38:25.44,DEBUG,classical_split_filter#14,produced 1 subtask(s) with mean weight 299879

2020-12-05T16:38:26.25,DEBUG,classical_split_filter#13,produced 4 subtask(s) with mean weight 299330

2020-12-05T16:38:58.71,INFO,MainProcess/21324,classical 20% [45 submitted, 0 queued]

2020-12-05T16:39:06.22,DEBUG,classical_split_filter#7,produced 7 subtask(s) with mean weight 300000

2020-12-05T16:39:18.97,INFO,MainProcess/21324,classical 21% [52 submitted, 0 queued]

2020-12-05T16:39:48.04,INFO,MainProcess/21324,classical 23% [52 submitted, 0 queued]

2020-12-05T16:40:36.97,INFO,MainProcess/21324,classical 25% [52 submitted, 0 queued]

2020-12-05T16:41:11.71,INFO,MainProcess/21324,classical 26% [52 submitted, 0 queued]

2020-12-05T16:41:46.44,INFO,MainProcess/21324,classical 28% [52 submitted, 0 queued]

2020-12-05T16:42:14.21,INFO,MainProcess/21324,classical 30% [52 submitted, 0 queued]

2020-12-05T16:42:26.55,INFO,MainProcess/21324,classical 32% [52 submitted, 0 queued]

2020-12-05T16:42:47.46,INFO,MainProcess/21324,classical 34% [52 submitted, 0 queued]

2020-12-05T16:48:44.55,INFO,MainProcess/21324,classical 36% [52 submitted, 0 queued]

2020-12-05T16:51:23.57,INFO,MainProcess/21324,classical 38% [52 submitted, 0 queued]

2020-12-05T17:04:19.42,INFO,MainProcess/21324,classical 40% [52 submitted, 0 queued]

2020-12-05T17:06:30.45,INFO,MainProcess/21324,classical 42% [52 submitted, 0 queued]

2020-12-05T17:13:56.47,INFO,MainProcess/21324,classical 44% [52 submitted, 0 queued]

2020-12-05T17:19:55.92,INFO,MainProcess/21324,classical 46% [52 submitted, 0 queued]

2020-12-05T17:26:04.10,INFO,MainProcess/21324,classical 48% [52 submitted, 0 queued]

2020-12-05T17:30:23.19,INFO,MainProcess/21324,classical 50% [52 submitted, 0 queued]

2020-12-05T17:34:32.25,INFO,MainProcess/21324,classical 51% [52 submitted, 0 queued]

2020-12-05T17:36:27.16,INFO,MainProcess/21324,classical 53% [52 submitted, 0 queued]

2020-12-05T17:39:44.31,INFO,MainProcess/21324,classical 55% [52 submitted, 0 queued]

2020-12-05T17:46:50.94,INFO,MainProcess/21324,classical 57% [52 submitted, 0 queued]

2020-12-05T18:03:24.38,INFO,MainProcess/21324,classical 59% [52 submitted, 0 queued]

2020-12-05T18:11:19.21,INFO,MainProcess/21324,classical 61% [52 submitted, 0 queued]

2020-12-05T18:11:53.07,INFO,MainProcess/21324,classical 63% [52 submitted, 0 queued]

2020-12-05T18:18:23.30,INFO,MainProcess/21324,classical 65% [52 submitted, 0 queued]

2020-12-05T18:21:20.04,INFO,MainProcess/21324,classical 67% [52 submitted, 0 queued]

2020-12-05T18:24:44.79,INFO,MainProcess/21324,classical 69% [52 submitted, 0 queued]

2020-12-05T18:31:24.89,INFO,MainProcess/21324,classical 71% [52 submitted, 0 queued]

2020-12-05T18:39:02.73,INFO,MainProcess/21324,classical 73% [52 submitted, 0 queued]

2020-12-05T18:50:09.44,INFO,MainProcess/21324,classical 75% [52 submitted, 0 queued]

2020-12-05T18:51:09.02,INFO,MainProcess/21324,classical 76% [52 submitted, 0 queued]

2020-12-05T18:52:40.62,INFO,MainProcess/21324,classical 78% [52 submitted, 0 queued]

2020-12-05T18:56:10.69,INFO,MainProcess/21324,classical 80% [52 submitted, 0 queued]

2020-12-05T18:57:06.19,INFO,MainProcess/21324,classical 82% [52 submitted, 0 queued]

2020-12-05T18:57:52.52,INFO,MainProcess/21324,classical 84% [52 submitted, 0 queued]

2020-12-05T18:59:34.86,INFO,MainProcess/21324,classical 86% [52 submitted, 0 queued]

2020-12-05T19:01:14.43,INFO,MainProcess/21324,classical 88% [52 submitted, 0 queued]

2020-12-05T19:03:28.90,INFO,MainProcess/21324,classical 90% [52 submitted, 0 queued]

2020-12-05T19:04:18.85,INFO,MainProcess/21324,classical 92% [52 submitted, 0 queued]

2020-12-05T19:12:03.47,INFO,MainProcess/21324,classical 94% [52 submitted, 0 queued]

2020-12-05T19:17:24.95,INFO,MainProcess/21324,classical 96% [52 submitted, 0 queued]

2020-12-05T19:20:37.73,INFO,MainProcess/21324,classical 98% [52 submitted, 0 queued]

2020-12-05T19:24:55.00,INFO,MainProcess/21324,classical 100% [52 submitted, 0 queued]

2020-12-05T19:24:55.02,INFO,MainProcess/21324,Received {'pmap': '69.55 MB', 'calc_times': '34.4 KB', 'extra': '5.28 KB', 'rup_data': '658 B'} in 9992 seconds

2020-12-05T19:24:55.17,INFO,MainProcess/21324,There are 4096 realization(s)

2020-12-05T19:24:55.18,INFO,MainProcess/21324,Storing by_task information

2020-12-05T19:24:55.22,INFO,MainProcess/21324,Effective number of ruptures: 2_399_540/5_368_679

2020-12-05T19:24:55.23,INFO,MainProcess/21324,Effective number of sites per rupture: 114

2020-12-05T19:24:57.21,INFO,MainProcess/21324,Building hazard statistics

2020-12-05T19:24:57.29,INFO,MainProcess/21324,Sent 12 tasks, 5.77 MB in 0 seconds

2020-12-05T19:25:46.82,INFO,MainProcess/21324,build_hazard 4% [13 submitted, 8 queued]

2020-12-05T19:26:27.95,INFO,MainProcess/21324,build_hazard 9% [14 submitted, 7 queued]

2020-12-05T19:26:27.97,INFO,MainProcess/21324,build_hazard 14% [15 submitted, 6 queued]

2020-12-05T19:26:27.99,INFO,MainProcess/21324,build_hazard 19% [16 submitted, 5 queued]

2020-12-05T19:26:28.01,INFO,MainProcess/21324,build_hazard 23% [17 submitted, 4 queued]

2020-12-05T19:26:28.03,INFO,MainProcess/21324,build_hazard 28% [18 submitted, 3 queued]

2020-12-05T19:26:28.06,INFO,MainProcess/21324,build_hazard 33% [19 submitted, 2 queued]

2020-12-05T19:26:28.09,INFO,MainProcess/21324,build_hazard 38% [20 submitted, 1 queued]

2020-12-05T19:26:28.12,INFO,MainProcess/21324,build_hazard 42% [21 submitted, 0 queued]

2020-12-05T19:26:28.13,INFO,MainProcess/21324,build_hazard 47% [21 submitted, 0 queued]

2020-12-05T19:26:28.15,INFO,MainProcess/21324,build_hazard 52% [21 submitted, 0 queued]

2020-12-05T19:26:28.16,INFO,MainProcess/21324,build_hazard 57% [21 submitted, 0 queued]

2020-12-05T19:26:28.18,INFO,MainProcess/21324,build_hazard 61% [21 submitted, 0 queued]

2020-12-05T19:26:47.10,INFO,MainProcess/21324,build_hazard 66% [21 submitted, 0 queued]

2020-12-05T19:27:02.73,INFO,MainProcess/21324,build_hazard 71% [21 submitted, 0 queued]

2020-12-05T19:27:23.05,INFO,MainProcess/21324,build_hazard 76% [21 submitted, 0 queued]

2020-12-05T19:27:23.06,INFO,MainProcess/21324,build_hazard 80% [21 submitted, 0 queued]

2020-12-05T19:27:23.09,INFO,MainProcess/21324,build_hazard 85% [21 submitted, 0 queued]

2020-12-05T19:27:23.12,INFO,MainProcess/21324,build_hazard 90% [21 submitted, 0 queued]

2020-12-05T19:27:23.14,INFO,MainProcess/21324,build_hazard 95% [21 submitted, 0 queued]

2020-12-05T19:27:23.17,INFO,MainProcess/21324,build_hazard 100% [21 submitted, 0 queued]

2020-12-05T19:27:23.19,INFO,MainProcess/21324,Received {'hcurves-rlzs': '1.53 GB', 'hmaps-rlzs': '24.95 MB', 'hcurves-stats': '1.53 MB', 'hmaps-stats': '145.63 KB'} in 145 seconds

2020-12-05T19:27:23.24,INFO,MainProcess/21324,The maximum hazard map values are {'PGA': 1.5785413, 'SA(0.1)': 3.5344646, 'SA(0.2)': 3.8607497, 'SA(0.3)': 3.82982, 'SA(0.5)': 2.893993, 'SA(0.6)': 2.4748278, 'SA(1.0)': 1.7096001, 'SA(10.0)': 0.27088043, 'SA(2.0)': 1.2240895, 'SA(5.0)': 0.41390702}

2020-12-05T19:27:23.59,INFO,MainProcess/21324,Building risk inputs from 4096 realization(s)

2020-12-05T19:27:23.83,INFO,MainProcess/21324,Built 135 risk inputs

2020-12-05T19:27:24.58,INFO,MainProcess/21324,Sending hazard for 2000 assets, 2 sites

2020-12-05T19:27:24.76,INFO,MainProcess/21324,Sending hazard for 2000 assets, 2 sites

2020-12-05T19:27:24.95,INFO,MainProcess/21324,Sending hazard for 2000 assets, 2 sites

2020-12-05T19:27:25.14,INFO,MainProcess/21324,Sending hazard for 2000 assets, 2 sites

2020-12-05T19:27:25.40,INFO,MainProcess/21324,Sending hazard for 2000 assets, 2 sites

2020-12-05T19:27:25.63,INFO,MainProcess/21324,Sending hazard for 2000 assets, 2 sites

2020-12-05T19:27:25.86,INFO,MainProcess/21324,Sending hazard for 1986 assets, 2 sites

2020-12-05T19:27:26.12,INFO,MainProcess/21324,Sending hazard for 1910 assets, 2 sites

2020-12-05T19:27:26.40,INFO,MainProcess/21324,Sending hazard for 1818 assets, 2 sites

2020-12-05T19:27:26.69,INFO,MainProcess/21324,Sending hazard for 1728 assets, 2 sites

2020-12-05T19:27:26.97,INFO,MainProcess/21324,Sending hazard for 2452 assets, 3 sites

2020-12-05T19:27:27.32,INFO,MainProcess/21324,Sending hazard for 2319 assets, 3 sites

2020-12-05T19:27:27.69,INFO,MainProcess/21324,Sending hazard for 2187 assets, 3 sites

2020-12-05T19:27:28.13,INFO,MainProcess/21324,Sending hazard for 2103 assets, 3 sites

2020-12-05T19:27:28.58,INFO,MainProcess/21324,Sending hazard for 2074 assets, 3 sites

2020-12-05T19:27:29.01,INFO,MainProcess/21324,Sending hazard for 1993 assets, 3 sites

2020-12-05T19:27:29.45,INFO,MainProcess/21324,Sending hazard for 1951 assets, 3 sites

2020-12-05T19:27:29.86,INFO,MainProcess/21324,Sending hazard for 2502 assets, 4 sites

2020-12-05T19:27:30.33,INFO,MainProcess/21324,Sending hazard for 2436 assets, 4 sites

2020-12-05T19:27:30.75,INFO,MainProcess/21324,Sending hazard for 2244 assets, 4 sites

2020-12-05T19:27:31.19,INFO,MainProcess/21324,Sending hazard for 2153 assets, 4 sites

2020-12-05T19:27:31.61,INFO,MainProcess/21324,Sending hazard for 2069 assets, 4 sites

2020-12-05T19:27:32.03,INFO,MainProcess/21324,Sending hazard for 2412 assets, 5 sites

2020-12-05T19:27:32.43,INFO,MainProcess/21324,Sending hazard for 2260 assets, 5 sites

2020-12-05T19:27:32.85,INFO,MainProcess/21324,Sending hazard for 2254 assets, 6 sites

2020-12-05T19:27:33.24,INFO,MainProcess/21324,Sending hazard for 2509 assets, 8 sites

2020-12-05T19:27:33.67,INFO,MainProcess/21324,Sending hazard for 2524 assets, 12 sites

2020-12-05T19:27:34.11,INFO,MainProcess/21324,Sending hazard for 2533 assets, 20 sites

2020-12-05T19:27:34.51,INFO,MainProcess/21324,Sending hazard for 639 assets, 18 sites

2020-12-05T19:27:34.93,INFO,MainProcess/21324,Sent 29 tasks, 63.72 MB in 10 seconds

Sujan Raj Adhikari

unread,
Dec 5, 2020, 10:47:14 PM12/5/20
to openqua...@googlegroups.com
I attempted twice same thing after another 4 hours see the chain log

2020-12-05T21:05:52.44,INFO,MainProcess/12700,sadhika6 running C:\Users\sadhika6\AppData\Local\Temp\tmpjrzgo25d\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard_risk\SHM6_450.mps.ini [--hc=None]

2020-12-05T21:05:52.46,INFO,MainProcess/12700,Using engine version 3.10.1

2020-12-05T21:05:53.99,WARNING,MainProcess/12700,Using 12 cores on molnar-2

2020-12-05T21:05:54.01,INFO,MainProcess/12700,Validated CanadaSHM6trial_W_CANADA_simplified_collapsedRates_source_model_logic_tree.xml in 0.01 seconds

2020-12-05T21:05:54.11,INFO,MainProcess/12700,Checksum of the input files: 4200604625

2020-12-05T21:05:54.14,INFO,MainProcess/12700,Reading the risk model if present

2020-12-05T21:06:26.17,INFO,MainProcess/12700,Reading C:\Users\sadhika6\AppData\Local\Temp\tmpjrzgo25d\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard_risk\Exposure_MV.xml

2020-12-05T21:06:29.11,INFO,MainProcess/12700,Associated 61056 assets to 123 sites

2020-12-05T21:06:30.46,INFO,MainProcess/12700,Reducing risk model from 7260 to 592 taxonomies

2020-12-05T21:06:30.67,INFO,MainProcess/12700,Potential number of logic tree paths = 4_096

2020-12-05T21:06:30.68,INFO,MainProcess/12700,1 effective smlt realization(s)

2020-12-05T21:06:30.71,INFO,MainProcess/12700,Reading the source model(s) in parallel

2020-12-05T21:06:30.93,INFO,MainProcess/12700,read_source_model 100% [1 submitted, 0 queued]

2020-12-05T21:06:30.94,INFO,MainProcess/12700,Received {'C:\\Users\\sadhika6\\AppData\\Local\\Temp\\tmpjrzgo25d\\Users\\sadhika6\\Desktop\\oqPSHA\\Metro\\Hazard_risk\\ssms\\simplifiedCollapsed\\sourceModels\\CanadaSHM6trial_W_CANADA_simplified_collapsedRates.xml': '92.32 KB'} in 0 seconds

2020-12-05T21:06:31.02,INFO,MainProcess/12700,There are 81 sources

2020-12-05T21:06:31.05,INFO,MainProcess/12700,Checking the sources bounding box

2020-12-05T21:06:31.09,ERROR,MainProcess/12700,source ACMWA: The buffer of 600 km is too large, the bounding box is larger than half the globe: 219 degrees

2020-12-05T21:06:31.21,INFO,MainProcess/12700,Rupture floating factor = 2.92

2020-12-05T21:06:31.23,INFO,MainProcess/12700,Rupture spinning factor = 6.12

2020-12-05T21:06:31.24,INFO,MainProcess/12700,You are not using the pointsource_distance approximation:

https://docs.openquake.org/oq-engine/advanced/common-mistakes.html#pointsource-distance

2020-12-05T21:06:31.45,INFO,MainProcess/12700,gzipping the input files

2020-12-05T21:06:35.18,INFO,MainProcess/12700,Storing risk model

2020-12-05T21:06:38.16,INFO,MainProcess/12700,Weighting the sources

2020-12-05T21:06:53.12,INFO,MainProcess/12700,tot_weight=14_408_660, max_weight=300_000

2020-12-05T21:06:53.13,INFO,MainProcess/12700,TRT = Subduction IntraSlab30

2020-12-05T21:06:53.14,INFO,MainProcess/12700,max_dist=4.85->400 ... 7.25->400, gsims=4, weight=15_900, blocks=1

2020-12-05T21:06:53.16,INFO,MainProcess/12700,TRT = Subduction IntraSlab55

2020-12-05T21:06:53.17,INFO,MainProcess/12700,max_dist=4.85->400 ... 7.65->400, gsims=4, weight=303_852, blocks=1

2020-12-05T21:06:53.21,INFO,MainProcess/12700,TRT = Stable Shallow Crust

2020-12-05T21:06:53.23,INFO,MainProcess/12700,max_dist=4.85->600 ... 7.95->600, gsims=16, weight=10_364_864, blocks=5

2020-12-05T21:06:53.24,INFO,MainProcess/12700,TRT = Subduction Interface

2020-12-05T21:06:53.25,INFO,MainProcess/12700,max_dist=7.05->1000 ... 9.45->1000, gsims=4, weight=267_972, blocks=1

2020-12-05T21:06:53.28,INFO,MainProcess/12700,TRT = Active Shallow Crust

2020-12-05T21:06:53.30,INFO,MainProcess/12700,max_dist=4.85->400 ... 8.45->400, gsims=4, weight=3_456_072, blocks=7

2020-12-05T21:06:53.31,INFO,MainProcess/12700,Estimated upper memory limit per core:

123 sites x 400 levels x 16 gsims x 1 src_multiplicity * 8 bytes = 6.01 MB

2020-12-05T21:06:53.36,INFO,MainProcess/12700,classical 6% [15 submitted, 0 queued]

2020-12-05T21:06:54.53,DEBUG,classical_split_filter#1,produced 1 subtask(s) with mean weight 299160

2020-12-05T21:06:54.64,INFO,MainProcess/12700,classical 12% [16 submitted, 0 queued]

2020-12-05T21:06:55.12,INFO,MainProcess/12700,classical 18% [16 submitted, 0 queued]

2020-12-05T21:06:55.41,INFO,MainProcess/12700,classical 25% [16 submitted, 0 queued]

2020-12-05T21:06:55.50,DEBUG,classical_split_filter#5,produced 11 subtask(s) with mean weight 297774

2020-12-05T21:06:55.55,DEBUG,classical_split_filter#11,produced 5 subtask(s) with mean weight 299264

2020-12-05T21:06:55.81,DEBUG,classical_split_filter#12,produced 4 subtask(s) with mean weight 299316

2020-12-05T21:06:56.20,DEBUG,classical_split_filter#9,produced 4 subtask(s) with mean weight 299445

2020-12-05T21:06:56.66,DEBUG,classical_split_filter#14,produced 1 subtask(s) with mean weight 299879

2020-12-05T21:06:57.51,DEBUG,classical_split_filter#13,produced 4 subtask(s) with mean weight 299330

2020-12-05T21:07:39.21,DEBUG,classical_split_filter#7,produced 7 subtask(s) with mean weight 300000

2020-12-05T21:09:46.99,INFO,MainProcess/12700,classical 26% [52 submitted, 0 queued]

2020-12-05T21:10:19.41,INFO,MainProcess/12700,classical 28% [52 submitted, 0 queued]

2020-12-05T21:10:49.53,INFO,MainProcess/12700,classical 30% [52 submitted, 0 queued]

2020-12-05T21:11:03.58,INFO,MainProcess/12700,classical 32% [52 submitted, 0 queued]

2020-12-05T21:11:21.49,INFO,MainProcess/12700,classical 34% [52 submitted, 0 queued]

2020-12-05T21:17:27.77,INFO,MainProcess/12700,classical 36% [52 submitted, 0 queued]

2020-12-05T21:19:42.31,INFO,MainProcess/12700,classical 38% [52 submitted, 0 queued]

2020-12-05T21:32:24.71,INFO,MainProcess/12700,classical 40% [52 submitted, 0 queued]

2020-12-05T21:34:28.83,INFO,MainProcess/12700,classical 42% [52 submitted, 0 queued]

2020-12-05T21:41:29.53,INFO,MainProcess/12700,classical 44% [52 submitted, 0 queued]

2020-12-05T21:46:14.73,INFO,MainProcess/12700,classical 46% [52 submitted, 0 queued]

2020-12-05T21:52:33.81,INFO,MainProcess/12700,classical 48% [52 submitted, 0 queued]

2020-12-05T21:56:20.39,INFO,MainProcess/12700,classical 50% [52 submitted, 0 queued]

2020-12-05T22:01:26.43,INFO,MainProcess/12700,classical 51% [52 submitted, 0 queued]

2020-12-05T22:02:42.21,INFO,MainProcess/12700,classical 53% [52 submitted, 0 queued]

2020-12-05T22:05:44.73,INFO,MainProcess/12700,classical 55% [52 submitted, 0 queued]

2020-12-05T22:12:46.72,INFO,MainProcess/12700,classical 57% [52 submitted, 0 queued]

2020-12-05T22:29:41.69,INFO,MainProcess/12700,classical 59% [52 submitted, 0 queued]

2020-12-05T22:36:51.08,INFO,MainProcess/12700,classical 61% [52 submitted, 0 queued]

2020-12-05T22:38:40.77,INFO,MainProcess/12700,classical 63% [52 submitted, 0 queued]

2020-12-05T22:43:31.62,INFO,MainProcess/12700,classical 65% [52 submitted, 0 queued]

2020-12-05T22:46:43.68,INFO,MainProcess/12700,classical 67% [52 submitted, 0 queued]

2020-12-05T22:50:27.14,INFO,MainProcess/12700,classical 69% [52 submitted, 0 queued]

2020-12-05T22:56:31.08,INFO,MainProcess/12700,classical 71% [52 submitted, 0 queued]

2020-12-05T23:03:46.30,INFO,MainProcess/12700,classical 73% [52 submitted, 0 queued]

2020-12-05T23:15:10.21,INFO,MainProcess/12700,classical 75% [52 submitted, 0 queued]

2020-12-05T23:15:17.89,INFO,MainProcess/12700,classical 76% [52 submitted, 0 queued]

2020-12-05T23:18:00.61,INFO,MainProcess/12700,classical 78% [52 submitted, 0 queued]

2020-12-05T23:20:31.92,INFO,MainProcess/12700,classical 80% [52 submitted, 0 queued]

2020-12-05T23:20:52.40,INFO,MainProcess/12700,classical 82% [52 submitted, 0 queued]

2020-12-05T23:21:37.33,INFO,MainProcess/12700,classical 84% [52 submitted, 0 queued]

2020-12-05T23:24:02.68,INFO,MainProcess/12700,classical 86% [52 submitted, 0 queued]

2020-12-05T23:25:32.76,INFO,MainProcess/12700,classical 88% [52 submitted, 0 queued]

2020-12-05T23:27:36.31,INFO,MainProcess/12700,classical 90% [52 submitted, 0 queued]

2020-12-05T23:28:48.10,INFO,MainProcess/12700,classical 92% [52 submitted, 0 queued]

2020-12-05T23:36:01.07,INFO,MainProcess/12700,classical 94% [52 submitted, 0 queued]

2020-12-05T23:40:50.11,INFO,MainProcess/12700,classical 96% [52 submitted, 0 queued]

2020-12-05T23:44:02.27,INFO,MainProcess/12700,classical 98% [52 submitted, 0 queued]

2020-12-05T23:49:31.65,INFO,MainProcess/12700,classical 100% [52 submitted, 0 queued]

2020-12-05T23:49:31.66,INFO,MainProcess/12700,Received {'pmap': '69.55 MB', 'calc_times': '34.4 KB', 'extra': '5.28 KB', 'rup_data': '658 B'} in 9758 seconds

2020-12-05T23:49:31.81,INFO,MainProcess/12700,There are 4096 realization(s)

2020-12-05T23:49:31.83,INFO,MainProcess/12700,Storing by_task information

2020-12-05T23:49:31.88,INFO,MainProcess/12700,Effective number of ruptures: 2_399_540/5_368_679

2020-12-05T23:49:31.88,INFO,MainProcess/12700,Effective number of sites per rupture: 114

2020-12-05T23:49:33.75,INFO,MainProcess/12700,Building hazard statistics

2020-12-05T23:49:33.83,INFO,MainProcess/12700,Sent 12 tasks, 5.77 MB in 0 seconds

2020-12-05T23:50:22.01,INFO,MainProcess/12700,build_hazard 4% [13 submitted, 8 queued]

2020-12-05T23:51:02.49,INFO,MainProcess/12700,build_hazard 9% [14 submitted, 7 queued]

2020-12-05T23:51:02.51,INFO,MainProcess/12700,build_hazard 14% [15 submitted, 6 queued]

2020-12-05T23:51:02.54,INFO,MainProcess/12700,build_hazard 19% [16 submitted, 5 queued]

2020-12-05T23:51:02.56,INFO,MainProcess/12700,build_hazard 23% [17 submitted, 4 queued]

2020-12-05T23:51:02.58,INFO,MainProcess/12700,build_hazard 28% [18 submitted, 3 queued]

2020-12-05T23:51:02.60,INFO,MainProcess/12700,build_hazard 33% [19 submitted, 2 queued]

2020-12-05T23:51:02.63,INFO,MainProcess/12700,build_hazard 38% [20 submitted, 1 queued]

2020-12-05T23:51:02.66,INFO,MainProcess/12700,build_hazard 42% [21 submitted, 0 queued]

2020-12-05T23:51:02.67,INFO,MainProcess/12700,build_hazard 47% [21 submitted, 0 queued]

2020-12-05T23:51:02.69,INFO,MainProcess/12700,build_hazard 52% [21 submitted, 0 queued]

2020-12-05T23:51:02.71,INFO,MainProcess/12700,build_hazard 57% [21 submitted, 0 queued]

2020-12-05T23:51:02.72,INFO,MainProcess/12700,build_hazard 61% [21 submitted, 0 queued]

2020-12-05T23:51:20.53,INFO,MainProcess/12700,build_hazard 66% [21 submitted, 0 queued]

2020-12-05T23:51:35.76,INFO,MainProcess/12700,build_hazard 71% [21 submitted, 0 queued]

2020-12-05T23:51:55.83,INFO,MainProcess/12700,build_hazard 76% [21 submitted, 0 queued]

2020-12-05T23:51:55.84,INFO,MainProcess/12700,build_hazard 80% [21 submitted, 0 queued]

2020-12-05T23:51:55.86,INFO,MainProcess/12700,build_hazard 85% [21 submitted, 0 queued]

2020-12-05T23:51:55.88,INFO,MainProcess/12700,build_hazard 90% [21 submitted, 0 queued]

2020-12-05T23:51:55.89,INFO,MainProcess/12700,build_hazard 95% [21 submitted, 0 queued]

2020-12-05T23:51:55.91,INFO,MainProcess/12700,build_hazard 100% [21 submitted, 0 queued]

2020-12-05T23:51:55.92,INFO,MainProcess/12700,Received {'hcurves-rlzs': '1.53 GB', 'hmaps-rlzs': '24.95 MB', 'hcurves-stats': '1.53 MB', 'hmaps-stats': '145.63 KB'} in 142 seconds

2020-12-05T23:51:55.97,INFO,MainProcess/12700,The maximum hazard map values are {'PGA': 1.5785413, 'SA(0.1)': 3.5344646, 'SA(0.2)': 3.8607497, 'SA(0.3)': 3.82982, 'SA(0.5)': 2.893993, 'SA(0.6)': 2.4748278, 'SA(1.0)': 1.7096001, 'SA(10.0)': 0.27088043, 'SA(2.0)': 1.2240895, 'SA(5.0)': 0.41390702}

2020-12-05T23:51:56.31,INFO,MainProcess/12700,Building risk inputs from 4096 realization(s)

2020-12-05T23:51:56.55,INFO,MainProcess/12700,Built 135 risk inputs

2020-12-05T23:51:57.28,INFO,MainProcess/12700,Sending hazard for 2000 assets, 2 sites

2020-12-05T23:51:57.46,INFO,MainProcess/12700,Sending hazard for 2000 assets, 2 sites

2020-12-05T23:51:57.66,INFO,MainProcess/12700,Sending hazard for 2000 assets, 2 sites

2020-12-05T23:51:57.84,INFO,MainProcess/12700,Sending hazard for 2000 assets, 2 sites

2020-12-05T23:51:58.03,INFO,MainProcess/12700,Sending hazard for 2000 assets, 2 sites

2020-12-05T23:51:58.25,INFO,MainProcess/12700,Sending hazard for 2000 assets, 2 sites

2020-12-05T23:51:58.50,INFO,MainProcess/12700,Sending hazard for 1986 assets, 2 sites

2020-12-05T23:51:58.77,INFO,MainProcess/12700,Sending hazard for 1910 assets, 2 sites

2020-12-05T23:51:59.03,INFO,MainProcess/12700,Sending hazard for 1818 assets, 2 sites

2020-12-05T23:51:59.26,INFO,MainProcess/12700,Sending hazard for 1728 assets, 2 sites

2020-12-05T23:51:59.53,INFO,MainProcess/12700,Sending hazard for 2452 assets, 3 sites

2020-12-05T23:51:59.88,INFO,MainProcess/12700,Sending hazard for 2319 assets, 3 sites

2020-12-05T23:52:00.23,INFO,MainProcess/12700,Sending hazard for 2187 assets, 3 sites

2020-12-05T23:52:00.66,INFO,MainProcess/12700,Sending hazard for 2103 assets, 3 sites

2020-12-05T23:52:01.13,INFO,MainProcess/12700,Sending hazard for 2074 assets, 3 sites

2020-12-05T23:52:01.58,INFO,MainProcess/12700,Sending hazard for 1993 assets, 3 sites

2020-12-05T23:52:01.99,INFO,MainProcess/12700,Sending hazard for 1951 assets, 3 sites

2020-12-05T23:52:02.41,INFO,MainProcess/12700,Sending hazard for 2502 assets, 4 sites

2020-12-05T23:52:02.84,INFO,MainProcess/12700,Sending hazard for 2436 assets, 4 sites

2020-12-05T23:52:03.30,INFO,MainProcess/12700,Sending hazard for 2244 assets, 4 sites

2020-12-05T23:52:03.71,INFO,MainProcess/12700,Sending hazard for 2153 assets, 4 sites

2020-12-05T23:52:04.16,INFO,MainProcess/12700,Sending hazard for 2069 assets, 4 sites

2020-12-05T23:52:04.61,INFO,MainProcess/12700,Sending hazard for 2412 assets, 5 sites

2020-12-05T23:52:05.03,INFO,MainProcess/12700,Sending hazard for 2260 assets, 5 sites

2020-12-05T23:52:05.44,INFO,MainProcess/12700,Sending hazard for 2254 assets, 6 sites

2020-12-05T23:52:05.89,INFO,MainProcess/12700,Sending hazard for 2509 assets, 8 sites

2020-12-05T23:52:06.33,INFO,MainProcess/12700,Sending hazard for 2524 assets, 12 sites

2020-12-05T23:52:06.77,INFO,MainProcess/12700,Sending hazard for 2533 assets, 20 sites

2020-12-05T23:52:07.18,INFO,MainProcess/12700,Sending hazard for 639 assets, 18 sites

2020-12-05T23:52:07.61,INFO,MainProcess/12700,Sent 29 tasks, 63.72 MB in 10 seconds


image.png

check id 45 was the previous after I restarted the Openquake after 1 hour


the new id 50 also gives the same message . it still on for last 1 hour with the same message. what is this bug.... or some mistake in my calculation





   Best Regards
Sujan Raj Adhikari
a: University of Western Ontario London On, Canada
e: asu...@gmail.com w: www.asujan.net.np
m: 4033978011 p: 4033978011
facebook icon twitter icon youtube icon linkedin icon 

Don’t print everything.....please do protect trees...GO GREEN...save energy


--
You received this message because you are subscribed to a topic in the Google Groups "OpenQuake Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openquake-users/-zwU6yOf2pY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openquake-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openquake-users/0a074c05-ef09-4577-ad90-612423270075n%40googlegroups.com.

peter...@gmail.com

unread,
Dec 7, 2020, 3:36:48 AM12/7/20
to OpenQuake Users
Hi,

on our server machine under windows similar things happen.
I guess it is the parallel computing which also causes the calculation to hang without any reason - at least nothing is shown
in the console, all the subprocesses have processor usage 0% and nothing happens for hours.

That is why I was forced to switch to Ubuntu run in virtualbox.
However when testing now it seems to me like putting a line like:
concurrent_tasks = 20
(we have 44 physical cores, 88 logical and 350GB RAM)
sometimes does the trick and the calculation does not hang.

Please try putting the line there If it does not help, otherwise I would also as like to ask the developers if they have any idea
how to avoid these "hanging" states...

Thank you
Peter


Dátum: nedeľa 6. decembra 2020, čas: 4:47:14 UTC+1, odosielateľ: asu...@gmail.com

Sujan Raj Adhikari

unread,
Dec 7, 2020, 8:46:16 AM12/7/20
to openqua...@googlegroups.com, peter...@gmail.com

Hi Peter
Nothing change i updated as you suggested .. i reduce the assets to 80, check the console log . something from last 1 hour

2020-12-07T13:06:41.31,INFO,MainProcess/10676,sadhika6 running C:\Users\sadhika6\AppData\Local\Temp\tmpg2y6y8rg\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard_risk\New450risk.mps.ini [--hc=24]

2020-12-07T13:06:41.34,INFO,MainProcess/10676,Using engine version 3.10.1

2020-12-07T13:06:42.69,WARNING,MainProcess/10676,Using 12 cores on molnar-2

2020-12-07T13:06:42.70,INFO,MainProcess/10676,Checksum of the input files: 4013716580

2020-12-07T13:06:42.75,INFO,MainProcess/10676,Reading the risk model if present

2020-12-07T13:07:12.69,INFO,MainProcess/10676,Reading C:\Users\sadhika6\AppData\Local\Temp\tmpg2y6y8rg\Users\sadhika6\Desktop\oqPSHA\Metro\Hazard_risk\Exposure_MV.xml

2020-12-07T13:07:12.73,INFO,MainProcess/10676,Associated 80 assets to 1 sites

2020-12-07T13:07:12.76,INFO,MainProcess/10676,Reducing risk model from 7260 to 80 taxonomies

2020-12-07T13:07:13.03,INFO,MainProcess/10676,Storing risk model

2020-12-07T13:07:13.52,INFO,MainProcess/10676,Building risk inputs from 4096 realization(s)

2020-12-07T13:07:13.54,INFO,MainProcess/10676,Built 1 risk inputs

2020-12-07T13:07:14.51,INFO,MainProcess/10676,Sending hazard for 80 assets, 1 sites

2020-12-07T13:07:15.01,INFO,MainProcess/10676,Sent 1 tasks, 1.77 MB in 0 seconds

   
Best Regards
Sujan Raj Adhikari
a: University of Western Ontario London On, Canada
e: asu...@gmail.com w: www.asujan.net.np
m: 4033978011 p: 4033978011
facebook icon twitter icon youtube icon linkedin icon 

Don’t print everything.....please do protect trees...GO GREEN...save energy

peter...@gmail.com

unread,
Dec 7, 2020, 11:29:41 PM12/7/20
to OpenQuake Users
I guess it is the situation described also here:

" Hence, since recent versions of the engine we are no longer returning data from the tasks via celery/rabbitmq: instead, we use zeromq. This is hidden from the user, but internally the engine keeps track of all tasks that were submitted and waits until they send the message that they finished. If one tasks runs out of memory badly and never sends the message that it finished, the engine may hang, waiting for the results of a task that does not exist anymore. You have to be careful. What we did in our cluster is to set some memory limit on the celery user with the cgroups technology, so that an out of memory task normally fails in a clean way with a Python MemoryError, sends the message that it finished and nothing particularly bad happens. Still, in situations of very heavy load the OOM killer may enter in action aggressively and the main calculation may hang: in such cases you need a good sysadmin having put in place some monitor mechanism, so that you can see if the OOM killer entered in action and then you can kill the main process of the calculation by hand. There is not much more that can be done for really huge calculations that stress the hardware as much as possible. You must be prepared for failures."

It seems to me like this hanging much often happens on windows, never had this problem on Ubuntu...
Could you please Michelle share some tricks how to trial and error resolve the situation, when the hang occurs?

Thank you very much
Peter

Dátum: pondelok 7. decembra 2020, čas: 14:46:16 UTC+1, odosielateľ: asu...@gmail.com

Sujan Raj Adhikari

unread,
Dec 8, 2020, 6:52:36 PM12/8/20
to OpenQuake Users
have any one gone through this ... help needed

Sujan Raj Adhikari

unread,
Dec 9, 2020, 8:53:47 AM12/9/20
to OpenQuake Users
still waiting for help

Michele Simionato

unread,
Dec 10, 2020, 1:54:19 AM12/10/20
to OpenQuake Users
On Tuesday, December 8, 2020 at 5:29:41 AM UTC+1 peter...@gmail.com wrote:
I guess it is the situation described also here:

" Hence, since recent versions of the engine we are no longer returning data from the tasks via celery/rabbitmq: instead, we use zeromq. This is hidden from the user, but internally the engine keeps track of all tasks that were submitted and waits until they send the message that they finished. If one tasks runs out of memory badly and never sends the message that it finished, the engine may hang, waiting for the results of a task that does not exist anymore. You have to be careful. What we did in our cluster is to set some memory limit on the celery user with the cgroups technology, so that an out of memory task normally fails in a clean way with a Python MemoryError, sends the message that it finished and nothing particularly bad happens. Still, in situations of very heavy load the OOM killer may enter in action aggressively and the main calculation may hang: in such cases you need a good sysadmin having put in place some monitor mechanism, so that you can see if the OOM killer entered in action and then you can kill the main process of the calculation by hand. There is not much more that can be done for really huge calculations that stress the hardware as much as possible. You must be prepared for failures."

It seems to me like this hanging much often happens on windows, never had this problem on Ubuntu...
Could you please Michelle share some tricks how to trial and error resolve the situation, when the hang occurs?

Possibly zeromq works better on linux than on windows, so in case of large calculations it is more likely to hang on windows than on linux.
Notice that we ALWAYS recommended to use linux for serious calculations. I think very unlikely that we will even improve the situation on windows
(unless somebody sponsors us with a 5 digit check just to do that). 

On the other hand, recent versions of windows have a linux subsystem. Perhaps by installing the engine on the linux subsystem the situation will improve, but I have idea if that is the case.

          Michele

Sujan Raj Adhikari

unread,
Dec 10, 2020, 6:29:39 AM12/10/20
to OpenQuake Users
so installing  Windows Subsystem for Linux  and running it . will solve the problem or i need the whole Linux system please highlight.  never used the Linux before.
sujan

Reply all
Reply to author
Forward
0 new messages