Risk calculation in Ubuntu

[Sujan Raj Adhikari]

It seems it also froze in ubuntu like same as Windows too. i run this one yesterday night around 9:30 pm. It's 11.30 am now. any thoughts. I am running in the ubuntu system not in virtual box. 

2020-12-18T03:09:41.51,INFO,MainProcess/73668,asujan running /tmp/tmpgolk3rcg/home/asujan/Documents/work_PHSA_New/RISK/OQrisk.mps.ini [--hc=7]

2020-12-18T03:09:41.53,INFO,MainProcess/73668,Using engine version 3.10.1

2020-12-18T03:09:42.34,WARNING,MainProcess/73668,Using 8 cores on asujan-ubuntu

2020-12-18T03:09:42.37,INFO,MainProcess/73668,Checksum of the input files: 1143302558

2020-12-18T03:09:42.40,INFO,MainProcess/73668,Reading the risk model if present

2020-12-18T03:11:05.00,INFO,MainProcess/73668,Reading /tmp/tmpgolk3rcg/home/asujan/Documents/work_PHSA_New/RISK/Exposure_MV.xml

2020-12-18T03:11:11.66,INFO,MainProcess/73668,Associated 61056 assets to 123 sites

2020-12-18T03:11:15.33,INFO,MainProcess/73668,Reducing risk model from 7260 to 592 taxonomies

2020-12-18T03:11:16.38,INFO,MainProcess/73668,Storing risk model

2020-12-18T03:11:20.16,INFO,MainProcess/73668,Building risk inputs from 4096 realization(s)

2020-12-18T03:11:20.76,INFO,MainProcess/73668,Built 135 risk inputs

2020-12-18T03:11:23.58,INFO,MainProcess/73668,Sending hazard for 3000 assets, 3 sites

2020-12-18T03:11:23.99,INFO,MainProcess/73668,Sending hazard for 3000 assets, 3 sites

2020-12-18T03:11:24.41,INFO,MainProcess/73668,Sending hazard for 3000 assets, 3 sites

2020-12-18T03:11:24.85,INFO,MainProcess/73668,Sending hazard for 3000 assets, 3 sites

2020-12-18T03:11:25.29,INFO,MainProcess/73668,Sending hazard for 2944 assets, 3 sites

2020-12-18T03:11:26.42,INFO,MainProcess/73668,Sending hazard for 3675 assets, 4 sites

2020-12-18T03:11:27.35,INFO,MainProcess/73668,Sending hazard for 3277 assets, 4 sites

2020-12-18T03:11:28.31,INFO,MainProcess/73668,Sending hazard for 3076 assets, 4 sites

2020-12-18T03:11:29.34,INFO,MainProcess/73668,Sending hazard for 3578 assets, 5 sites

2020-12-18T03:11:30.71,INFO,MainProcess/73668,Sending hazard for 3370 assets, 5 sites

2020-12-18T03:11:31.86,INFO,MainProcess/73668,Sending hazard for 3239 assets, 5 sites

2020-12-18T03:11:32.99,INFO,MainProcess/73668,Sending hazard for 3703 assets, 6 sites

2020-12-18T03:11:34.06,INFO,MainProcess/73668,Sending hazard for 3392 assets, 6 sites

2020-12-18T03:11:35.50,INFO,MainProcess/73668,Sending hazard for 3676 assets, 7 sites

2020-12-18T03:11:36.57,INFO,MainProcess/73668,Sending hazard for 3775 assets, 8 sites

2020-12-18T03:11:37.74,INFO,MainProcess/73668,Sending hazard for 3493 assets, 9 sites

2020-12-18T03:11:38.81,INFO,MainProcess/73668,Sending hazard for 3762 assets, 14 sites

2020-12-18T03:11:40.26,INFO,MainProcess/73668,Sending hazard for 3796 assets, 30 sites

2020-12-18T03:11:41.49,INFO,MainProcess/73668,Sending hazard for 300 assets, 13 sites

2020-12-18T03:11:43.16,INFO,MainProcess/73668,Sent 19 tasks, 46.31 MB in 19 seconds

[Sujan Raj Adhikari]

i used the following code.  i have 123 sites and 60000 building..

[General]

description = TsawwassenLossEstimation

calculation_mode = classical_risk

random_seed = 23

optimize_same_id_sources = True

[Geometry]

sites_csv = sites_NBCC2020localities_west.csv

[Exposure]

exposure_file = Exposure_MV.xml

[Loss_Calculation_Vulnerability]

structural_vulnerability_file = vulnerabilitymodel_structural.xml

nonstructural_vulnerability_file = vulnerabilitymodel_nonstructural.xml

contents_vulnerability_file = vulnerabilitymodel_contents.xml

insured_losses = False

[Risk_calculation]

asset_hazard_distance = 20

individual_curves = true

[Risk_outputs]

export_dir = /tmp

avg_losses = true

risk_investigation_time = 1

conditional_loss_poes = 0.000404, 0.001025, 0.002105, 0.004453, 0.0101645, 0.013767

[peter...@gmail.com]

Hi,

I would like to kindly ask Michele to give some advice how to fine tune parameters like concurrent_tasks to make the calculation run.

As mentioned before, the reason why it is hanging is most probably one or more of the subprocesses ended due to insufficient memory

but the main process did not get info about that.

Would be great if there was say some rule of thumb how to tweak multiprocessing so that the trial and error would go faster.

Thank you very much

Peter

Dátum: piatok 18. decembra 2020, čas: 17:13:52 UTC+1, odosielateľ: asu...@gmail.com

[Michele Simionato]

On Friday, December 18, 2020 at 5:37:00 PM UTC+1 peter...@gmail.com wrote:

Hi,

I would like to kindly ask Michele to give some advice how to fine tune parameters like concurrent_tasks to make the calculation run.

As mentioned before, the reason why it is hanging is most probably one or more of the subprocesses ended due to insufficient memory

but the main process did not get info about that.

Would be great if there was say some rule of thumb how to tweak multiprocessing so that the trial and error would go faster.#

Unfortunately there is no magic recipe. Most of my time in GEM is spent making big calculations to run, by tuning the parameters

and by refactoring the calculators until they work. It is hard work and a time-consuming process.

Increasing the number of concurrent tasks can decrease the memory usage, but even increase it, if there are too many tasks piling up.

Also, the performance of some calculations are very dependent on the machine you are using (if it has a fast memory or a slow memory,

if it has a big or small L3 cache, if it has few powerful CPUs or a lot poor CPUs, etc).

Sorry if I am not being useful, but this is as things are.

            Michele