Running more than 65536 sites in event-based calculation

[mal...@sunstonerisk.com]

Hi All,

Is there any way I can run more than 65536 sites at a time for an event-based calculation?

ValueError: The event based calculator is restricted to 65536 sites

Thanks

Malcolm

[Michele Simionato]

Not in a single calculation. What we do is to aggregate the assets on a grid so that there are less than 65536 sites.

For instance we put the all of Canada in a grid of 10 km and it took only 20,000 sites. Mnd that most of Canada is

empty and the grid does not need to fill all the points, only the points were there are assets.

   Michele

[mal...@sunstonerisk.com]

OK thanks Michele. This is already aggregated (for a very large region), so I'll need to break up the job into smaller units.

Is this limitation something that could be set by the user in a future version? The 65536 number implies it might be due to the use of 16-bit integers somewhere in the code though.

[Michele Simionato]

On Wednesday, October 24, 2018 at 11:12:34 AM UTC+2, mal...@sunstonerisk.com wrote:

OK thanks Michele. This is already aggregated (for a very large region), so I'll need to break up the job into smaller units.

Is this limitation something that could be set by the user in a future version? The 65536 number implies it might be due to the use of 16-bit integers somewhere in the code though.

The problem is that when there a lot of sites the calculation will likely fail for other reasons (out of memory, data transfer, etc) so I put this strict limit on purpose,

to get an early error instead of an error after hours/days of calculation. I do not want to remove this limit because it is actually helpful.

If, in the far future, the engine performance will improve by order of magnitudes and computing more than 65536 sites will become feasible, I will raise the limit

to 4 billion sites. It would be pretty easy to do, but I do see that we are close to that point at the moment.

       Michele

[mal...@sunstonerisk.com]

I see in the python code that the warning relates to possible exceedance of limits of the dtypes in gmv_dt, where sid is a 32-bit integer so can go higher than 65536. I might relax this constraint and see what happens. It's running on a large server, but I will take your warning onboard!

[Michele Simionato]

On Wednesday, October 24, 2018 at 3:12:37 PM UTC+2, mal...@sunstonerisk.com wrote:

I see in the python code that the warning relates to possible exceedance of limits of the dtypes in gmv_dt, where sid is a 32-bit integer so can go higher than 65536. I might relax this constraint and see what happens. It's running on a large server, but I will take your warning onboard!

Notice that there is another limit on the size of the stored GMFs, so you could fail while storing the GMFs. Let me know how it goes,

         Michele 

PS: the limit on the GMF size is because otherwise the risk part will become too slow or go out of memory.

[Hyeuk Ryu]

Hi Michele,

Sorry to interrupt the discussion, but I'd like to know how to aggregate assets on a grid. Do you have a python script for the job? 

Thanks.

Hyeuk

[Michele Simionato]

On Thursday, October 25, 2018 at 7:10:28 AM UTC+2, Hyeuk Ryu wrote:

Hi Michele,

Sorry to interrupt the discussion, but I'd like to know how to aggregate assets on a grid. Do you have a python script for the job? 

The engine automatically grids the assets on a grid if your job.ini specifies a region_grid_spacing parameter at least since version 3.1.

  Michele

[mal...@sunstonerisk.com]

I tried running > 65536 sites but the run failed due to integer overflows. Memory usage was fine.

If I breakup the sites into separate runs, what are the random seeds that I need to set in an event-based hazard simulation to ensure that I can combine the separate runs by eid afterwards?

[Michele Simionato]

On Thursday, October 25, 2018 at 12:21:04 PM UTC+2, mal...@sunstonerisk.com wrote:

I tried running > 65536 sites but the run failed due to integer overflows. Memory usage was fine.

If I breakup the sites into separate runs, what are the random seeds that I need to set in an event-based hazard simulation to ensure that I can combine the separate runs by eid afterwards?

Unfortunately, there is no supported way to go over the  65536 sites limit. We may consider doing that in the future, but not in the near future.

[Michele Simionato]

On Wednesday, October 24, 2018 at 10:58:46 AM UTC+2, mal...@sunstonerisk.com wrote:

Since yesterday the engine recognizes a parameter `max_num_sites` that you can set in the job.in filei.

The default is still 65,536, but you can raise the limit to any value up to 2**32 sites. However I did not

add any new optimization, so computations with large number of sites at best will be inefficient and at

worse will run out of memory. The new parameter will be available in the next release of the engine,

due for mid December.

HTH,

                          Michele

[shubha...@gmail.com]

Hello Michele, 

Thanks for the update regarding adding a parameter 'max_num_sites' that we can set in the job.ini file.

Having said that, it would be really nice if you can let me know in which section we need to add that parameter. By section, i mean to say for e.g. [general],[geometry] ,[logic_tree] and others

Thanks,
Shubham

[Michele Simionato]

Any section is good, but I would suggest the section [calculation].

[shubha...@gmail.com]

Hello Michele,

Thanks for your prompt response.

I tried running the engine by adding  following query in the [calculation] section but got a message saying :

  

Warning : root: The parameter 'max_num_sites' is unknown,ignoring

And therefore, i got stuck again with the same error. I guess, Its something related to the openquake version. I'm currently using OpenQuake_Engine_3.2.0-1 version.

----------------------------------------

[Michele Simionato]

On Friday, December 14, 2018 at 10:28:12 AM UTC+1, shubha...@gmail.com wrote:

Hello Michele,

Thanks for your prompt response.

I tried running the engine by adding  following query in the [calculation] section but got a message saying :

  

Warning : root: The parameter 'max_num_sites' is unknown,ignoring

And therefore, i got stuck again with the same error. I guess, Its something related to the openquake version. I'm currently using OpenQuake_Engine_3.2.0-1 version.

Yes, the new parameter is in the current master, so you need to install it or just wait for the release of engine 3.3, which is feature complete but it may be released next year

because there is a lot to document and we are close to the Christmas period. 

   Michele

[mal...@sunstonerisk.com]

I very much need to find a workaround for running large numbers of sites (>200k) for many years (10k).

I was unable to make shorter runs with all sites but have successfully made multiple hazard runs with identical model setup (including same random, master and ses seeds) but with different sites.

In the output hdf5 files I have many matching events.eid across the different runs (>50%). Can I assume they from the same rupture i.e. combine the outputs by eid? I will plot the combined event shakemaps to check, but wanted to get your input.

Thanks

Malcolm

[michele....@globalquakemodel.org]

On Friday, May 3, 2019 at 7:20:38 AM UTC+2, mal...@sunstonerisk.com wrote:

I very much need to find a workaround for running large numbers of sites (>200k) for many years (10k).

I was unable to make shorter runs with all sites but have successfully made multiple hazard runs with identical model setup (including same random, master and ses seeds) but with different sites.

In the output hdf5 files I have many matching events.eid across the different runs (>50%). Can I assume they from the same rupture i.e. combine the outputs by eid? I will plot the combined event shakemaps to check, but wanted to get your input.

No, you cannot assume that. However, I have good news for you. Since 6 months ago, when you first asked about the 65536 sites limit, there has been a lot of progress on the engine. Now there is no limit on the number of sites and I have seen a real calculation for Canada with half million sites running to completion in a few hours. Of course it requires memory and computational power, but it can be done, while before it was impossible. But there is a trick to it. You should use the new ebrisk calculator, introduced experimentally in engine version 3.4. In engine 3.5 (the current master) such calculator has been improved, so that now it also manages the coefficient of variations and can be used as a fulll replacement for the old event based risk calculator. If you want to try it I would recommend using the current master. Also, I would recommend double checking the results in small calculations with the old calculator, since this is a new and shiny and not battle tested as the old one.

HTH,

   Michele

[mal...@sunstonerisk.com]

Great thanks Michele. That is good news.

I will give the ebrisk calculator in v3.5 a try.

Malcolm

[Peter Pažák]

Hi Michele,

this however contradicts what we have been told before. We are running Canada (970k sites) split into 19 parts, always full source model and the same ses seed.

This should ensure the ruptures and events are the same for all runs, so we should be able to join the parts of event footprints (IMT's) calculated in

each run by eid - I checked several events if they have same rupture parameters and continuous footprints through boundary and everything seemed fine.

If this would not be true, we have problem in the calculations.

Thank you for clarification

Peter

Dňa piatok, 3. mája 2019 8:53:28 UTC+2 michele....@globalquakemodel.org napísal(-a):

[mal...@sunstonerisk.com]

That's good to hear Peter as I'm unable to use the ebrisk calculator as I only need access to the hazard.

I was just reading the version 3.4 release notes and it mentioned that for a multi-part run that all ruptures need to be calculated in the first run and used for subsequent runs, but I do not see how to use previously computed ruptures in a hazard run. From what you say, it looks like that setting the same random seeds between runs might be enough to enable matching of the events between runs using the event ID.

[michele....@globalquakemodel.org]

On Friday, May 3, 2019 at 4:50:01 PM UTC+2, Peter Pažák wrote:

Hi Michele,

this however contradicts what we have been told before. We are running Canada (970k sites) split into 19 parts, always full source model and the same ses seed.

This should ensure the ruptures and events are the same for all runs, so we should be able to join the parts of event footprints (IMT's) calculated in

each run by eid - I checked several events if they have same rupture parameters and continuous footprints through boundary and everything seemed fine.

If this would not be true, we have problem in the calculations.

Thank you for clarification

Peter

To run Canada with event_based_risk you should follow this (very inconvenient) procedure:

1. set ground_motion_fields=true and run a calculation for ALL of Canada. The you will have the full rupture set computed consistently. Let's say that the calculation ID is 1000.

2. split the exposure in 19 blocks and run all 19 calculations starting from the same precomputed ruptures, i.e. run

  $ oq engine --run job1.ini --hazard-calculation-id=1000

  $ oq engine --run job2.ini --hazard-calculation-id=1000

  ...

  $ oq engine --run job19.ini --hazard-calculation-id=1000

3. collect the losses manually from the 19 datastores.

Wit the ebrisk calculator instead the procedure is to run one calculation with the full exposure:

$ oq engine --run job.ini  (tha'ts all!)

Implementing the ebrisk calculator took 6 months, so in the past we were recommending to split the calculations because we could not ask the users to wait for 6 months.

But now the work is done, even if I am sure there will be some points to refine, and I would recommend to use the new calculator.

           Michele

[Peter Pažák]

Ok, we are just calculating hazard using OQ, loss is calculated in our software. Is there a problem connecting hazard through eid if just a part of the sites is used each time but full source model and the same ses seed? Thank you

[michele....@globalquakemodel.org]

On Sunday, May 5, 2019 at 8:19:01 AM UTC+2, Peter Pažák wrote:

Ok, we are just calculating hazard using OQ, loss is calculated in our software. Is there a problem connecting hazard through eid if just a part of the sites is used each time but full source model and the same ses seed?

Not necessarily, it could be than in a specific version of the engine the approach works and you get the same ruptures and same GMFs by splitting and not splitting the sites, but it is an implementation detail, it is not

guaranteed behavior: it could easily stop working in the next version, so you should be careful. Producing the full set of ruptures in a single run and then computing the GMFs for different blocks of sites in subsequent runs is how the splitting is supposed to work. It is also more efficient.

                 Michele

[mal...@vsamp.com]

Not necessarily, it could be than in a specific version of the engine the approach works and you get the same ruptures and same GMFs by splitting and not splitting the sites, but it is an implementation detail, it is not

guaranteed behavior: it could easily stop working in the next version, so you should be careful. Producing the full set of ruptures in a single run and then computing the GMFs for different blocks of sites in subsequent runs is how the splitting is supposed to work. It is also more efficient.

I have been doing the same as Peter up until now - running different sites with the same seeds and it has been working OK. I had assumed using --hazard-calculation-id just uses the GMFs from a previous run and not the ruptures, which was not appropriate when trying to run a large number of sites all with specified site conditions. I will switch over to the method that you propose Michele, which would be more efficient as it does not need to calculate the ruptures each time.

I will do the initial run using a grid of sites to calculate the ruptures. Would you have any recommendations regarding the grid resolution to ensure that the ruptures are spatially complete?

[michele....@globalquakemodel.org]

On Sunday, May 5, 2019 at 8:44:19 AM UTC+2, mal...@vsamp.com wrote:

I will do the initial run using a grid of sites to calculate the ruptures. Would you have any recommendations regarding the grid resolution to ensure that the ruptures are spatially complete?

The way we do it for Canada (and all countries which are mostly empty) is to start from the exposure. Given the exposure and a file with the vs30 values with the command `oq prepare_site_model`

you can generate a site_model.csv file for all Canada. You can choose the grid spacing depending on how fine your exposure is and how many sites you want to generate. The site_model.csv file

will contain only points on the grid when there are assets, so it will be a lot smaller than the full grid. Once you have the site_model.csv file you can split it manually in say 19 files and then run the 19

subcalculations. In the calculation 0 (the one producing the ruptures) instead use the full file and remember to set ground_motion_fields = false.

Here is the help for prepare_site_model in engine 3.4:

$ oq prepare_site_model --help

usage: oq prepare_site_model [-h] [-e EXPOSURE_XML [EXPOSURE_XML ...]] [-1]

                             [-2] [-3] [-g 0] [-s 5] [-o site_model.csv]

                             vs30_csv [vs30_csv ...]

Prepare a sites.csv file from an exposure xml file, a vs30 csv file and a grid

spacing which can be 0 (meaning no grid). For each asset site or grid site the

closest vs30 parameter is used. The command can also generate (on demand) the

additional fields z1pt0, z2pt5 and vs30measured which may be needed by your

hazard model, depending on the required GSIMs.

positional arguments:

  vs30_csv              files with lon,lat,vs30 and no header

optional arguments:

  -h, --help            show this help message and exit

  -e EXPOSURE_XML [EXPOSURE_XML ...], --exposure-xml EXPOSURE_XML [EXPOSURE_XML ...]

                        exposure(s) in XML format

  -1, --z1pt0           build the z1pt0

  -2, --z2pt5           build the z2pt5

  -3, --vs30measured    build the vs30measured

  -g 0, --grid-spacing 0

                        grid spacing in km (the default 0 means no grid)

  -s 5, --site-param-distance 5

                        sites over this distance are discarded

  -o site_model.csv, --output site_model.csv

                        output file

[mal...@sunstonerisk.com]

Thanks Michele.

I prepare the site model as XML files outside of OQ, so that part is all fine.

My first attempt at the new workflow was unsuccessful.

I have 360k sites for which I want event-based hazard only. Each site has unique site conditions.

I first ran a subset of 32k sites covering the full region to get the ruptures, but with ground_motion_fields=false. This is my analysis 0.

Each subsequent run, with different site_model_file XML containing a subset of the 360k sites and ground_motion_fields=true always produced ground motion at the locations of the 32k original sites only (analysis 0) i.e. the output locations were identical in each analysis and the new site_model_file was ignored.

[mal...@sunstonerisk.com]

OK reading manual again I'm guessing:

I use all 360k sites in "site_model_file" for the initial run to get the ruptures.

For each subsequent run I enter the subset of sites using "sites_csv".

[mal...@sunstonerisk.com]

Unfortunately this doesn't work either. In the runs used to calculate the GMFs from the pre-calculated ruptures I get the error:

ValueError: Please set a value for 'reference_vs30_value', this is required by the GSIM

This seems to imply that it is not recognising the site model used in the first ruptures hazard run.

Adding the reference in the GMF runs to the site_model_file that was used to calculate the ruptures gives a different error:

ValueError: The sites are overdetermined

I'm going to resort back to the original method that both Peter and I seem to be able to run successfully of doing independent runs across the exposure groups but with the same random seeds, then matching events by eid.

[michele....@globalquakemodel.org]

You cannot have both a site_model.csv and a sites.csv. You should remove the sites.csv and split the site_model.csv in the subsequent runs.

                Michele 

[mal...@sunstonerisk.com]

You cannot have both a site_model.csv and a sites.csv. You should remove the sites.csv and split the site_model.csv in the subsequent runs.

Thanks but I tried that and I always got the GMF at the sites used in the original ruptures run (see my post from today 2:20pm).

This was using an XML site model using "site_model_file". Is site_model.csv something different?

[michele....@globalquakemodel.org]

The workflow of splitting the sites/site model is not really supported. It should work, but there are no tests checking that, so it can stop working at every new release.

The reason is that I think that spending time on the splitting (which is a hack) would be a waste: it would be much better to spend the same amount of time optimizing the event based calculator

so that it can run all sites in a single calculation. This is what I did for the ebrisk calculator in the last 6 months. For the event based hazard calculator the work to do would be a lot less, a few weeks and not

months, but it still requires a paying sponsor to be started. So, don't hold your breath on this front, paying sponsors have not appeared yet. Same thing for hazard disaggregation.

Instead, we will keep working on risk.

    Michele

[mal...@sunstonerisk.com]

OK makes sense. I'm doing what I can to help on the sponsor side ;)

v3.5 is able to handle a larger number of sites than v3.3. I can run 1k years of hazard sims with 360k sites, so can split by years rather than sites.

[Raghucharan M C]

Dear sir,

I am computing UHS from classical PSHA for 54 sites.

I have stored the lon lat of 54 sites in "hazard_stations.csv" file and calling it in [geometry] section of job.ini file

I also have "site_model_file = site_model.csv" in [site_params] section of job.ini file

during a run, the program is giving an error

"ValueError: The sites are overdetermined"

After going through your reply, I have removed the "hazard_stations.csv" file and copied all 54 sites [geometry] section.

However, I am still getting the same error.

I request you to suggest where am I going wrong?

[Michele Simionato]

On Friday, July 26, 2019 at 8:41:31 AM UTC+2, Raghucharan M C wrote:

Dear sir,

I am computing UHS from classical PSHA for 54 sites.

I have stored the lon lat of 54 sites in "hazard_stations.csv" file and calling it in [geometry] section of job.ini file

I also have "site_model_file = site_model.csv" in [site_params] section of job.ini file

during a run, the program is giving an error

"ValueError: The sites are overdetermined"

After going through your reply, I have removed the "hazard_stations.csv" file and copied all 54 sites [geometry] section.

However, I am still getting the same error.

I request you to suggest where am I going wrong?

Perhaps you also have a line sites = ... in your job.ini or a region.

There must be only one way to specify the sites, either with sites, sites_csv or site_model or a grid.

[Raghucharan M C]

Thank you, sir, it's working

Kind Regards,

Raghucharan M.C.

Ph.D. Student

Department of Civil Engineering 

IIT Hyderabad.

--
You received this message because you are subscribed to a topic in the Google Groups "OpenQuake Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openquake-users/RSQkrGEKALc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openquake-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openquake-users/8c000ec7-38a3-420e-97a3-842dd64519af%40googlegroups.com.

[Michele Simionato]

On Friday, July 26, 2019 at 3:47:54 PM UTC+2, Raghucharan M C wrote:

Dear Sir,

I am enclosing a figure showing the 2% PGA levels across districts along with the boundaries of area source.

I observed that farther stations from the source have higher PGA values, which is counter-intuitive.

I am unable to find where I am committing the mistake.

I request your suggestions.

I have no way of knowing what is wrong if I do not have the input files you are using. But even in that it case, it is your job as a Ph. D. student to figure out where the problem is. If you are able to determine for sure that the problem is a bug in the engine, then submit a bug report with all the relevant informations and then it will become my job.

          Michele

[homos...@gmail.com]

Hi Michele,

I am running OQ 3.6 and still get the error "ValueError: You cannot compute the GMFs for 205560 > 65536 sites"

I also used the source code for 3.7 (the master), and get the same error. I added also max_num_sites=205560 in the job file and get the same error.

Any help or suggestion?

Thanks

Hossein

[Michele Simionato]

On Thursday, September 19, 2019 at 10:49:43 PM UTC+2, homos...@gmail.com wrote:

Hi Michele,

I am running OQ 3.6 and still get the error "ValueError: You cannot compute the GMFs for 205560 > 65536 sites"

I also used the source code for 3.7 (the master), and get the same error. I added also max_num_sites=205560 in the job file and get the same error.

 

The name of the parameter is max_sites_per_gmf, not max_num_sites. You can set it to a number larger than 205560 and then the engine will try to compute the GMFs for 205560 sites and will fail later.

There is no way you can compute and store the GMFs for a model of that size (say Canada). The ebrisk calculator can do the calculation, but it does not store the GMFs. If you need to store the GMFs you have to

perform some serious reduction of your model, like reducing the number of realizations and the investigation time by orders of magnitudes to be able to manage 205560 sites.

                  Michele

[homos...@gmail.com]

Thanks Michele, 

I tried it for China and it worked (with 5 realizations). OQ created a 6 GB HDF5 file. 

Thanks

Hossein