[Bug 1097676] [NEW] oq-engine: Investigation - Does the current site->source filtering mechanism scale with 500k-1m sites?

[Lars Butler]

Public bug reported:

We want to know how well oq-engine can scale with 500k-1m sites, with
respect to source->site distance filtering. In some recent performance
optimizations [1] and profiling [2] I've been working on, filtering did
not appear to be a major bottleneck with ~100k sites. However, we should
run similar tests with 500k-1m and see if there are any major
bottlenecks, and then optimize as necessary. One potential optimization
is defined in bug # 1096915.


[1] bug # 1094297
[2] https://launchpadlibrarian.net/126993750/prof_opt.log

** Affects: openquake
Importance: Undecided
Status: New

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:
New

Bug description:
We want to know how well oq-engine can scale with 500k-1m sites, with
respect to source->site distance filtering. In some recent performance
optimizations [1] and profiling [2] I've been working on, filtering
did not appear to be a major bottleneck with ~100k sites. However, we
should run similar tests with 500k-1m and see if there are any major
bottlenecks, and then optimize as necessary. One potential
optimization is defined in bug # 1096915.


[1] bug # 1094297
[2] https://launchpadlibrarian.net/126993750/prof_opt.log

To manage notifications about this bug go to:
https://bugs.launchpad.net/openquake/+bug/1097676/+subscriptions

[Lars Butler]

** Changed in: openquake
Status: New => In Progress

** Changed in: openquake
Importance: Undecided => High

** Changed in: openquake
Assignee: (unassigned) => Lars Butler (lars-butler)

** Changed in: openquake
Milestone: None => 0.9.1

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

The source model was trimmed down to a single source, containing only
the second simple fault source from the demo:

<?xml version="1.0" encoding="UTF-8"?>

<nrml xmlns="http://openquake.org/xmlns/nrml/0.4" xmlns:gml="http://www.opengis.net/gml">
<sourceModel name="DISS Fault Model">
<simpleFaultSource id="CHCS001" name="Eastern Valais" tectonicRegion="Active Shallow Crust">
<simpleFaultGeometry>
<gml:LineString>
<gml:posList>7.6773 46.2612 7.7135 46.2853 7.7498 46.3094 7.7860 46.3335 7.8222 46.3576 7.8717 46.3785 7.9211 46.3995 7.9706 46.4204 8.0231 46.4423 8.0756 46.4641 8.1280 46.4860</gml:posList>
</gml:LineString>
<dip>65.0</dip>
<upperSeismoDepth>5.0</upperSeismoDepth>
<lowerSeismoDepth>20.0</lowerSeismoDepth>
</simpleFaultGeometry>
<magScaleRel>WC1994</magScaleRel>
<ruptAspectRatio>1.0</ruptAspectRatio>
<truncGutenbergRichterMFD aValue="3.7804" bValue="1.0000" minMag="5.0000" maxMag="5.9000"/>
<rake>-90.0</rake>
</simpleFaultSource>
</sourceModel>
</nrml>

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Note: I will be using a fresh database for each test case, at least in
the first round of tests. Later, I would like to run them all in
sequence and see if there is any major performance difference with lots
pre-existing DB contents.

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

These cases are based on the simple_fault_demo_hazard
(https://github.com/gem/oq-
engine/blob/ed2fe2519ceb91a44d52a73f524dc677d7f85f80/demos/simple_fault_demo_hazard/job.ini),
with modified geometry to increase the number of sites.

The `number_of_logic_tree_samples` is also reduced to 0, to specify end-
branch enumeration. For this configuration, this will result in only 1
logic tree realization.

All post-processing options are enabled. This also helps to get an idea of how well post-processing performs with different numbers of sites. The following settings are used:
mean_hazard_curves = true
quantile_hazard_curves = 0.15 0.85
poes_hazard_maps = 0.1 0.2

As a separate test, it would also helpful to test post-processing with a
job containing many logic tree samples. I will leave that for another
time.


There are 3 cases. In the first, the goal is to run with ~100k sites, in the second, ~500k, and in the third, ~1m.

These cases are run with the `--no-distribute` option enabled. This
means that everything is computed using a single process; no
parallelization is employed for performing these calculations.

----------
Case 1:

region = 4.5 43.8, 4.5 48.5, 10.5 48.5, 10.5 43.8
# km
region_grid_spacing = 1.5

Total sites: 107,057

----------
Case 2:

region = 4.5 43.8, 4.5 48.5, 10.5 48.5, 10.5 43.8
# km
region_grid_spacing = 0.69

Total sites: 506,505

----------
Case 3:

region = 4.5 43.8, 4.5 48.5, 10.5 48.5, 10.5 43.8
# km
region_grid_spacing = 0.49

Total sites: 1,004,604

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Update: I decided to go with `number_of_logic_tree_samples = 2`, to make
for a more realistic scenario for post-processing (means, quantiles,
etc.).

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Here is an example of the command I'm using to run the tests:
$ python -m cProfile -s time bin/openquake --rh=demos/simple_fault_demo_hazard/job.ini --log-file=test1.log --no-distribute --exports=xml >> test1.prof

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Update:

It seems that the post-processing is in SERIOUS need of optimization.
The first test case completed the main calculation in about 5 minutes,
but the post-processing ran for almost an hour and didn't yet complete.

For the scope of these tests, I'm going to omit all post-processing.
There's a separate bug for this: bug # 1087661.

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Test 2 input files.

** Attachment added: "test2.tgz"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3482976/+files/test2.tgz

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Test 1 input files.

** Attachment added: "test1.tgz"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3482975/+files/test1.tgz

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Profiling report from the first test.

** Attachment added: "test1.prof"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3482973/+files/test1.prof

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

OQ Engine log from the first test.

** Attachment added: "test1.log"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3482974/+files/test1.log

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Test 3 input files.

** Attachment added: "test3.tgz"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3482977/+files/test3.tgz

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Test 2 oq log.

** Attachment added: "test2.log"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3483755/+files/test2.log

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Test 3 oq log.

** Attachment added: "test3.log"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3483757/+files/test3.log

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Test 3 profiling report.

** Attachment added: "test3.prof"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3483756/+files/test3.prof

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

Test 2 profiling report.

** Attachment added: "test2.prof"
https://bugs.launchpad.net/openquake/+bug/1097676/+attachment/3483754/+files/test2.prof

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

After analyzing these reports, I propose the following:

- cache sites of interest geometry in hazard_calculation table (possible optimization to Polygon.discretize())
- more efficient NRML hazard curve serialization (optimization to export())
- store hazard curve progress matrices as pickled lists, instead of numpy arrays, which could reduce storage requirements and DB select/insert/update time (possible optimization to DB queries)

Also, we need more detailed information from the database. I want to re-run all tests
- with query logging enabled (min_duration_statement)
- check check DB and table sizes
- dump job_stats and job_phase_stats

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

In Progress

[Lars Butler]

We have created separate bug reports for optimizing separate areas of
the Classical hazard calculation. They are as follows:

- export: bug # 1099763
- post-execution: bug # 1100332
- post-processing: bug # 1087661

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

Fix Committed

[Lars Butler]

The actual optimization work will be done in other bugs, so I'll close
this issue now.

** Changed in: openquake
Status: In Progress => Fix Committed

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

Fix Committed

[Lars Butler]

** Tags added: hazard optimization

--
You received this bug notification because you are subscribed to
OpenQuake.
Matching subscriptions: openquake-bugs
https://bugs.launchpad.net/bugs/1097676

Title:
oq-engine: Investigation - Does the current site->source filtering
mechanism scale with 500k-1m sites?

Status in OpenQuake:

Fix Committed