Exporting ground motion fields in csv from large database in an event-based PSHA

66 views
Skip to first unread message

Crescenzo Petrone

unread,
Feb 2, 2017, 9:58:01 AM2/2/17
to OpenQuake Users, Matthia...@willistowerswatson.com
Dear OpenQuake developers,
I am recently using OpenQuake to run an event-based PSHA with a single area source by using OpenQuake engine 2.0 on Ubuntu Linux Release 16.04. 
A typical analysis is run with "oq engine --run xxx.ini --nd --exports csv". The analysis is successful and realizations, rup_data and sescollection are successfully exported in csv format during the analysis. I need then to export the GMFs separately by using the command "oq engine --export-output 83 /xxx/oq_output", where 83 is the output ID associated to the GMFs. 
However, when the database is too large (larger than 1GB) the output extraction fails and openquake shows "25260 Killed". Is there any alternative way of extracting the output for such large databases?
Moreover, even if the output extraction is successful, the GMFs are exported in xml format. Would it be possible to have them in csv format?

Thank you very much in advance for your help.
Kind regards,
Crescenzo Petrone

Michele Simionato

unread,
Feb 3, 2017, 5:48:01 AM2/3/17
to OpenQuake Users, crescenz...@hotmail.com
First of all, I suggest you to install the latest version of the engine. From version 2.2 it is possible to export the GMFs in .npz format which is the most efficient format we have for large datasets.
The .npz format is a binary format that you can read from Python with numpy (google for numpy npz). We also have a command (`oq to_hdf5`) to convert it into hdf5, if you prefer that format.
The command is to give is something like

$ oq engine --export-output 83 /xxx/oq_output --exports=npz

The format of the array is still undocumented. Using the binary format  requires being able to write some code, but that's life, you cannot pretend to manage gigabytes of data in XML or CSV.

Please do not use "oq engine --run xxx.ini --exports csv" if you are producing large amounts of GMFs. Also, why are you using the flag `--nd`? You will just waste your cores.

crescenz...@hotmail.com

unread,
Feb 3, 2017, 9:02:45 AM2/3/17
to OpenQuake Users, crescenz...@hotmail.com
Thank you Michele for your detailed reply.
Unfortunately I am not administrating the box where openquake is installed and I started running analyses before the latest release (two weeks ago if I am not wrong). 
Irrespective of csv output format, the ground motion fields are not exported to any other format if the data produced is too large (with 2.0 version). I think the only option left (besides installing the new version) is to extract GMFs from the hdf5 files. Please may you correct me if I am wrong?
Thanks for advising about "--nd" and thanks for your time,
Crescenzo

Michele Simionato

unread,
Feb 3, 2017, 11:34:12 AM2/3/17
to OpenQuake Users, crescenz...@hotmail.com


Il giorno venerdì 3 febbraio 2017 15:02:45 UTC+1, crescenz...@hotmail.com ha scritto:
Thank you Michele for your detailed reply.
Unfortunately I am not administrating the box where openquake is installed and I started running analyses before the latest release (two weeks ago if I am not wrong). 
Irrespective of csv output format, the ground motion fields are not exported to any other format if the data produced is too large (with 2.0 version). I think the only option left (besides installing the new version) is to extract GMFs from the hdf5 files. Please may you correct me if I am wrong?

You are not wrong. Please be careful because the format used in the datastore (i.e. the .hdf5 files) is not meant to stay the same across versions.

crescenz...@hotmail.com

unread,
Feb 3, 2017, 11:36:53 AM2/3/17
to OpenQuake Users, crescenz...@hotmail.com
Thanks Michele! Best wishes,
Crescenzo
Reply all
Reply to author
Forward
0 new messages