Execution speed of OQ

[hasan mousavi]

Dear all,
I installed OpenQuake on  Laptop with Ubuntu 12.04 32bit machine. (system Memory:  2.0 GiB, Processor: Intel® Core™2 Duo CPU T9300 @ 2.50GHz × 2 ). Running OQ to compute Hazard curves for my study area takes too much time, about 24 hours (with minimum resolution: REGION_GRID_SPACING = 0.1, AREA_SOURCE_DISCRETIZATION = 0.1, NUMBER_OF_LOGIC_TREE_SAMPLES = 1).

I have access to a relatively strong server with Fedora Linux. I can't change its operating system.
- Can I use it to run Program faster?
- If it is executable, What is the procedure performed?
- Was OQ written parallel so that it can be run in parallel?
- Are there other ways in which the program executes faster?
Beast Regards,
Hasan

[Lars Butler]

Hi Hasan,

Technically, it should be possible to run OQ Engine on Fedora Linux, although we do not officially support that platform yet. To do so, you'll basically need to get the source code from oq-engine, oq-nrmllib, oq-hazardlib, and oq-risklib. See https://github.com/gem/oq-engine, https://github.com/gem/oq-nrmllib, https://github.com/gem/oq-hazardlib, and https://github.com/gem/oq-risklib. Then, you'll need to install all of the dependencies, such Celery, RabbitMQ, Postgres/PostGIS, numpy, scipy, shapely, etc. I will warn you that this can be quite a time consuming task. I recommend that you ask the administrator of the Fedora server for assistance.

OQ Engine was written such the calculations can be distributed across many worker machines and processors. If, for example, your Fedora server has 16 CPU cores, you can run 16 worker nodes. For classical hazard calculations, the number of seismic sources in your source model determines the overall workload. You can configure the number of sources per task/work item: https://github.com/gem/oq-engine/blob/master/openquake.cfg#L64.

It also worth noting that in the last 2 months, we have made some significant optimizations to the classical hazard calculator in computation time and memory consumption. Getting the latest source code or installation the latest packages will yield these benefits.

I hope that will help you get started.

Cheers,

Lars

--
You received this message because you are subscribed to the Google Groups "OpenQuake Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openquake-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Lars Butler]

Hi Hasan,

Also, can you please tell us which version of OQ Engine you're using? You can check it by typing "openquake --version".

Thanks,

Lars

On Sun, Feb 24, 2013 at 3:58 PM, hasan mousavi <s_hasan...@yahoo.com> wrote:

[hasan mousavi]

Hi Lars,
Tanks a lot,
On my laptop, Openquake version is 0.8.3, released 2012-10-23T08:31:12Z
I have 2 option for install new version of OQ:
1- use of PC (OS: Linux Ubuntu 12.04, 32bit, Cpu: core 2 quad 2.67 8400, Ram: 4Gig)
2- use of server (OS: Fedora 13, 64 bit, Cpu: 8 cores, Ram: 16Gig)
Can you please tell me which one is better for my work.

Cheers,
Hasan

[Lars Butler]

Hi Hasan,

In terms of hardware, the Fedora server is better. However, Fedora 13 is almost 3 years old and you will have to make sure that dependency packages are available for that platform. This can be a lot of work. Again, I recommend that you consult with the administrator of that machine.

Cheers,

Lars

[hasan mousavi]

Hi Lars,
Now, i get packages of OQ from https://github.com/gem/oq-engine/wiki/Ubuntu-12.04, but its version is 0.8.3, released 2012-10-23T08:31:12Z.
How can I give new source code or install the latest packages to consumption of computation time.
Regards,
Hasan

[Lars Butler]

Hi Hasan,

You can take the source directly from github. The following guide was written for Ubuntu 12.04, but it can be used as a starting point for running the oq-engine on other Linux distributions: https://github.com/gem/oq-engine/wiki/Running-the-OpenQuake-Engine-from-a-git-repository

Start from there and post back if you have questions.

Cheers,

Lars

[hasan mousavi]

Dear Lars
I try to install OQ on other laptop and another PC with ubuntu 12.04 from https://github.com/gem/oq-engine/wiki/Ubuntu-12.04. but it give some error message:
mohamad@mohamad-VGN-FW45GJ-B:~$ cd /usr/openquake/
mohamad@mohamad-VGN-FW45GJ-B:/usr/openquake$ celeryd &
[1] 2881
mohamad@mohamad-VGN-FW45GJ-B:/usr/openquake$ [2013-03-08 08:30:17,343: WARNING/MainProcess] 

 -------------- celery@mohamad-VGN-FW45GJ-B v2.4.6
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      amqp://guest@localhost:5672//
- ** ----------   . loader:      celery.loaders.default.Loader
- ** ----------   . logfile:     [stderr]@WARNING
- ** ----------   . concurrency: 2
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
 --------------   . celery:      exchange:celery (direct) binding:celery
                 

[2013-03-08 08:30:17,353: WARNING/MainProcess] celery@mohamad-VGN-FW45GJ-B has started.

mohamad@mohamad-VGN-FW45GJ-B:/usr/openquake$ openquake --config-file=iranhaz3/config.gem --output-type xml --log-level progress
Ignoring GMF_RANDOM_SEED in classical, it's meaningful only in  scenario_damage, scenario, event_based
Traceback (most recent call last):
  File "/usr/bin/openquake", line 9, in <module>
    load_entry_point('openquake==0.8.3', 'console_scripts', 'openquake')()
  File "/usr/lib/pymodules/python2.7/openquake/bin/oqscript.py", line 192, in main
    args.config_file, ajob, user_name, args.force_inputs)
  File "/usr/lib/pymodules/python2.7/openquake/engine.py", line 996, in import_job_profile
    job_profile = _prepare_job(params, sections, user_name, job, force_inputs)
  File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 217, in inner
    res = func(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/openquake/engine.py", line 718, in _prepare_job
    job_profile = _get_job_profile(calc_mode, job_type, owner)
  File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 217, in inner
    res = func(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/openquake/engine.py", line 706, in _get_job_profile
    job_profile.save()
  File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 553, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 1436, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
django.db.utils.DatabaseError: invalid byte sequence for encoding "UTF8": 0x00

mohamad@mohamad-VGN-FW45GJ-B:/usr/openquake$
please give me suggestion to solve it.
hasan

[Lars Butler]

Hi Hasan,

This error is related to a Django bug, and it requires a patch. I have added an entry to our FAQ page to address this. Please see https://github.com/gem/oq-engine/wiki/FAQ. (The answer to your question is right at the top of the page.)

Cheers,

Lars