Install the OQ from source without admin privileges

85 views
Skip to first unread message

Hyeuk Ryu

unread,
Jul 2, 2015, 3:54:13 AM7/2/15
to openqua...@googlegroups.com
Hi, 

I was trying to follow the steps in the instruction (https://github.com/gem/oq-engine/blob/master/doc/engine/Installing-the-OpenQuake-Engine-from-source-code.md), but I couldn't since I do not have admin privileges.
Is there any way to install the OQ at ubuntu from source without admin privileges? 
Thanks.

Regards,

Hyeuk

Daniele Viganò

unread,
Jul 2, 2015, 4:07:59 AM7/2/15
to openqua...@googlegroups.com
Dear Hyeuk,

the oq-hazardlib and oq-risklib can be installed/used without any admin privilege; for the oq-engine admin privileges are required to setup PostgreSQL. You can either:
  • ask your sysadmin to setup PostgreSQL for you (and make sure that you have all the system deps in place, like RabbitMQ). You don't need privileges to use the oq-engine, also celery runs as a normal user
  • don't use the system installation of PostgreSQL and install your own instance of PostgreSQL in your home. This means compile PostgreSQL (and PostGIS) from scratch, changing the PREFIX during the ./configure: it is not an easy task and requires lot of build dependencies

Best regards,
Daniele

--
You received this message because you are subscribed to the Google Groups "OpenQuake Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openquake-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
DANIELE VIGANÒ | System Administrator | Skype dennyv85 | +39 0382 5169882
GLOBAL EARTHQUAKE MODEL | working together to assess risk

Hyeuk Ryu

unread,
Jul 2, 2015, 4:37:36 AM7/2/15
to openqua...@googlegroups.com
Hi Daniele, 

Thanks for your kind advice. 
If the PostgreSQL is once set up by admin (or openquake is installed by admin), then any user can set up its own OQ by following the instruction?
Thanks.

Regards,

Hyeuk 

Daniele Viganò

unread,
Jul 2, 2015, 10:50:57 AM7/2/15
to openqua...@googlegroups.com
Dear Hyeuk,


On 02/07/15 10:37, Hyeuk Ryu wrote:
Hi Daniele, 

Thanks for your kind advice. 
If the PostgreSQL is once set up by admin (or openquake is installed by admin), then any user can set up its own OQ by following the instruction?
if each user needs a separated installation of the Engine yes, it can be done. But please take into account that each user should start its own celery instance and use a custom queue (the queue name can be changed in  https://github.com/gem/oq-engine/blob/master/openquake.cfg#L40) and probably (if he/she needs to run different version of the Engine) also a different DB each.

If you need just to run calculations (and not to change the code), a single installation can be used by all the system users, even if it's done using git sources (you have only to make sure that the users have set the custom PYTHONPATH and added the oq-engine/bin to theri PATH).

What are your users needs? What they will do with the engine? (run calculation, develop new code, develop new models ...).

Cheers,
Daniele

Christoph Scheingraber

unread,
Nov 25, 2015, 5:20:20 AM11/25/15
to OpenQuake Users
Hi Daniele,

on our local linux server I can't get a system-wide database. The last couple of days I tried hard to go down the path of installing my own instance of PostgreSQL and PostGIS from source with user privileges only. Using a local python environment, I installed

pip install numpy decorator celery django scipy cython h5py psutil shapely mock concurrent futures

pip install nose


+ hazardlib speedups

 

wget https://ftp.postgresql.org/pub/source/v9.4.5/postgresql-9.4.5.tar.gz

tar -xzf postgresql-9.4.5.tar.gz

cd post*

./configure --prefix=$HOME/local/postgres/ --with-python

make

make install

 

cd ~/local/src

git clone https://github.com/OSGeo/proj.4

cd proj.4

./autogen.sh

./configure --prefix=/home/scheingraber/local/proj4

make

make install

 

(gdal did not work for me now, so I used --without-raster below in postgis configure. Will that be a problem?)

wget http://download.osgeo.org/gdal/2.0.1/gdal-2.0.1.tar.gz

tar -xzf gdal*

cd gdal*

./configure --prefix=/home/scheingraber/local/gdal

make

make install

 

wget http://download.osgeo.org/postgis/source/postgis-2.2.0.tar.gz

tar -xzf postgis*

cd postgis*

./configure --prefix=/home/scheingraber/local/postgis/ --with-projdir=/home/scheingraber/local/proj4 --without-raster

 

make

make install



I adapted the oq_create_db script to create a local DB, however I am not totally sure if this went down correctly. Rabbitmq is still missing and I am just trying out if the local PostgreSQL server is working with oq-engine:

(python)scheingraber@coredump:~$ oq-engine --lhc
Traceback (most recent call last):
  File "/home/scheingraber/local/gem/oq-engine/bin/oq-engine", line 640, in <module>
    main()
  File "/home/scheingraber/local/gem/oq-engine/bin/oq-engine", line 549, in main
    list_calculations('hazard')
  File "/home/scheingraber/local/gem/oq-engine/bin/oq-engine", line 311, in list_calculations
    if len(jobs) == 0:
  File "/export/data/scheingraber/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/export/data/scheingraber/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/export/data/scheingraber/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/export/data/scheingraber/python/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/export/data/scheingraber/python/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/export/data/scheingraber/python/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/export/data/scheingraber/python/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "uiapi.oq_job" does not exist
LINE 1: ..."."relevant", "uiapi"."oq_job"."ds_calc_dir" FROM "uiapi"."o...
                                                             ^

(python)scheingraber@coredump:~$ 


Do you have an idea what could be wrong?

Best,
Chris

Daniele Viganò

unread,
Nov 25, 2015, 5:33:17 AM11/25/15
to OpenQuake Users
Dear Christoph,



On Wednesday, November 25, 2015 at 11:20:20 AM UTC+1, Christoph Scheingraber wrote:
Hi Daniele,

on our local linux server I can't get a system-wide database. The last couple of days I tried hard to go down the path of installing my own instance of PostgreSQL and PostGIS from source with user privileges only.

if you need to build the full stack from scratch you can have a look at this script: https://github.com/daniviga/openquake-centos/blob/release-1.3/bin/deploy-openquake-centos6.sh

It was made for OpenQuake 1.3 so it does not work with the current release/master (it's missing, for example, some new python dependencies); the project has also been dismissed. I would not recommend a full installation from scratch since we cannot guarantee that numbers are correct (because different libraries version) so if you want to go in that way you'll need to run the full set of tests after every upgrade.

 

 

(gdal did not work for me now, so I used --without-raster below in postgis configure. Will that be a problem?)


No, that's ok. We do not use any raster functionality in PostGIS.
 


This error means that the tables have not been created. Have you run oq-engine --upgrade-db after the database creation (with oq_create_db)?

Cheers,
Daniele

Christoph Scheingraber

unread,
Nov 25, 2015, 6:20:25 AM11/25/15
to OpenQuake Users
Running oq-engine --upgrade-db I get

(python)scheingraber@coredump:~/local/gem/oq-engine/bin$ oq-engine --upgrade-db
INFO:root:Creating the initial schema from 0000-base_schema.sql
ERROR:root:Error executing /home/scheingraber/local/gem/oq-engine/openquake/engine/db/schema/upgrades/0000-base_schema.sql
Traceback (most recent call last):
  File "/home/scheingraber/local/gem/oq-engine/bin/oq-engine", line 640, in <module>
    main()
  File "/home/scheingraber/local/gem/oq-engine/bin/oq-engine", line 493, in main
    conn, extract_scripts='read_scripts')
  File "/home/scheingraber/local/gem/oq-engine/openquake/engine/db/upgrade_manager.py", line 365, in what_if_I_upgrade
    upgrader = UpgradeManager.instance(conn, pkg_name)
  File "/home/scheingraber/local/gem/oq-engine/openquake/engine/db/upgrade_manager.py", line 311, in instance
    upgrader.init(conn)
  File "/home/scheingraber/local/gem/oq-engine/openquake/engine/db/upgrade_manager.py", line 162, in init
    apply_sql_script(conn, os.path.join(self.upgrade_dir, base))
  File "/home/scheingraber/local/gem/oq-engine/openquake/engine/db/upgrade_manager.py", line 95, in apply_sql_script
    conn.cursor().execute(sql)
psycopg2.ProgrammingError: function addgeometrycolumn(unknown, unknown, unknown, integer, unknown, integer) does not exist
LINE 61: SELECT AddGeometryColumn('hzrdi', 'site_model', 'location', ...
                ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

(python)scheingraber@coredump:~/local/gem/oq-engine/bin$ 

Apparently at this point the PostGIS installation can not be found?

Daniele Viganò

unread,
Nov 25, 2015, 7:34:41 AM11/25/15
to openqua...@googlegroups.com
Yes, it looks like that the database you are using is missing the PostGIS extensions.

A couple of more information:
  1. Which Linux distribution are you running?
  2. Is it possible for you to ask your IT to just install the base dependencies, create a database one time and then let you to install and run OQ as normal user? If you cannot install OQ as root, I would suggest to install at least rabbitmq and PostgreSQL systemwide, create a single database (maybe with strict permissions/ACLs) and then install everything else from sources/pip.
  3. What about alternative solutions like using an LXC or a set of Docker containers?

Michele Simionato

unread,
Nov 25, 2015, 12:31:05 PM11/25/15
to OpenQuake Users
Another option is to not install the engine and only use oq-lite. Then the dependencies are easy (no database, no rabbitmq, no celery, no Django). The problem is that oq-lite is not documented and it is not yet a full replacement for the engine. But it can get you started and it is the future anyway. If you go in that direction don't install anything: clone the oq-hazardlib.oq-risklib repositories, set the PYTHONPATH and work from the sources.

Christoph Scheingraber

unread,
Nov 30, 2015, 6:30:12 AM11/30/15
to OpenQuake Users
Hi Daniele,

thanks for your reply.

1.
scheingraber@pacha:~$ cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
BUG_REPORT_URL="https://bugs.debian.org/"

scheingraber@pacha:~$ cat /proc/version
Linux version 3.16.0-4-amd64 (gcc ver
 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09)

2. At the university, the administrator now agreed to this procedure. I think I won't try installing the dependencies locally anymore. Thanks for your help! At the MR, we'll meet with IT this week and see if we can get a machine that we may administer ourself.

(3. They said no to lxc at the university and at MR)

Cheers,
Chris

Christoph Scheingraber

unread,
Dec 9, 2015, 9:47:19 AM12/9/15
to OpenQuake Users
  1. What about alternative solutions like using an LXC or a set of Docker containers?

Hi Daniele,

this is about the MR installation, I had another meeting with IT today. They'll provide me with a CentOS 7 machine and I should install openquake using docker/lxc. The links to the containers provided in

are down. Can you fix that/upload new containers for 1.6?

Thanks,
Chris

Daniele Viganò

unread,
Dec 11, 2015, 11:05:47 AM12/11/15
to OpenQuake Users
Dear Chris,

currently LXC and OVA for 1.6 are not available because we decided to "discard" release 1.6 and looking forward to 1.7 which will be released early next week.

By the way a produced a couple of LXC based on current master (which is more or less what 1.7 will be). One LXC is based on Ubuntu 14.04 and the other one on CentOS 7. Both are using binary packages provided by our PPA and COPR repos respectively.

http://www.globalquakemodel.org/lxc/master/

As usual user is "openquake" and password the same. In the openquake home folder you'll find the README. Celery and the WebUI are started at boot by supervisord.
By default these containers are meant to be used with the linuxcontainers (aka LXC) userland tools (using EPEL: yum install lxc) but can be easily ported to other LXC drivers (i.e. libvirtd) and meant to be put in /var/lib/lxc. To exctract use the following command:

cd /var/lib/lxc
sudo tar --numeric-owner -xjpvf /path/to/tar/bz2

then you can start them with

lxc-start -n openquake-engine[-c7] -d

The default config (in /var/lib/lxc/containername/config) uses 'virbr0' as the network bridge and DHCP.

If you need further, more detailed information let me know!

Cheers,
Daniele
Reply all
Reply to author
Forward
0 new messages