Re: Issues detecting PostGIS version in openquake [SEC=UNOFFICIAL]

[Daniele Viganò]

Dear Stephen,

thanks for your feedback and sorry for the late response, yesterday I was off.

I still need some information to help you with this issue:

• Which OS are you running? I suppose RHEL 7
• How did you installed all the dependencies? Yum? Have been manually compiled?
• How did you installed the oq-engine and its libraries? RPM from yum/copr (we have experimental RPMs for RHEL 7 and its clones starting from OQ Engine 1.5.0) or manually using git/zip files?
• Which version of these dependencies are you running?
• Django
• Postgres (It should be 9.3)
  
• Postgis (it looks 2.1.3)
  

You can get them running

• yum list installed | grep -Ei "django|postgres|postgis"
• pip freeze | grep -i "django" (in case Django has been installed/upgraded using pip)
  

These information can help me in reproducing your bug and to have a clear overview of your setup.

On Tue, Sep 29, 2015 at 2:37 AM, <S...@ga.gov.au> wrote:

Hi Daniele,

 

We’re having some problems with OpenQuake not picking up the PostGIS version. We tried your suggestion of

 

 

1) test your Postgis setup

sudo -u postgres psql -d openquake2 -c "SELECT postgis_lib_version();"

 

2) force the PostGIS version in Django

​​

echo "

​​

POSTGIS_VERSION = (2, 0, 3)" >> /path/to/oq-engine/openquake/engine/local_settings.py

 

/path/to/oq-engine/ should be the path of the oq-engine sources (it looks like you are not using packages) and (2,0,3) must be replaces with your PostGIS version.

 

​​

Our PostGIS version is 2.1.3, so it should be sufficient, but we are still having some issues. How do we ensure that local_settings.py is picked up? I’d downloaded the sources as a zip and tried working through the installation instructions for Red Hat 7/CentOS 7 to come up with something I could install in /usr/local and have everyone use. This may not be the most optimal approach.

​​This approach is fine. We use the same on our cluster, where all the dependencies are from vendor binary packages, but the OQ Engine is installed from source in /usr/local. If you are running 'python setup.py install' you don't need any further step since all the code will be placed in the system and python PATH. If you want all the code in a single place (on our cluster we use /usr/local/openquake/oq-{engine,hazardlib,risklib} you need three extra steps:

1. Compile the speedups (see https://github.com/gem/oq-hazardlib/wiki/Installing-C-extensions-from-git-repository)
   
2. Set the PYTHONPATH to PYTHONPATH="/usr/local/openquake/oq-hazardlib:/usr/local/openquake/oq-risklib:/usr/local/openquake/oq-engine"
3. Set the PATH to PATH="/usr/local/openquake/oq-engine/bin:/usr/local/openquake/oq-risklib/bin:$PATH"

​​Our PostGIS version is 2.1.3, so it should be sufficient, but we are still having some issues. How do we ensure that local_settings.py is picked up?

​You try to put the​ POSTGIS_VERSION in the settings.py instead of local_settings.py in a case that something is overriding it:

echo "​​POSTGIS_VERSION = (2, 1, 3)" >> /usr/local/openquake/oq-engine/openquake/engine/settings.py

​Best regards,

Daniele

(I'm adding our forum in CC ​since this thread has been started there and could be useful for other users; feel free to reply me directly if you cannot disclose all the information requested)

--

DANIELE VIGANÒ | System Administrator | Skype dennyv85 | +39-0382-5169882

GLOBAL EARTHQUAKE MODEL | working together to assess risk

GEM - globalquakemodel.org | T - @GEMwrld | F - GEMwrld

[Daniele Viganò]

Hyeuk,
thank you very much for your prompt response.

I'm currently on a train headed to the office, I'll try to reproduce the error as soon as I'll be at my desk.

Cheers,

Daniele

On Wed, Sep 30, 2015 at 8:03 AM, <Hyeu...@ga.gov.au> wrote:

Hi Daniele,

 

Stephen is away at the moment, so I provide some of the answers as below.

Thanks.

 

Regards,

 

Hyeuk

 

From: Daniele Viganò [mailto:daniele...@globalquakemodel.org]
Sent: Wednesday, 30 September 2015 3:57 PM
To: Hocking Stephen
Cc: Ryu Hyeuk; openqua...@googlegroups.com
Subject: Re: Issues detecting PostGIS version in openquake [SEC=UNOFFICIAL]

 

Dear Stephen,

thanks for your feedback and sorry for the late response, yesterday I was off.

I still need some information to help you with this issue:

• Which OS are you running? I suppose RHEL 7

[u65242@rhe-compute1:/usr/share/openquake/engine]$uname -a

Linux rhe-compute1.ga.gov.au 3.10.0-229.11.1.el7.x86_64 #1 SMP Wed Jul 22 12:06:11 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

[u65242@rhe-compute1:/usr/share/openquake/engine]$cat /etc/redhat-release

Red Hat Enterprise Linux Server release 7.1 (Maipo)

• How did you installed all the dependencies? Yum? Have been manually compiled?
• How did you installed the oq-engine and its libraries? RPM from yum/copr (we have experimental RPMs for RHEL 7 and its clones starting from OQ Engine 1.5.0) or manually using git/zip files?
• Which version of these dependencies are you running?

• Django
• Postgres (It should be 9.3)
• Postgis (it looks 2.1.3)

You can get them running

• yum list installed | grep -Ei "django|postgres|postgis"

[u65242@rhe-compute1:/usr/share/openquake/engine]$yum list installed | grep -Ei "django|postgres|postgis"

*Note* Spacewalk repositories are not listed below. You must run this command as root to access Spacewalk repositories.

postgis.x86_64                2.1.3-5.el7.centos      @openquake-rhel7

postgresql.x86_64             9.2.13-1.el7_1          @rhel-x86_64-server-7

postgresql-devel.x86_64       9.2.13-1.el7_1          @rhel-x86_64-server-7

postgresql-libs.x86_64        9.2.13-1.el7_1          @rhel-x86_64-server-7

postgresql-server.x86_64      9.2.13-1.el7_1          @rhel-x86_64-server-7

python-django.noarch          1.6.11-1.el7            @epel7_channel

python-django-bash-completion.noarch

 

• pip freeze | grep -i "django" (in case Django has been installed/upgraded using pip)

[u65242@rhe-compute1:/usr/share/openquake/engine]$pip freeze | grep -i "django"

Django==1.6.11

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
-------------------------------------------------------------------------------------------------------------------------

[Daniele Viganò]

Dear all,

I was unable to reproduce the error in our test environment (it's a systemd-nspawn/machinectl container, we can pass it to you if your IT wants to have a look), but I have some ideas:

1) let's see if python is picking the right settings.py. What's the output of this command?

python -c "from openquake.engine import settings; print settings.__file__"

it should returns /usr/lib/python2.7/site-packages/openquake/engine/settings.pyc if the packages have been used, or /path/to/oq-engine/openquake/engine/settings.pyc if the sources are used. Make sure you have added POSTGIS_VERSION = (2, 1, 3) in the settings.py printed by that command (and please remove other installations, if present, to avoid conflicts).

2) was maybe PostgreSQL already on the system before the installation of PostGIS has been done? My idea is that for some reasons Django is looking at the default template, instead of the real OQ DB, and the template does not have the PostGIS extensions (because it has been installed later).

File "/usr/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/operations.py", line 258, in spatial_version
    'template?' % self.connection.settings_dict['NAME']

I'm trying to reproduce this condition and I'll let you know.

3) the only slight difference that I see for now between your system and our is a slightly newer Django package we have (python-django-1.6.11-2.el7 vs python-django-1.6.11-1.el7) which contains a backported patch for a CVE, but I don't think it's the cause of your issue.

I'll keep you updated on my progress.

Thanks,
Daniele

--

DANIELE VIGANÒ | System Administrator | Skype dennyv85 | +39 0382 5169882

[Daniele Viganò]

I also don't remember if we already tried to recreate the database. Before making new test please run:

cd /tmp && sudo -u postgres dropdb openquake2

this will drop the openquake2 database (be careful, all its data will be lost). It should return nothing on screen. Make sure celery has been stopped before and that there are no open connections (like psql connected to the openquake2 db).

cd /tmp && sudo -u postgres oq_create_db

this will re-create the database structure. This command will fail if the database has not been correctly removed first. The output should be

.. creating roles ..
.. Current table spaces: hzrdi_ts hzrdr_ts pg_default pg_global riski_ts riskr_ts uiapi_ts
.. Creating database openquake2 ..

(cd /tmp is used only to avoid some warnings that could be triggered if the command is run from a folder where the postgres user has no read access)

You will then need to populate the database with (oq-engine will ask you to do so if this step is missing):

oq-engine --upgrade-db

Best,
Daniele

--

DANIELE VIGANÒ | System Administrator | Skype dennyv85 | +39 0382 5169882

[Hyeuk Ryu]

Hi, 

Please see my answer below. 

Thanks.

Regards,

Hyeuk

On Thursday, 1 October 2015 16:02:55 UTC+10, Daniele Viganò wrote:

Dear all,

I was unable to reproduce the error in our test environment (it's a systemd-nspawn/machinectl container, we can pass it to you if your IT wants to have a look), but I have some ideas:

1) let's see if python is picking the right settings.py. What's the output of this command?

python -c "from openquake.engine import settings; print settings.__file__"

it should returns /usr/lib/python2.7/site-packages/openquake/engine/settings.pyc if the packages have been used, or /path/to/oq-engine/openquake/engine/settings.pyc if the sources are used. Make sure you have added POSTGIS_VERSION = (2, 1, 3) in the settings.py printed by that command (and please remove other installations, if present, to avoid conflicts).

[u65242@rhe-compute1:~]$python -c "from openquake.engine import settings; print settings.__file__"

/usr/lib/python2.7/site-packages/openquake/engine/settings.pyc

When I checked the /usr/lib/python2.7/site-packages/openquake/engine/settings.py there is no POSTGIS_VERSION. As I don't have admin access, I am not able to add it to the file. Is there any way for each user to maintain its own db?

[Daniele Viganò]

Dear Hyeuk,

On 01/10/15 08:28, Hyeuk Ryu wrote:

[u65242@rhe-compute1:~]$python -c "from openquake.engine import settings; print settings.__file__"

/usr/lib/python2.7/site-packages/openquake/engine/settings.pyc

When I checked the /usr/lib/python2.7/site-packages/openquake/engine/settings.py there is no POSTGIS_VERSION. As I don't have admin access, I am not able to add it to the file. Is there any way for each user to maintain its own db?

ok, the problem is that you have multiple installation of the OpenQuake Engine (and its libraries), at least one from binary packages and one from source. What you need to do is ask your sysadmin to remove the packages without touching the dependencies and the database:

sudo yum erase python-oq-*

This will allow you to use your own copy in /usr/local (or even in your home dir) without the need to be root. root is then needed if you decide to destroy and or/recreate the openquake2 database (but your sysadmin can change the sudo policy to allow you to do that, and just that). Take into account that upgrading to a newer version of the Engine does not require root, since you are using it from sources and the DB is migrated by oq-engine itself (with --upgrade-db). Also celery does not need root privileges (it should NOT be executed as root).

Please, after the removal of the packages, try a calculation without the POSTGIS_VERSION in the settings.py and if you still have the issue please put it into the settings.py; now python should read the right one and it should workaround the issue.

Please also remember that if you are using the Engine from sources you need to have a valid PYTHONPATH and PATH set in the system bashrc or just in your ~/.bashrc; since you will have the python-oq-risklib package removed you'll need also to use the demos located in the oq-risklib repo (/usr/local/something/oq-risklib/demos) instead of the ones located in /usr/share/doc.

Let me know if you need more detailed instructions.

Best,