problems with the self-installable binary distribution on RedHat 6.2

82 views
Skip to first unread message

Ádám Zsarnóczay

unread,
Sep 29, 2016, 1:18:27 PM9/29/16
to OpenQuake Users
Dear Developers,

After the release of OpenQuake 2.0 I had been looking forward to finally installing OpenQuake on our HPC Cluster. The cluster runs Red Hat Enterprise 6.2.

I don't have root permissions, so as far as I understand my only option is the self-installable binary distribution. I downloaded the installer and ran it following the commands at https://github.com/gem/oq-engine/blob/master/doc/installing/linux-generic.md

When asked about where to install OpenQuake I tried using the default folder by simply leaving the field blank and also tried installing to a specific folder. This does not seem to affect the results.

During installation I get the following message many times:

/home/1/psha01/OpenQuake/openquake/bin/sed: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/1/psha01/OpenQuake/openquake/bin/sed)

The version of GLIBC is 2.12 on our computer. Because I have no root permissions, I cannot update it. However, the installer finishes with the message: "Installation completed", so I assumed I might get away with this GLIBC issue and continued.

I started the environment using env.sh and tried to simply run "oq engine --help" only to get the message "oq: command not found". Although I am far from understanding everything that happens here, I assume that the reason is that the PATH given in env.sh is incorrect. The PATH uses such a PREFIX:

PREFIX=/tmp/build-openquake-dist/qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq/openquake

This is not my OpenQuake installation folder. If I modify env.sh and change PREFIX to point to the installation folder and reload the environment, then I can run oq without a problem. However, I immediately get a different error message:

/home/1/psha01/OpenQuake/openquake/bin/oq: /tmp/build-openquake-dist/qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq: bad interpreter: No such file or directory

I realized that this path with lots of q characters is hard coded into a lot of the OpenQuake files. I also found it on github in this file: https://github.com/gem/oq-installers/blob/master/unix/opt/install.sh

At this point I do not see if my problem stems from not having the proper GLIBC version or making a mistake during installation or a possible bug in the program. I would really appreciate your help.

Thanks and kind regards,
Adam

Daniele Viganò

unread,
Sep 30, 2016, 5:26:21 AM9/30/16
to openqua...@googlegroups.com

Dear Adam,

thanks for your feedback.


On 29/09/16 19:18, Ádám Zsarnóczay wrote:
During installation I get the following message many times:

/home/1/psha01/OpenQuake/openquake/bin/sed: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/1/psha01/OpenQuake/openquake/bin/sed)

The version of GLIBC is 2.12 on our computer. Because I have no root permissions, I cannot update it. However, the installer finishes with the message: "Installation completed", so I assumed I might get away with this GLIBC issue and continued.

the OpenQuake standalone installer requires GLIBC/LIBC >= 2.15 as stated in the documentation:

https://github.com/gem/oq-engine/blob/engine-2.0/doc/installing/linux-generic.md#requirements

Requirements are:

  • GNU/Linux with bash
  • libc version at least 2.15
  • 4 GB of RAM (8 GB recommended)
  • 1.2 GB of free disk space
I started the environment using env.sh and tried to simply run "oq engine --help" only to get the message "oq: command not found". Although I am far from understanding everything that happens here, I assume that the reason is that the PATH given in env.sh is incorrect. The PATH uses such a PREFIX:

PREFIX=/tmp/build-openquake-dist/qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq/openquake

You get this wrong path (actually, it's a placeholder long enough to do nott cause buffer overflows after the sed run) since sed had fail because the previous error.

Next version of the installer will be built on top of GLIBC 2.12 and it should work fine on RHEL 6 too.

If you can give me a couple of days, I will try to build a standalone distribution based on current master that should work fine on your environment.


Regards,
Daniele

--
DANIELE VIGANÒ | System Administrator | Skype dennyv85 | +39 0382 5169882
GLOBAL EARTHQUAKE MODEL | working together to assess risk

Daniele Viganò

unread,
Sep 30, 2016, 11:19:29 AM9/30/16
to openqua...@googlegroups.com

Dear Adam,

at the following link you'll find a version of the OpenQuake Engine 2.0.1 standalone installer that should work fine on your RHEL 6:

http://ftp.openquake.org/rhel/staging/openquake-setup-linux64-5cd099f.run

I want also to remind you that this way of installing OpenQuake is meant to be used as single user, single node setup only. Even if it is installed on a cluster, only the master node computational power and memory will be actually used.

Let me know if it works.

Best regards,
Daniele

Ádám Zsarnóczay

unread,
Oct 4, 2016, 7:55:45 AM10/4/16
to OpenQuake Users
Dear Daniele,

thank you for your prompt and helpful reply.

The custom installer you uploaded worked perfectly and now I can run OpenQuake on our HPC. However, it runs only on the master node - exactly as you told me in your previous message. What is the reason behind this behavior?

Is it possible to improve on the situation without having root permissions? I imagine something like the oq-lite package that - as far as I understand - is a simple python package that I might be able to install on any OS.

Thanks again for your help!

Best regards,
Adam

Daniele Viganò

unread,
Oct 4, 2016, 10:18:09 AM10/4/16
to openqua...@googlegroups.com

Dear Adam,

the official and (only) supported way to install an OpenQuake Engine cluster is using our Ubuntu 14.04, Ubuntu 16.04 or RHEL (and clones) 7 binary packages but, yes, root access is required.

OpenQuake does not use MPI, but it uses RabbitMQ + Celery to parallelize tasks across network and multiple nodes. A copy of the Engine and its libs must be installed on each worker node and they must be able to communicate through the network.

In theory it is possible to setup an OQ cluster even without root access using our stand-alone installer as a starting point.

What you have to do is:

In any case, you cannot do it if:

This solution is not simple and it may not be worth the effort. How big is your master node?


On 04/10/16 13:55, Ádám Zsarnóczay wrote:
I imagine something like the oq-lite package that - as far as I understand - is a simple python package that I might be able to install on any OS.

There no oq-lite anymore, I would say. Everything has been merged in the "oq" command which is now "modular". Based on its configuration and some of the extra dependencies availability it can either work on a multi-core, single machine (on any OS: Linux, Mac, Windows) and on a Linux HPC cluster (having, precisely, RabbitMQ and Celery, with its deps, installed).

Best regards,

Ádám Zsarnóczay

unread,
Oct 5, 2016, 8:52:24 AM10/5/16
to OpenQuake Users
Dear Daniele,

thank you for the detailed explanation.
I am afraid that I will need root permissions, because I do not have SSH access to the workers and as far as I understand, workers use a clone of the disk space of the head node, so there is no way to install OQ distributions on the workers.

Let me give you a short description of our HPC:
We have a head and 30 computing nodes. All nodes are made of two 6 core Intel Xeon CPUs and 48 GB of memory. The head node has 34 TB disk space and stores all the projects. The project directories are available to the computing nodes through a 10 GB/s ethernet network using an NFS file system. The nodes have 500 GB disks for temporary storage. Multi-threaded computation is run through a low latency, 40 GB/s InfiniBand network. Execution is scheduled by HTCondor. There are several running environments available including OpenMPI, MVAPICH2 and CUDA (for a couple of TESLA cards), but I see no sign of RabbitMQ and/or Celery at the moment.

Provided that the admin of our HPC is willing to help and configure OQ, RabbitMQ, Celery and its dependencies using his root access, is it possible to set them up on our system? I am concerned that the HTCondor and MPI we use might interfere with your approach to parallelization.

Thanks for your help.

Best regards,
Adam
Reply all
Reply to author
Forward
0 new messages