Python Version Dependancy

[Shaun Dudley]

Sysadmin question regarding our OpenQuake installation on 12.04LTS. 

Some of our analysts have expressed an interest in expanding their use of Python scripting to manipulate the CSV output from OpenQuake. This would be independent from OpenQuake itself although on the same system that runs OpenQuake. 

Is OpenQuake dependent on the version of Python that's installed on the system? Can Python be upgraded to a newer version than the version that ships with Ubuntu 12.04 without impacting OpenQuake?

A bit of digging around in a crash course on Python installation suggests that only the newer version of PIP is tolerant or can be manipulated to use a mirrored PyPA site that sits behind the corporate firewall. From what I can tell, version 1.0 of PIP that ships with Ubuntu 12.04 is incapable of using a local PIP resource and hardcoded to hit the public site. Due to security restrictions as part of a typical corporate firewall arrangement, we mirror the upstream Ubuntu repositories to a dedicated system behind the corporate firewall and internal systems then use this system to install OpenQuake and updates. I need to employ a similar process for the Python packaging system and need the flexibility of the newer PIP client. 

Thanks,

Shaun   

[Daniele Viganò]

Dear Shaun,

On 24/04/15 23:24, Shaun Dudley wrote:

Sysadmin question regarding our OpenQuake installation on 12.04LTS. 

Some of our analysts have expressed an interest in expanding their use of Python scripting to manipulate the CSV output from OpenQuake. This would be independent from OpenQuake itself although on the same system that runs OpenQuake.
Is OpenQuake dependent on the version of Python that's installed on the system? Can Python be upgraded to a newer version than the version that ships with Ubuntu 12.04 without impacting OpenQuake?

you'll find an overview of the Engine required libraries with their versions here: https://github.com/gem/oq-engine/wiki/Technology-stack-and-requirements
OpenQuake requires python 2.7, so it not possible upgrade/downgrade Python to another release; you can upgrade the python libraries unless the OpenQuake Engine requirements are satisfied. However this is not the recommended way, since the OpenQuake Ubuntu 12.04 packages are tested against official Ubuntu packages versions.

A bit of digging around in a crash course on Python installation suggests that only the newer version of PIP is tolerant or can be manipulated to use a mirrored PyPA site that sits behind the corporate firewall. From what I can tell, version 1.0 of PIP that ships with Ubuntu 12.04 is incapable of using a local PIP resource and hardcoded to hit the public site. Due to security restrictions as part of a typical corporate firewall arrangement, we mirror the upstream Ubuntu repositories to a dedicated system behind the corporate firewall and internal systems then use this system to install OpenQuake and updates. I need to employ a similar process for the Python packaging system and need the flexibility of the newer PIP client.

If the problem is only PIP you can safely install a new version of it. It can be done following this guide: https://pip.pypa.io/en/latest/installing.html#install-pip. But my strong suggestion is to keep the OS as it is, then create a virtual environment (http://docs.python-guide.org/en/latest/dev/virtualenvs/) where to upgrade PIP and install all the new versions of the libraries you need; an example:

$ sudo apt-get install python-virtualenv
$ virtualenv --system-site-packages ~/my/new/env/path
$ source ~/my/new/env/path/bin/activate
$ python get-pip.py --proxy="[user:passwd@]proxy.server:port"
$ pip install -U numpy
$ python myscript.py

Another option would be upgrade to Ubuntu 14.04 LTS (Trusty). We are in a process of releasing packages for 14.04; they will be ready by the first week of May with the ongoing OpenQuake Engine 1.4. Ubuntu 14.04 has a more fresh an updated set of python packages.

For any further explanation don't hesitate to ask.

Cheers,
Daniele

Thanks,

Shaun  

--

DANIELE VIGANÒ | System Administrator | Skype dennyv85 | +39-0382-5169882

GLOBAL EARTHQUAKE MODEL | working together to assess risk

GEM - globalquakemodel.org | T - @GEMwrld | F - GEMwrld

[Michele Simionato]

It is not clear to me if you are talking about the dependency from Python version or about the dependency from Python libraries. If you are talking about the latter, Daniele already answered you; instead if the question is "can I install Python 3.X on the same machine where the engine is installed" the answer is yes. Python 3 is totally decoupled from Python 2 and there is no interference between the two.

[Shaun Dudley]

Thanks Daniele and Michele. Yes, what I'm asking is if I can install the newer version of Python (which I think includes PIP by default) alongside the version of Python that is bundled with Ubuntu 12.04. Apologies for the lake of clarity in the original question, I'm in a position where the Python components are a bit foreign to me and I'm learning as I go. These systems are in a corporate environment and trying to abide by mirroring repositories to an internal system and the OQ systems have no direct access through the firewall presents an even greater challenge during the learning curve. Daniele's response helps in understanding how OQ uses Python and given the close proximity to version 1.4's release date on 14.04, I think I'll present these options to the team and see how they'd like to proceed.

I've had great success with incorporating the OQ PPA into our existing upstream Ubuntu internal apt-mirror update mechanism where internal Ubuntu systems then have their apt-sources file modified via Puppet to use the internal mirrored version of the upstream sites. I was hoping to stay as close as possible to this configuration architecture on the PIP front as well. Bandersnatch seems to be the suggested tool and appears to have worked with respect to mirroring the upstream Python packages index, just need a version of PIP that can be manipulated via shell environmental variables or a conf file, to query the internal PIP repository vs trying to hit the primary site on the internet.  

Thanks again for taking the time to provide a thorough response, it helps immensely in providing support for the team using OQ. 

Shaun