The repository oq-commonlib is gone

[Michele Simionato]

Dear OpenQuake users,

as of today the repository https://github.com/gem/oq-commonlib

has been merged into the repository https://github.com/gem/oq-risklib

and removed from GitHub. What does it mean?

It means that if you have a source installation you must:

1. remove the oq-commonlib directory from your PYTHONPATH

2. remove the oq-commonlib directory (this is not strictly necessary,

   but there is no need to keep an obsolete repository)

3. do a `git pull` in your oq-risklib directory to get the new version of risklib

If you are using the daily packages you must uninstall the package

python-oq-commonlib and upgrade the package python-oq-risklib,

which will contain the new commonlib.

Your scripts using commonlib or risklib do not need to be changed.

The plan for the future is to merge oq-hazardlib and oq-risklib in

a single repository (tentative name: oq-libs) whereas the repository

for the engine will stay separated.

Why is this being done?

Because we want to implement light versions of the engine calculators

inside oq-libs and move the QA tests there. Then with a single `pip install` an
user will have all the libraries that he needs to write both hazard and risk

calculators and this will work on different platforms, without the

need to install the engine. The removal of oq-commonlib is a first step
in that direction.

Internally the libraries risklib/hazardlib/commonlib will stay separated as 

they are now. We are merging the repositories, not the libraries.

That's all, please let us know if you have any issue with the upgrade,

                 Michele

[Lukas Heiniger]

Hi Michele

Thanks for the info, I was wondering where it had gone. It's a bit unfortunate for people who don't run the latest commit though. I just wanted to upgrade my sources to release-1.2 but now I'm basically forced to use the latest version. I usually prefer tagged versions since I'm assuming those are reasonably tested/stable.

Best

Lukas

[Michele Simionato]

In the case of the OpenQuake engine and its libraries, the latest version is always the most tested and stable.

We are using a continuous integration approach, so every night there is a new version of the libraries that passes all the tests of the previous versions

and possibly more. So the recommend approach is to stay updated to the latest master.

[Lukas Heiniger]

Thanks for the prompt response, Michele. To be frank, I don't think keeping up with the latest master is a viable strategy for us. Why have releases in the first place if they become obsolete with the next commit? I'm aware that you're doing CI which is great, but IMHO CI is not a replacement for proper release management for a number of reasons:

 - APIs are changing frequently on the master branch, so code that integrates with OQ libraries directly has to catch up constantly.

 - The documentation usually lags behind the code, which is totally fine as long as it catches up at release time.

 - CI can only catch so many regressions, there are always a couple of things that will slip through the net. Maybe the size of your user base and the maturity of the project don't warrant long term support with bugfix releases yet, but I'm sure they will at some point.

Don't get me wrong, you guys are doing a absolutely fantastic job and I appreciate every update. But in my opinion, as more and more people start to depend on your code, I think you have to become careful with introducing large changes like this.

Best

Lukas

[Michele Simionato]

Lukas, you are making reasonable points, but please keep in mind that the OpenQuake code base is relatively young and subject to change. We try very hard to keep compatibility

of the input and output formats, so that you can run today a computation using the same input files you used 3-4 years ago (except for the .ini file which had some slight changes); moreover also the files produced are compatible with the past. On the other hand we feel free to change to underlying implementation as we think best.

We would like to know how people are using our software. We have the impression that most users use the engine as a command-line tool and do not use the libraries directly.

If this is a false impression and if there are people using the libraries directly, please let us know. We can tell you which are the libraries which are more or less stable and

which one are the ones that are in state of flux. Right now for instance a lot of development is going on in the commonlib libraries, and a lot of code is being refactored and

moved from the engine into commonlib, with the goal of decoupling the scientific logic from the database. So right now it would not be a good idea to depend on code which

is currently on the engine, because it will likely disappear in the next future. Also we want to merge together the risklib and hazardlib repositories and change the repository name.

So right now it is a period of big changes of the internals. For people using the engine as a command-line tool instead very little is changing. Just some options have been deprecated

but they continue to work.

[thomas ulrich]

Hi Michele,

 

I think the wiki

https://github.com/gem/oq-engine/wiki/Installing-the-OpenQuake-Engine-Nightly

have to be fixed.

 

sudo apt-get install python-oq-* is not working anymore because of “python-oq-commonlib”

“The following packages have unmet dependencies:

python-oq-risklib : Conflicts: python-oq-commonlib”

 

The I propose to change it like this:

sudo apt-get install python-oq-hazardlib python-oq python-oq-engine python-oq-nrmllib python-oq-risklib

 

 

Thomas.

[Daniele Viganò]

Dear Thomas,

thanks for the feedback. We have just removed oq-commonlib from our PPA repo to avoid the conflict.

Cheers,

Daniele

[thomas ulrich]

Dear Daniele,

I also noticed when running my old openquake 1.0 files that:

The coordinates format have to be slightly changed:
ValueError: Could not convert to coordinates: sites=107.68, 10.6296
now: sites=107.68 10.6296

And that a name attibute is mandatory in areaSource:
 <areaSource tectonicRegion="Active Shallow Crust" id="NZ_Vietnam_Z34fus_NZ4">
KeyError: 'name'
now: <areaSource tectonicRegion="Active Shallow Crust" name="NZ_Vietnam_Z34fus_NZ4" id="NZ_Vietnam_Z34fus_NZ4">

Maybe these modifications should be made explicite in:
https://github.com/gem/oq-engine/wiki/Incompatible-changes-between-OpenQuake-1.0-and-1.2

Thomas.

[Michele Simionato]

On Wednesday, November 26, 2014 3:23:18 PM UTC+1, thomas ulrich wrote:

Dear Daniele,

I also noticed when running my old openquake 1.0 files that:

The coordinates format have to be slightly changed:
ValueError: Could not convert to coordinates: sites=107.68, 10.6296
now: sites=107.68 10.6296

This is more of a bug fix. A comma inside a couple of coordinates is wrong, and the old engine worked by accident.

You can use commas to separate *pairs* of coordinates:

sites = x1 y1, x2 y2, ... 

And that a name attibute is mandatory in areaSource:
 <areaSource tectonicRegion="Active Shallow Crust" id="NZ_Vietnam_Z34fus_NZ4">
KeyError: 'name'
now: <areaSource tectonicRegion="Active Shallow Crust" name="NZ_Vietnam_Z34fus_NZ4" id="NZ_Vietnam_Z34fus_NZ4">

Maybe these modifications should be made explicite in:
https://github.com/gem/oq-engine/wiki/Incompatible-changes-between-OpenQuake-1.0-and-1.2

Thomas.

We could actually relax this validation and assign a dummy name to unnamed sources, since the calculation does not use the name. Let me ask to our scientific staff.