multiple versions of libraries?

[Chris Barker]

So I am in lib version hell -- specifically libjpeg, but I'm sure it's not just that.

I've noticed a lot of activity recently with "pinning" packages to particular jpeg libs, as, I take it, there are compatibility issues. That seems the right hting.

But now we have all sorts of packages, some pinned to libjpeg8, some to libjpeg9, and some (presumably) not pinned to a particular version, but linked to a particular version...

For the "not pinned" ones the solution is to do more pinning -- a bit tricky with packages from default,s but what can we do? (OK, add them to conda forge).

But we now have a LOT of packages on conda-forge (and defaults), and it seems someone, at some point, decided that they wanted jpeg9 (I wonder if there was a good reason for that ? -- but in any case, upgrades will happen, and there will be compelling reasons to adopt them).

So:

* a jpeg 9 package is built.

* some package(s) that used libjpeg is updated to use jpeg9, and appropriately pinned.

all is well for the user that needed those particular packages.

But: now we have a whole bunch of packages that rely on jpeg8, and a bunch that rely on jpeg9 -- and you can't install these at the same time. And as we all have our own random assortment of package that are mixed and matched, and some may come from defaults (Or our own channels) we have an unholy mess.

conda handles this with python and numpy by actually pinning the versions are part of the package, so you can have two builds of a given package that supports different numpy versions, and conda-forge has infrastructure to build a whole matrix. 

But numpy and python are special cased, so hard to extend this -- and even if we did, there would be exponential (or is it geometric?) growth in the size of the build matrix, which is absolutely unsustainable. So we need another solution.

Python itself does not support having multiple versions of package installed at once -- personally, I think it should, but that was rejected years ago, and folks seem to think that virtual environments solved that problem. So that's not going to happen.

But AFAICT, there is no problem with multiple versions of compiled libs being installed at once -- you install libjpeg8 and libjpeg9, and any package that depends on one or the other is linked to the correct one. This happens all the time at the OS level (unless some idiot decided not to embed any version info in a Windows dll -- I'm looking at you: netcdf.dll)

And, in fact, there is no reason not to have python2 and python3 installed at once.

However, conda does not support this (AFAICT), but we could fudge our way around it by embedding a version in the name, so instead of:

jpeg version 8d

and 

jpeg version 9b

we'd have:

jpeg8 version 8d.x.y

jpeg9 version 9.d 

etc.

This would get ugly at some point, I suppose, but I think support for the older ones would simply get dropped, so there wouldn't be a huge pile up.

BTW -- it would be really nice to have python2 and python3 packages that could co-exist in one conda environment, though that would probably add a substantial level of complication.

NOTE: I can probably solve the libjpeg problem by updating one or two packages -- but this is going to be a pain every time any lib gets an upgrade!

(of course, this could get pretty ugly with exponential growth, too -- so we'd have to be circumspect about which packages got "versioned names")

Thoughts? am I missing something??

Also -- maybe it would be better for conda itself to grow a way to handle multiple versions at one -- some kind of "top level versions can co-exist" flag -- though I have no idea how big a change that would be to the whole conda infrastructure.

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Jan Schulz]

Hi,
On 3 November 2016 at 17:36, Chris Barker <chris....@noaa.gov> wrote:
[...]

> However, conda does not support this (AFAICT), but we could fudge our way
> around it by embedding a version in the name, so instead of:

> we'd have:
> jpeg8 version 8d.x.y
> jpeg9 version 9.d

this is basically this proposal in libpng:

https://github.com/conda-forge/libpng-feedstock/issues/7
https://github.com/conda-forge/conda-forge.github.io/issues/157

IMO this also how every linux distribution has "fixed" this problem.

Jan
--
Jan Schulz
mail: ja...@gmx.net
web: http://www.katzien.de

[Chris Barker]

On Thu, Nov 3, 2016 at 12:04 PM, Jan Schulz <ja...@gmx.net> wrote:

> However, conda does not support this (AFAICT), but we could fudge our way
> around it by embedding a version in the name, so instead of:
> we'd have:
> jpeg8 version 8d.x.y
> jpeg9 version 9.d

this is basically this proposal in libpng:

https://github.com/conda-forge/libpng-feedstock/issues/7
https://github.com/conda-forge/conda-forge.github.io/issues/157

of course this has already been thought out -- and it looks like it's done for png, yes?

and now I know that such discussions should be looked for in the

conda-forge.github.io

issues.

Thanks,

   -CHB