Gensim library on Anaconda

[Michel Fombellida]

My Python environment is Anaconda and their Gensim library is 0.13.3 ;  see https://anaconda.org/anaconda/gensim

Any known reason why they have not upgraded to Gensim 0.13.4 yet ?

[Lev Konstantinovskiy]

Hi Michel,

Thanks for spotting it!

Unfortunately Anaconda is a private closed source product which we have no control over.

Best way to contact Anaconda is to raise an issue in the Anaconda support mailing list.

 https://groups.google.com/a/continuum.io/forum/#!forum/anaconda

Regards

Lev

[Michel Fombellida]

Hi Lev,

Thanks for the feedback, I shall post a question on their support mailing list but before I do... is it normal that your github page (https://github.com/RaRe-Technologies/gensim) shows 0.13.3 as the current release in the readme.md ?

see the screenshot below :

Le mardi 10 janvier 2017 10:49:56 UTC+1, Michel Fombellida a écrit :

[Gordon Mohr]

It's not quite fair to describe Anaconda as "a private closed source product" – the package manager and core packages are all open source. There are some advanced options & upgrades requiring a license/subscription, but they're easy to avoid. 

I tend to prefer the 'miniconda' installer, so that my environments have only the packages I consciously choose, but I still benefit from the optimizations available in Conda packages (like the Intel MKL-enhanced numpy/scipy/scikit-learn). I've also found the easy switching between Python 2.x and 3.x interpreters it offers helpful. 

To be sure to get the latest gensim, after installing the other core libraries that you want to get from Conda packages, you can just use `pip install gensim` as with any other Python (virtual) environment. That will pull the latest gensim from PyPI, instead of whatever often-older gensim version is in the Conda package repository. 

- Gordon

[Radim Řehůřek]

That's interesting. Where is the Anaconda's source code hosted, how can we open a PR there to update their gensim ourselves?

-rr

[Michel Fombellida]

I am using the Anaconda environment which is actually part of another package (the MS Cognitive Toolkit, CNTK), I tried "pip install gensim" in this environment but it fails with dependencies issues. As I have little time to spend on this I would prefer to rely on packages that are already integrated.
I posted the question on the Anaconda mailing list... let's see what they say.

[Gordon Mohr]

The conda package manager source is at:

    https://github.com/conda/conda

Tools/instructions for building your own packages are at:

    https://github.com/conda/conda-build

    http://conda.pydata.org/docs/building/bpp.html

The 'Anaconda Distribution' is just what Continuum Analytics makes automatically available as installer-bundled packages, and from their repository. The recipes that seem to populate <https://repo.continuum.io/pkgs/> are all at:

    https://github.com/ContinuumIO/anaconda-recipes

...and there's a separate issues-only project for raising Anaconda distribution/installer matters at:

    https://github.com/ContinuumIO/anaconda-issues/issues

HOWEVER, given that installation from PyPI via `pip install gensim` works fine in conda environments, I don't know strong reasons to add another 'official' maintained place to get gensim. 

My suggestion would be to recommend that conda users, including those who've pre-installed the giant ball of Anaconda core packages, just ignore the 'gensim' that's in the conda repositories. (And, uninstall it if they've previously run `conda install gensim`). Install gensim from PyPI instead. 

We might also ask Continuum to either update their Anaconda-distribution mirror-package more rapidly after gensim PyPI releases, or that they phase-out `gensim` from their mirrors entirely, so there's no risk of getting a lagged version, and users then naturally rely on PyPI instead. But asking for (or getting) that change shouldn't be strictly necessary, as long as there's clear guidance from gensim's docs about preferred installation sources. 

- Gordon

[Lev Konstantinovskiy]

The "Anaconda distribution" which Continuum occasionally updates with new releases is the closed source product that I was referring to. It seems fair that it is Continuum and not Gensim community that tests and makes sure that Gensim works with the specific packages that Continuum has selected to be in the Anaconda distribution. Hopefully the errors that Michel has encountered with 'pip install gensim' will be resolved by Continuum when gensim is updated to 0.13.4 in the Anaconda Distribution. I would prefer for Gensim to stay in the Anaconda distribution and that we keep pinging Continuum to update it via the Anaconda mailing list.

[Gordon Mohr]

It's hard to find any parts of the "Anaconda Distribution" for which source isn't available and open-licensed, so I still don't think it is accurate to describe it as a "closed source product". 

Michael, without seeing the errors it's unclear what "dependencies issues" might be coming up with a plain `pip install gensim`, which is how I've successfully installed gensim in conda environments. It could be specific to the MS CNTK setup you're using. If there are problems, it might be as simple as installing a few key packages first via `conda`, then gensim via PIP – perhaps faster and less time-consuming than awaiting Continuum's update of Anaconda repositories. 

- Gordon

[Parkway]

I asked similar questions on the Anaconda user group in December and got a response from Continuum - https://groups.google.com/a/continuum.io/forum/#!searchin/anaconda/parkway%7Csort:date/anaconda/Q50Yl1V2q9g/yNdqhll4CQAJ

[Michel Fombellida]

Thanks Gordon, I eventually managed.

I initially tried "pip install gensim" without first doing a "conda uninstall gensim"... and it caused problems.

This time following your advice I first removed gensim with "conda uninstall gensim" and then did "pip install gensim and I now have 0.13.4.1 correctly installed !   

[Radim Řehůřek]

Thanks for the links Gordon.

Agreed with Lev -- if it's as easy as opening a PR / raising an issue at https://github.com/ContinuumIO/anaconda-recipes/tree/master/gensim , we can simply ping Continuum for an update from time to time.

-rr

[Lev Konstantinovskiy]

Raised an issue with Anaconda on github and made it a part of our release process.