Gensim support for Apple M1 chips

[Mohammad Momayezi]

Is gensim going to support Apple M1 chips anytime soon? I can import almost all my required data science modules (numpy, pandas, scipy, ...) without any problem in my miniforge environment which has a python 3.9.1 installed and everything runs natively on the M1 chip. The only exception for me is gensim! right now I'm receiving "PackagesNotFoundError" error by running conda install -c conda-forge gensim.

[Gordon Mohr]

Of course it'd be great to support it, but unless you donate me an M1 Mac, there are limits to the debugging/tuning I can do. 

Can you show more of the errors you get when trying to use it? How about if you do a checkout from the Github of the 'develop' code, and do a pip install from a local directory? (Which errors, if any, do you see in that scenario?)

- Gordon

[Mohammad Momayezi]

Haha! Didn't know it's going to take this much effort on your side that it needs to be compensated with an M1 Mac! 😄 but I'll consider it! 😜

I can't install it! The screen shot after I run <conda install -c conda-forge gensim> is attached.

I also tried <pip install --pre --upgrade gensim> to install the gensim 4.0.0 beta, it was having a problem installing scipy and it was continuously trying older versions of scipy without any success. (The screenshot is attached.) I abrupted the process and install scipy manually, then tired <pip install --pre --upgrade gensim> again and it was successful! Weird!

However this doesn't change the fact that it's still not working on anaconda.

[Mohammad Momayezi]

Hi. I still can't install gensim on my MacBook M1 by running the following command in my miniforge environment:

conda install -c conda-forge gensim

still the same PackagesNotFoundError!

PackagesNotFoundError: The following packages are not available from current channels:

  - gensim

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64

  - https://conda.anaconda.org/conda-forge/noarch

  - https://repo.anaconda.com/pkgs/main/osx-arm64

  - https://repo.anaconda.com/pkgs/main/noarch

  - https://repo.anaconda.com/pkgs/r/osx-arm64

  - https://repo.anaconda.com/pkgs/r/noarch

I was able to install it using pip in a pip environment but still can't install it in my conda environment.

[Radim Řehůřek]

Hi,

Gensim supports only the official Python Package Index (PyPI) = pip.

I'm not sure how the conda environment works (and have no interest in it) – maybe try asking the conda devs?

Best,

Radim

[Gordon Mohr]

Eventually the conda maintainers may add it to conda-forge. 

I've never had any problem using `pip` to install Gensim into a conda environment, from PyPI. Did you try that?

(There might still be issues running on the M1 - I don't think any of Gensim's build/test environment tests on M1 Macs. So it might work, or hit new errors. If `pip` into a non-conda venv seems to work, that's a good sign.)

- Gordon

[Ben Reaves]

It looks like the error is from BASS which is used by numpy which is used by gensim. Quickest fix would probably be to reinstall numpy, being careful to get a version that does not use BASS. Might need to recompile it. Remind me not to upgrade my 6-year-old Macbook Pro to an M1-based

--
You received this message because you are subscribed to the Google Groups "Gensim" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gensim+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gensim/94f9a14b-e412-4b35-85c8-1f5e17983b85n%40googlegroups.com.

--

_____________________________________________________________________

Ben Reaves

--

[Mohammad Momayezi]

@Gordon Mohr, gensim 4.0.1 already exists in conda-forge packages here:

https://anaconda.org/conda-forge/gensim

The problem is it can't be installed. Seems the version doesn't satisfy the needs to be placed in osx-arm64.

As I said it works in my pip environment natively but it takes toll to change my environment every time I want to use gensim.

@Ben Reaves Thanks! How can I make sure?

[Gordon Mohr]

Have you tried `pip install gensim` from inside the conda environment? If that fails, with what error?

- Gordon

[Ben Reaves]

Another M1 user reported problems with numpy, and this posting shows how he recompiled it

https://stackoverflow.com/questions/65336789/numpy-build-fail-in-m1-big-sur-11-1

That's all I could find for now. Please let us know, because, if my current laptop fails, I'll want to buy another and Apple is strongly pushing m1.

To view this discussion on the web visit https://groups.google.com/d/msgid/gensim/cc24405c-8b20-4d14-befb-ac4af5e0826fn%40googlegroups.com.

[Mohammad Momayezi]

@Gordon Mohr It's a very bad idea and it's highly advised not to mix pip and conda in an environment, so I didn't.

[Gordon Mohr]

Though I know there can be problems between package managers, and conda packages generally prefer to have their requirements satisfied by oher conda packages, I've not seen such "highly advised not to mix" guidance - except against haphazard/unmotivated mixing. 

To the contrary, whenever the conda-repos lag PyPI in including a package or recent version, installing as much as you can via conda, then installing the missing package via pip *at the end*, has been recommended, and reliably worked. (This is also how I've typically done Gensim source development: start with a conda environment with as many requirements pre-installed, then do an editable pip install from a local git-clone source directory.)

This 2018  article from Anaconda Inc – https://www.anaconda.com/blog/using-pip-in-a-conda-environment – specifically endorses this style of use when needing packages not in conda-format – which would include packages/versions only at PyPI, or private/local under-development packages. (For these cases, it also recommends using isolated, rather than hierarchically-dependent, conda environments. That's something I consider a good practice always, as any use or change to a shared root environment increases the potential for confusion in other dependent environments, even if not mixing package sources.)

Note that `conda list` happily shows pip-installed packages (unless using a non-default flag), and since Conda 4.6 (January 2019) conda includes expanded support for mixing `pip` & `conda` packages under a new 'pip_interop_enabled` option. (But, the tactic of install-pip-packages-after-conda-packages described above has worked since long before this new option's arrival.)

If you're having problems running Gensim that are unique to a conda environment, while pip-installed Gensim works, it's reasonable to try rolling the involved packages of the dependency-tree, starting from the leaves, over to pip-versions, iteratively. Eventually, you may find the package or packages where the functionality-divergence starts.

At that point, you may have found a hybrid-configuration that's functional for your purposes, or at least a clear map to which packages' conda-versions need to be fixed. Here, the problems appear to arise in Numpy/Conda, rather than Gensim. Maybe you strongly prefer the conda-packaged Numpy, for things like its default use of the IntelMKL library - though that library's performance benefits might no longer apply using a non-Intel M1 chip. 

It's certainly possible that Gensim may also need tweaks to run on a new chip, but if other M1 users report it already installs/works via PyPI, that further suggests the problem is within areas of Numpy/Conda maintenance.

- Gordon

[Mohammad Momayezi]

Thank you Gordon! The document you mentioned also "highly advises not to mix" pip and conda! And it suggests staying away from mixing as much as possible and mix only if you're forced to (also in that case by following the practices which mentions and you also mentioned). I'm also using Spyder IDE and also the IDE highly suggests (strictly!) no to mix pip and conda, here'se the link:

https://www.youtube.com/watch?v=Ul79ihg41Rs

Anyway again thank you for your comprehensive response, but I rather not to risk this. Even if it works by mixing pip and conda, making this work solely on conda shouldn't be out of reach.

Regards,

[Gordon Mohr]

If you are in the situation where the packages you want work from PyPI, but don't work from conda repos, you *are* in the situation where mixing packages may solve your problem, and then that article is recommending ways to do that safely/effectively.

But, if you don't want to try any of the multiple possible workarounds suggested upthread - you just want `conda install -c conda-forge gensim` to work – you should just wait for, or poke, the various conda/conda-forge/numpy/etc maintainers to fix things for you. They'll get around to it eventually!

- Gordon

[Mohammad Momayezi]

👍🏻

[ben.r...@gmail.com]

Conda is great for programs that don't rely on the newest features of rapidly changing modules. But for my current project, which requires Gensim and Spacy which do significant upgrades frequently, I use pyenv. If you want to avoid "polluting" your conda environment with pip installs of newer versions than are in conda-forge, I would recommend you use a virtual environment, see pyenv.

Do this before reinstalling or modifying numpy. 

[ben.r...@gmail.com]

conda-env may be better if you're experimenting on your PC and want many of the other conda environment modules like scipy etc., use

conda create --name myenv [1]

But if you're writing production code, or you want others to easily install on a different machine, you should write it as a module, use setup.py, and, to be sure that setup.py includes all of your dependencies properly, test it in a smaller python environment. For this, pyenv or venv[2] is better.

[1] https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment

[2] https://docs.python.org/3/tutorial/venv.html

[Mohammad Momayezi]

Thanks Ben!

[rohit bharadwaj]

Hi all, The channels used by Miniforge (which I have installed on my M1 MacBook Air are the following:

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64

  - https://conda.anaconda.org/conda-forge/noarch

There is no gensim package in these channels, I'm guessing that gensim has not been ported over to M1 based chips? because I'm getting PackagesNotFoundError: The following packages are not available from current channels:

is there any plans to port Gensim to support M1chips? or do we have to contact Miniforge maintainers to include gensim package in their channels?

  - gensim

[Radim Řehůřek]

Hi,

we (maintainers of Gensim) have no capacity to support 3rd party ecosystems like conda etc. So contacting their maintainers to package & include Gensim in their product is a great idea.

Let us know how it went!

Cheers,

Radim

[rohit bharadwaj]

Hi there, 

Thanks for the reply. I opened an issue on their GitHub to include gensim in mini-forge conda channel (I want to go through this way because there are already many packages in their channel which are compatible with M1 chips). You can view it here: https://github.com/conda-forge/gensim-feedstock/issues/62

In the GitHub it lists that the maintainers of gensim-feedstock are:

1. @mpenkov
   
2. @souravsingh
   

I haven't got a response yet on my issue which was opened 4 days ago, could you please let me know the progress with this, maybe I could try to help with the testing (do let me know what to do)

[rohit bharadwaj]

Hi Radim,

I'm sorry to disturb you and I know you might be busy with lots of stuff, but it has been 4 days since you commented on the issue on that GitHub repo but there is no response on it yet. 

It would be really helpful to have gensim available through conda channels because almost all the popular python packages which a data scientist might use is available through conda for M1 Mac chips. I tried to pip install but it is giving me some Cython errors and its not getting installed properly, could you please look into this. I have to use Google Colab to run my experiments for now.

Thanks,

Rohit