Gensim on Raspberry Cluster

126 views
Skip to first unread message

Cássio Rampelotto Dias

unread,
Apr 2, 2019, 11:18:43 AM4/2/19
to Gensim
I'm encountering some difficulties with dependencies to install gensim on ARM architecture of raspberry cluster.
Does Gensim work with ARM?

Gordon Mohr

unread,
Apr 2, 2019, 4:37:01 PM4/2/19
to Gensim
I'm not sure it's been tested, but I wouldn't expect problems, as long as there's Python & typical OS build tools. What errors/blocks did you encounter?

- Gordon

Cássio Rampelotto Dias

unread,
Apr 2, 2019, 5:11:32 PM4/2/19
to Gensim
Thanks for posting. We're trying to run gensim on a raspberry cluster, but we're having problems with numpy and scypy dependencies. There is always an error in the end in these dependencies. In this case, it emerged doubt if it is possible to run Gensim in raspberry, there is very little of this on the web.

Gordon Mohr

unread,
Apr 2, 2019, 5:42:58 PM4/2/19
to Gensim
Which specific errors do you see with numpy and scipy, after what specific method of installations in what environment/OS, and do these errors occur on installation or upon use? 

Each of those libraries has a test option – `numpy.test('full')` or `scipy.test('full')`. Do such self-tests succeed or show useful errors?

What do the numpy/scipy forums/issue-trackers show for the specific error messages you've seen?

(My hunch is your problem is solvable, but will require far more precision/specificity in describing steps attempted and blocks encountered.)

- Gordon

Cássio Rampelotto Dias

unread,
Apr 4, 2019, 3:58:18 PM4/4/19
to Gensim
So

The architecture I'm using is Linux raspberrypi 4.14.79-v7 + # 1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU / Linux
Raspbian in RP 3B +

I am using PIP for installation and the errors start as described below:


Building wheels for collected packages: gensim, numpy, scipy, smart-open, bz2file
  Running setup.py bdist_wheel for gensim ... done
  Stored in directory: /home/pi/.cache/pip/wheels/d3/8c/19/c7b397f19e37b1230795b1f119147696203abc58d367b93303
  Running setup.py bdist_wheel for numpy ... done
  Stored in directory: /home/pi/.cache/pip/wheels/8c/a8/49/e458f0fdbc4fe3759be6b9371e172b3f0e82c09f5a750e977e
  Running setup.py bdist_wheel for scipy ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-eAya8s/scipy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpkANQrDpip-wheel- --python-tag cp27:
  Running from scipy source directory.
  lapack_opt_info:
  lapack_mkl_info:
    libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
    NOT AVAILABLE
  
  openblas_lapack_info:
    libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
    NOT AVAILABLE
  
  system_info:
    NOT AVAILABLE
  
  atlas_3_10_threads_info:
  Setting PTATLAS=ATLAS
    libraries tatlas,tatlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries tatlas,tatlas not found in /usr/lib
    libraries lapack_atlas not found in /usr/lib
    libraries tatlas,tatlas not found in /usr/lib/arm-linux-gnueabihf
    libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
  <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
    NOT AVAILABLE
  
  atlas_3_10_info:
    libraries satlas,satlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries satlas,satlas not found in /usr/lib
    libraries lapack_atlas not found in /usr/lib
    libraries satlas,satlas not found in /usr/lib/arm-linux-gnueabihf
    libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
  <class 'numpy.distutils.system_info.atlas_3_10_info'>
    NOT AVAILABLE
  
  atlas_threads_info:
  Setting PTATLAS=ATLAS
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/lib
    libraries lapack_atlas not found in /usr/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/lib/arm-linux-gnueabihf
    libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
  <class 'numpy.distutils.system_info.atlas_threads_info'>
    NOT AVAILABLE
  
  atlas_info:
    libraries f77blas,cblas,atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries f77blas,cblas,atlas not found in /usr/lib
    libraries lapack_atlas not found in /usr/lib
    libraries f77blas,cblas,atlas not found in /usr/lib/arm-linux-gnueabihf
    libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
  <class 'numpy.distutils.system_info.atlas_info'>
    NOT AVAILABLE
  
  /usr/lib/python2.7/dist-packages/numpy/distutils/system_info.py:572: UserWarning:
      Atlas (http://math-atlas.sourceforge.net/) libraries not found.
      Directories to search for the libraries can be specified in the
      numpy/distutils/site.cfg file (section [atlas]) or by setting
      the ATLAS environment variable.
    self.calc_info()
  lapack_info:
    libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
    NOT AVAILABLE
  
  /usr/lib/python2.7/dist-packages/numpy/distutils/system_info.py:572: UserWarning:
      Lapack (http://www.netlib.org/lapack/) libraries not found.
      Directories to search for the libraries can be specified in the
      numpy/distutils/site.cfg file (section [lapack]) or by setting
      the LAPACK environment variable.
    self.calc_info()
  lapack_src_info:
    NOT AVAILABLE
  
  /usr/lib/python2.7/dist-packages/numpy/distutils/system_info.py:572: UserWarning:
      Lapack (http://www.netlib.org/lapack/) sources not found.
      Directories to search for the sources can be specified in the
      numpy/distutils/site.cfg file (section [lapack_src]) or by setting
      the LAPACK_SRC environment variable.
    self.calc_info()
    NOT AVAILABLE
  
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-eAya8s/scipy/setup.py", line 492, in <module>
      setup_package()
    File "/tmp/pip-build-eAya8s/scipy/setup.py", line 488, in setup_package
      setup(**metadata)
    File "/usr/lib/python2.7/dist-packages/numpy/distutils/core.py", line 135, in setup
      config = configuration()
    File "/tmp/pip-build-eAya8s/scipy/setup.py", line 395, in configuration
      raise NotFoundError(msg)
  numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
  
  ----------------------------------------
  Failed building wheel for scipy
  Running setup.py clean for scipy
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-eAya8s/scipy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  
  `setup.py clean` is not supported, use one of the following instead:
  
    - `git clean -xdf` (cleans all files)
    - `git clean -Xdf` (cleans all versioned files, doesn't touch
                        files that aren't checked into the git repo)
  
  Add `--force` to your command to use it anyway if you must (unsupported).
  
  
  ----------------------------------------
  Failed cleaning build dir for scipy
  Running setup.py bdist_wheel for smart-open ... done
  Stored in directory: /home/pi/.cache/pip/wheels/f7/a6/ff/9ab5842c14e50e95a06a4675b0b4a689c9cab6064dac2b01d0
  Running setup.py bdist_wheel for bz2file ... done
  Stored in directory: /home/pi/.cache/pip/wheels/81/75/d6/e1317bf09bf1af5a30befc2a007869fa6e1f516b8f7c591cb9
Successfully built gensim numpy smart-open bz2file
Failed to build scipy
Installing collected packages: numpy, scipy, six, boto, docutils, jmespath, python-dateutil, urllib3, botocore, futures, s3transfer, boto3, bz2file, idna, chardet, certifi, requests, smart-open, gensim
  Running setup.py install for scipy ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-eAya8s/scipy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-GHcWhp-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    
    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:
    
      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install scipy`   (last SciPy release on PyPI)
    
    
    Running from scipy source directory.
    lapack_opt_info:
    lapack_mkl_info:
    customize UnixCCompiler
      libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
      NOT AVAILABLE
    
    openblas_lapack_info:
    customize UnixCCompiler
    customize UnixCCompiler
      libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
      NOT AVAILABLE
    
    openblas_clapack_info:
    customize UnixCCompiler
    customize UnixCCompiler
      libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
      NOT AVAILABLE
    
    atlas_3_10_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
    customize UnixCCompiler
      libraries tatlas,tatlas not found in /usr/lib/arm-linux-gnueabihf
    <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
      NOT AVAILABLE
    
    atlas_3_10_info:
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
    customize UnixCCompiler
      libraries satlas,satlas not found in /usr/lib/arm-linux-gnueabihf
    <class 'numpy.distutils.system_info.atlas_3_10_info'>
      NOT AVAILABLE
    
    atlas_threads_info:
    Setting PTATLAS=ATLAS
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
    customize UnixCCompiler
      libraries ptf77blas,ptcblas,atlas not found in /usr/lib/arm-linux-gnueabihf
    <class 'numpy.distutils.system_info.atlas_threads_info'>
      NOT AVAILABLE
    
    atlas_info:
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/local/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/lib
    customize UnixCCompiler
      libraries lapack_atlas not found in /usr/lib/arm-linux-gnueabihf
    customize UnixCCompiler
      libraries f77blas,cblas,atlas not found in /usr/lib/arm-linux-gnueabihf
    <class 'numpy.distutils.system_info.atlas_info'>
      NOT AVAILABLE
    
    accelerate_info:
      NOT AVAILABLE
    
    /home/pi/.local/lib/python2.7/site-packages/numpy/distutils/system_info.py:638: UserWarning:
        Atlas (http://math-atlas.sourceforge.net/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [atlas]) or by setting
        the ATLAS environment variable.
      self.calc_info()
    lapack_info:
    customize UnixCCompiler
      libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/arm-linux-gnueabihf']
      NOT AVAILABLE
    
    /home/pi/.local/lib/python2.7/site-packages/numpy/distutils/system_info.py:638: UserWarning:
        Lapack (http://www.netlib.org/lapack/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [lapack]) or by setting
        the LAPACK environment variable.
      self.calc_info()
    lapack_src_info:
      NOT AVAILABLE
    
    /home/pi/.local/lib/python2.7/site-packages/numpy/distutils/system_info.py:638: UserWarning:
        Lapack (http://www.netlib.org/lapack/) sources not found.
        Directories to search for the sources can be specified in the
        numpy/distutils/site.cfg file (section [lapack_src]) or by setting
        the LAPACK_SRC environment variable.
      self.calc_info()
      NOT AVAILABLE
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-eAya8s/scipy/setup.py", line 492, in <module>
        setup_package()
      File "/tmp/pip-build-eAya8s/scipy/setup.py", line 488, in setup_package
        setup(**metadata)
      File "/home/pi/.local/lib/python2.7/site-packages/numpy/distutils/core.py", line 137, in setup
        config = configuration()
      File "/tmp/pip-build-eAya8s/scipy/setup.py", line 395, in configuration
        raise NotFoundError(msg)
    numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-eAya8s/scipy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-GHcWhp-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-eAya8s/scipy/


Looking for some solutions in forums, according to guidelines

I installed scipy with apt-get successfully, but when I return to install gensim, again the same error occurs.

Thanks a lot for some help

Benedict Holland

unread,
Apr 4, 2019, 6:41:51 PM4/4/19
to gen...@googlegroups.com
Do they go away on Python3? 

Also, those libraries might not get installed by default or have availability on python2.7. 

Thanks,
~Ben

--
You received this message because you are subscribed to the Google Groups "Gensim" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gensim+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gordon Mohr

unread,
Apr 4, 2019, 9:56:21 PM4/4/19
to Gensim
I'm not sure how close your Linux is to Debian/Ubuntu. I would avoid mixing installation methods (apt-get vs pip) for things that are Python packages (like scipy) - and personally I'd prefer pip, especially if using virtual environments. (Virtual environments are usually a good idea when doing development, to silo the configuration needed for specific projects, but if you're just trying to prep these systems for deployment of something elsewhere, maybe you're OK changing the system Python's installed packages.)

The main gist of the error you've shown - "numpy.distutils.system_info.NotFoundError: No lapack/blas resources found" – is that scipy is missing a prerequisite lapack/blas resource. Such native libraries can and should be installed by the native OS `apt-get` or similar, whenever possible. So mainly I'd look for ways to install those prerequisites on your machine. If your Linux is Debian/Ubuntu related, with the right things in package repositories, that might be something like what's recommended in this other SO answer (<https://stackoverflow.com/a/31840553/130288>):

    sudo apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran

So my suggested strategy would be:

- start fresh, undoing anything attempted so far, perhaps even via a re-imaging of the target system
- ensure Python pip is available, and use a virtual environment if appropriate
- using the proper Python (environment), attempt a narrow "pip install scipy"
- if it fails, note the error, and if as above, try using apt-get to install the lapack/blas/etc support libraries that scipy needs
- then, try "pip install scipy" again; if that works, then move on to pip-install things (like gensim) that rely on it

- Gordon
Reply all
Reply to author
Forward
0 new messages