Can't run cython after upgrading scipy, on MacOS 10.12.5

[Peter]

This is going to be a bit long, sorry. You might want to skip to the end first, where I give the actual error message. The rest of the post is background, but probably necessary background.

I've got a new-ish Mac running MacOS 10.12.5. It came with python, numpy, scipy (albeit relatively old: python 2.7.10 and scipy 0.13); and I added cython. Since then I've been using cython painlessly like always.

The problem came yesterday when I needed to use a package in scipy that wasn't added until scipy version 0.14. And it turns out that, on this version of MacOS, it won't let me upgrade scipy, even as sudo. But there is a workaround which I found suggested here: http://docs.python-guide.org/en/latest/starting/install3/osx/ - install python3, which will give me separate python2 and python3 commands. Then I can leave the command "python" for system programs, and do all my personal work using python2 (or python3 if I want). Then when I upgraded scipy, both python2 and python3 pick up scipy 0.19 (but the original "python" does not). The download also aliases pip2 and pip3 to be the python "pip" command for updating python2 and python3 respectively.

These new python downloads didn't have cython. So I downloaded it again. Here is an edited transcript (**** is where I have edited out my username):

$ python2

Python 2.7.14 (default, Sep 25 2017, 09:53:22)

[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cython
Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
ImportError: No module named cython

$ pip2 install Cython

Collecting Cython

  Downloading Cython-0.27.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.6MB)
    100% |████████████████████████████████| 4.6MB 234kB/s
Installing collected packages: Cython
Exception:
Traceback (most recent call last):
[most of traceback redacted]
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/distlib/util.py", line 407, in write_binary_file
    with open(path, 'wb') as f:
IOError: [Errno 13] Permission denied: '/usr/local/bin/cython'


$ sudo pip2 install Cython
Password:

The directory '/Users/****/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/****/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: Cython in /usr/local/lib/python2.7/site-packages


$ python2

Python 2.7.14 (default, Sep 25 2017, 09:53:22)

[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cython
import cython

So cython appears to have been installed, and I can run cython so long as I'm not using numpy. But when I try to call cimport numpy, I get problems. I use the same format of compile file as I've been using for years:

from distutils.core import setup
from Cython.Build import cythonize
import numpy as np
setup( include_dirs = [np.get_include()],
       ext_modules = cythonize("cyrun_diffusion.pyx")
     )

I compiled with the command:

$ python2 cyrun_diffusion_compile.py build_ext --inplace > cyrun_diffusion_compile.log 2>&1

and I get this error message (i.e. the contents of cyrun_diffusion_compile.log):

running build_ext
building 'cyrun_diffusion' extension
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c cyrun_diffusion.c -o build/temp.macosx-10.12-x86_64-2.7/cyrun_diffusion.o
cyrun_diffusion.c:538:10: fatal error: 'numpy/arrayobject.h' file not found
#include "numpy/arrayobject.h"
         ^
1 error generated.
error: command 'clang' failed with exit status 1

I assume this is in reaction to the line "cimport numpy as np" in my source file.

So finally here's my question: why can't Cython find numpy? And what should I try to do to fix it?

[Matthew Brett]

Hi,

Does your /usr/local/python2 have numpy?

python2 -c 'import numpy'

? By the way, I highly recommend pip user installs, which don't need
system permissions. Although you might want to put this in your
`.bash_profile` file:

export PY_USER_BIN=$($python2 -c 'import site; print(site.USER_BASE + "/bin")')
PATH=$PY_USER_BIN:$PATH

Cheers,

Matthew

[Hai Nguyen]

Hi, my reply does not directly solve your issue: 

I always use python from Miniconda distribution, it is much much more reliable than system python. 

Hai

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Barker]

On Thu, Oct 26, 2017 at 6:42 AM, Hai Nguyen <nha...@gmail.com> wrote:

So finally here's my question: why can't Cython find numpy? And what should I try to do to fix it?

In sort: the python that comes with OS-X should be considered a system utility -- it is NOT designed to be upgraded or added to or anything. You will be MUCH happier if you start with an independent Python install, and don't try to mess the system one.

I always use python from Miniconda distribution, it is much much more reliable than system python. 

miniconda is a great option. (or the full Anaconda distribution)

Personally, I recommend using miniconda, and then adding the conda-forege cahnnel to your install:

conda config --add channels conda-forge

That will give you access to a large collection of up to date packages.

Another option is to install the python from python.org, and then use pip to install the other stuff.

conda supports a wider variety of complex packages, but pip provides the core scipy stack and cython for the pyton.org build, so that may be a fine solution for you.

BTW, if you are starting fresh -- use Python 3.6 -- it really is the way of the future, and I think finally the present :-)

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Peter]

Thanks Hai and Chris. I solved the problem by installing miniconda. By default miniconda installs to ~/miniconda2, so I've just put ~/miniconda2/bin at the front of my PATH, so all previous python upgrades are now moot. (I assume the "2" is for python 2, so may yet move to python 3).

If this screws up "system" python, I don't particularly care :)

[Chris Barker]

On Thu, Oct 26, 2017 at 4:54 PM, Peter <ozpeter...@gmail.com> wrote:

Thanks Hai and Chris. I solved the problem by installing miniconda. By default miniconda installs to ~/miniconda2, so I've just put ~/miniconda2/bin at the front of my PATH, so all previous python upgrades are now moot. (I assume the "2" is for python 2, so may yet move to python 3).

 

If this screws up "system" python, I don't particularly care :)

yes, you do -- the system python is used by who knows what system utilities -- which is why you don't want to mess with it.

But miniconda will not screw it up -- that's kind of the point.

Yes, miniconda2 is the miniconda with python2.

Again, if you don't have legacy code to deal with, I'd start with python3. But you can actually create a python3 environment even in a conda based on python2:

conda create -n py3 python=3

with create it.

Then you can run

source activate py3

and you'll have python3 -- in there, you can install Cython and whatever else you need.

-CHB

 

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Peter]

On Saturday, October 28, 2017 at 10:26:59 AM UTC+10:30, Chris Barker wrote:

On Thu, Oct 26, 2017 at 4:54 PM, Peter <ozpeter...@gmail.com> wrote:

Thanks Hai and Chris. I solved the problem by installing miniconda. By default miniconda installs to ~/miniconda2, so I've just put ~/miniconda2/bin at the front of my PATH, so all previous python upgrades are now moot. (I assume the "2" is for python 2, so may yet move to python 3).

 

If this screws up "system" python, I don't particularly care :)

yes, you do -- the system python is used by who knows what system utilities -- which is why you don't want to mess with it.

Perhaps I was being a bit dramatic, but my point is that without cython my work is severely hampered. Existing critical code was literally taking 100 times as long running plain python. (Isn't cython great!) So if putting miniconda at the front of PATH was dangerous, and I had the choice between a maybe broken machine (due to system python upgrades) or a certainly broken machine (without cython); I'd choose the former without hesitation.

Anyway, thanks again, and I'm glad fiddling PATH to point at miniconda is safe.
 
And as for python3, it's certainly an option. I've run a few programs and it seems there's very little to change, just my "raise" statements.

[Chris Barker - NOAA Federal]

And as for python3, it's certainly an option. I've run a few programs and it seems there's very little to change, just my "raise" statements.

And making print a function :-)

Despite all the hysteria, it really is the same language.

If you do continue with py2, I recommend using the primary __future__ imports to make your code 2/3 compatible.

And follow advice to make your Cython code compatible— not hard, but you need to be careful with strings and bytes objects.

-CHB

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

[Peter]

Not for me: I never liked the "print" syntax so I've been using "sys.stdout.write()" instead of "print" for years. I came to regard "print" as a hack, albeit an important hack which makes life easier for beginners.