[cython-users] cimport numpy fails to find arrayobject.h

[RJVB]

Hello,

Here's a little mystery to me, mysterious in that the issue apparently
affects only me, even with a freshly installed "stock" Enthought
Python 2.6 distribution on MSWinXP.

When one installs numpy, the headers go somewhere rather deep inside
the site-packages directory. But when one references numpy in Cython
code, and does a

cimport numpy as nd

the resulting C code does an

#include "numpy/arrayobject.h"

which of course causes the compiler to complain about a missing header
file. The issue can be triggered simply by loading/building the
numpy_test.pyx file inside tests/run of the distribution tree.

On Linux, MacOSX and Cygwin, the problem is solved by putting a
symlink to the numpy header file directory in the directory where
Python.h lives. But that isn't possible of course under MSWin.

I see just now that runtests.py from the source distribution invokes
numpy.get_include() and adds the result to some header file include
path. Why does this not happen automatically, possibly triggered by a
'cimport numpy' statement?

TIA,
René


--
Subscription settings: http://groups.google.com/group/cython-users/subscribe?hl=en

[Dag Sverre Seljebotn]

RJVB wrote:
> Hello,
>
> Here's a little mystery to me, mysterious in that the issue apparently
> affects only me, even with a freshly installed "stock" Enthought
> Python 2.6 distribution on MSWinXP.
>

It affects everyone. The C include path is considered an orthogonal
issue to compiling Cython code, i.e. it should be part of your build
system. Search for distutils, Cython, numpy, get_include.

Perhaps we could add this as a feature, some sort of pragma in the pxd
files suggesting to build tools to e.g. call "numpy.get_include"...I
suggested this earlier but was downvoted at the time.

Dag Sverre

[RJVB]

On Apr 23, 7:33 pm, Dag Sverre Seljebotn <da...@student.matnat.uio.no>
wrote:

>
> It affects everyone. The C include path is considered an orthogonal
> issue to compiling Cython code, i.e. it should be part of your build
> system. Search for distutils, Cython, numpy, get_include.
>
> Perhaps we could add this as a feature, some sort of pragma in the pxd
> files suggesting to build tools to e.g. call "numpy.get_include"...I
> suggested this earlier but was downvoted at the time.

I can see how this would be a standpoint for a "classical" build
system, using a setup.py file.

But I can't reconcile it with how I perceive the pyximport extension,
which I think is supposed to allow transparent importing of pyx files.
If there's a way to configure the importing options, adding to the
include_dirs path would have to be done there, but is there such a
mechanism?

It's not like it was difficult to add the include_dir once I traced
where the distutils.core.Extension is constructed in pyximport.py ...

R.B.

[Dag Sverre Seljebotn]

RJVB wrote:
> On Apr 23, 7:33 pm, Dag Sverre Seljebotn <da...@student.matnat.uio.no>
> wrote:
>
>> It affects everyone. The C include path is considered an orthogonal
>> issue to compiling Cython code, i.e. it should be part of your build
>> system. Search for distutils, Cython, numpy, get_include.
>>
>> Perhaps we could add this as a feature, some sort of pragma in the pxd
>> files suggesting to build tools to e.g. call "numpy.get_include"...I
>> suggested this earlier but was downvoted at the time.
>>
>
> I can see how this would be a standpoint for a "classical" build
> system, using a setup.py file.
>
> But I can't reconcile it with how I perceive the pyximport extension,
> which I think is supposed to allow transparent importing of pyx files.
> If there's a way to configure the importing options, adding to the
> include_dirs path would have to be done there, but is there such a
> mechanism?
>

You are right, pyximport is underdeveloped and needs some improvement in
this area. Part of the reason is that none of the core developers use
pyximport on a daily basis (AFAIK).

I believe there's some configuration files etc. possible which it loads
(or flags you pass to install(), or something). But I totally agree that
some information in numpy.pxd understandable to pyximport is the way to
go. (I just don't have the time to champion it on the mailing list and
implement it..)

Dag Sverre

[josef...@gmail.com]

On Fri, Apr 23, 2010 at 1:53 PM, Dag Sverre Seljebotn

<da...@student.matnat.uio.no> wrote:
> RJVB wrote:
>>
>> On Apr 23, 7:33 pm, Dag Sverre Seljebotn <da...@student.matnat.uio.no>
>> wrote:
>>
>>>
>>> It affects everyone. The C include path is considered an orthogonal
>>> issue to compiling Cython code, i.e. it should be part of your build
>>> system. Search for distutils, Cython, numpy, get_include.
>>>
>>> Perhaps we could add this as a feature, some sort of pragma in the pxd
>>> files suggesting to build tools to e.g. call "numpy.get_include"...I
>>> suggested this earlier but was downvoted at the time.
>>>
>>
>> I can see how this would be a standpoint for a "classical" build
>> system, using a setup.py file.
>>
>> But I can't reconcile it with how I perceive the pyximport extension,
>> which I think is supposed to allow transparent importing of pyx files.
>> If there's a way to configure the importing options, adding to the
>> include_dirs path would have to be done there, but is there such a
>> mechanism?
>>
>
> You are right, pyximport is underdeveloped and needs some improvement in
> this area. Part of the reason is that none of the core developers use
> pyximport on a daily basis (AFAIK).

from mailing list last November:
-------------------

>> How do I tell pyximport where the numpy files are?
>>
>
> Enter python, import numpy, print numpy.get_include(), and then add
> that path under the [build_ext] section in distutils.cfg, like this:
>
> [build_ext]
> include_dirs= <NUMPY INCLUDE DIR>
> compiler=mingw32

With this it works without problems, additionally build_dir can
be set as an option to pyximport.install.
-----------

I don't know if anything has changed in the meantime

Josef

[RJVB]

On Apr 23, 7:53 pm, Dag Sverre Seljebotn <da...@student.matnat.uio.no>
wrote:

> (or flags you pass to install(), or something). But I totally agree that
> some information in numpy.pxd understandable to pyximport is the way to
> go. (I just don't have the time to champion it on the mailing list and
> implement it..)

So do I, I'm going to try and champion python+Cython as a viable,
intuitive and (hopefully) bulletproof approach at work, that won't
leave me much time to delve deeply into pyximport's implementation.

I don't see anything pertaining to a pyximport configuration
mechanism, apart from the arguments that can be passed to
pyximport.install().

So, for anyone stumbling upon this thread looking for a quick
solution, open site-packages/pyximport/pyximport.py and add

## RJVB
try:
from numpy import get_include as numpy_get_include
numpy_include_dir = [numpy_get_include()]
except:
numpy_include_dir = []


for instance above the definition for get_distutils_extension, and
modify that function so it reads

def get_distutils_extension(modname, pyxfilename):
## RJVB removed stale code
extension_mod,setup_args = handle_special_build(modname,
pyxfilename)
if not extension_mod:
from distutils.extension import Extension
## RJVB: added the include_dirs argument
extension_mod = Extension(name = modname,
sources=[pyxfilename], include_dirs=numpy_include_dir)
return extension_mod,setup_args

and cross your fingers you'll never trigger a special build ;)

R.

[RJVB]

On Apr 23, 8:05 pm, josef.p...@gmail.com wrote:
[,,,]

> > Enter python, import numpy, print numpy.get_include(), and then add
> > that path under the [build_ext] section in distutils.cfg, like this:
>
> > [build_ext]
> > include_dirs= <NUMPY INCLUDE DIR>
> > compiler=mingw32

I guess that would also be an option.

> With this it works without problems, additionally build_dir can
> be set as an option to pyximport.install.

Yes, but that's where the compilation results go. I agree one will
probably not want them to go into ~/.pyxbld ;)


R

[Lisandro Dalcin]

On 23 April 2010 15:15, RJVB <rjvb...@gmail.com> wrote:
>
> Yes, but that's where the compilation results go. I agree one will
> probably not want them to go into ~/.pyxbld ;)
>

Why not? What's wrong with putting them in your OS-designated home directory?


--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

[RJVB]

On Apr 23, 8:20 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:

> On 23 April 2010 15:15, RJVB <rjvber...@gmail.com> wrote:
>
> Why not? What's wrong with putting them in your OS-designated home directory?

For me, that's a starting point. Just how often do you actually do
things there?

My source code lives in directories that are organised by theme,
topic, whatever. Depending on the project I'm working on, I can have
files that do the same thing in different ways, or just about any
other reason why I wouldn't want a compiled pyx file to go into a
system-wide location. Code I write that has a more general
applicability goes into site-packages.

R.

[Dag Sverre Seljebotn]

RJVB wrote:
> On Apr 23, 8:20 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:
>
>> On 23 April 2010 15:15, RJVB <rjvber...@gmail.com> wrote:
>>
>> Why not? What's wrong with putting them in your OS-designated home directory?
>>
>
> For me, that's a starting point. Just how often do you actually do
> things there?
>
> My source code lives in directories that are organised by theme,
> topic, whatever. Depending on the project I'm working on, I can have
> files that do the same thing in different ways, or just about any
> other reason why I wouldn't want a compiled pyx file to go into a
> system-wide location. Code I write that has a more general
> applicability goes into site-packages.
>

I guess the point is that the .so files can be considered temporary
results needed for linking. It's a cache. In the spirit of pyximport
they could go to /tmp/pyxbld as well (well, the home dir means you don't
need to recompile stuff when you reboot).

If one wants to actually have the .so files in a location where you can
depend on their existence, then one really should look into build systems.

Dag Sverre

[Dag Sverre Seljebotn]

Dag Sverre Seljebotn wrote:
> RJVB wrote:
>> On Apr 23, 8:20 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:
>>
>>> On 23 April 2010 15:15, RJVB <rjvber...@gmail.com> wrote:
>>>
>>> Why not? What's wrong with putting them in your OS-designated home
>>> directory?
>>>
>>
>> For me, that's a starting point. Just how often do you actually do
>> things there?
>>
>> My source code lives in directories that are organised by theme,
>> topic, whatever. Depending on the project I'm working on, I can have
>> files that do the same thing in different ways, or just about any
>> other reason why I wouldn't want a compiled pyx file to go into a
>> system-wide location. Code I write that has a more general
>> applicability goes into site-packages.
>>
> I guess the point is that the .so files can be considered temporary
> results needed for linking. It's a cache. In the spirit of pyximport
> they could go to

Not linking, but loading into Python.

Dag Sverre

[RJVB]

On Apr 23, 8:33 pm, Dag Sverre Seljebotn <da...@student.matnat.uio.no>
wrote:
>

> If one wants to actually have the .so files in a location where you can
> depend on their existence, then one really should look into build systems.

I don't think I agree. I see it as something complementary to a build
system that feeds into a global repository. The files could be
temporary for all I care (on a fast enough machine). Look at it this
way. Do you have an argument to put all your .py files in a single,
central repository? I guess not ... I for one don't. I could easily
have 36 different optimise.py files, all slightly different, tuned to
a specific problem living in a given directory. Cython allows my to
improve performance by creating the corresponding optimise.pyx file.
Then, I can import this code transparently, say in an encapsulating
python framework. Why would the .so, .dll or .pyd file that gets
created and imported behind my back go somewhere where it can replace
an already existing file? Or where, worse, it gets imported instead of
the intended library, if it happens to be newer than the local
optimise.pyx file?

BTW, the build directory that holds the intermediaries and results of
a classical build process also does not go into $HOME, does it?

Anyway, I might not be hammering on this if I hadn't learned to live
with local _pyxbld directories in a previous release, but I've gotten
to prefer things like that.

R

[Lisandro Dalcin]

I think you have a point. Given the recent features introduced in
Python 3 (related to local __pycache__ directories), I think we
should follow that route and use a local __pyxcache__ directory for
caching .so/.pyd files.

--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

[Robert Bradshaw]

On Apr 23, 2010, at 10:53 AM, Dag Sverre Seljebotn wrote:

> RJVB wrote:
>> On Apr 23, 7:33 pm, Dag Sverre Seljebotn
>> <da...@student.matnat.uio.no>
>> wrote:
>>
>>> It affects everyone. The C include path is considered an orthogonal
>>> issue to compiling Cython code, i.e. it should be part of your build
>>> system. Search for distutils, Cython, numpy, get_include.
>>>
>>> Perhaps we could add this as a feature, some sort of pragma in the
>>> pxd
>>> files suggesting to build tools to e.g. call "numpy.get_include"...I
>>> suggested this earlier but was downvoted at the time.
>>>
>>
>> I can see how this would be a standpoint for a "classical" build
>> system, using a setup.py file.
>>
>> But I can't reconcile it with how I perceive the pyximport extension,
>> which I think is supposed to allow transparent importing of pyx
>> files.
>> If there's a way to configure the importing options, adding to the
>> include_dirs path would have to be done there, but is there such a
>> mechanism?
>>
> You are right, pyximport is underdeveloped and needs some
> improvement in this area. Part of the reason is that none of the
> core developers use pyximport on a daily basis (AFAIK).

I would say that pyximport is very useful, but not production ready
yet (at least for anything involving dependancies). Other than getting
it to work for an example and shipping it, I've never used it myself
(99% of my Cython is in the Sage notebook or Sage codebase itself,
which handles all the build details).

> I believe there's some configuration files etc. possible which it
> loads (or flags you pass to install(), or something). But I totally
> agree that some information in numpy.pxd understandable to pyximport
> is the way to go. (I just don't have the time to champion it on the
> mailing list and implement it..)

It would be great if this kind of metadata could be put into the .pyx
files themselves. While working on Sage's build system, I wrote up
some thoughts at http://wiki.cython.org/enhancements/distutils_preprocessing
, which I think would be applicable to pyximport as well.

- Robert

[RJVB]

Well, nice to see in any event that the question I hesitated to asked
spurned some constructive discussion! :)

R.