Getting includes from a package to be installed from requirements.txt

[Andrei Pashkin]

Is there an idiom for using header files from packages that go into "install_requires" in setup.py? In the docs I've found this:

include_path = [numpy.get_include()]

https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#configuring-the-c-build

But it implies that numpy is already installed when python setup.py install is executed.

[mattip]

Could you expand on your use case? Why would you want to use numpy's headers if you do not have numpy installed? You can run `pip install` before `setup.py` (`setup.py` is generally meant to be run as part of `pip install` and not on its own)

[Andrei Pashkin]

Basically I want to dynamically link my extension to a third-party extension that is shipped in a package from requirements.txt. It also contains headers which I want to use when building the extension in my package.

[Chris Barker]

On Fri, Mar 27, 2020 at 10:47 PM Andrei Pashkin <andrew....@gmx.co.uk> wrote:

Basically I want to dynamically link my extension to a third-party extension that is shipped in a package from requirements.txt. It also contains headers which I want to use when building the extension in my package.

well, numpy as get_include() exactly because there is no standard way to do this in Python.

If the third-party extension is required, then it should be installed before you run setup.py, and you'll need to do some trickery to discover its headers, if it doesn't provide a utility like numpy's.

probably something making use of relative paths from the __file__ of the package.

this is all assuming that the third=party extension ships its headers in the package. If not, then you'll need to do something completely different.

-CHB

 

On Friday, March 27, 2020 at 12:21:48 PM UTC+3, mattip wrote:

On Thursday, 26 March 2020 14:56:15 UTC+2, Andrei Pashkin wrote:

Is there an idiom for using header files from packages that go into "install_requires" in setup.py? In the docs I've found this:

include_path = [numpy.get_include()]

https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#configuring-the-c-build

But it implies that numpy is already installed when python setup.py install is executed.

Could you expand on your use case? Why would you want to use numpy's headers if you do not have numpy installed? You can run `pip install` before `setup.py` (`setup.py` is generally meant to be run as part of `pip install` and not on its own)

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/f78db257-b798-49aa-aa03-8480ba0863a6%40googlegroups.com.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Jason Madden]

I've dealt with this in a few different packages including gevent (needs greenlet's headers to build, uses Cython) and most recently, zope.container (needs persistent and zope.proxy to build, pure C). Both are installed in a wide variety of environments using a variety of different tools of different versions (including plain pip and zc.buildout).

To make this work well, it's helpful if the dependencies do two things: install their headers using *both* the `headers=` setup() argument *and* as package data. 

Then, in the project that needs them, you need to arrange for the dependencies to be installed at build time. You can use PEP 518's support to declare these dependencies in pyproject.toml's `[build-system]requires` key, but widest compatibility (e.g., with zc.buildout) still requires declaring them in the `setup_requires=` setup() argument. Or you can just document that for the user, but pip's build-time isolation that goes along with pyproject.toml makes that more challenging these days.

Finally, when setup.py runs, you need to discover the location of these headers and add them to the `Extension`. This has to be carefully deferred until setup() is actually running and not before. The easiest way I've seen to do this is to pass a custom object to the `include_dirs` argument of `Extension`  In zope.container and a few other projects, this is done by using an object that either does this lookup when its __str__ is called (as in BTrees, which is older), or, in zope.container, acts as a proxy. 

The source for what zope.container does is https://github.com/zopefoundation/zope.container/blob/1381b11e3195b30c40b68b3c1e1a3160d0a8004a/setup.py#L32-L128, and BTrees is https://github.com/zopefoundation/BTrees/blob/078ba608bfc966bd4e4ca381347e1e7d5dcec4ae/setup.py#L69-L92

Hope this helps!