A workaround for specifying compile-time dependencies in setuptools

[Matthew Honnibal]

Python packaging is quite a mess.  The standard library module, distutils, lacks a lot of features, but the replacement, setuptools, brought a lot of its own issues. For a long time setuptools seemed to go out of its way to break Cython compilation, by silently renaming .pyx files to .c files if Pyrex was not installed. This has since been fixed, but the word "Cython" still occurs exactly 0 times in the setuptools documentation.

A key feature that distutils lacks is dependency resolution: if someone wants to install your package, you want specified dependencies to be downloaded and installed automatically. This is unsupported by distutils. There's a "require" keyword argument to distutils.core.setup, but it does absolutely nothing.

The setuptools.setup function supports two keyword arguments for dependencies: setup_requires and install_requires. The packages you specify in install_requires will be installed, but the setup_requires packages will only be downloaded and inserted into sys.path, so that they can be imported during your setup.py script. They will not be available at runtime.

Frustratingly, if you specify the same package in setup_requires and install_requires, it will not be installed:

https://bitbucket.org/pypa/setuptools/issue/209

https://bitbucket.org/pypa/setuptools/issue/80/packages-listed-in-setup_requires-are

I'm shipping a package written in Cython, and it requires certain C header files to be present at compile time. I'd like my users to be able to write "pip install mylibrary". I don't want them to have to write "pip install numpy && pip install mylibrary". If the dependency resolution system is broken, it's broken all the way down the tree. Any library that imports my library will also suffer from this problem.

In the absence of a good solution, here's *a* solution:

https://github.com/syllog1sm/headers_workaround

I uploaded a separate package to PyPi, which distributes the headers I require. I then specify this package as the setup_requires dependency, and add some code to my setup.py file telling the headers_workaround module which headers to install.

Obviously, this is unideal, but it's been working for me. If anyone has better ideas, I'd love to hear them!

Best,
Matt.

[Björn Dahlgren]

On Sunday, 11 January 2015 21:28:01 UTC+1, Matthew Honnibal wrote:

 I don't want them to have to write "pip install numpy && pip install mylibrary". If the dependency resolution system is broken, it's broken all the way down the tree. Any library that imports my library will also suffer from this problem.

is instructing them to run: "pip install -r requirements.txt" acceptable?
Then you just need to make sure your setup.py don't try to import numpy for e.g. "python setup.py egg_info"
I ran into this and got help here:
https://github.com/pypa/pip/issues/25#issuecomment-31218582

Best,
Björn