Distributing modules

[Geert Jansen]

Hi,

I'm following the docs on how to distribute a module that uses CFFI. This has the effect that the setup script becomes dependent on cffi. Is there a way in which the cffi package can be automatically installed by pip when installing an sdist from PyPI, *before* pip attempts to run the setup script?

I did a hack where I manually run easy_install if cffi is not available. But it's not very pretty. See:

  https://github.com/geertj/gruvi/blob/master/setup.py

Is there a better way to do this?

Regards,

Geert

[Armin Rigo]

Hi Geert,

On Wed, Aug 21, 2013 at 5:37 PM, Geert Jansen
<geert....@ravellosystems.com> wrote:
> I'm following the docs on how to distribute a module that uses CFFI. This
> has the effect that the setup script becomes dependent on cffi. Is there a
> way in which the cffi package can be automatically installed by pip when
> installing an sdist from PyPI, *before* pip attempts to run the setup
> script?
>
> I did a hack where I manually run easy_install if cffi is not available. But
> it's not very pretty. See:

Ah, good question. Actually, this is something we missed so far. I
don't have a good solution. I suppose that doing it properly would
require some refactorings of how setup.py is usually written, but even
so it's not clear how to do it.

As an "appetizer", here is how it would most likely look like in the
future CFFI 1.0: you'd write a separate module that is *not* imported
by your program:

ffi = FFI()
ffi.cdef("...lots of lines...")
ffi.make_library("foo", "...lines like in ffi.verify...")

And you would run this explicitly in order to make or re-make the
"foo.so" module. Then your program would simply say:

ffi = FFI("foo")

Which avoids all issues like computing checksums to know if it looks
like we need to rebuild something, as well as needing to re-parse the
cdef, which can take a significant time at import.

In the model above, setup.py would only reference the separate module
by *name*. When you run setup.py it works even if cffi is not
installed yet, as long as it installs it before trying to import the
separate module. Again, however, how to do it exactly is not very
clear so far.


A bientôt,

Armin.

[Geert Jansen]

Hi Armin,

On Friday, August 23, 2013 12:57:11 AM UTC-7, Armin Rigo wrote:

[snip: appetizer]

In the model above, setup.py would only reference the separate module
by *name*.  When you run setup.py it works even if cffi is not
installed yet, as long as it installs it before trying to import the
separate module.  Again, however, how to do it exactly is not very
clear so far.

Just to make sure I understand: make_library() does need to be called at some point during setup.py, right? Just doing it on first import is not going to work because the install might be done as root and the import may be as a user. Not sure if you were suggesting the former, or the latter.

Setuptools has the "setup_requires" keyword that allows you to specify early dependencies. But pip doesn't honor it. So I'm not sure it can be fixed. Pip seems to install dependencies *after* the package itself. But setup.py would need to have access to cffi to install the extension.

For now I am requiring cffi at the top of my setup script. If it's not there I print an error message asking the user to install it, and exit. I'm not installing it automatically anymore, I think that was misguided.

Regards, 

Geert

[Armin Rigo]

Hi Geert,

On Sun, Aug 25, 2013 at 3:32 PM, Geert Jansen <gee...@gmail.com> wrote:
> Just to make sure I understand: make_library() does need to be called at
> some point during setup.py, right? Just doing it on first import is not
> going to work because the install might be done as root and the import may
> be as a user. Not sure if you were suggesting the former, or the latter.

Yes: the idea would be to run what I called the "separate module" during
setup.py, and not import it at all from the regular program.

> Setuptools has the "setup_requires" keyword that allows you to specify early
> dependencies. But pip doesn't honor it. So I'm not sure it can be fixed. Pip
> seems to install dependencies *after* the package itself. But setup.py would
> need to have access to cffi to install the extension.

Argh, unhappiness :-( It seems that on this front, we cannot really
do better that what you did, which is to explain to the user that he
needs to run "pip install cffi" explicitly first...


A bientôt,

Armin.

[Armin Rigo]

Hi again,

On Sun, Aug 25, 2013 at 7:36 PM, Armin Rigo <ar...@tunes.org> wrote:
> Argh, unhappiness :-( It seems that on this front, we cannot really
> do better that what you did, which is to explain to the user that he
> needs to run "pip install cffi" explicitly first...

But see also: http://stackoverflow.com/questions/10335371/pip-does-not-install-dependency-declared-in-setup-requires-parameter
. Does it work for you?


A bientôt,

Armin.

[Geert Jansen]

On Sunday, August 25, 2013 8:36:47 PM UTC+2, Armin Rigo wrote:

> Setuptools has the "setup_requires" keyword that allows you to specify early
> dependencies. But pip doesn't honor it. So I'm not sure it can be fixed. Pip
> seems to install dependencies *after* the package itself. But setup.py would
> need to have access to cffi to install the extension.

Argh, unhappiness :-(  It seems that on this front, we cannot really
do better that what you did, which is to explain to the user that he
needs to run "pip install cffi" explicitly first...

For reference, I found a way to make things a little better although it's quite hacky and there are still cased where this doesn't work.

When pip installs from requirements.txt, it will first run "python setup.py egg_info" on all dependencies to find indirect dependencies. Then it will run "python setup.py install" to install. Both are run in the order specified in requirements.txt.

Therefore, if you:

 * List *both* pycparser and cffi as the first lines in requirements.txt.
 * Write your setup script so that "egg_info" does not need cffi (install obviously needs cffi, for ext_modules).

This will ensure that when pip runs "python setup.py install" on your package, cffi will already be there.

The once case where this breaks down is when you "pip install" a distribution from PyPI whose setup script directly needs cffi. Pip will unfortunately put the distribution itself as the first requirement and run its setup script. Installing a distribution from PyPI that depends on a distribution whose setup script requires cffi should work, if you list all transitive dependencies (so pycparser, cffi, and the distribution itself) in the setup.py file.

Regards,

Geert