Lazy ffi.include for out-of-line dlopen'd modules

[Sean Vig]

Hi,

I'm wondering if it would be possible to introduce a feature to generate a cffi module using the out-of-line format that tries to ffi.include an external ffi in basically a try/except block where the included ffi itself may not be available at build time. I know this wouldn't be possible for a compiled cffi module, but when dlopen'ing it should be possible in theory. This is a problem for us with qtile, as we use cairo with XCB support through cairocffi, which requires xcffib to be installed before cairocffi. Unfortunately, standard installation methods (pip, setup.py install) with qtile don't install xcffib in a way that can be found when installing cairocffi, so the cairocffi ffi module can't ffi.include the xcffib ffi module. In cases like this, we have to run extra checks on install to flag users to uninstall cairocffi, clear pip caches, and re-install in the correct order by hand, and despite adding these instructions, this has led to many people struggling to get it working. I'm not sure the complexity of what I'm proposing, and it really only applies to small corner cases like this, but any help or guidance here would be appreciated, thanks.

Sean

[Armin Rigo]

Hi Sean,

Can you describe in more details the original problem? It seems that
you're saying "we had problem X, we fixed it with workaround Y, which
added problem Z, now how do we fix problem Z?". Usually this kind of
problem is best addressed by either fixing problem X or giving a
motivated, detailed explanation of why solving X is not possible.

Maybe it's as easy as moving some dependencies from
"install_requires=..." to "setup_requires=...". I guess it is not,
but I'd like to be sure.


A bientôt,

Armin.

[Sean Vig]

Hi Armin,

Thanks for the response and for not getting back faster. We "addressed" the original problem by catching it at install time and telling people how they might fix it, by basically re-installing everything in the right order by hand, but there are still lots of people that struggle to get it right. It'd definitely be nice if we can find a way to fix the original problem.

The problem with moving to setup_requires is the ultimate culprit is an optional dependency. Basically we ffi.include cairocffi and cairocffi can optionally ffi.include xcffib, but we require this optional dependency. If we could move xcffib to a required dependency of cairocffi in setup_requires, that might fix it, but that doesn't really make sense in this case, as cairo may not be built with xcb support, e.g. on Windows.

Sean

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "python-cffi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-cffi/wS7cvcj32o8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-cffi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Armin Rigo]

Hi Sean,

On 6 July 2015 at 22:25, Sean Vig <sean....@gmail.com> wrote:
> The problem with moving to setup_requires is the ultimate culprit is an
> optional dependency. Basically we ffi.include cairocffi and cairocffi can
> optionally ffi.include xcffib, but we require this optional dependency.

I'm unsure how cairocffi's "optional dependency" is done. Sorry if
I'm still not understanding. Can you give an artificial,
self-contained, complete example? That would help tremendously.
Failing that, maybe just step-by-step instructions for me to reproduce
the problem (but be sure to say exactly what must be or not be
installed at every step). I guess with enough efforts I could
reverse-engineer it from the error message that tells people about the
problem, but any help is appreciated :-)


A bientôt,

Armin.

[Sean Vig]

Cairocffi uses "extras_require" [1], I'll see if I can't get a minimal example working. Doing a `pip install cairocffi[xcb]` installs it with xcb support, but requiring `cairocffi[xcb]` in qtile does not get xcffib installed such that cairocffi can be built right.

Sean

[1] https://github.com/SimonSapin/cairocffi/blob/master/setup.py#L64

A bientôt,

Armin.

[Armin Rigo]

Hi Sean,

On 7 July 2015 at 00:33, Sean Vig <sean....@gmail.com> wrote:
> Cairocffi uses "extras_require" [1], I'll see if I can't get a minimal
> example working. Doing a `pip install cairocffi[xcb]` installs it with xcb
> support, but requiring `cairocffi[xcb]` in qtile does not get xcffib
> installed such that cairocffi can be built right.

This look entirely like a pip/setuptools problem to me. Maybe
`extras_require` is installed too late? I'd try to understand what
goes on in which order.


A bientôt,

Armin.