[Python-Dev] libffi inclusion in python

[Maciej Fijalkowski]

Hi

libffi has bugs sometimes (like this
http://bugs.python.org/issue17580). Now this is a thing that upstream
fixes really quickly, but tracking down issues on bugs.python.org is
annoying (they never get commited as quickly as the upstream). is
there a good reason why cpython has it's own copy of libffi? I
understand historical reasons, but PyPy gets along relying on the
system library, so maybe we can kill the inclusion.

Cheers,
fijal
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Stefan Behnel]

Maciej Fijalkowski, 18.04.2013 13:41:

> PyPy gets along relying on the system library

Depends on what systems you want to support, I guess.

Stefan

[Benjamin Peterson]

2013/4/18 Maciej Fijalkowski <fij...@gmail.com>:

> Hi
>
> libffi has bugs sometimes (like this
> http://bugs.python.org/issue17580). Now this is a thing that upstream
> fixes really quickly, but tracking down issues on bugs.python.org is
> annoying (they never get commited as quickly as the upstream). is
> there a good reason why cpython has it's own copy of libffi? I
> understand historical reasons, but PyPy gets along relying on the
> system library, so maybe we can kill the inclusion.

IIRC, it had (has?) some custom windows patches, which no one knows
whether they're relevant or not.

I personally would love to see all libraries that we have copies of in
the repo killed.


--
Regards,
Benjamin

[Ned Deily]

In article
<CAPZV6o_HRmkU_=1MZPJDzJGZOBTwBZne...@mail.gmail.com>,
Benjamin Peterson <benj...@python.org> wrote:
> 2013/4/18 Maciej Fijalkowski <fij...@gmail.com>:

> > libffi has bugs sometimes (like this
> > http://bugs.python.org/issue17580). Now this is a thing that upstream
> > fixes really quickly, but tracking down issues on bugs.python.org is
> > annoying (they never get commited as quickly as the upstream). is
> > there a good reason why cpython has it's own copy of libffi? I
> > understand historical reasons, but PyPy gets along relying on the
> > system library, so maybe we can kill the inclusion.
>
> IIRC, it had (has?) some custom windows patches, which no one knows
> whether they're relevant or not.

The cpython copy also has custom OS X patches. I've never looked at
them so I don't have a feel for how much work would be involved in
migrating to current upstream. If it's just a matter of supporting
universal builds, it could be done with some Makefile hacking to do a
lipo dance. Ronald, any additional thoughts?

http://bugs.python.org/issue15194

Currently, for the OS X installer builds, we build a number of
third-party libs that are either missing from OS X (like lzma) or are
too out-of-date on the oldest systems we support. It would be useful to
generalize the third-party lib support and move it out of the installer
build process so that all builds could take advantage of the libs if
needed. libffi could be added to those. Of course, that wouldn't help
for Windows builds.

--
Ned Deily,
n...@acm.org

[Maciej Fijalkowski]

On Thu, Apr 18, 2013 at 2:47 PM, Benjamin Peterson <benj...@python.org> wrote:
> 2013/4/18 Maciej Fijalkowski <fij...@gmail.com>:
>> Hi
>>
>> libffi has bugs sometimes (like this
>> http://bugs.python.org/issue17580). Now this is a thing that upstream
>> fixes really quickly, but tracking down issues on bugs.python.org is
>> annoying (they never get commited as quickly as the upstream). is
>> there a good reason why cpython has it's own copy of libffi? I
>> understand historical reasons, but PyPy gets along relying on the
>> system library, so maybe we can kill the inclusion.
>
> IIRC, it had (has?) some custom windows patches, which no one knows
> whether they're relevant or not.
>
> I personally would love to see all libraries that we have copies of in
> the repo killed.

Upstream seems to be incredibly responsive. Why not merge them there?
Is it just "we can't patch distutils" crap? If we *really* need to
provide libraries for systems that don't have them, how about simply
having a separate download for those systems instead of using it on
systems that *do* have them and in fact *do* have them in a better
version.

Note that the recent patches (one for 32bit, one for ARM) are easily
security holes. I'm not 100% sure if segfault caused by stack
misalignment is an exploitable thing, but you can definitely cause a
segfault on any ctypes code, even if perfectly safe.

Cheers,
fijal

[Thomas Heller]

>>> libffi has bugs sometimes (like this
>>> http://bugs.python.org/issue17580). Now this is a thing that upstream
>>> fixes really quickly, but tracking down issues on bugs.python.org is
>>> annoying (they never get commited as quickly as the upstream). is
>>> there a good reason why cpython has it's own copy of libffi? I
>>> understand historical reasons, but PyPy gets along relying on the
>>> system library, so maybe we can kill the inclusion.
>>
>> IIRC, it had (has?) some custom windows patches, which no one knows
>> whether they're relevant or not.

The only windows patch that is most certainly not in upstream is the
ability to check and correct the stack pointer after a windows function
call depending on the calling convention (stdcall or cdecl).

Since this is windows 32-bit only (on windows 64-bit these calling
conventions are aliases for the same thing), it would probably be good
to remove the distinction between them for function calls.

> Upstream seems to be incredibly responsive. Why not merge them there?

At the time when ctypes was implemented, this was different. They
didn't even do libffi releases, IIRC.

Thomas

[Maciej Fijalkowski]

On Thu, Apr 18, 2013 at 9:22 PM, Thomas Heller <the...@ctypes.org> wrote:
>>>> libffi has bugs sometimes (like this
>>>> http://bugs.python.org/issue17580). Now this is a thing that upstream
>>>> fixes really quickly, but tracking down issues on bugs.python.org is
>>>> annoying (they never get commited as quickly as the upstream). is
>>>> there a good reason why cpython has it's own copy of libffi? I
>>>> understand historical reasons, but PyPy gets along relying on the
>>>> system library, so maybe we can kill the inclusion.
>>>
>>>
>>> IIRC, it had (has?) some custom windows patches, which no one knows
>>> whether they're relevant or not.
>
>
> The only windows patch that is most certainly not in upstream is the
> ability to check and correct the stack pointer after a windows function
> call depending on the calling convention (stdcall or cdecl).
>
> Since this is windows 32-bit only (on windows 64-bit these calling
> conventions are aliases for the same thing), it would probably be good
> to remove the distinction between them for function calls.
>
>
>
>> Upstream seems to be incredibly responsive. Why not merge them there?
>
>
> At the time when ctypes was implemented, this was different. They
> didn't even do libffi releases, IIRC.
>
> Thomas

Cool.

Note that I don't ask "why the hell was it included", I assume there
was the right choice at a time, I ask "can we remove it now?" which
seems to be mostly yes.

Cheers,
fijal

[Ronald Oussoren]

On 18 apr. 2013, at 18:09, Ned Deily <n...@acm.org> wrote:

> In article
> <CAPZV6o_HRmkU_=1MZPJDzJGZOBTwBZne...@mail.gmail.com>,
> Benjamin Peterson <benj...@python.org> wrote:
>> 2013/4/18 Maciej Fijalkowski <fij...@gmail.com>:
>>> libffi has bugs sometimes (like this
>>> http://bugs.python.org/issue17580). Now this is a thing that upstream
>>> fixes really quickly, but tracking down issues on bugs.python.org is
>>> annoying (they never get commited as quickly as the upstream). is
>>> there a good reason why cpython has it's own copy of libffi? I
>>> understand historical reasons, but PyPy gets along relying on the
>>> system library, so maybe we can kill the inclusion.
>>
>> IIRC, it had (has?) some custom windows patches, which no one knows
>> whether they're relevant or not.
>
> The cpython copy also has custom OS X patches. I've never looked at
> them so I don't have a feel for how much work would be involved in
> migrating to current upstream. If it's just a matter of supporting
> universal builds, it could be done with some Makefile hacking to do a
> lipo dance. Ronald, any additional thoughts?

It is probably just a matter of supporting universal builds, but I haven't checked the state of upstream libffi in the last couple of years.

The libffi_osx tree is a fork from upstream that started around the time of OSX 10.4, and possibly earlier. As Thomas mentioned the upstream maintainer weren't very responsive in the past, at the time I hacked on libffi (IIRC for Darwin/i386 support) it wasn't even clear how the maintainers could be reached.

Stripping libffi from python's source tree would be fine by me, but would require testing with upstream libffi. AFAIK system libffi on osx wouldn't be goog enough, it doesn't work properly with clang.

Ronald

>
> http://bugs.python.org/issue15194
>
> Currently, for the OS X installer builds, we build a number of
> third-party libs that are either missing from OS X (like lzma) or are
> too out-of-date on the oldest systems we support. It would be useful to
> generalize the third-party lib support and move it out of the installer
> build process so that all builds could take advantage of the libs if
> needed. libffi could be added to those. Of course, that wouldn't help
> for Windows builds.
>
> --
> Ned Deily,
> n...@acm.org
>
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

[Dirkjan Ochtman]

On Thu, Apr 18, 2013 at 11:17 PM, Ronald Oussoren
<ronaldo...@mac.com> wrote:
> Stripping libffi from python's source tree would be fine by me, but would require testing with upstream libffi. AFAIK system libffi on osx wouldn't be goog enough, it doesn't work properly with clang.

If you mean http://bugs.python.org/issue17136, I think that has been
fixed in libffi upstream?

Cheers,

Dirkjan

[Ronald Oussoren]

On 19 Apr, 2013, at 10:13, Dirkjan Ochtman <dir...@ochtman.nl> wrote:

> On Thu, Apr 18, 2013 at 11:17 PM, Ronald Oussoren
> <ronaldo...@mac.com> wrote:
>> Stripping libffi from python's source tree would be fine by me, but would require testing with upstream libffi. AFAIK system libffi on osx wouldn't be goog enough, it doesn't work properly with clang.
>
> If you mean http://bugs.python.org/issue17136, I think that has been
> fixed in libffi upstream?

Appearently, but not in /usr/lib/libffi.dylib (at least nog when I checked it last time, which was at least one minor release of 10.8 ago). Which means that the binary installers will still have to use a bundled copy of libffi, but as Ned wrote that's not really a problem because the script that builds those installers already includes some other libraries that are not available on OSX or are too old.

That said, I haven't tested yet if upstream libffi actually works although others seem to have.

Ronald