CFFI release 1.15.0

[Matt Davis]

Hi all,

CFFI release 1.15.0 is available on PyPI (pip install cffi==1.15.0) and on the release-1.15 branch at https://foss.heptapod.net/pypy/cffi.

Highlights of the 1.15 release include:

* Python 3.10 support + wheels for common platforms

* Linux wheels standardized on embedded libffi 3.4.2

* MacOS arm64 (Apple Silicon/M1) wheels, finally!

* Initial support for Windows on ARM (no CI or wheels yet)

* Various docs and test updates

Please try out the new wheels in particular, and file any issues at https://foss.heptapod.net/pypy/cffi/-/issues.

Cheers!

-Matt Davis

[Alex Gaynor]

Hi Matt,

With 1.15.0 we're seeing hangs in pyca/cryptography's tests on macOS
with Python 3.6. 3.6 on Linux/Windows are fine, and 3.10 on macOS is
also ok. You can see the issue on
https://github.com/pyca/cryptography/pull/6418.

Unfortunately I don't have a macOS system to test locally and pin down
where it's hanging -- but we have isolated that varying the cffi
version between 1.15.0 and 1.14.6 is all it takes to reproduce this.
Were there any changes in 1.15.0 that might impact locking or
destructors or anything like that?

Alex

> --
> -- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "python-cffi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/8a1d0420-2e9f-4edb-98e8-dcd61dd9aaf4n%40googlegroups.com.



--
All that is necessary for evil to succeed is for good people to do nothing.

[Armin Rigo]

Hi Alex,

On Sat, 16 Oct 2021 at 21:32, Alex Gaynor <alex....@gmail.com> wrote:
> Were there any changes in 1.15.0 that might impact locking or
> destructors or anything like that?

No. Basically this release only adds support for new architectures
and updated wheel-building procedures. The core C code did not change
at all apart from the bump in the version number.


A bientôt,
Armin

[Alex Gaynor]

Did the macOS wheels change in any significant way, since they were
already a supported platform? New libffi version or something?

Alex

> --
> -- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "python-cffi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/CAMSv6X2-97-ySgxcD4RRcUu_Rbk0wvjFLV3S5v1xi_c2E2ci5Q%40mail.gmail.com.

[Matt Davis]

Would've been nice to know during the prerelease phase... ;)

I can't think of any notable changes to the Intel Mac wheel build that would've caused a problem; they're still being built on a 10.15 worker against the default SDK. The Mac wheels have always just dynamically linked to the system libffi anyway- I'm hoping to change everything in a future release to statically link the extensions against a self-built libffi (since IIUC static linking is forbidden against most Apple frameworks); that should hopefully eliminate any weird conflicts. The only other possibility I can think of is maybe your test environment has a homebrew-installed libffi that's getting used instead of the system's libffi?

Since I'm not well-versed in cffi's actual guts, I'm probably not going to be much help in debugging the underlying problem, though if you can point me at a repro, I can make a CI job that anyone could run and ssh into to debug...

-Matt

You received this message because you are subscribed to a topic in the Google Groups "python-cffi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-cffi/MENXcRm6Xn4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-cffi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/CAFRnB2U%2Bj41UvFkTHLXjsvCKGahPc-wOsoDuVm%3DcwAMD%3DjzL_g%40mail.gmail.com.

[Matt Davis]

Oh, going back and looking at the 1.14 build stuff, there is one difference: AZP still has a MacOS 10.14 build worker (that IIRC they've deprecated), which is what was used for the 1.14 wheels. I moved everything to GHA for 1.15, which only has 10.15 and 11.0 MacOS workers available. At a glance, I don't think there was a change to system framework's libffi between 10.14 and 10.15, but that could also account for a change in behavior if there was (all the more reason to statically link, as that chasm grows and the M1 stuff in particular has to be built against a much newer MacOS SDK).

[Alex Gaynor]

Yes, sorry we didn't catch this sooner. I'm not entirely sure how we'd
have caught this short of running all builds in our CI against the
pre-release though, since it only impacts Python 3.6 on macOS :-/

Getting a minimal reproducer is going to be a challenge since it's
specific to macOS, but I'll see what can be done.

Alex

> To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/CAJD0SsiGV5NBZ-vaJT%2BwswJGo-uRpOxPgu9WiDuh1AWnQfwteA%40mail.gmail.com.

[Matt Davis]

I'm trying to reproduce it locally, but there are so many layers that I've thus far been unsuccessful. We also don't know how many versions it affects since the CI is only testing 3.6- if it's just 3.6, less bad than if it's 3.6-3.9... If I'm unsuccessful in reproing locally, I'll just fork the cryptography repo and debug the remote worker before it starts the tests. 

To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/CAFRnB2WfTWxd%3DhBPFp3GQ17Za6FACJVaYbAMESqDQPbvNDihpw%40mail.gmail.com.

[Matt Davis]

I'm able to reproduce a hang locally- it appears to be in tests/hazmat/backends/test_openssl_memleak.py in `TestAssertNoMemoryLeaks.test_errors()` - it only hangs on Python 3.6, but it crashes Python on exit for all the other versions (for both old and new cffi). Not sure if that's the desired behavior or not, but that's what's causing the hang anyway.

[Alex Gaynor]

Huh. That's definitely not supposed to be crashing! How are you
telling that it's crashing?

Alex

> To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/CAJD0SsgbvCrDRBrPGCFgPm6E%2BXCLeFS2nLGOV%3DOWmfcjSE5%3Dww%40mail.gmail.com.

[Matt Davis]

When running it interactively- the test passes, but the Python subprocess the test creates crashes on exit for all versions, even on really old cffi builds.

https://github.com/nitzmahone/cryptography/runs/3934299702?check_suite_focus=true <-- here's a super stripped down run that uses a timeout plugin to keep going if that test hangs...

I haven't tried a full run under my fork yet, but skipping that test locally seems to let things move along nicely.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/CAFRnB2Wrs8RhNGbQRG6tC64SGpd%2BNaTdj%2BkXJkUKAXV-4Y9jGQ%40mail.gmail.com.

[Matt Davis]

... you get the Mac bugreport window when it crashes interactively- skipping just `test_errors` lets the rest of the memory leak test set run fine on 3.6.

[Matt Davis]

Just to sew up this thread- looks like the cffi 1.15.0 Mac wheels are OK, it was a misbehaving cryptography test that's actually been quietly crashing the hosting Python subprocess after passing the test. Something changed just enough that instead of crashing, it hung on one particular Python- the test has been adjusted and it looks like all is well.

Thanks everyone!

-Matt