[Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Christian Heimes]

Hi,

I'm still working on a ssl module PEP for 3.7 [1], but it's probably not
going to be finished before beta 1 deadline. I have a bunch of fixes and
improvements for the ssl module in queue, most of them require OpenSSL
1.0.2 features. The features are also present and working properly since
LibreSSL 2.5.3


If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
bunch of useful goodies like proper hostname verification [2], proper
fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
and PrivateKey types (support loading certs and keys from file and
memory) [4], and simplified cipher suite configuration [5]. I can
finally clean up _ssl.c during the beta phase, too.


OpenSSL 1.0.1 is out of support since December 2016, 0.9.8 since 2015.
These versions haven't received any security updates for more than a year!

All major Linux and BSD distributions have at least 1.0.2 [6]. The only
relevant exception is Ubuntu 14.04 LTS, because Travis CI is running
14.04. PR 3562 [7] contains a PoC to compile a custom build of OpenSSL
on Travis. Builds are cached.

Regards,
Christian

[1] https://github.com/tiran/peps/blob/sslmodule37/pep-9999.txt
[2] https://bugs.python.org/issue31399
[3] https://bugs.python.org/issue32185
[4] https://bugs.python.org/issue18369
[5] https://bugs.python.org/issue31429
[6] https://gist.github.com/tiran/c5409bbd60a5f082f654d967add8cc79
[7] https://github.com/python/cpython/pull/3462

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Antoine Pitrou]

On Sat, 13 Jan 2018 13:54:33 +0100
Christian Heimes <chri...@python.org> wrote:
>
> If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
> bunch of useful goodies like proper hostname verification [2], proper
> fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
> and PrivateKey types (support loading certs and keys from file and
> memory) [4], and simplified cipher suite configuration [5]. I can
> finally clean up _ssl.c during the beta phase, too.

Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
this.

We'll have to deal with the complaints of users of Debian oldstable,
CentOS 6 and RHEL 6, though.

Regards

Antoine.

[Christian Heimes]

On 2018-01-13 14:23, Antoine Pitrou wrote:
> On Sat, 13 Jan 2018 13:54:33 +0100
> Christian Heimes <chri...@python.org> wrote:
>>
>> If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
>> bunch of useful goodies like proper hostname verification [2], proper
>> fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
>> and PrivateKey types (support loading certs and keys from file and
>> memory) [4], and simplified cipher suite configuration [5]. I can
>> finally clean up _ssl.c during the beta phase, too.
>
> Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
> this.
>
> We'll have to deal with the complaints of users of Debian oldstable,
> CentOS 6 and RHEL 6, though.

It's more of an issue for Travis CI. The Python 3.7-dev target won't
have a functional ssl module. Travis either has to update their build
base to 16.04, provide a custom build of OpenSSL, or all packages have
to use a container. [1]

Christian

[1]
https://github.com/travis-ci/travis-ci/issues/5821#issuecomment-214452987

[Antoine Pitrou]

On Sat, 13 Jan 2018 15:49:21 +0100

Christian Heimes <chri...@python.org> wrote:
> On 2018-01-13 14:23, Antoine Pitrou wrote:
> > On Sat, 13 Jan 2018 13:54:33 +0100
> > Christian Heimes <chri...@python.org> wrote:
> >>
> >> If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
> >> bunch of useful goodies like proper hostname verification [2], proper
> >> fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
> >> and PrivateKey types (support loading certs and keys from file and
> >> memory) [4], and simplified cipher suite configuration [5]. I can
> >> finally clean up _ssl.c during the beta phase, too.
> >
> > Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
> > this.
> >
> > We'll have to deal with the complaints of users of Debian oldstable,
> > CentOS 6 and RHEL 6, though.
>
> It's more of an issue for Travis CI. The Python 3.7-dev target won't
> have a functional ssl module. Travis either has to update their build
> base to 16.04, provide a custom build of OpenSSL, or all packages have
> to use a container. [1]

That's Travis-CI's problem. And hopefully they'll migrate to Ubuntu
16.04 soon (it's almost 2 years old...).

Regards

Antoine.

[Brett Cannon]

On Sat, Jan 13, 2018, 05:24 Antoine Pitrou, <soli...@pitrou.net> wrote:

On Sat, 13 Jan 2018 13:54:33 +0100
Christian Heimes <chri...@python.org> wrote:
>
> If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
> bunch of useful goodies like proper hostname verification [2], proper
> fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
> and PrivateKey types (support loading certs and keys from file and
> memory) [4], and simplified cipher suite configuration [5]. I can
> finally clean up _ssl.c during the beta phase, too.

Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
this.

+1 from me as well for the improved security.

-Brett

We'll have to deal with the complaints of users of Debian oldstable,
CentOS 6 and RHEL 6, though.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

[Christian Heimes]

On 2018-01-13 21:02, Brett Cannon wrote:
> +1 from me as well for the improved security.

Thanks, Brett!

How should we handle CPython's Travis CI tests? The 14.04 boxes have
OpenSSL 1.0.1. To the best of my knowledge, Travis doesn't offer 16.04.
We could either move to container-based testing with a 16.04 container,
which would give us 1.0.2 Or we could compile our own copy of OpenSSL
with my multissl builder and use some rpath magic.

In order to test all new features, Ubuntu doesn't cut it. Even current
snapshot of Ubuntu doesn't contain OpenSSL 1.1. Debian Stretch or Fedora
would do the trick, though.

Maybe Barry's work on official test container could leveraged testing?

Regards,
Christian

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Steven D'Aprano]

On Sat, Jan 13, 2018 at 02:23:19PM +0100, Antoine Pitrou wrote:
> On Sat, 13 Jan 2018 13:54:33 +0100
> Christian Heimes <chri...@python.org> wrote:
> >
> > If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
> > bunch of useful goodies like proper hostname verification [2], proper
> > fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
> > and PrivateKey types (support loading certs and keys from file and
> > memory) [4], and simplified cipher suite configuration [5]. I can
> > finally clean up _ssl.c during the beta phase, too.
>
> Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
> this.
>
> We'll have to deal with the complaints of users of Debian oldstable,
> CentOS 6 and RHEL 6, though.

It will probably be more work for Christian, but is it reasonable to
keep support for the older versions of OpenSSL, but make the useful
goodies conditional on a newer version?



--
Steve

[Gregory P. Smith]

On Sat, Jan 13, 2018 at 4:34 PM Steven D'Aprano <st...@pearwood.info> wrote:

On Sat, Jan 13, 2018 at 02:23:19PM +0100, Antoine Pitrou wrote:
> On Sat, 13 Jan 2018 13:54:33 +0100
> Christian Heimes <chri...@python.org> wrote:
> >
> > If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
> > bunch of useful goodies like proper hostname verification [2], proper
> > fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
> > and PrivateKey types (support loading certs and keys from file and
> > memory) [4], and simplified cipher suite configuration [5]. I can
> > finally clean up _ssl.c during the beta phase, too.
>
> Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
> this.
>
> We'll have to deal with the complaints of users of Debian oldstable,
> CentOS 6 and RHEL 6, though.

It will probably be more work for Christian, but is it reasonable to
keep support for the older versions of OpenSSL, but make the useful
goodies conditional on a newer version?

I don't think it is worth spending our limited engineering time supporting an unsupported library version.  Leave that burden to stale distro maintainers who continue to choose dangerously stale software versions if they ironically want to use something as modern as 3.7 on top of an ancient set of libraries.

+1 from me for requiring OpenSSL >= 1.0.2 in Python 3.7.

-gps

[Brett Cannon]

On Sat, Jan 13, 2018, 14:45 Christian Heimes, <chri...@python.org> wrote:

On 2018-01-13 21:02, Brett Cannon wrote:
> +1 from me as well for the improved security.

Thanks, Brett!

How should we handle CPython's Travis CI tests? The 14.04 boxes have
OpenSSL 1.0.1. To the best of my knowledge, Travis doesn't offer 16.04.
We could either move to container-based testing with a 16.04 container,
which would give us 1.0.2 Or we could compile our own copy of OpenSSL
with my multissl builder and use some rpath magic.

In order to test all new features, Ubuntu doesn't cut it. Even current
snapshot of Ubuntu doesn't contain OpenSSL 1.1. Debian Stretch or Fedora
would do the trick, though.

Maybe Barry's work on official test container could leveraged testing?

My guess is we either move to containers on Travis, see if we can manually install -- through apt or something -- a newer version of OpenSSL, or we look at alternative CI options.

-Brett

Regards,
Christian

[Paul G]

One thing to note is that if getting Travis working with Python 3.7 is a pain, a huge number of libraries on PyPI probably just won't test against Python 3.7, which is not a great situation to be in.

It's probably worth contacting Travis to give them a head's up and see how likely it is that they'll be able to support Python 3.7 if it requires a newer version of these libraries.

[Oleg Broytman]

On Sun, Jan 14, 2018 at 02:16:53AM +0000, Brett Cannon <br...@python.org> wrote:
> My guess is we either move to containers on Travis, see if we can manually
> install -- through apt or something -- a newer version of OpenSSL

OpenSSL 1.0.2 cannot be installed with apt on Trusty but I think it
can be compiled from sources.

> -Brett

Oleg.
--
Oleg Broytman http://phdru.name/ p...@phdru.name
Programmers don't die, they just GOSUB without RETURN.

[Terry Reedy]

On 1/13/2018 3:02 PM, Brett Cannon wrote:
>
>
> On Sat, Jan 13, 2018, 05:24 Antoine Pitrou, <soli...@pitrou.net
> <mailto:soli...@pitrou.net>> wrote:
>
> On Sat, 13 Jan 2018 13:54:33 +0100
> Christian Heimes <chri...@python.org

> <mailto:chri...@python.org>> wrote:
> >
> > If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I
> can land
> > bunch of useful goodies like proper hostname verification [2], proper
> > fix for IP address in SNI TLS header [3], PEP 543 compatible
> Certificate
> > and PrivateKey types (support loading certs and keys from file and
> > memory) [4], and simplified cipher suite configuration [5]. I can
> > finally clean up _ssl.c during the beta phase, too.
>
> Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
> this.
>
>
> +1 from me as well for the improved security.

FWIW, given that I will not be doing any of the work, +1 from me also.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Christian Heimes]

On 2018-01-14 01:03, Steven D'Aprano wrote:
> On Sat, Jan 13, 2018 at 02:23:19PM +0100, Antoine Pitrou wrote:
>> On Sat, 13 Jan 2018 13:54:33 +0100
>> Christian Heimes <chri...@python.org> wrote:
>>>
>>> If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land
>>> bunch of useful goodies like proper hostname verification [2], proper
>>> fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate
>>> and PrivateKey types (support loading certs and keys from file and
>>> memory) [4], and simplified cipher suite configuration [5]. I can
>>> finally clean up _ssl.c during the beta phase, too.
>>
>> Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
>> this.
>>
>> We'll have to deal with the complaints of users of Debian oldstable,
>> CentOS 6 and RHEL 6, though.
>
> It will probably be more work for Christian, but is it reasonable to
> keep support for the older versions of OpenSSL, but make the useful
> goodies conditional on a newer version?

It's much more than just goodies. For example the
X509_VERIFY_PARAM_set1_host() API fixes a whole lot of issues with
ssl.match_hostname(). The feature is OpenSSL 1.0.2+ and baked into the
certificate validation system. I don't see a realistic way to perform
the same task with 1.0.1.

Christian

[Christian Heimes]

On 2018-01-14 03:48, Paul G wrote:
> One thing to note is that if getting Travis working with Python 3.7 is a
> pain, a huge number of libraries on PyPI probably just won't test
> against Python 3.7, which is not a great situation to be in.
>
> It's probably worth contacting Travis to give them a head's up and see
> how likely it is that they'll be able to support Python 3.7 if it
> requires a newer version of these libraries.

Unless my proposal isn't rejected, I'll contact Travis CI tomorrow.

[Antoine Pitrou]

On Sat, 13 Jan 2018 23:45:07 +0100
Christian Heimes <chri...@python.org> wrote:
> On 2018-01-13 21:02, Brett Cannon wrote:
> > +1 from me as well for the improved security.
>
> Thanks, Brett!
>
> How should we handle CPython's Travis CI tests? The 14.04 boxes have
> OpenSSL 1.0.1. To the best of my knowledge, Travis doesn't offer 16.04.
> We could either move to container-based testing with a 16.04 container,
> which would give us 1.0.2 Or we could compile our own copy of OpenSSL
> with my multissl builder and use some rpath magic.

I don't think you need some rpath magic, just set LD_LIBRARY_PATH to
the right value.

Regards

Antoine.

[Christian Heimes]

On 2018-01-14 11:17, Antoine Pitrou wrote:
> On Sat, 13 Jan 2018 23:45:07 +0100
> Christian Heimes <chri...@python.org> wrote:
>> On 2018-01-13 21:02, Brett Cannon wrote:
>>> +1 from me as well for the improved security.
>>
>> Thanks, Brett!
>>
>> How should we handle CPython's Travis CI tests? The 14.04 boxes have
>> OpenSSL 1.0.1. To the best of my knowledge, Travis doesn't offer 16.04.
>> We could either move to container-based testing with a 16.04 container,
>> which would give us 1.0.2 Or we could compile our own copy of OpenSSL
>> with my multissl builder and use some rpath magic.
>
> I don't think you need some rpath magic, just set LD_LIBRARY_PATH to
> the right value.

I prefer LD_RUN_PATH because it adds rpath to the ELF header of shared
libraries and binaries.

https://github.com/python/cpython/pull/5180

Christian

[Matt Billenstein]

Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has
deprecated (and no longer ships the header files for) the version shipped with
recent versions of osx.

Perhaps this is an option to support the various flavors of Linux as well?

m

> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> https://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: https://mail.python.org/mailman/options/python-dev/matt%40vazor.com


--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/

[Christian Heimes]

On 2018-01-14 09:24, Matt Billenstein wrote:
> Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has
> deprecated (and no longer ships the header files for) the version shipped with
> recent versions of osx.
>
> Perhaps this is an option to support the various flavors of Linux as well?

AFAK Apple has decided to compile and statically link CPython's ssl with
an ancient, customized LibreSSL version. Cory posted [1] a couple of
months ago

Can confirm: macOS 10.13 will ship a Python linked against LibreSSL
2.2.7. A downside: this continues to use the TEA, meaning you cannot
choose to distrust the system roots with it.

For TEA, see Hynek's blog post [2]


I'm not going to add OpenSSL sources or builds to CPython. We just got
rid of copies of libffi and other 3rd party dependencies. Crypto and TLS
libraries are much, MUCH more complicated to handle than libffi. It's a
constant moving targets of attacks. Vendors and distributions also have
different opinions about trust store and policies.

Let's keep build dependencies a downstream and vendor problem.

Christian

[1] https://twitter.com/lukasaoz/status/872085966579802112
[2] https://hynek.me/articles/apple-openssl-verification-surprises/

[Ned Deily]

On Jan 14, 2018, at 08:39, Christian Heimes <chri...@python.org> wrote:
> On 2018-01-14 09:24, Matt Billenstein wrote:
>> Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has
>> deprecated (and no longer ships the header files for) the version shipped with
>> recent versions of osx.
>>
>> Perhaps this is an option to support the various flavors of Linux as well?
>
> AFAK Apple has decided to compile and statically link CPython's ssl with
> an ancient, customized LibreSSL version. Cory posted [1] a couple of
> months ago

I think you're conflating some things here. Apple has not yet shipped a
version of Python 3 with macOS so the fact that Apple now links their
version of Python2.7 with a "private" copy of LibreSSL is irrelevant.
(It's private in the sense that they don't ship the header files for it;
the shared libs are there just for the use of the open source products
they ship with macOS that don't yet use the macOS native crypto APIs,
products like Python and Perl.)

What Matt is likely thinking of is the Python 3 versions provided by the
python.org macOS binary installers where we do build and link with our
own 1.0.2 (and soon 1.1.0 for 3.7) versions of OpenSSL. Currently,
the OpenSSL (and several other third-party libs such as libxz which
is not shipped by Apple) are built as part of the installer build
script in the Mac section of the source repo. I would like to
refactor and generalize that so those third-party libs
could optionally be used for non-installer builds as well. But, in
any case, we don't have much choice for the installer builds until
such time as cPython has support for the Apple-provided crypto APIs.

> I'm not going to add OpenSSL sources or builds to CPython. We just got
> rid of copies of libffi and other 3rd party dependencies. Crypto and TLS
> libraries are much, MUCH more complicated to handle than libffi. It's a
> constant moving targets of attacks. Vendors and distributions also have
> different opinions about trust store and policies.
>
> Let's keep build dependencies a downstream and vendor problem.

That's not always an option, unfortunately.

--
Ned Deily
n...@python.org -- []

[Wes Turner]

FWIW, anaconda and conda-forge currently have 1.0.2 X

https://anaconda.org/anaconda/openssl

https://anaconda.org/conda-forge/openssl

Support for Apple SecureTransport is part of the TLS module. IDK how far along that work is (whether it'll be ready for 3.7 beta 1)?

https://github.com/python/peps/blob/master/pep-0543.rst

https://www.python.org/dev/peps/pep-0543/

http://markmail.org/search/?q=list%3Aorg.python+PEP+543+TLS

 

> I'm not going to add OpenSSL sources or builds to CPython. We just got
> rid of copies of libffi and other 3rd party dependencies. Crypto and TLS
> libraries are much, MUCH more complicated to handle than libffi. It's a
> constant moving targets of attacks. Vendors and distributions also have
> different opinions about trust store and policies.
>
> Let's keep build dependencies a downstream and vendor problem.

That's not always an option, unfortunately.

--
  Ned Deily
  n...@python.org -- []

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com

[Christian Heimes]

Yeah, that sounds useful for macOS and Windows development. Back when I
was doing more Windows stuff, I used our buildbot scripts to provide
local builds of dependencies such as expat and OpenSSL.

>> I'm not going to add OpenSSL sources or builds to CPython. We just got
>> rid of copies of libffi and other 3rd party dependencies. Crypto and TLS
>> libraries are much, MUCH more complicated to handle than libffi. It's a
>> constant moving targets of attacks. Vendors and distributions also have
>> different opinions about trust store and policies.
>>
>> Let's keep build dependencies a downstream and vendor problem.
>
> That's not always an option, unfortunately.

For Python.org macOS and Windows installers, I'm considering us as our
own downstream vendors. I should rather say "Steve and you" instead of
us. You are both doing the heavy lifting. Thanks for you hard work. :)

Christian

[Matt Billenstein]

On Sun, Jan 14, 2018 at 10:54:57AM -0500, Ned Deily wrote:
> On Jan 14, 2018, at 08:39, Christian Heimes <chri...@python.org> wrote:
> > On 2018-01-14 09:24, Matt Billenstein wrote:
> >> Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has
> >> deprecated (and no longer ships the header files for) the version shipped with
> >> recent versions of osx.
> >>
> >> Perhaps this is an option to support the various flavors of Linux as well?
> >
> > AFAK Apple has decided to compile and statically link CPython's ssl with
> > an ancient, customized LibreSSL version. Cory posted [1] a couple of
> > months ago
>

> What Matt is likely thinking of is the Python 3 versions provided by the
> python.org macOS binary installers where we do build and link with our
> own 1.0.2 (and soon 1.1.0 for 3.7) versions of OpenSSL.

Yes, referring to the Python3 python.org installers -- I'm seeing this practice
of bundling libs (particularly ssl) become more common as operating system
support lags behind. In my mind it becomes easier to bundle deps in a binary
installer across the board (Linux, OSX, Windows) rather than rely on whatever
version the operating system provides.

m

--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/

[Stephen J. Turnbull]

Matt Billenstein writes:

> In my mind it becomes easier to bundle deps in a binary installer
> across the board (Linux, OSX, Windows) rather than rely on whatever
> version the operating system provides.

Thing is, as Christian points out, TLS is a rapidly moving target.
Every Mac OS or iOS update seems to link to a dozen CVEs for TLS
support. We can go there if we have to, but it's often hard to go
back when vendor support catches up to something reasonable. I think
this is something for Ned and Christian and Steve to negotiate, since
they're the ones who are most aware of the tradeoffs and bear the
costs.

[Steve Dower]

From my perspective, we can’t keep an OpenSSL-like API and use Windows platform libraries (we could do a requests-like API easily enough, but even urllib3 is painfully low-level).

 

We have to continue shipping our own copy of OpenSSL on Windows. Nothing to negotiate here except whether OpenSSL releases should trigger a Python release, and I think that decision can stay with the RM.

 

Good luck solving macOS :o)

 

Cheers,

Steve

 

Top-posted from my Windows phone

Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

 

[Christian Heimes]

On 2018-01-16 08:08, Steve Dower wrote:
> From my perspective, we can’t keep an OpenSSL-like API and use Windows

> platform libraries (we *could* do a requests-like API easily enough, but

> even urllib3 is painfully low-level).
>
>  
>
> We have to continue shipping our own copy of OpenSSL on Windows. Nothing
> to negotiate here except whether OpenSSL releases should trigger a
> Python release, and I think that decision can stay with the RM.

3.7 will no longer use static linking. We can offer out-of-bounds
updates of the OpenSSL DLLs. And by "we", I'm talking about you. :)

Christian

[Wes Turner]

On Tuesday, January 16, 2018, Steve Dower <steve...@python.org> wrote:

From my perspective, we can’t keep an OpenSSL-like API and use Windows platform libraries (we could do a requests-like API easily enough, but even urllib3 is painfully low-level).

Support for Windows SChannel and Apple SecureTransport is part of the TLS module.

IDK how far along that work is (whether it'll be ready for 3.7 beta 1)? Or where those volunteering to help with the TLS module can send PRs?

https://github.com/python/peps/blob/master/pep-0543.rst

https://www.python.org/dev/peps/pep-0543/

http://markmail.org/search/?q=list%3Aorg.python+PEP+543+TLS

https://www.python.org/dev/peps/pep-0543/#interfaces

 

We have to continue shipping our own copy of OpenSSL on Windows. Nothing to negotiate here except whether OpenSSL releases should trigger a Python release, and I think that decision can stay with the RM.

 

Good luck solving macOS :o)

 

Cheers,

Steve

 

Top-posted from my Windows phone

 

From: Stephen J. Turnbull
Sent: Tuesday, January 16, 2018 17:45
To: Matt Billenstein
Cc: Christian Heimes; pytho...@python.org
Subject: Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >=2.5.3

 

Matt Billenstein writes:

 

> In my mind it becomes easier to bundle deps in a binary installer

> across the board (Linux, OSX, Windows) rather than rely on whatever

> version the operating system provides.

 

Thing is, as Christian points out, TLS is a rapidly moving target.

Every Mac OS or iOS update seems to link to a dozen CVEs for TLS

support.  We can go there if we have to, but it's often hard to go

back when vendor support catches up to something reasonable.  I think

this is something for Ned and Christian and Steve to negotiate, since

they're the ones who are most aware of the tradeoffs and bear the

costs.

 

 

 

_______________________________________________

Python-Dev mailing list

Pytho...@python.org

https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

 

[Christian Heimes]

On 2018-01-16 12:28, Wes Turner wrote:
>
>
> On Tuesday, January 16, 2018, Steve Dower <steve...@python.org

> <mailto:steve...@python.org>> wrote:
>
> From my perspective, we can’t keep an OpenSSL-like API and use

> Windows platform libraries (we *could* do a requests-like API easily
> enough, but even urllib3 is painfully low-level).____

>
> Support for Windows SChannel and Apple SecureTransport is part of the
> TLS module.
>
> IDK how far along that work is (whether it'll be ready for 3.7 beta 1)?
> Or where those volunteering to help with the TLS module can send PRs?

You are misunderstanding the goal of PEP 543. It's not about providing
implementations of various backends. The PEP merely defines an minimal
abstraction layer. Neither the PEP nor the API are finalized or complete
yet, too Some parts of the PEP must be changed before it can be
finalized. Cory and I are discussion the matter.

Python 3.7's ssl module won't be compatible with PEP 543. For 3.8 it
*might* be possible to provide a 543 compatible implementation on top of
the ssl module.

I will not work on SChannel or SecureTransport, since I have neither
expertise, knowledge, interest or resources to work on other
implementations. AFAIK Steve would rather plug in Windows' cert
validation API into OpenSSL than to provide another TLS implementation.
For Apple ... no clue. How about you contact Apple support?

Regards,
Christian

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Wes Turner]

On Tuesday, January 16, 2018, Christian Heimes <chri...@python.org> wrote:

On 2018-01-16 12:28, Wes Turner wrote:
>
>
> On Tuesday, January 16, 2018, Steve Dower <steve...@python.org
> <mailto:steve...@python.org>> wrote:
>
>     From my perspective, we can’t keep an OpenSSL-like API and use
>     Windows platform libraries (we *could* do a requests-like API easily
>     enough, but even urllib3 is painfully low-level).____
>
> Support for Windows SChannel and Apple SecureTransport is part of the
> TLS module.
>
> IDK how far along that work is (whether it'll be ready for 3.7 beta 1)?
> Or where those volunteering to help with the TLS module can send PRs?

You are misunderstanding the goal of PEP 543. It's not about providing
implementations of various backends. The PEP merely defines an minimal
abstraction layer. Neither the PEP nor the API are finalized or complete
yet, too Some parts of the PEP must be changed before it can be
finalized. Cory and I are discussion the matter.

Python 3.7's ssl module won't be compatible with PEP 543. For 3.8 it
*might* be possible to provide a 543 compatible implementation on top of
the ssl module.

Got it. Thanks! 

I will not work on SChannel or SecureTransport, since I have neither
expertise, knowledge, interest or resources to work on other
implementations. AFAIK Steve would rather plug in Windows' cert
validation API into OpenSSL than to provide another TLS implementation.
For Apple ... no clue. How about you contact Apple support?

A HUP to their seclist about this work awhile back doesn't seem to have upgraded OpenSSL. 

Presumably there's another mailing list thread or GitHub issue for PEP 543 interface and implementation development.

 

Regards,
Christian

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com

[Christian Heimes]

FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I have
created a daily cronjob to populate Travis' cache with OpenSSL builds.
Until the cache is filled, Linux CI will take an extra 5 minute.

Christian

[1] https://github.com/python/cpython/pull/5180

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Steve Dower]

Honestly, I’d rather plug into the WinHTTP API and just not even bother with sockets :)

 

Certificate validation is about the only thing broken in OpenSSL on Windows (as far as not working well with system config), and it’s relatively easy to replace with a couple of API calls. Now that we don’t statically link OpenSSL anymore, it can be done easily with ctypes, so I’ll probably put out a package for it sometime soon.

 

Top-posted from my Windows phone

 

From: Christian Heimes
Sent: Tuesday, January 16, 2018 22:52
To: pytho...@python.org
Subject: Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >=2.5.3

 

On 2018-01-16 12:28, Wes Turner wrote:

Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

 

[Steve Dower]

I think you mean out-of-band updates, and by “you” I'm going to pretend you mean PyCA ;)

 

Top-posted from my Windows phone

 

From: Christian Heimes
Sent: Tuesday, January 16, 2018 21:42
To: pytho...@python.org
Subject: Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL>=2.5.3

 

On 2018-01-16 08:08, Steve Dower wrote:

Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

 

[Christian Heimes]

On 2018-01-16 22:47, Steve Dower wrote:
> I think you mean out-of-band updates, and by “you” I'm going to pretend
> you mean PyCA ;)

Err, yes :)

[Christian Heimes]

On 2018-01-16 21:17, Christian Heimes wrote:
> FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I have
> created a daily cronjob to populate Travis' cache with OpenSSL builds.
> Until the cache is filled, Linux CI will take an extra 5 minute.

I have messed up my initial research. :( When I was checking LibreSSL
and OpenSSL for features, I draw a wrong conclusion. LibreSSL is *not*
OpenSSL 1.0.2 compatible. It only implements some of the required
features from 1.0.2 (e.g. X509_check_hostname) but not
X509_VERIFY_PARAM_set1_host.

X509_VERIFY_PARAM_set1_host() is required to perform hostname
verification during the TLS handshake. Without the function, I'm unable
to fix Python's hostname matching code [1]. LibreSSL upstream knows
about the issue since 2016 [2]. I have opened another bug report [3].

We have two options until LibreSSL has addressed the issue:

1) Make the SSL module more secure, simpler and standard conform
2) Support LibreSSL

I started a vote on Twitter [4]. So far most people prefer security.

Christian

[1] https://bugs.python.org/issue31399
[2] https://github.com/pyca/cryptography/issues/3247
[3] https://github.com/libressl-portable/portable/issues/381
[4] https://twitter.com/reaperhulk/status/953991843565490176

[Wes Turner]

Is there a build flag or a ./configure-time autodetection that would allow for supporting LibreSSL while they port X509_VERIFY_PARAM_set1_host?

Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com

[Christian Heimes]

On 2018-01-18 19:42, Wes Turner wrote:
> Is there a build flag or a ./configure-time autodetection that would
> allow for supporting LibreSSL while they port X509_VERIFY_PARAM_set1_host?

X509_VERIFY_PARAM_set1_host() is a fundamental and essential piece in
the new hostname verification code. I cannot replace
ssl.match_hostname() easily without the API. There might be a way to add
a callback, but it would take a couple of days of R&D to implement it.
It won't be finished for beta1 feature freeze.

Christian

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Wes Turner]

LibreSSL is not a pressing need for me; but fallback to the existing insecure check if LibreSSL is present shouldn't be too difficult?

On Thursday, January 18, 2018, Christian Heimes <chri...@python.org> wrote:

Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com

[Christian Heimes]

On 2018-01-18 20:54, Wes Turner wrote:
> LibreSSL is not a pressing need for me; but fallback to the existing
> insecure check if LibreSSL is present shouldn't be too difficult?

Please give it a try and report back. Patches welcome :)

Christian

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Chris Jerdonek]

On Thu, Jan 18, 2018 at 7:34 AM Christian Heimes <chri...@python.org> wrote:

On 2018-01-16 21:17, Christian Heimes wrote:
We have two options until LibreSSL has addressed the issue:

1) Make the SSL module more secure, simpler and standard conform
2) Support LibreSSL

I started a vote on Twitter [4]. So far most people prefer security.

It’s not exactly the most balanced (neutral) presentation of a ballot question though. :)

—Chris

Christian

[1] https://bugs.python.org/issue31399
[2] https://github.com/pyca/cryptography/issues/3247
[3] https://github.com/libressl-portable/portable/issues/381
[4] https://twitter.com/reaperhulk/status/953991843565490176

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

[Christian Heimes]

On 2018-01-18 21:49, Chris Jerdonek wrote:
>
> On Thu, Jan 18, 2018 at 7:34 AM Christian Heimes <chri...@python.org
> <mailto:chri...@python.org>> wrote:
>
> On 2018-01-16 21:17, Christian Heimes wrote:
> We have two options until LibreSSL has addressed the issue:
>
> 1) Make the SSL module more secure, simpler and standard conform
> 2) Support LibreSSL
>
> I started a vote on Twitter [4]. So far most people prefer security.
>
>
> It’s not exactly the most balanced (neutral) presentation of a ballot
> question though. :)

It's more venting than voting :)

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Nathaniel Smith]

On Jan 18, 2018 07:34, "Christian Heimes" <chri...@python.org> wrote:

On 2018-01-16 21:17, Christian Heimes wrote:
> FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I have
> created a daily cronjob to populate Travis' cache with OpenSSL builds.
> Until the cache is filled, Linux CI will take an extra 5 minute.

I have messed up my initial research. :( When I was checking LibreSSL
and OpenSSL for features, I draw a wrong conclusion. LibreSSL is *not*
OpenSSL 1.0.2 compatible. It only implements some of the required
features from 1.0.2 (e.g. X509_check_hostname) but not
X509_VERIFY_PARAM_set1_host.

X509_VERIFY_PARAM_set1_host() is required to perform hostname
verification during the TLS handshake. Without the function, I'm unable
to fix Python's hostname matching code [1]. LibreSSL upstream knows
about the issue since 2016 [2]. I have opened another bug report [3].

We have two options until LibreSSL has addressed the issue:

1) Make the SSL module more secure, simpler and standard conform
2) Support LibreSSL

There are tons of different SSL libraries out there that we could theoretically support, but don't. IIUC, the reasons we started supporting LibreSSL in the first place were:

- it claimed to be OpenSSL compatible, so supporting it is supposed to be "free"

- when it started (just after heartbleed), there was a lot of uncertainty about whether OpenSSL would remain a viable option, and LibreSSL looked like it might become the new de facto standard.

Now it's a few years later, and things have turned out differently: they aren't compatible in practice, and OpenSSL has turned things around so that it's clearly ahead of LibreSSL technically (cleaner API, new features like TLS 1.3, ...), and in terms of developer momentum.

We have *very* few people qualified to maintain the ssl module, so given the new landscape I think we should focus on keeping our core OpenSSL support solid and not worry about LibreSSL. If LibreSSL wants to be supported as well then – like any other 2nd tier platform – they need to find someone to do the work. And if people are worried about supporting more diversity in SSL implementations, then PEP 543 is probably the thing to focus on.

-n

[Steve Holden]

On Fri, Jan 19, 2018 at 12:09 AM, Nathaniel Smith <n...@pobox.com> wrote:

On Jan 18, 2018 07:34, "Christian Heimes" <chri...@python.org> wrote:

On 2018-01-16 21:17, Christian Heimes wrote:
> FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I have
> created a daily cronjob to populate Travis' cache with OpenSSL builds.
> Until the cache is filled, Linux CI will take an extra 5 minute.

I have messed up my initial research. :( When I was checking LibreSSL
and OpenSSL for features, I draw a wrong conclusion. LibreSSL is *not*
OpenSSL 1.0.2 compatible. It only implements some of the required
features from 1.0.2 (e.g. X509_check_hostname) but not
X509_VERIFY_PARAM_set1_host.

X509_VERIFY_PARAM_set1_host() is required to perform hostname
verification during the TLS handshake. Without the function, I'm unable
to fix Python's hostname matching code [1]. LibreSSL upstream knows
about the issue since 2016 [2]. I have opened another bug report [3].

We have two options until LibreSSL has addressed the issue:

1) Make the SSL module more secure, simpler and standard conform
2) Support LibreSSL

​[...]

 

We have *very* few people qualified to maintain the ssl module, so given the new landscape I think we should focus on keeping our core OpenSSL support solid and not worry about LibreSSL. If LibreSSL wants to be supported as well then – like any other 2nd tier platform – they need to find someone to do the work. And if people are worried about supporting more diversity in SSL implementations, then PEP 543 is probably the thing to focus on.

​Given the hard limit on resources it seems only sensible to focus on the "industry standard" library​. I'm rather disappointed that LibreSSL isn't a choice, but given the lack of compatibility that's hardly Python's problem.

[Christian Heimes]

On 2018-01-19 10:43, Steve Holden wrote:
> On Fri, Jan 19, 2018 at 12:09 AM, Nathaniel Smith <n...@pobox.com
> <mailto:n...@pobox.com>> wrote:
>
> On Jan 18, 2018 07:34, "Christian Heimes" <chri...@python.org

Thanks!

I'd prefer to support LibreSSL, too. Paul Kehrer from PyCA summed up the
issue with LibreSSL nicely:

> It was marketed as an API compatible drop-in replacement and it is
failing in that capacity. Additionally, it is missing features needed to
improve the security and ease the maintenance burden of CPython’s dev team.

Since I haven given up on LibreSSL, I spent some time and implemented
some autoconf magic in https://github.com/python/cpython/pull/5242. It
checks for the presence of libssl and X509_VERIFY_PARAM_set1_host()
function family:

...
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
...

The ssl module will regain compatibility with LibreSSL as soon as a new
release provides the necessary functions.

Christian

[Christian Heimes]

No core developer has vetoed against my proposal. I also spoke to
several members of Python Cryptographic Authority and Python Packaging
Authority. They are all in favor of my proposal, too.

There I have decided to move forward and require OpenSSL 1.0.2 API. This
means Python 3.7 temporarily suspends support for LibreSSL until
https://github.com/libressl-portable/portable/issues/381 is resolved. I
have appended a lengthy explanation to my LibreSSL ticket, too.

I also informed LibreSSL developers that Python 3.8 will most likely
require an OpenSSL 1.1 compatible API. With OpenSSL 1.0.2 support, I can
drop a considerable amount of legacy code, e.g. custom thread locking,
initialization and a bunch of shim functions.

Regards,