[Python-Dev] Security releases of CPython

[Michał Górny]

Hello,

I'm the primary maintainer of CPython packages in Gentoo. I would like
to discuss possible improvement to the release process in order to
accelerate releasing security fixes to users.


I feel that vulnerability fixes do not make it to end users fast enough.
For example, according to the current release schedules for 3.9 and 3.8,
the bugfix releases are planned two months apart. While the release is
expected to happen in the next few days, both versions are known to be
vulnerable for almost a month!

Ironically, things look even worse for security-supported versions.
Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
schedule (planned for Jan 15th), and they are known to be vulnerable
since mid-October.

In my opinion, this causes three issues:

1. Users using official releases are exposed to security vulnerabilities
for prolonged periods of time.

2. When releases happen, security fixes are often combined with many
other changes. This causes problems for distribution maintainers who, on
one hand, would like to deploy the security fixes to production versions
ASAP, and on the other, would prefer that the new version remained in
testing for some time due to the other changes.

3. Effectively, it means that distribution developers need to track
and backport security fixes themselves. In the end, this means a lot of
duplicate work.


I think that security fixes are important enough to justify not sticking
to a strict release schedule. Therefore, I would like to propose that if
vulnerability fixes are committed, new releases are made
as frequently as necessary and as soon as possible (i.e. possibly giving
some time for testing) rather than according to a strict schedule.

Furthermore, I think that at least for branches that are in higher level
of maintenance than security, it could make sense to actually make
security releases (e.g. 3.9.1.x) that would include only security fixes
without other changes.


What do you think?

--
Best regards,
Michał Górny
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VKIRQTSQCO3MTXS57MDGJE6FH3PJCEBI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Terry Reedy]

On 2/11/2021 3:23 PM, Michał Górny wrote:
> Hello,
>
> I'm the primary maintainer of CPython packages in Gentoo. I would like
> to discuss possible improvement to the release process in order to
> accelerate releasing security fixes to users.
>
> I feel that vulnerability fixes do not make it to end users fast enough.

When we introduce a bad regression in a release, including a
severe-enough security vulnerability, and discover it soon enough, we
have sometimes made immediately releases.

Beyond that, your proposal to add several releases per Python version,
perhaps double what we do now, runs up against the limits of volunteer
time and effort. As it is now, becoming a release manager is a 7 year
commitment, with at least one release about every other month for the
first 4. There are also the 2 people who produce the Windows and macOS
installers. I believe each has done it for at least 7 or 8 years
already. Releases are not just a push of a button. Make the release
job too onerous, and there might not be any more volunteers.

> For example, according to the current release schedules for 3.9 and 3.8,
> the bugfix releases are planned two months apart. While the release is
> expected to happen in the next few days, both versions are known to be
> vulnerable for almost a month!
>
> Ironically, things look even worse for security-supported versions.
> Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
> schedule (planned for Jan 15th), and they are known to be vulnerable
> since mid-October.
>
> In my opinion, this causes three issues:
>
> 1. Users using official releases are exposed to security vulnerabilities
> for prolonged periods of time.
>
> 2. When releases happen, security fixes are often combined with many
> other changes. This causes problems for distribution maintainers who, on
> one hand, would like to deploy the security fixes to production versions
> ASAP, and on the other, would prefer that the new version remained in
> testing for some time due to the other changes.
>
> 3. Effectively, it means that distribution developers need to track
> and backport security fixes themselves. In the end, this means a lot of
> duplicate work.

Perhaps people doing duplicate work could get together to eliminate some of
the duplication. It should be possible to filter security commits from the
python-checkins list by looking at the news entries and if need be, the bpo
issues.

> I think that security fixes are important enough to justify not sticking
> to a strict release schedule. Therefore, I would like to propose that if
> vulnerability fixes are committed, new releases are made
> as frequently as necessary and as soon as possible (i.e. possibly giving
> some time for testing) rather than according to a strict schedule.
>
> Furthermore, I think that at least for branches that are in higher level
> of maintenance than security, it could make sense to actually make
> security releases (e.g. 3.9.1.x) that would include only security fixes
> without other changes.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4S5LOTVJZUA5PQ5TRGQFCWHTGA4BOXBO/

[Gregory P. Smith]

On Thu, Feb 11, 2021 at 8:29 PM Terry Reedy <tjr...@udel.edu> wrote:

On 2/11/2021 3:23 PM, Michał Górny wrote:
> Hello,
>
> I'm the primary maintainer of CPython packages in Gentoo. I would like
> to discuss possible improvement to the release process in order to
> accelerate releasing security fixes to users.
>
> I feel that vulnerability fixes do not make it to end users fast enough.

When we introduce a bad regression in a release, including a
severe-enough security vulnerability, and discover it soon enough, we
have sometimes made immediately releases.

Indeed. What Terry said. There is risk in doing that. It means the solution is more rushed and hasn't been vetted nearly as well. So without a compelling severe reason we're unlikely to.

I would expect distros shipping their own python runtime packages to track security issues and their fixes and backport those (generally easy) to their own packages before an official release is ready anyways. The big distros definitely do this. How to deal with this is up to the individual policies of any given OS distro's package owners.

-gps

[Victor Stinner]

Hi Michał,

I created https://python-security.readthedocs.io/ website to track
known Python vulnerabilities to help me checking if fixes are
backported to all supported Python branches. I'm maintaing this list
manually, it's far from being complete, and likely outdated.

I also created https://github.com/vstinner/check_python_vuln project
which implements runtime checks for a few Python vulnerabilities. It
should help users and Linux packagers to check if their Python has
known vulnerabilities or not.

On Thu, Feb 11, 2021 at 9:44 PM Michał Górny <mgo...@gentoo.org> wrote:
> I feel that vulnerability fixes do not make it to end users fast enough.
> For example, according to the current release schedules for 3.9 and 3.8,
> the bugfix releases are planned two months apart. While the release is
> expected to happen in the next few days, both versions are known to be
> vulnerable for almost a month!

We are doing our best to fix all known security vulnerabilities in all
maintained Python versions (3.6, 3.7, 3.8, 3.9 and master: 5
branches). But we have no policy to define the severity of these
vulnerabilities. IMO sometimes we are a little bit too conservative
when marking some issues are "security" issues.

For example, the https://bugs.python.org/issue41944 "CJK codecs tests
call eval() on content received via HTTP" issue only impacts users
running the Python test suite.
https://python-security.readthedocs.io/vuln/cjk-codec-download-eval.html
gives the timeline, extract:

2020-10-05 (+0 days): Reported (email sent to the PSRT list)
(...)
2020-10-22 (+17 days): CVE-2020-27619 published
2020-12-07 (+63 days): Python 3.9.1 released
2020-12-21 (+77 days): Python 3.8.7 released

IMO here the delay is reasonable (~2 months for 3.8 and 3.9 versions)
for this kind of vulnerability.

HTTP header and email header injections vulnerabilities are bad, but
they can be prevented by applications. For example, good user inputs
validation should prevent such vulnerability. Example of HTTP header
injection timeline:
https://python-security.readthedocs.io/vuln/http-header-injection-method.html

Do you have a specific kind of vulnerability in mind that would
require a faster release?

> 2. When releases happen, security fixes are often combined with many
> other changes. This causes problems for distribution maintainers who, on
> one hand, would like to deploy the security fixes to production versions
> ASAP, and on the other, would prefer that the new version remained in
> testing for some time due to the other changes.

We attempt to avoid incompatible changes in 3.x.y bug fix releases. If
it happens, we can consider to revert it on a case by case basis.
Usually, it is discussed before merging a change.

Usually, the incompatible changes that we allow are justified... to
fix security issues :-)

Do you have a specific example of problematic incompatible change in
mind? For me, the largest change was a Python 2.7 release which
started to check TLS certificates on HTTP... It was to make Python
more secure to protect users :-)

> Furthermore, I think that at least for branches that are in higher level
> of maintenance than security, it could make sense to actually make
> security releases (e.g. 3.9.1.x) that would include only security fixes
> without other changes.

IMO that's too much work for everybody. developers, testers and packagers.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PBZADDEVMQNWAHMR6XZD3BAYVP3OE3HM/

[Michał Górny]

On Tue, 2021-02-16 at 00:53 +0100, Victor Stinner wrote:
> Hi Michał,
>
> I created https://python-security.readthedocs.io/ website to track
> known Python vulnerabilities to help me checking if fixes are
> backported to all supported Python branches. I'm maintaing this list
> manually, it's far from being complete, and likely outdated.
>
> I also created https://github.com/vstinner/check_python_vuln project
> which implements runtime checks for a few Python vulnerabilities. It
> should help users and Linux packagers to check if their Python has
> known vulnerabilities or not.

Thank you.

> On Thu, Feb 11, 2021 at 9:44 PM Michał Górny <mgo...@gentoo.org> wrote:
> > I feel that vulnerability fixes do not make it to end users fast enough.
> > For example, according to the current release schedules for 3.9 and 3.8,
> > the bugfix releases are planned two months apart. While the release is
> > expected to happen in the next few days, both versions are known to be
> > vulnerable for almost a month!
>
> We are doing our best to fix all known security vulnerabilities in all
> maintained Python versions (3.6, 3.7, 3.8, 3.9 and master: 5
> branches). But we have no policy to define the severity of these
> vulnerabilities. IMO sometimes we are a little bit too conservative
> when marking some issues are "security" issues.

I generally try to avoid making assumptions about severity of security
bugs, and treat all of them the same.

> For example, the https://bugs.python.org/issue41944 "CJK codecs tests
> call eval() on content received via HTTP" issue only impacts users
> running the Python test suite.
> https://python-security.readthedocs.io/vuln/cjk-codec-download-eval.html
> gives the timeline, extract:
>
> 2020-10-05 (+0 days): Reported (email sent to the PSRT list)
> (...)
> 2020-10-22 (+17 days): CVE-2020-27619 published
> 2020-12-07 (+63 days): Python 3.9.1 released
> 2020-12-21 (+77 days): Python 3.8.7 released
>
> IMO here the delay is reasonable (~2 months for 3.8 and 3.9 versions)
> for this kind of vulnerability.

I have ignored this one as well, since we are running the test suite
with network access disabled. I can't complain for this one but I don't
know whether it doesn't impact other people's workflows. Please correct
me if I'm missing something but if a distro packager uses a workflow
like 'build, run tests, install' (this is what we do), I believe that
this vulnerability could be used to inject malware into Python packages
distributed with a lot of distributions.

> HTTP header and email header injections vulnerabilities are bad, but
> they can be prevented by applications. For example, good user inputs
> validation should prevent such vulnerability. Example of HTTP header
> injection timeline:
> https://python-security.readthedocs.io/vuln/http-header-injection-method.html
>
> Do you have a specific kind of vulnerability in mind that would
> require a faster release?

As I said above, I try not to make assumptions about severity. However,
I personally wouldn't rely on people having their scripts fully
protected or actually doing anything to workaround vulnerabilities
in CPython. If we have a fix handy, it is our responsibility to deploy
it.

> > 2. When releases happen, security fixes are often combined with many
> > other changes. This causes problems for distribution maintainers who, on
> > one hand, would like to deploy the security fixes to production versions
> > ASAP, and on the other, would prefer that the new version remained in
> > testing for some time due to the other changes.
>
> We attempt to avoid incompatible changes in 3.x.y bug fix releases. If
> it happens, we can consider to revert it on a case by case basis.
> Usually, it is discussed before merging a change.
>
> Usually, the incompatible changes that we allow are justified... to
> fix security issues :-)
>
> Do you have a specific example of problematic incompatible change in
> mind? For me, the largest change was a Python 2.7 release which
> started to check TLS certificates on HTTP... It was to make Python
> more secure to protect users :-)

I didn't mean intended incompatibilities. I meant the possibility of
new bugs. I can't say I recall such a problem with CPython but that
doesn't mean that there weren't sneaky issues in the past. We have
a testing process in place to give braver users a chance to discover
such problems.

> > Furthermore, I think that at least for branches that are in higher level
> > of maintenance than security, it could make sense to actually make
> > security releases (e.g. 3.9.1.x) that would include only security fixes
> > without other changes.
>
> IMO that's too much work for everybody. developers, testers and packagers.

I do realize that's extra work for people responsible for making
the release but packagers are doing the work already (or should be doing
it).

--
Best regards,
Michał Górny

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/IH7QSDNS6GGLGQNITB5RMNORZCMOMYIF/

[Michał Górny]

On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
> On 2/11/2021 3:23 PM, Michał Górny wrote:
> > Hello,
> >
> > I'm the primary maintainer of CPython packages in Gentoo. I would like
> > to discuss possible improvement to the release process in order to
> > accelerate releasing security fixes to users.
> >
> > I feel that vulnerability fixes do not make it to end users fast enough.
>
> When we introduce a bad regression in a release, including a
> severe-enough security vulnerability, and discover it soon enough, we
> have sometimes made immediately releases.
>
> Beyond that, your proposal to add several releases per Python version,
> perhaps double what we do now, runs up against the limits of volunteer
> time and effort. As it is now, becoming a release manager is a 7 year
> commitment, with at least one release about every other month for the
> first 4. There are also the 2 people who produce the Windows and macOS
> installers. I believe each has done it for at least 7 or 8 years
> already. Releases are not just a push of a button. Make the release
> job too onerous, and there might not be any more volunteers.

While I understand your concerns and sympathize with them, I don't think
it's fair to play the volunteer card here. After all, I am a volunteer
too, and many distribution packagers are as well. We also have limited
time and many other duties, and it just feels unfair for you to be
dumping the security work on us.

The thing is, CPython upstream has the opportunity to do the central
work that benefits everyone. You get the first indication
of the vulnerability, you have the first fix and backport to all
branches. It is only the natural next step to make a release with it.
While I realize it means more work, not doing that means more work for
a number of other people who are impacted by your prior work.

That said, if the regular release process is too much work, then maybe
a process similar to security-supported branches would work? After all,
I think it reasonable to assume that a large number of CPython users are
using it via distribution packages and having Python 3.8.x.y source-only
release will be already a big help to people making these packages.

> > For example, according to the current release schedules for 3.9 and 3.8,
> > the bugfix releases are planned two months apart. While the release is
> > expected to happen in the next few days, both versions are known to be
> > vulnerable for almost a month!
> >
> > Ironically, things look even worse for security-supported versions.
> > Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
> > schedule (planned for Jan 15th), and they are known to be vulnerable
> > since mid-October.
> >
> > In my opinion, this causes three issues:
> >
> > 1. Users using official releases are exposed to security vulnerabilities
> > for prolonged periods of time.
> >
> > 2. When releases happen, security fixes are often combined with many
> > other changes. This causes problems for distribution maintainers who, on
> > one hand, would like to deploy the security fixes to production versions
> > ASAP, and on the other, would prefer that the new version remained in
> > testing for some time due to the other changes.
> >
> > 3. Effectively, it means that distribution developers need to track
> > and backport security fixes themselves. In the end, this means a lot of
> > duplicate work.
>
> Perhaps people doing duplicate work could get together to eliminate some of
> the duplication. It should be possible to filter security commits from the
> python-checkins list by looking at the news entries and if need be, the bpo
> issues.

This makes me think of the 'systemd-stable' repository (where another
maintainer is making bugfix releases to previous versions of systemd).
I suppose this could work as a last resort but only if it was officially
supported and endorsed by CPython. However, at this point it's not
really clear why not go the one step further and make the official
releases.

--
Best regards,
Michał Górny

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/I52QGQVA55BXF425V5IZLF3GYYPWLOKW/

[Brett Cannon]

On Fri, Feb 19, 2021 at 2:16 AM Michał Górny <mgo...@gentoo.org> wrote:

On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
> On 2/11/2021 3:23 PM, Michał Górny wrote:
> > Hello,
> >
> > I'm the primary maintainer of CPython packages in Gentoo. I would like
> > to discuss possible improvement to the release process in order to
> > accelerate releasing security fixes to users.
> >
> > I feel that vulnerability fixes do not make it to end users fast enough.
>
> When we introduce a bad regression in a release, including a
> severe-enough security vulnerability, and discover it soon enough, we
> have sometimes made immediately releases.
>
> Beyond that, your proposal to add several releases per Python version,
> perhaps double what we do now, runs up against the limits of volunteer
> time and effort.  As it is now, becoming a release manager is a 7 year
> commitment, with at least one release about every other month for the
> first 4.  There are also the 2 people who produce the Windows and macOS
> installers.  I believe each has done it for at least 7 or 8 years
> already.  Releases are not just a push of a button.  Make the release
> job too onerous, and there might not be any more volunteers.

While I understand your concerns and sympathize with them, I don't think
it's fair to play the volunteer card here.  After all, I am a volunteer
too, and many distribution packagers are as well.  We also have limited
time and many other duties, and it just feels unfair for you to be
dumping the security work on us.

I also don't think it's fair to say anyone is "dumping security work" on you. Volunteerism is about accepting that I can't ask anything of you while you can't ask anything of me. We all do our best here, but it's making choices with very finite resources and our release and security approach is the one we have made based on what we have to work with. If people would like to see more frequent releases then it will most likely require people volunteering to help with the security workload as well as with the release process, both of which are rather thankless jobs unfortunately. But simply because we have differing views on how to handle security doesn't mean anyone is actively "dumping" anything on anyone.

 

The thing is, CPython upstream has the opportunity to do the central
work that benefits everyone.  You get the first indication
of the vulnerability, you have the first fix and backport to all
branches.  It is only the natural next step to make a release with it.
While I realize it means more work, not doing that means more work for
a number of other people who are impacted by your prior work.

That said, if the regular release process is too much work, then maybe
a process similar to security-supported branches would work?  After all,
I think it reasonable to assume that a large number of CPython users are
using it via distribution packages and having Python 3.8.x.y source-only
release will be already a big help to people making these packages.

But a very large number of people get their releases from python.org, so if we do source-only releases without pushing out an install that then causes a schism between what users get on one platform versus another that goes beyond simply the delivery mechanism (and this isn't hypothetical; downstream packages by Linux distros have caused issues for users due to differing from what python.org pushes out, so we do need to consider this impact).

-Brett

[Terry Reedy]

On 2/19/2021 5:11 AM, Michał Górny wrote:
> On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:

>> Releases are not just a push of a button. Make the release
>> job too onerous, and there might not be any more volunteers.
>
> While I understand your concerns and sympathize with them,

Your accusations of unfairness in response to my polite, volunteered
response, suggest otherwise.

> I don't think it's fair to play the volunteer card here.

...

> it just feels unfair for you to be dumping the security work on us.

Which I did not do. Bye.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/UOWWSWZ3O4NV6PIQBMF2KKBVSTK6HXL3/

[Stestagg]

The thing that stood out from this conversation, for me, is: Releases are too hard, and there’s a risk of not having enough volunteers as a result.

How hard is it to fix that? 

Steve

[Christian Heimes]

On 19/02/2021 23.22, Stestagg wrote:
> The thing that stood out from this conversation, for me, is: Releases
> are too hard, and there’s a risk of not having enough volunteers as a
> result.
>
> How hard is it to fix that? 

Actually it's easy to fix!

The PSF needs needs sufficient money to hire a couple of people, so the
PSF can turn release management and security maintenance from unpaid
volunteers into paid fulltime jobs.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4INCHKPOGKMTDQ4IDS4RDR33KJ4RR2H5/

[Stestagg]

On Fri, 19 Feb 2021 at 22:50, Christian Heimes <chri...@python.org> wrote:

On 19/02/2021 23.22, Stestagg wrote:
> The thing that stood out from this conversation, for me, is: Releases
> are too hard, and there’s a risk of not having enough volunteers as a
> result.
>
> How hard is it to fix that? 

Actually it's easy to fix!

The PSF needs needs sufficient money to hire a couple of people, so the
PSF can turn release management and security maintenance from unpaid
volunteers into paid fulltime jobs.

That’s certainly one option. I’d be super surprised if that were the only one. 

What were the budget requirements for this? How far from the target is the PSF?  Is there a plan to hit it?

Are there no technical solutions that might help reduce the cost?

[Brett Cannon]

There is no specific drive to hire someone to target security and/or release management at the moment. We just got enough funding for the first time to hire a dev-in-residence for Python itself to try to help tackle our 1.4K PR backlog in some fashion that they won't be dedicated to security or release management.

 

Are there no technical solutions that might help reduce the cost?

If you would like to help out, you can speak with the release managers to see if they could use help in some way; same goes for the security team.

-Brett

 

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/55GFSMWJOG7P6NWU4KDHMKJN2OGX5YSG/

[Steve Holden]

On Fri, Feb 19, 2021 at 10:53 PM Christian Heimes <chri...@python.org> wrote:

On 19/02/2021 23.22, Stestagg wrote:
> The thing that stood out from this conversation, for me, is: Releases
> are too hard, and there’s a risk of not having enough volunteers as a
> result.
>
> How hard is it to fix that? 

Actually it's easy to fix!

For some value of "easy"  that involves raising a six-figure dollar sum annually ...

The PSF needs needs sufficient money to hire a couple of people, so the
PSF can turn release management and security maintenance from unpaid
volunteers into paid fulltime jobs.

Oh, is that all? Sustainability of the PSF, as has been shown over its 20 years of existence, is not an easy thing to achieve. It was hit by the financial crisis in 2008 and again by the coronavirus crisis last year, both things that affected all foundations.

If you plan to bring this kind of money in and rely on it, you have to ensure it comes from sources that can't just be switched off when budgets tighten. It could be done, but "easy" sounds like exaggeration to me. Unless your suggestions were joking, but I saw no smiley ...

Kind regards,

Steve

[Michał Górny]

On Fri, 2021-02-19 at 17:03 -0500, Terry Reedy wrote:
> On 2/19/2021 5:11 AM, Michał Górny wrote:
> > On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
>
> > > Releases are not just a push of a button. Make the release
> > > job too onerous, and there might not be any more volunteers.
> >
> > While I understand your concerns and sympathize with them,
>
> Your accusations of unfairness in response to my polite, volunteered
> response, suggest otherwise.
>
> > I don't think it's fair to play the volunteer card here.
> ...
> > it just feels unfair for you to be dumping the security work on us.
>
> Which I did not do. Bye.

I'm sorry, I did not mean to insult you. That's just how I felt after
reading the response.

--
Best regards,
Michał Górny

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/GJVNYYMEE73GGYELTRR66XUF2EPAJNYF/

[Victor Stinner]

On Thu, Feb 11, 2021 at 9:44 PM Michał Górny <mgo...@gentoo.org> wrote:

> I feel that vulnerability fixes do not make it to end users fast enough.

I think that it's time to put that into perspective with past vulnerabilities.

Ok, let me look at the timeline of the discussed vulnerability, ctypes
CVE-2021-3177:
https://python-security.readthedocs.io/vuln/ctypes-buffer-overflow-pycarg_repr.html

2021-01-16: Python issue bpo-42938 reported by Jordy Zomer
...
2021-01-18 (+2 days): commit c347cbe (branch 3.9)
2021-01-18 (+2 days): commit ece5dfd (branch 3.8)
2021-01-19 (+3 days): CVE-2021-3177 published
...
2021-02-19 (+34 days): Python 3.8.8 released
2021-02-19 (+34 days): Python 3.9.2 released

Ok. What about vulnerabilities fixes released last years?

"HTTP header injection in urllib, urrlib2, httplib and http.client modules"
https://python-security.readthedocs.io/vuln/http-header-injection.html
2017-09-19 (+1030 days): Python 3.3.7 released

"CGI directory traversal"
https://python-security.readthedocs.io/vuln/cgi-directory-traversal-is_cgi.html
2011-05-09 (+1158 days): CVE-2011-1015 published
2013-04-07 (+1857 days): Python 3.2.4 released
2013-04-07 (+1857 days): Python 3.3.1 released

"httplib unlimited read"
https://python-security.readthedocs.io/vuln/httplib-unlimited-read.html
2011-06-11 (+652 days): Python 2.7.2 released
2011-06-11 (+652 days): Python 3.1.4 released

"rgbimg and imageop overflows"
https://python-security.readthedocs.io/vuln/rgbimg-imageop-overflows.html
2008-12-19 (+460 days): Python 2.5.3 released

So the CVE-2021-3177 fix was delivery between 14x and 55x faster than
the other listed fixes (I picked a few worst cases to put numbers in
perspective).

Congrats to the core developers for fixing the vulnerability in only
*3* days and to release manager for releasing *4* (!) Python versions
(3.6.13, 3.7.10, 3.8.8, 3.9.2) in only 34 days!

I would like to highlight that exploiting a directory traversal or
HTTP header injection is really trivial. Once you find a pattern to
explore the filesystem / inject a HTTP header, the exploit is 100%
reliable.

On the other side, there is no known exploit for ctypes CVE-2021-3177
and ctypes is rarely used. I read that Django's GIS uses ctypes and
floats, but so far nobody shows that PyCArg_repr() is called, and
nobody published an exploit.

To write a CVE-2021-3177 exploit, you must create a 64-bit floating
point number (only 8 bytes!) which becomes valid machine code, and
this code should allow to take control on the machine, once it's
formatted as decimal. For example, PyCArg_repr(123.5) writes "123.5\0"
string into the stack memory. but I don't think that it's valid x86-64
machine code. It is also hard to write a reliable exploit by injecting
machine code in the stack memory.

---

Nowadays it's way more difficult than 10 years ago to write an exploit
using a stack overflow, C compilers provide multiple hardening layers:
- FORTIFY_SOURCE,
- Control flow Enforcement Technology (Intel CET),
- Address Space Layout Randomization (ASLR),
- stack protector,
- Position Independent Executable (PIE),
- etc.

See https://wiki.debian.org/Hardening for example of C flags and
linker flags for harderning.

Did anyone notice that Red Hat Entreprise Linux 8 (RHEL) is *not*
affected by the ctypes CVE-2021-3177 vulnerability thanks to
hardening?

"Red Hat Enterprise Linux 8: python36:3.6/python36: Not affected"
and
"This flaw could have had a higher Impact, however our packages are
compiled with FORTIFY_SOURCE, which provides runtime protection to
some memory and string functions and prevents this flaw from actually
overwriting the buffer and potentially executing code."
=> https://access.redhat.com/security/cve/cve-2021-3177

I suggest you checking how your operating system built your Python
executable, and libpython if Python is built in shared mode: hardening
can prevent the ctypes vulnerabiliy, but also against *future*
vulnerabilities!

For example, Fedora 33 builds Python 3.9 with the following C flags:

-Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2
-fcf-protection
-fstack-clash-protection
-fstack-protector-strong
-fPIC
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 => add the -fPIE flag
(...)

And linker flags:

-Wl,-z,now
-Wl,-z,relro
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld => add the -pie flag
(...)

You can inspect hardening options on a ELF binary using hardening-check tool:

$ hardening-check /usr/bin/python3.9
/usr/bin/python3.9:
Position Independent Executable: yes
Stack protected: no, not found!
Fortify Source functions: unknown, no protectable libc functions used
Read-only relocations: yes
Immediate binding: yes
Stack clash protection: unknown, no -fstack-clash-protection instructions found
Control flow integrity: yes

$ hardening-check /usr/lib64/libpython3.9.so.1.0
/usr/lib64/libpython3.9.so.1.0:
Position Independent Executable: no, regular shared library (ignored)
Stack protected: yes
Fortify Source functions: yes (some protected functions found)
Read-only relocations: yes
Immediate binding: yes
Stack clash protection: unknown, no -fstack-clash-protection instructions found

Note: I don't know if this tool is reliable, but at least I can see
that multiple hardening options are enabled ;-)


On Fedora and RHEL 8, packages are built with the "annobin" extension
which records C and linker flags (and much more information). So you
check hardening using annocheck tool:

$ annocheck /usr/lib64/libpython3.6m.so.1.0
annocheck: Version 9.50.
Hardened: libpython3.6m.so.1.0: PASS.

$ annocheck /usr/lib64/libpython3.6m.so.1.0 -v|grep -vE '(skip|warn|info):'
annocheck: Version 9.50.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: One dynamic
section/segment found.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: Stack not executable.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: DT_RPATH/DT_RUNPATH
absent or rooted at /usr.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No RWX segments found.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No text relocations found.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No thread
cancellation problems.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: GOT/PLT relocations
are read only.
Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS.

More information about annobin:
https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-binaries/

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XF33457KYRJPUZAN4C4E3KCNOA7TDL6S/

[Mats Wichmann]

On 2/19/21 11:55 PM, Steve Holden wrote:

> The PSF needs needs sufficient money to hire a couple of people, so the
> PSF can turn release management and security maintenance from unpaid
> volunteers into paid fulltime jobs.
>
>
> Oh, is that all? Sustainability of the PSF, as has been shown over its
> 20 years of existence, is not an easy thing to achieve. It was hit by
> the financial crisis in 2008 and again by the coronavirus crisis last
> year, both things that affected all foundations.
>
> If you plan to bring this kind of money in and rely on it, you have to
> ensure it comes from sources that can't just be switched off when
> budgets tighten. It could be done, but "easy" sounds like exaggeration
> to me. Unless your suggestions were joking, but I saw no smiley ...

Steve's comments probably need no reinforcement, but I can speak as
someone who's been funded by fees collected from motivated companies,
and had the tap turn off. In my cases (more than one) they were
consortia where the members committed a set fee yearly, and then one
year: we've decided not to renew, from one or more... It's actually
easier to raise funds for a one-time campaign than one that is committed
to last for several years, in my limited experience on the fund-raising
side.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VMPYGK6MDPFUCNSNB62BFCVEEDUQW4D6/

[Jim J. Jewett]

Looking at the other replies, I'm wondering if you fully understand python's variant of version numbering.

I suggest we change the announcement template from:

"Python 3.9.2 is the newest major release of the Python programming language, and it contains many new features and optimizations."

to:

"Python 3.9 was the newest major release of the Python programming language, and it contains many new features and optimizations. Python 3.9.2 is a bugfix and security release; it has no new features relative to 3.9."



3.9.1 (and 3.9.27, if that ever happens) are supposed to the be the same as 3.9.0, except with bugs fixed.

Because "a feature is just a bug with tenure", there comes a time when non-security bugs stop being fixed. There isn't a hard-and-fast rule on when that is, but within a year or two. Looking back, a .6 release was unlikely to contain much beyond security. Even before that time, CPython still tries to err on the overly-cautious side of "can this bug-fix be backported", because of an overly-optimistic assessment ~20 years ago. (A harmless feature was added in a backwards-compatible way, but for a while instructions had to specify a bugfix version as well.)

In theory, someone could release 3.9.0s1, 3.9.0s2, ... 3.9.1s1 ... but what would be the point? 3.9.1s2 would have contained exactly the same changes as 3.9.2rc, which apparently didn't get picked up much. The difference between 3.9.2rc and 3.9.2 does include a non-security bugfix -- a part of the Windows API that was advertised as working will now actually work. Is that really adding much extra upgrade risk?

Of course, to get these extra releases, someone will have to be more careful about deciding what counts as a security fix vs a regular bugfix, which is already sometimes fuzzy. And realistically, it is *only* the security fixes that are likely to be a problem for working code ... Even if the releases were trivial, how much value would that actually provide?

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/WDBRC4WSOYSHMQVYDWHWSO2ITH4YGRQC/

[Mike Miller]

On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:

> ... Releases are not just a push of a button.

On 2021-02-19 15:05, Stestagg wrote:
> > The thing that stood out from this conversation, for me, is: Releases
> > are too hard, and there’s a risk of not having enough volunteers as a
> > result.
> >
> > How hard is it to fix that?
>

> Are there no technical solutions that might help reduce the cost?

Sounds like automating until it is "just a push of a button," should be a goal.
According to Victor there has been progress, but always room for more.

On 2021-02-19 18:36, Brett Cannon wrote:
> There is no specific drive to hire someone to target security and/or release
> management at the moment. We just got enough funding for the first time to hire
> a dev-in-residence for Python itself to try to help tackle our 1.4K PR backlog
> in some fashion that they won't be dedicated to security or release management.

Looking at the PSF Annual report from a normal year it also looks like there is
substantial revenue, income, and assets available. There are significant
expenses as well.

(And of course income down for ~two years, but it should return at some point.)

I would argue that security releases are of higher importance than most,
including sponsorship programs and grants, and that a mild change of priorities
is in order. This is not to say (of course) that any other categories are not
important, simply that having machines and networks pwned while being sponsored
or educated is a devil's bargain.

Such a position may not require a hundred-thousand a year salary either. A
skilled contractor could improve automation, while a stipend might be enough to
ensure releases get out within a ~week if they need to.

-Mike

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/MWDLKT3H4CXVPGSSMYSXTRBI33637LWA/

[Stephen J. Turnbull]

Mike Miller writes:

> Sounds like automating until it is "just a push of a button,"
> should be a goal. According to Victor there has been progress, but
> always room for more.

When XEmacs was releasing betas regularly, everything from tagging the
local authoritative repo to pushing to the public repo to making
tarballs to signing and authenticating them to Mad-Libbing the form
for announcement was invocation of a single script. "Somehow" the
typical release was a me-day's work and three calendar days (and unlike
GNU Emacs with its "kitchen sink" standard library or Python with its
batteries from CR-25s to Prius power replacements, XEmacs was just the
Lisp interpreter and basic editor functions).

What takes the time?

1. A courtesy pre-announcement to core devs that the release was
coming (yes, it's on the schedule, no, they don't check their
calendars).
2. Triaging incomplete tasks other folks labeled as "important" or above.
3. Writing news (somebody always forgets, and typically the choice was
write it myself or delay the release).
4. Triaging last-minute "urgent" commits. (They rarely were.)
5. Triaging and where possible fixing test failures.
6. Rolling back premature commits.
7. Scheduling future work that didn't *need* to be in a beta release
(we had a couple of developers who were trying but couldn't write
documentation grammatically to save their commits; it just didn't
make sense to force them to do work I could do in 1/10 the effort).
8. Coordination with the dev who built Windows installers. (A few
Windows users could get pretty cranky if they didn't get an
installer within a couple hours after the Unix-oriented tarball
releases.)

The Python process is more disciplined, and has a lot of automation to
help with the QA stuff. But there's always last-minute "people work",
which is what Brooks would call an "essential difficulty." It's
pretty irreducible. I would imagine the same is true for Python RMs.

> I would argue that security releases are of higher importance than
> most,

*Some* of them are, but this just isn't true most of the time. Take
the very recent CVE-2021-3177, rated 9.8 ("critical", 9.8 out of 10, I
believe) because it *might* lead to remote code execution (RCE) under
(implausible, AFAICS) conditions in current Python. There is no RCE
exploit for arbitrary Python webapps: there are a bunch of other
conditions that must be satisfied. It might be possible to create a
denial of service for some webapps, but that still depends on only
somewhat more plausible conditions.

Then, why 9.8? Well, "remote code execution". The critical rating is
a *threat* assessment, based on *everything* going right for the
attacker, not a *risk* assessment, based on likelihood and degree of
damage. Most reported vulnerabilities have no known exploit and
potential RCE is not the common case: risk is quite small.

Such vulnerabilities must be fixed: as time goes on, risk increases.
Not just more time = more time for the black hats, but the hazard rate
increases too. The particular currently implausible kill chain may be
generalized or others may arise in future versions of Python, or more
webapps may incorporate the risky behavior, etc. But there's no
reason to believe it will be a known exploit soon, and anybody with a
million dollars at risk can (and should) keep a couple of hours of
junior developer time on call to assess risk and apply the patch if
needed. Not to mention that in 2025 there will still be a bunch of
vulnerable 3.8 and 3.9 installations online, which are still whatever
the patchlevel was when they were originally installed.

I believe that had they let this one wait until the scheduled release,
the odds are 1000:1 that *no harm whatsoever* would have been done in
that window. Nevertheless, the relevant release managers (3.8, 3.9,
and Windows) chose to make a quick release. I think "customer
relations" (which are important, I'm not deprecating them!) had as
much to do with the decision as the assessment of risk to Python
users. I am not going to second-guess the actual decision, but I do
think that this is very much a case by case issue not very different
from any "urgent" defect, and in many cases "wait for scheduled
release" is going to be a no-brainer, and in others that will be the
outcome chosen.

Some of the time, there will be an emergency release. That has been
demonstrated to be possible and realistic, even in a case like this
where the need is not open-and-shut. I don't see a need for anything
except more love for the release managers. Of course IWBNI we had the
resources to pay release managers. But we're doing OK.

Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LJ3E2UQBPDSFEH7ULNF3QVOYEQYRSAGI/

[Wes Turner]

The default Make flags differ from platform to platform (and compiler
to compiler, IIRC) as well. Thanks for this overview of RHEL/Fedora
Python build security flags.

( Containers are the easiest way to get per- python interpreter
SELinux contexts ( in order to limit the impact of exploitation of a
vulnerability in CPython that an application exposes to users. FWIW,
FWIU, OpenShift is the only k8s platform that does per-container
contexts; and only RHEL/Fedora have a container-selinux?
https://src.fedoraproject.org/rpms/container-selinux/blob/rawhide/f/container-selinux.spec
)

https://github.com/python/miss-islington is the backport bot.

https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.9.2.ebuild

These are the patches necessary for conda-forge:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml

These are the only patches necessary on fedora:
https://src.fedoraproject.org/rpms/python3.9/tree/rawhide

--
Wes Turner
https://westurner.org
https://wrdrd.com/docs/consulting/knowledge-engineering

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/E7B6OMRC6CPUM3L2TUXGF25PKHYG2XPM/