Impending silent breakage of pip / macOS likely to cause severe confusion

106 views
Skip to first unread message

Matthew Brett

unread,
Apr 6, 2018, 11:26:13 AM4/6/18
to pypa-dev
Hi,

Please do let me know if this is not the right place to discuss this,
but I think we're about to cause a storm of user distress on macOS,
and I am wondering if there is anything we can do to reduce it.

The situation for users of macOS < 10.13 is the following:

Until recently, pip, used with default command line options, worked as
expected, with no warnings, other than the usual (and usually ignored)
general message to upgrade pip if you are on an old version.

As of 8th April, pip will break, largely in silence. As used with
default command line options, pip will stop seeing anything on pypi.
It will tell the user that it can't find packages that do exist on
pypi, that that users are up to date for packages that have later
versions on pypi - including pip. There are no messages to warn the
user what has happened or how to fix it. It is easy to think you've
upgraded to the latest version, when you have not.

The only way that the user will know what has happened, is if they
think to use the -v flag for the pip install, or if they are on a
Python mailing list, and have been paying some attention.

It seems to me that a significant majority of users are not going to
be in either category, and this is going to be a significant cause of
confusion and pain. I don't think our users will think well of us,
when they discover the problem.

The brownout period isn't going to help much, because the breakage is
largely silent.

Is there any way of generating an informative warning and instructions
*for the default use of pip install*?

Cheers,

Matthew

Donald Stufft

unread,
Apr 6, 2018, 11:49:48 AM4/6/18
to Matthew Brett, pypa-dev

On Apr 6, 2018, at 11:25 AM, Matthew Brett <matthe...@gmail.com> wrote:


As of 8th April, pip will break, largely in silence.  As used with
default command line options, pip will stop seeing anything on pypi.
It will tell the user that it can't find packages that do exist on
pypi, that that users are up to date for packages that have later
versions on pypi - including pip.  There are no messages to warn the
user what has happened or how to fix it.  It is easy to think you've
upgraded to the latest version, when you have not.


This isn’t quite accurate:

This is not a blanket issue for anyone on macOS < 10.13. It *only* affects people using a Python that links against the ancient OpenSSL provided by Apple, of which the #1 cause is System Python on macOS < 10.13, however it is also the 2.7 Python.org macOS installer on any version of Python. All other forms of getting Python, to my knowledge, link against a newer version of OpenSSL (or against LibreSSL in System Python on macOS 10.13). [1]

This affects a small percentage of users, Of the downloads from PyPI in the last 7 days that originated from a macOS system, 6% of them would have failed, 94% of them would have succeeded.

Once we are at 100% unavailability, we can ask our CDN to disable TLSv1.0 and TLSv1.1 completely, which will raise an OpenSSL error message instead of a HTTP error message. That will look like (ignore the index-url, used that because it already has TLSv1.0/1.1 disabled completely):

    $ /usr/local/bin/python2.7 -m pip install --index-url https://files.pythonhosted.org/ --upgrade pip
    Could not fetch URL https://files.pythonhosted.org/pip/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping
    Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/2.7.14_10_6/lib/python2.7/site-packages
    You are using pip version 9.0.1, however version 9.0.3 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.


Finally, even If the above weren’t true, there really isn’t much we can do either way. We can’t go back in time and change the already released version of pip, and we only have so many ways that PyPI can signal an error to pip one is using an HTTP status code (what the brownouts do now) and one is rejecting the connection with a TLS error (what the above snippet does). Beyond that, any improvements require getting users to upgrade their version of pip, and if we can get them to upgrade for a better error message, presumably we can get them to upgrade for the SecureTransport fallback in 9.0.3+.


[1] You can see more information about which Python.org macOS installers are affected here: https://github.com/pypa/warehouse/issues/3293#issuecomment-378468534

- Donald

Matthew Brett

unread,
Apr 6, 2018, 12:31:25 PM4/6/18
to Donald Stufft, pypa-dev
Hi,

On Fri, Apr 6, 2018 at 4:49 PM, Donald Stufft <don...@stufft.io> wrote:
>
> On Apr 6, 2018, at 11:25 AM, Matthew Brett <matthe...@gmail.com> wrote:
>
>
> As of 8th April, pip will break, largely in silence. As used with
> default command line options, pip will stop seeing anything on pypi.
> It will tell the user that it can't find packages that do exist on
> pypi, that that users are up to date for packages that have later
> versions on pypi - including pip. There are no messages to warn the
> user what has happened or how to fix it. It is easy to think you've
> upgraded to the latest version, when you have not.
>
>
> This isn’t quite accurate:
>
> This is not a blanket issue for anyone on macOS < 10.13. It *only* affects
> people using a Python that links against the ancient OpenSSL provided by
> Apple, of which the #1 cause is System Python on macOS < 10.13, however it
> is also the 2.7 Python.org macOS installer on any version of Python. All
> other forms of getting Python, to my knowledge, link against a newer version
> of OpenSSL (or against LibreSSL in System Python on macOS 10.13). [1]

I think that list is for current installers - then there's the tail of
older installs, of which I am one (a Python.org 3.5 install). Only
relevant, because this is going to be a complex patchwork of affected
users.

> This affects a small percentage of users, Of the downloads from PyPI in the
> last 7 days that originated from a macOS system, 6% of them would have
> failed, 94% of them would have succeeded.

6% of very confused users is a lot. But - I guess some proportion of
that 100% is continuous integration installs? These will tend to
have very up to date Pythons. Is there any way to estimate how many
non-bot users will be affected?

> Once we are at 100% unavailability, we can ask our CDN to disable TLSv1.0
> and TLSv1.1 completely, which will raise an OpenSSL error message instead of
> a HTTP error message. That will look like (ignore the index-url, used that
> because it already has TLSv1.0/1.1 disabled completely):
>
> $ /usr/local/bin/python2.7 -m pip install --index-url
> https://files.pythonhosted.org/ --upgrade pip
> Could not fetch URL https://files.pythonhosted.org/pip/: There was a
> problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION]
> tlsv1 alert protocol version (_ssl.c:661) - skipping
> Requirement already up-to-date: pip in
> /Library/Frameworks/Python.framework/Versions/2.7.14_10_6/lib/python2.7/site-packages
> You are using pip version 9.0.1, however version 9.0.3 is available.
> You should consider upgrading via the 'pip install --upgrade pip'
> command.

Yes - that's better than the current silent break, but still it's a
shame that the message is so obscure given the long lead-in.

> Finally, even If the above weren’t true, there really isn’t much we can do
> either way. We can’t go back in time and change the already released version
> of pip, and we only have so many ways that PyPI can signal an error to pip
> one is using an HTTP status code (what the brownouts do now) and one is
> rejecting the connection with a TLS error (what the above snippet does).
> Beyond that, any improvements require getting users to upgrade their version
> of pip, and if we can get them to upgrade for a better error message,
> presumably we can get them to upgrade for the SecureTransport fallback in
> 9.0.3+.

I was hoping against hope that there was another way of generating an
informative error from the server that would force itself on the
user's attention, perhaps by subverting another mechanism. Is there
really no way for pypi to transmit an informative error when it
detects a request from an affected system?

Cheers,

Matthew

Donald Stufft

unread,
Apr 6, 2018, 2:21:42 PM4/6/18
to Matthew Brett, pypa-dev
Not that I am aware of. Note— that is 6% of downloads, not 6% of users, that could have been a whole lot users each downloading a single package, or one user downloading thousands, we can’t tell.
No, there’s not. pip makes HTTP requests and there’s no place for extra metadata attached those requests except in the HTTP status code (which as you noted, pip swallows by default because historically we didn’t know if the URL was expected to work or not). The simple API wasn’t really designed, it evolved out of the primordial ooze.


Matthew Brett

unread,
Apr 6, 2018, 4:29:08 PM4/6/18
to Donald Stufft, pypa-dev
Hi again,
Ouch - it does look like we're in a very difficult situation, which
might apply to a significant number of users.

I think the worst option is what we have ramping up in the brownout,
which is the silent failure to upgrade or find packages. Yes, you
get something informative with the -v option if you think to try it,
but I think the confusion from the not -v state is of much greater
harm than the help from the -v message.

The better option, I believe, is just to go straight to the SSL error.
At least that will set people Googling. They might be annoyed, but at
least they won't be mystified.

Ideally, that would happen as late as possible, to give people the
maximum chance to upgrade with the usual pip upgrade commands. One of
the nasty features of this one is that it breaks the normal pip
upgrade method. Can we push back the SSL error until it's forced upon
us in June?

Cheers,

Matthew

Donald Stufft

unread,
Apr 6, 2018, 4:55:18 PM4/6/18
to Matthew Brett, pypa-dev

> On Apr 6, 2018, at 4:28 PM, Matthew Brett <matthe...@gmail.com> wrote:
>
>
> Ouch - it does look like we're in a very difficult situation, which
> might apply to a significant number of users.
>
> I think the worst option is what we have ramping up in the brownout,
> which is the silent failure to upgrade or find packages. Yes, you
> get something informative with the -v option if you think to try it,
> but I think the confusion from the not -v state is of much greater
> harm than the help from the -v message.
>
> The better option, I believe, is just to go straight to the SSL error.
> At least that will set people Googling. They might be annoyed, but at
> least they won't be mystified.
>
> Ideally, that would happen as late as possible, to give people the
> maximum chance to upgrade with the usual pip upgrade commands. One of
> the nasty features of this one is that it breaks the normal pip
> upgrade method. Can we push back the SSL error until it's forced upon
> us in June?
>

Not without delaying the launch of Warehouse until after June (at which point we will have already run out of MOSS grant money and won’t have people with paid, dedicated time on hand to sort out any issues that crop up from the launch). This is because portions of Warehouse are already mandatory TLSv1.2 and that can’t be rolled back (Warehouse is new enough on Fastly it never had TLSv1.0/1.1 support).

It’s not great, but it’s the least bad option we have right now.

Matthew Brett

unread,
Apr 6, 2018, 5:07:10 PM4/6/18
to Donald Stufft, pypa-dev
OK - so our hard deadline is the planned Warehouse launch on April
16th? I would argue for going straight to the SSL error at that
point, and turning off the current brownout and April 8th TLS 1.0 shut
down. Is that possible? Do other Macolytes agree with me that that
would be less confusing? In the mean time, would it be possible to
put out some big announcements following up on the originals, giving
the SSL error, to seed Google searches, and prime memories?

Cheers,

Matthew

Chris Jerdonek

unread,
Apr 6, 2018, 5:36:50 PM4/6/18
to Donald Stufft, Matthew Brett, pypa-dev
On Fri, Apr 6, 2018 at 11:21 AM Donald Stufft <don...@stufft.io> wrote:
No, there’s not. pip makes HTTP requests and there’s no place for extra metadata attached those requests except in the HTTP status code (which as you noted, pip swallows by default because historically we didn’t know if the URL was expected to work or not). The simple API wasn’t really designed, it evolved out of the primordial ooze.

Would it make sense to open an issue for future versions of pip to allow such metadata to be attached and displayed, or is there already such an issue?

—Chris





Erik Bray

unread,
Apr 9, 2018, 1:33:03 PM4/9/18
to Chris Jerdonek, Donald Stufft, Matthew Brett, pypa-dev
I was going to suggest the same--while it would be too late to help in
this particular case (and as Donald already convincingly it explained
it probably won't have huge impact), this case, others I can think of
before it, and others that are likely to occur in the future would
have been well-served by the ability of PyPI administrators to set
arbitrary broadcast messages (a MotD if you will) to send along with
HTTP responses from PyPI (they could even go in an HTTP header,
perhaps).

Best,
E

Donald Stufft

unread,
Apr 9, 2018, 2:28:57 PM4/9/18
to Erik Bray, Chris Jerdonek, Matthew Brett, pypa-dev
It wouldn’t be a pip issue, it’d be a distutils-sig discussion and an amendment to PEP 503. The primary concern would be that as we move to a world where metadata is signed to prevent a malicious repository from attacking users, that this is another avenue that an attacker could take to trick end users (imagine a MOTD that says to pip install malicious-package).

That doesn’t make it impossible, or a bad idea though! It just is something to consider, and the best avenue is distutils-sig.

Donald Stufft

unread,
Apr 9, 2018, 2:43:06 PM4/9/18
to Matthew Brett, pypa-dev

On Apr 6, 2018, at 5:06 PM, Matthew Brett <matthe...@gmail.com> wrote:

OK - so our hard deadline is the planned Warehouse launch on April
16th?   I would argue for going straight to the SSL error at that
point, and turning off the current brownout and April 8th TLS 1.0 shut
down.  Is that possible?   Do other Macolytes agree with me that that
would be less confusing?  In the mean time, would it be possible to
put out some big announcements following up on the originals, giving
the SSL error, to seed Google searches, and prime memories?


We’ve modified the plan so that instead of the brownout style error lasting until the 16th, we’re going to switch to the hard failure tomorrow with the 100% brownout failure happening today (and yesterday). We didn’t want to move straight to the hard failure incase we needed to roll it back for some reason. We don’t want to wait until the 16th to avoid lumping too many changes onto a single day (so we don’t have to deal with potential fallout of too many different changes on a single day).

Hopefully that works for everyone.

Chris Jerdonek

unread,
Apr 10, 2018, 12:59:47 AM4/10/18
to Donald Stufft, Erik Bray, Matthew Brett, pypa-dev
I raised this suggestion on distutils-sig, FYI:
—Chris


Sumana Harihareswara

unread,
Apr 17, 2018, 11:03:36 PM4/17/18
to Donald Stufft, Matthew Brett, pypa-dev
Users with older TLS support libraries (on various operating systems)
are currently seeking support as they discover breakage, and we're
seeing their support requests on IRC and in support requests filed as
issues in multiple GitHub repositories (for virtualenv, pip, Warehouse,
and pipenv, among others). I gathered some common symptoms and created a
help item in the pypi.org FAQ: https://pypi.org/help/#tls-deprecation .
Patch: https://github.com/pypa/warehouse/pull/3720 .

In March, I was deciding whether to make a big publicity push about the
TLS deprecation, and I decided not to spend several hours writing and
publicizing announcements about it (and to instead focus on other
Warehouse project management work). I made this decision (noted in
https://mail.python.org/pipermail/distutils-sig/2018-March/032067.html )
based on my own misunderstanding about urgency (I believe I had not yet
realized that the new deadline would be, at the latest, the Warehouse
cutover, not June 30th) and based on percentage-type predictions about
how few users would be affected by the cutover (I didn't think hard
enough about how many people that would represent, and how hard to
diagnose the breakage would be for them). I regret my earlier decision.
I have now in fact spent several hours supporting users, writing
documentation, and reaching out to people with platforms in the macOS
user community to raise awareness and perhaps speed a platform-level
solution, and I hope this partially makes up for my earlier mistake.

--
Sumana Harihareswara
Warehouse project manager
Changeset Consulting
https://changeset.nyc
Reply all
Reply to author
Forward
0 new messages