PyPI/Warehouse: infrastructure hardening & the CAPTCHA conundrum

[Sumana Harihareswara]

So we aren't quite at beta yet, but we'll be shouting about pypi.org *really soon*. We have nearly all the Warehouse improvements we need for beta, and nearly all the infrastructure improvements we believe we'll need for the switchover.

I'll tell you how you can help, then talk about the current state of things.

• The big blocker keeping us from beta: China & CAPTCHAs. Help advise us.
  
• Comment on a "needs discussion" issue.
  
• Help us with large-scope JavaScript issues, like our frontend testing approach.
  
• Please talk with Nicole about being a subject or interviewer for user tests.
  
• Tell me if you're planning to join us at sprints at PyCon or EuroPython.
  
• Check out our open good first Warehouse issues (we usually have 10+ open) and get started.
  

If you follow https://status.python.org/  you saw we did some load testing last week and learned from it! We redirected some traffic, for a few periods, for `pip install`, from the old server to Warehouse, and learned from it. For instance, people running Ubuntu 14.04 LTS (long term service release) are usually using a pretty old version of pip, and people on some versions of the Mac OS have older versions of Python and old versions of security-related libraries that don't support the version of TLS that we want them to use. Ernest, Donald and Dustin did a bunch of work addressing this, including Donald putting out pip 9.0.2.

(A thing to understand about Ernest's continuing work on PyPI and distribution infrastructure is that it's in a lot of places. It's cabotage & a test cabotage app, configuration with salt, conveyor, pip & get-pip, and he filed a bug in Kubernetes which I personally find particularly impressive. And it's in user-facing communication in IRC and GitHub comments and on our statuspage and Twitter, plus a lot of internal discussion with infrastructure colleagues. I have a harder time gathering links for Ernest's work for these emails than for my other teammates; regrets.)

As usual, a summary of the past week's work is in our meeting notes. We have new features like letting PyPI administrators add new trove classifiers easily, infrastructure improvements like this complexity reduction,  ton of polish and bug fixing around layout, description content types (Markdown!), a FAQ restructuring, a more useful collaboration page, etc. And we reviewed and merged a lot of volunteers' pull requests!

Thanks to our prolific volunteers:

• pgadige making sure an error message reflects whether you're on PyPI or Test PyPI
  

• waseem18 providing an error message for the password reset
  

• cryvate fixing form requirements for password reset
  

• waseem18 fixing disabled button CSS
  

• yeraydiazdiaz fixing modal window behavior, then refixing
  

• berkerpeksag adding a "public profile" link to the user dropdown
  

• Mariatta sending notification email when a project collaborator's added
  

• berkerpeksag hiding the "view project" button for no-release-yet projects in maintainers' project lists
  

• alexwlchan renaming a CSS class for consistency
  

• jMuzsik improving documentation of owners' and maintainers' privileges
  

• yeraydiazdiaz adding JavaScript validation to show the user if "new password" and "confirm new password" don't match
  

• alexwlchan documenting all the modifiers in our SASS directory
  

• alanbato and yeraydiazdiaz adding a check to stop someone from uploading a file whose blake2 hash matches an already-uploaded file
  

• cryvate improving sorting of package versions in our /simple/ API
  

• jMuzsik improving how PyPI links look on Twitter, adding an image to our Twitter cards
  

• 9999years updating the Python Packaging User Guide and sample project for Markdown/PEP 566
  

And thanks to our many bug reporters, especially those who helped us learn from our load tests.

Also, check out discussion on API key support/macaroons,  supporting GitHub-flavored Markdown as Description-Content-Type,  and project rating/ranking/stars.

And finally, we are ever closer to accepting PEP 541 (and planning followup tasks) and are testing our PEP 566 compliance. And I may start a PEP for a Python package index upload API specification.

More next week, as usual.

Thanks to Mozilla for their support for the PyPI & Warehouse work!

--

Sumana Harihareswara

Warehouse project manager

Changeset Consulting

s...@changeset.nyc