PEP 480 status, getting it ready to be reopened for discussion

[Sumana Harihareswara]

Summary: TUF and in-toto experts are taking a fresh look at PEP 480 to start re-circulating it for comment and approval, to reduce implementation delays.

Details:

Once the TUF implementation in PyPI is complete (maybe in a few months?), it will make sense to start implementing TUF support in pip, and to start work on securing the developer-uploading-to-PyPI toolchain.

There's no policy blocker to implementing TUF support in pip -- that's basically covered in PEP 458, which is already approved.

PEP 480  https://www.python.org/dev/peps/pep-0480/  suggests securing Python's developer-to-user package pipeline with cryptographically signed metadata (probably using TUF and in-toto). It is in Deferred status:

"Due to the amount of work required to implement this PEP, it is deferred until appropriate funding can be secured to implement the PEP."

but that's what the status for PEP 458 said, and then it turned out that funding wasn't the only issue -- there were wording and policy things that PEP authors needed to discuss with the Python community, too. (See
https://discuss.python.org/t/pep-458-secure-pypi-downloads-with-package-signing/2648
for the discussion.) For both PEP 458 and PEP 480, after a bunch of discussion from 2013 to 2018, there was no clear community consensus that said "yes we want this but we need to defer it for funding reasons" -- marking them as such was a private decision that one person made in early 2019.

So, to help avoid delays in implementation, now's a good time to take a fresh look at PEP 480, make necessary revisions, and circulate it for fresh approval. As such, Marina Moore has just updated it https://github.com/python/peps/pull/1681 and I've suggested a few further improvements https://github.com/python/peps/pull/1693 , including moving it out of Deferred and into Draft status. Once this is finished, I suggest the PEP authors start a thread on the Python Discourse to re-start the approval process.

The PSF's Packaging Working Group maintains a list of "fundable" improvements: https://github.com/psf/fundable-packaging-improvements and one of the criteria for getting on the list is: "there's already consensus among project maintainers, and if a PEP is involved it's been accepted." Once PEP 480 is accepted, someone can submit a pull request to add PEP 480 implementation to that list, which will help find implementation funding.

-Sumana

-- 
Sumana Harihareswara
Changeset Consulting
https://changeset.nyc

[Joshua Lock]

On 27 Oct 2020, at 21:27, Sumana Harihareswara <con...@changeset.nyc> wrote:

Summary: TUF and in-toto experts are taking a fresh look at PEP 480 to start re-circulating it for comment and approval, to reduce implementation delays. 

Cool stuff!

Details:

Once the TUF implementation in PyPI is complete (maybe in a few months?), it will make sense to start implementing TUF support in pip, and to start work on securing the developer-uploading-to-PyPI toolchain.

FYI a proof of concept and proposed design for the pip integration, along with some important discussion items, has been submitted to the pip community: https://github.com/pypa/pip/pull/9041

There's no policy blocker to implementing TUF support in pip -- that's basically covered in PEP 458, which is already approved.

PEP 480  https://www.python.org/dev/peps/pep-0480/  suggests securing Python's developer-to-user package pipeline with cryptographically signed metadata (probably using TUF and in-toto).

I just read through the PEP and in-toto is not explicitly mentioned. Is there a plan to include it in PEP 480? Or perhaps add in-toto as a follow-on/separate PEP? If the PyPI build farm discussed in "Appendix A: PyPI Build Farm and End-to-End Signing” is a reality, that would be a very compelling place to integrate in-toto.

So, to help avoid delays in implementation, now's a good time to take a fresh look at PEP 480, make necessary revisions, and circulate it for fresh approval. As such, Marina Moore has just updated it https://github.com/python/peps/pull/1681 and I've suggested a few further improvements https://github.com/python/peps/pull/1693 , including moving it out of Deferred and into Draft status. Once this is finished, I suggest the PEP authors start a thread on the Python Discourse to re-start the approval process.

I’m interested in helping get the PEP into shape before it is circulated for approval. What is the best way to collaborate on that? I don’t want to start submitting PRs to the peps repository without at least agreeing to direction with whomever is now leading the PEP authoring efforts. Is that Marina?

Some high-level proposals for changes to the PEP:

• I’d be inclined to split the implementation section of the PEP into two major sections: i) the changes required to warehouse, PyPI, and the administrator workflows. ii) the developer workflow, key management, and tool integrations.
• The current text of the PEP feels a little undecided in its recommendations. The general tone of the PEP implies that developer signing of packages should be required (for all newly uploaded packages?), but allows for a hybrid model of repository signed and optional developer signed packages. I think the PEP would be more persuasive and coherent if it advocated for a single approach throughout; with an appendix section suggesting the alternative, and what sections of the PEP would need to be addressed were the Python community to prefer the alternative.
• It might make sense to work with the BDFL delegate on which position to advocate for?
• The PEP recommends a key management solution like miniLock. But the miniLock repository has disappeared, and the implementations of miniLock in different languages appear to be mostly abandonware. I think the suggestion of using a system like miniLock is a good one. Are there problems with miniLock itself that resulted in it fading into obscurity? Is there a more appropriate/modern alternative? Is the fundamental construct used to derive secret keys in miniLock sound enough to use in the PEP? It would be good to have a concrete recommendation here.
• Relatedly, the PEP seems to suggest private keys per-machine for a developer, but doesn’t address how this might affect threshold computations for a delegation.
• Also, while the PEP suggests that signing keys should not be shared across machines, it makes some recommendations to supporting shared keys. It would probably benefit PEP readers if, as above with developer signed vs. hybrid, we pick one position and use it consistently throughout the PEP. With the alternative in an appendix.

I have several less significant changes noted that I can suggest either as a follow-up or directly against the peps repository, but wanted to propose the above first.

Somewhat related to this PEP, I see a bunch of security features coming to PyPI and I think it would be very valuable to PyPI users if the security of packages were more obvious. I think there are potentially four levels of trust:

1. PyPI signed packages, password authenticated upload (PEP 458)
2. PyPI signed packages, 2FA upload (though, afaik, this is not yet implemented)
3. Developer signed packages (PEP 480)
4. PyPI verified & developer signed packages (Build farm and/or in-toto)

It would be worth considering UX to indicate the trust, and even encourage developers to opt in to the trust model PyPI prefer. For example, if developer-signed packages are optional – signed packages could be sorted first in search results, and have a visual indicator of higher trustworthiness on the project listings. That might encourage greater adoption amongst package uploaders who might otherwise feel signing packages isn’t worth the effort.

Sumanah, any suggestions on how best to explore this idea with the PyPA community?

Look forward to collaborating on the PEP.

Thanks,

Joshua

-- 
Joshua Lock
VMware Open Source Technology Center

[Sumana Harihareswara]

I think Marina's the best person to align with on significant PEP collaboration, yes. Also, if you could review https://github.com/python/peps/pull/1693 with a suggested fix for the registration-related language, that would help me out!

That UX idea is neat and I think you should file a new issue for it in the Warehouse repository, and when you file it, you should link to https://github.com/pypa/warehouse/issues/6236 and https://github.com/pypa/warehouse/issues/194 .

[Trishank Kuppusamy]

On Tuesday, November 3, 2020 at 9:54:31 PM UTC+8 jl...@vmware.com wrote:

I just read through the PEP and in-toto is not explicitly mentioned. Is there a plan to include it in PEP 480? Or perhaps add in-toto as a follow-on/separate PEP? If the PyPI build farm discussed in "Appendix A: PyPI Build Farm and End-to-End Signing” is a reality, that would be a very compelling place to integrate in-toto.

It's very easy to add in-toto support. As long as PEP 458/480 support custom targets metadata, and we add in-toto verification on the pip side, there is not much to do on the PyPI side except to upload in-toto metadata, which I suppose is as easy as uploading any other distribution/release file. Please see ITE-2 for more details.

[Marina Moore]

On Tue, Nov 3, 2020 at 5:54 AM Joshua Lock <jl...@vmware.com> wrote:

So, to help avoid delays in implementation, now's a good time to take a fresh look at PEP 480, make necessary revisions, and circulate it for fresh approval. As such, Marina Moore has just updated it https://github.com/python/peps/pull/1681 and I've suggested a few further improvements https://github.com/python/peps/pull/1693 , including moving it out of Deferred and into Draft status. Once this is finished, I suggest the PEP authors start a thread on the Python Discourse to re-start the approval process.

I’m interested in helping get the PEP into shape before it is circulated for approval. What is the best way to collaborate on that? I don’t want to start submitting PRs to the peps repository without at least agreeing to direction with whomever is now leading the PEP authoring efforts. Is that Marina?

I'd be happy to take this on. It might be nice to start with a meeting with everyone interested in collaborating so that we can decide what needs to be done and break it down into prs from there.

 

Some high-level proposals for changes to the PEP:

• I’d be inclined to split the implementation section of the PEP into two major sections: i) the changes required to warehouse, PyPI, and the administrator workflows. ii) the developer workflow, key management, and tool integrations.

That sounds reasonable, I think that will make the implementation more clear.

• The current text of the PEP feels a little undecided in its recommendations. The general tone of the PEP implies that developer signing of packages should be required (for all newly uploaded packages?), but allows for a hybrid model of repository signed and optional developer signed packages. I think the PEP would be more persuasive and coherent if it advocated for a single approach throughout; with an appendix section suggesting the alternative, and what sections of the PEP would need to be addressed were the Python community to prefer the alternative.
• It might make sense to work with the BDFL delegate on which position to advocate for?

This is an important decision, especially because there may be some scalability problems if all developers sign packages with the current model as signed packages are all delegated from the claimed role. If a majority of developers are expected to sign packages, we should discuss what hashed bin delegations will look like with offline keys.

• The PEP recommends a key management solution like miniLock. But the miniLock repository has disappeared, and the implementations of miniLock in different languages appear to be mostly abandonware. I think the suggestion of using a system like miniLock is a good one. Are there problems with miniLock itself that resulted in it fading into obscurity? Is there a more appropriate/modern alternative? Is the fundamental construct used to derive secret keys in miniLock sound enough to use in the PEP? It would be good to have a concrete recommendation here.

I'll have to look more into this, but I remember that we referred to YubiHSM for key management in PEP 458

• Relatedly, the PEP seems to suggest private keys per-machine for a developer, but doesn’t address how this might affect threshold computations for a delegation.
• Also, while the PEP suggests that signing keys should not be shared across machines, it makes some recommendations to supporting shared keys. It would probably benefit PEP readers if, as above with developer signed vs. hybrid, we pick one position and use it consistently throughout the PEP. With the alternative in an appendix.

I'd argue that we can avoid all key sharing through the use of delegations. We should make sure that's clear.

This list of changes is a great place to start breaking down what we need to do going forward. I think we can also go through the sections of PEP 480 about the snapshot process and consistent snapshots to remove any redundancy with PEP 458.

Thanks,

Marina

[Joshua Lock]

On 3 Nov 2020, at 19:16, Marina Moore <mm9...@nyu.edu> wrote:

On Tue, Nov 3, 2020 at 5:54 AM Joshua Lock <jl...@vmware.com> wrote:

So, to help avoid delays in implementation, now's a good time to take a fresh look at PEP 480, make necessary revisions, and circulate it for fresh approval. As such, Marina Moore has just updated it https://github.com/python/peps/pull/1681 and I've suggested a few further improvements https://github.com/python/peps/pull/1693 , including moving it out of Deferred and into Draft status. Once this is finished, I suggest the PEP authors start a thread on the Python Discourse to re-start the approval process.

I’m interested in helping get the PEP into shape before it is circulated for approval. What is the best way to collaborate on that? I don’t want to start submitting PRs to the peps repository without at least agreeing to direction with whomever is now leading the PEP authoring efforts. Is that Marina?

I'd be happy to take this on. It might be nice to start with a meeting with everyone interested in collaborating so that we can decide what needs to be done and break it down into prs from there.

I’d be happy to join a meeting, or to try and kick things off asynchronously.

The workflow we had with PEP 458, where there was a fork of the PEP on GitHub with Issues enabled, worked quite well. Though I seem to recall the Python folks would have preferred to see more granular submissions to the upstream peps repository.

Some high-level proposals for changes to the PEP:

• I’d be inclined to split the implementation section of the PEP into two major sections: i) the changes required to warehouse, PyPI, and the administrator workflows. ii) the developer workflow, key management, and tool integrations.

That sounds reasonable, I think that will make the implementation more clear.

• The current text of the PEP feels a little undecided in its recommendations. The general tone of the PEP implies that developer signing of packages should be required (for all newly uploaded packages?), but allows for a hybrid model of repository signed and optional developer signed packages. I think the PEP would be more persuasive and coherent if it advocated for a single approach throughout; with an appendix section suggesting the alternative, and what sections of the PEP would need to be addressed were the Python community to prefer the alternative.
• It might make sense to work with the BDFL delegate on which position to advocate for?

This is an important decision, especially because there may be some scalability problems if all developers sign packages with the current model as signed packages are all delegated from the claimed role. If a majority of developers are expected to sign packages, we should discuss what hashed bin delegations will look like with offline keys.

There’s also the scaling factor of impact on the PyPI administrators. If *all* projects on PyPI require a delegation before they can make new releases (because developer signing is mandatory), that’s a lot of key maintenance work. There would also be an aspect of ongoing work helping developers revoke keys and setting up delegations for new projects.

• The PEP recommends a key management solution like miniLock. But the miniLock repository has disappeared, and the implementations of miniLock in different languages appear to be mostly abandonware. I think the suggestion of using a system like miniLock is a good one. Are there problems with miniLock itself that resulted in it fading into obscurity? Is there a more appropriate/modern alternative? Is the fundamental construct used to derive secret keys in miniLock sound enough to use in the PEP? It would be good to have a concrete recommendation here.

I'll have to look more into this, but I remember that we referred to YubiHSM for key management in PEP 458

PEP 458 uses YubiHSM and NitroKeys for the top-level roles. The miniLock suggestion in PEP 480 is to have a simple key management solution for developers signing their own packages. It’s probably not reasonable to expect all developers uploading packages to PyPI to have a security key.

The suggestion in the PEP to have mechanism for deriving keys from a passphrase is a good one, I think. But for the PEP to be actionable it would be nice to make this recommendation more actionable.

• Relatedly, the PEP seems to suggest private keys per-machine for a developer, but doesn’t address how this might affect threshold computations for a delegation.
• Also, while the PEP suggests that signing keys should not be shared across machines, it makes some recommendations to supporting shared keys. It would probably benefit PEP readers if, as above with developer signed vs. hybrid, we pick one position and use it consistently throughout the PEP. With the alternative in an appendix.

I'd argue that we can avoid all key sharing through the use of delegations. We should make sure that's clear.

This list of changes is a great place to start breaking down what we need to do going forward. I think we can also go through the sections of PEP 480 about the snapshot process and consistent snapshots to remove any redundancy with PEP 458.

Sounds good.

Joshua

[Trishank Kuppusamy]

On Wed, Nov 4, 2020 at 10:16 PM Joshua Lock <jl...@vmware.com> wrote:

There’s also the scaling factor of impact on the PyPI administrators. If *all* projects on PyPI require a delegation before they can make new releases (because developer signing is mandatory), that’s a lot of key maintenance work. There would also be an aspect of ongoing work helping developers revoke keys and setting up delegations for new projects.

In the Diplomat paper (Section 7), we talk about how to get 80% of the bang with 20% of the buck with policies such as targeting the top few % of most downloaded projects. Please do see the evaluation in the paper for more details, which will need an update for the most recent statistics.

 

PEP 458 uses YubiHSM and NitroKeys for the top-level roles. The miniLock suggestion in PEP 480 is to have a simple key management solution for developers signing their own packages. It’s probably not reasonable to expect all developers uploading packages to PyPI to have a security key.

The suggestion in the PEP to have mechanism for deriving keys from a passphrase is a good one, I think. But for the PEP to be actionable it would be nice to make this recommendation more actionable.

 Yes, we do need to think a bit more about how to make offline key management simple for both the offline delegations on PyPI, and developers.

[Sumana Harihareswara]

On Wednesday, November 4, 2020 at 9:16:44 AM UTC-5 Joshua wrote:

The workflow we had with PEP 458, where there was a fork of the PEP on GitHub with Issues enabled, worked quite well. Though I seem to recall the Python folks would have preferred to see more granular submissions to the upstream peps repository.

Please do as much as possible in https://github.com/python/peps and, every time there's a new pull request or a substantial new review of the pull request, mention it in the discuss.python.org thread . The workflow for PEP 458, in my view, didn't work well, in that it was fairly opaque to some of your colleagues in Python packaging and endangered the authors' credibility.

One of you should fairly immediately (as in, this week) start a thread in https://discuss.python.org/c/peps/19 announcing that you are updating the draft PEP and intend to re-submit it for comment and approval.

Do as much work as possible in discussions on Discourse, and be open to your colleagues in the larger packaging world doing things like suggesting specific textual changes and reviewing/asking for review in Discourse comments, even though we’re generally used to doing that work in GitHub pull request reviews when we work together on code.

When a community member asks a question or shares a criticism of the existing proposal, it’s better to have a back-and-forth conversation with lots of little iterative questions and answers, instead of the proposal author coming back with a multi-paragraph essay or proposal revision. Yes, even if that feels spammy to you at first and means the thread ends up being hundreds of posts.

-Sumana

[Marina Moore]

I will post to discuss later today.

Maybe we can use the issue tracker on the https://github.com/secure-systems-lab/peps to prioritize what work needs to be done, then open pull requests directly to python?

--
You received this message because you are subscribed to the Google Groups "The Update Framework (TUF)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theupdateframew...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/theupdateframework/046c2a5a-17bc-49cf-a075-1d898f4970d6n%40googlegroups.com.

[Marina Moore]

Link to the thread: https://discuss.python.org/t/pep-480-surviving-a-compromise-of-pypi-end-to-end-signing-of-packages/5666

[Joshua Lock]

On 4 Nov 2020, at 14:25, Trishank Kuppusamy <trishank....@datadoghq.com> wrote:

On Wed, Nov 4, 2020 at 10:16 PM Joshua Lock <jl...@vmware.com> wrote:

There’s also the scaling factor of impact on the PyPI administrators. If *all* projects on PyPI require a delegation before they can make new releases (because developer signing is mandatory), that’s a lot of key maintenance work. There would also be an aspect of ongoing work helping developers revoke keys and setting up delegations for new projects.

In the Diplomat paper (Section 7), we talk about how to get 80% of the bang with 20% of the buck with policies such as targeting the top few % of most downloaded projects. Please do see the evaluation in the paper for more details, which will need an update for the most recent statistics.

Nice! That aligns with some other research into securing the NPM ecosystem (in the npm case, it’s not that securing a smaller number of packages would get 80% band for 20% buck, but that most of the risk lies with a small number of maintainers and improving their security posture would get 80% bang for 20% buck).

Do you have any more detailed notes on how those numbers were generated (or scripts), so that we can update them?

Thanks,

Joshua

[Joshua Lock]

On 6 Nov 2020, at 14:41, Joshua Lock <jl...@vmware.com> wrote:

On 4 Nov 2020, at 14:25, Trishank Kuppusamy <trishank....@datadoghq.com> wrote:

On Wed, Nov 4, 2020 at 10:16 PM Joshua Lock <jl...@vmware.com> wrote:

There’s also the scaling factor of impact on the PyPI administrators. If *all* projects on PyPI require a delegation before they can make new releases (because developer signing is mandatory), that’s a lot of key maintenance work. There would also be an aspect of ongoing work helping developers revoke keys and setting up delegations for new projects.

In the Diplomat paper (Section 7), we talk about how to get 80% of the bang with 20% of the buck with policies such as targeting the top few % of most downloaded projects. Please do see the evaluation in the paper for more details, which will need an update for the most recent statistics.

Nice! That aligns with some other research into securing the NPM ecosystem (in the npm case, it’s not that securing a smaller number of packages would get 80% band for 20% buck, but that most of the risk lies with a small number of maintainers and improving their security posture would get 80% bang for 20% buck).

Citation needed: https://www.usenix.org/conference/usenixsecurity19/presentation/zimmerman and the tl;dr at https://blog.acolyer.org/2019/09/30/small-world-with-high-risks/

Joshua

[Trishank Kuppusamy]

On Fri, Nov 6, 2020 at 10:41 PM Joshua Lock <jl...@vmware.com> wrote:

Do you have any more detailed notes on how those numbers were generated (or scripts), so that we can update them?

Yes, sir: https://github.com/trishankkarthik/diplomat-experiments