K8s release managers updates

41 views
Skip to first unread message

Tim Pepper

unread,
Jul 22, 2019, 7:10:55 PM7/22/19
to kubernetes-se...@googlegroups.com, Caleb Miles, Stephen Augustus

During the past year as a release team member and SIG Release chair I've been working to get a better handle on the project's release process and enable a bit broader set of folks to participate.  Our goal in changes is to have better response times, more reliable processes, and not be burning folks out.

 

We've shifted from each patch release stream (ie: 1.12 and prior) having one person handling all of the patch/build/release flow for nine months to instead have a team collaborating across time zones on all supported branches:

https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md 

This team is staffed by folks who've previously been branch managers.  While a team, this is still a very limited set of hands on the steering wheel.  Today this is:

  • Aleksandra Malinowska (@aleksandra-malinowska)
  • Hannes Hörl (@hoegaarden)
  • Pengfei Ni (@feiskyer)
  • Tim Pepper (@tpepper)

all of whom are of course signed off on the embargo process.

 

But at the link above you'll also see reference to this newer page:

https://github.com/kubernetes/sig-release/blob/master/release-managers.md

and there you'll see quite a few more names and roles on our side.


And there we'd like to engage in conversation with the Product Security Committee and community on a couple "security release team" topics:

 

  1. Patch Release Team: The security release team includes the above four patch managers each by name, though management of their membership is delegated to us, with manager priv's on the security google group.  We have our own kubernetes-releas...@googlegroups.com nowadays.  Would it be better separation of concerns to have our list nested in the security list and us not have manager priv's on your Google Group?  And if so, should we add a security team member to our list's mgmt ACL for ya'll to be able to audit our membership?
  2. There are a few places additional folks on our side may to some extent come in contact with PSC embargoed information, while not formally members of the security release team.  We don't think any of these three sets of folks need explicitly included in the private groups, but we want to remind PSC they may come into security conversations and work, ask for an ACK that we can manage this on our side with all involved abiding the embargo process, and ask for any other guidance or requirement PSC might have:
    • SIG Release Chairs: Potentially can access the embargoed chain as Google Group and Slack admins for SIG Release's subprojects.  Also generally a privileged reviewer/approver set of folks in the build/release pipeline.
    • Branch Managers: Each release team lead is on the private security list for their three month release cycle.  Branch manager(s) within each release team are currently not on the security private list and this should be ok as they're primarily engaged in master branch public work.  But there is a chance late in a release cycle a CVE needs quick handling and the branch managers are pulled in.
    • Build Admins: There are a few parts of the build/release process which still require a Googler to press some buttons.  These individuals are pulled in at the point we are planning a date on which to make a release.  They generally do not have or need access to CVE detail information, but will be involved in the private release managers' discussions to insure specific patch artifacts are correctly built and published on the target date.
  1. Associates:  We're starting our own associate program.  We want to work additional folks through the progression from helper, to privileged on upcoming release (master alpha/beta/rc), to privileged on stable release branches.  This will give us a pipeline of helpers with time to establish trust.  By default the associates would not be involved with embargoed flows until they've graduated up into branch or patch release management so this is roughly as per today.  Are there additional considerations the PSC would like us to make here?

I've also noted a broken url in the Security Release Process document related to this so have also opened a PR to update the docs: https://github.com/kubernetes/security/pull/42

 

 

-- 

Tim Pepper

Orchestration & Containers Lead

VMware Open Source Technology Center

Tim Allclair

unread,
Jul 22, 2019, 7:31:59 PM7/22/19
to Tim Pepper, kubernetes-se...@googlegroups.com, Caleb Miles, Stephen Augustus
Thanks for bringing this up, Tim.

On Mon, Jul 22, 2019 at 4:10 PM 'Tim Pepper' via kubernetes-security-discuss <kubernetes-se...@googlegroups.com> wrote:

During the past year as a release team member and SIG Release chair I've been working to get a better handle on the project's release process and enable a bit broader set of folks to participate.  Our goal in changes is to have better response times, more reliable processes, and not be burning folks out.

 

We've shifted from each patch release stream (ie: 1.12 and prior) having one person handling all of the patch/build/release flow for nine months to instead have a team collaborating across time zones on all supported branches:

https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md 

This team is staffed by folks who've previously been branch managers.  While a team, this is still a very limited set of hands on the steering wheel.  Today this is:

  • Aleksandra Malinowska (@aleksandra-malinowska)
  • Hannes Hörl (@hoegaarden)
  • Pengfei Ni (@feiskyer)
  • Tim Pepper (@tpepper)

all of whom are of course signed off on the embargo process.

 

But at the link above you'll also see reference to this newer page:

https://github.com/kubernetes/sig-release/blob/master/release-managers.md

and there you'll see quite a few more names and roles on our side.


And there we'd like to engage in conversation with the Product Security Committee and community on a couple "security release team" topics:

 

  1. Patch Release Team: The security release team includes the above four patch managers each by name, though management of their membership is delegated to us, with manager priv's on the security google group.  We have our own kubernetes-releas...@googlegroups.com nowadays.  Would it be better separation of concerns to have our list nested in the security list and us not have manager priv's on your Google Group?  And if so, should we add a security team member to our list's mgmt ACL for ya'll to be able to audit our membership?
I suggest we create a new group dedicated to discussing security releases. This means release managers aren't burdened by triage noise, and also narrows the scope of people initially aware of sensitive vulnerabilities until we're ready to discuss the release. We could nest both the security group & release manager group in this list to reduce management overhead.
  1. There are a few places additional folks on our side may to some extent come in contact with PSC embargoed information, while not formally members of the security release team.  We don't think any of these three sets of folks need explicitly included in the private groups, but we want to remind PSC they may come into security conversations and work, ask for an ACK that we can manage this on our side with all involved abiding the embargo process, and ask for any other guidance or requirement PSC might have:
    • SIG Release Chairs: Potentially can access the embargoed chain as Google Group and Slack admins for SIG Release's subprojects.  Also generally a privileged reviewer/approver set of folks in the build/release pipeline.
    • Branch Managers: Each release team lead is on the private security list for their three month release cycle.  Branch manager(s) within each release team are currently not on the security private list and this should be ok as they're primarily engaged in master branch public work.  But there is a chance late in a release cycle a CVE needs quick handling and the branch managers are pulled in.
    • Build Admins: There are a few parts of the build/release process which still require a Googler to press some buttons.  These individuals are pulled in at the point we are planning a date on which to make a release.  They generally do not have or need access to CVE detail information, but will be involved in the private release managers' discussions to insure specific patch artifacts are correctly built and published on the target date.
Security releases are need-to-know, but if someone needs to know, they can be in the loop :)  That said, it's good to keep the scope in mind as these individuals are pulled into various parts of the process, and periodically audit anywhere that persistent or cross-cutting access is granted.
  1. Associates:  We're starting our own associate program.  We want to work additional folks through the progression from helper, to privileged on upcoming release (master alpha/beta/rc), to privileged on stable release branches.  This will give us a pipeline of helpers with time to establish trust.  By default the associates would not be involved with embargoed flows until they've graduated up into branch or patch release management so this is roughly as per today.  Are there additional considerations the PSC would like us to make here?
No, this sounds in-line with our own associate process. I agree that associates should not be a part of the regular security release group, but I'm open to them being pulled into specific releases on a 1-off basis, either because they have domain knowledge (see above), or to get experience with low-severity vulnerabilities.
 

I've also noted a broken url in the Security Release Process document related to this so have also opened a PR to update the docs: https://github.com/kubernetes/security/pull/42


Thanks!

 

 

-- 

Tim Pepper

Orchestration & Containers Lead

VMware Open Source Technology Center

--
You received this message because you are subscribed to the Google Groups "kubernetes-security-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-security...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-security-discuss/5BE25C0D-4C3D-426D-BAA8-6319B34B3890%40vmware.com.

Stephen Augustus

unread,
Jul 23, 2019, 12:32:02 AM7/23/19
to Tim Allclair, Tim Pepper, kubernetes-se...@googlegroups.com, Caleb Miles
To reframe this slightly, here are some near term goals in my head:
  • Ensure SIG Release Chairs own all SIG Release assets
  • Dedupe SIG Release communication channels where possible/appropriate
  • Grant access to release GCP projects via Google Groups, instead of per user
So concretely, what I'd like to do:

Does that all sound reasonable to get your help with?

-- Stephen

p.s. I'm not married to any of the mailing list names; we can definitely workshop them, if needed.

Stephen Augustus

unread,
Jul 25, 2019, 8:33:26 PM7/25/19
to Tim Allclair, Tim Pepper, kubernetes-se...@googlegroups.com, Caleb Miles
Just wanted to ping again on moving this forward.
Caleb and I agree to uphold the security embargo.

Screenshot from 2019-07-25 20-09-47.png


-- Stephen

Stephen Augustus

unread,
Jul 29, 2019, 2:34:20 PM7/29/19
to comm...@kubernetes.io, steering, Tim Allclair, Tim Pepper, kubernetes-se...@googlegroups.com, Caleb Miles
Community / Steering--

Can one of you lovely folks help out with creating release-...@kubernetes.io and adding me as a manager/owner for that (ste...@agst.us)?
Context is below, with active convo happening on https://github.com/kubernetes/sig-release/issues/732.

-- Stephen

Joe Beda

unread,
Aug 6, 2019, 1:56:24 PM8/6/19
to Stephen Augustus, comm...@kubernetes.io, steering, Tim Allclair, Tim Pepper, kubernetes-se...@googlegroups.com, Caleb Miles

Hey Stephen – I thought I created this yesterday.  You still need something here?  Or am I confused?

 

Joe

 

From: Stephen Augustus <Ste...@agst.us>
Date: Tuesday, August 6, 2019 at 10:03 AM
To: "comm...@kubernetes.io" <comm...@kubernetes.io>, steering <stee...@kubernetes.io>
Cc: Tim Allclair <tall...@google.com>, Tim Pepper <tpe...@vmware.com>, "kubernetes-se...@googlegroups.com" <kubernetes-se...@googlegroups.com>, Caleb Miles <cmi...@pivotal.io>
Subject: [k8s-steering] Re: K8s release managers updates

 

Just pinging again on this one.

 

-- Stephen

 

On Mon, Jul 29, 2019 at 2:33 PM Stephen Augustus <Ste...@agst.us> wrote:

Community / Steering--

 

Can one of you lovely folks help out with creating release-...@kubernetes.io and adding me as a manager/owner for that (ste...@agst.us)?

Context is below, with active convo happening on https://github.com/kubernetes/sig-release/issues/732.

 

-- Stephen

 

On Thu, Jul 25, 2019 at 8:32 PM Stephen Augustus <Ste...@agst.us> wrote:

Just wanted to ping again on moving this forward.

Caleb and I agree to uphold the security embargo.

 

--
You received this message because you are subscribed to the Google Groups "steering" group.
To unsubscribe from this group and stop receiving emails from it, send an email to steering+u...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/steering/CAFQm5yTkrX-R6pGfiZmqMCvkmtBn7nLy9gmPydQbD0JLsQ_uDQ%40mail.gmail.com.

Stephen Augustus

unread,
Aug 9, 2019, 7:00:12 PM8/9/19
to Joe Beda, comm...@kubernetes.io, steering, Tim Allclair, Tim Pepper, kubernetes-se...@googlegroups.com, Caleb Miles
Hey Joe,

We're all set on this[1]. I think it might've been a delayed email (maybe caught in the moderation queue)?
Thanks again!

-- Stephen

Stephen Augustus

unread,
Aug 9, 2019, 7:00:12 PM8/9/19
to kubernetes-se...@googlegroups.com, release-...@kubernetes.io, Kubernetes Release Managers (Private), kubernetes-sig-release, Kubernetes Release Team, Tim Pepper, Caleb Miles
(Community+Steering to bcc)

All,

We plan to move to the following Google Groups for Release Manager communications:
The motivations and action plan for this are captured below and in more detail on https://github.com/kubernetes/sig-release/issues/732.

As the changes have already been approved, I'm setting a short lazy consensus period for Wednesday, 8/7 at 6pm ET.
Please direct any questions/comments/concerns to the above issue, so we capture the conversation in one place.

-- Stephen

Stephen Augustus

unread,
Aug 9, 2019, 7:00:12 PM8/9/19
to kubernetes-se...@googlegroups.com, release-...@kubernetes.io, Kubernetes Release Managers (Private), kubernetes-sig-release, Kubernetes Release Team, Tim Pepper, Caleb Miles
Release Manager mailing list name changes are now complete.
Please use the following group addresses moving forward:
There is also one pending documentation update to approve: https://github.com/kubernetes/sig-release/pull/751

-- Stephen

Stephen Augustus

unread,
Aug 9, 2019, 7:00:12 PM8/9/19
to comm...@kubernetes.io, steering, Tim Allclair, Tim Pepper, kubernetes-se...@googlegroups.com, Caleb Miles
Just pinging again on this one.

-- Stephen

Jordan Liggitt

unread,
Aug 9, 2019, 8:35:40 PM8/9/19
to Stephen Augustus, kubernetes-se...@googlegroups.com, release-...@kubernetes.io, Kubernetes Release Managers (Private), kubernetes-sig-release, Kubernetes Release Team, release-mana...@kubernetes.io, Tim Pepper, Caleb Miles
No objection to removing PSC folks if the release-managers-private list can be managed by sig-release.

It seems like we're getting slightly into list sprawl... do we need a combined list or can we cc release-mana...@kubernetes.io into threads for security release discussion? I guess that's an autocomplete away from release-...@kubernetes.io, which isn't great



On Fri, Aug 9, 2019 at 8:29 PM Stephen Augustus <Ste...@agst.us> wrote:

Before granting release-mana...@kubernetes.io GCP access, I'd like to clean up membership leaving only Release Managers. 

IIUC, the PSC was only on this list to manage membership, which is no longer required.

Any issues with removing the following people?

Additionally, can we create security...@kubernetes.io, which would contain:

Per @tallclair's suggestion on the thread, this (and unnesting release-managers-private@ from security@) would allow us to minimize traffic to release-managers-private@ to be only release-focused.

Setting Tuesday, 8/13 at 5pm ET as the lazy consensus time.

-- Stephen

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAFQm5yR5dReUULo%2B5Gd0L_Hn7_rgzRdnxa4bj%2BbQ_T2rmef3pw%40mail.gmail.com.

Stephen Augustus

unread,
Aug 13, 2019, 7:11:11 PM8/13/19
to kubernetes-se...@googlegroups.com, steering, release-...@kubernetes.io, release-mana...@kubernetes.io, kubernetes-sig-release, Kubernetes Release Team, Tim Pepper, Caleb Miles, Jordan Liggitt
Hearing no objections and with the lazy consensus period passed, I'm going to move forward with tidying up membership on the release-managers-private group.

Steering--
Can you create security...@kubernetes.io and add me (Ste...@agst.us) as an owner?

-- Stephen

On Fri, Aug 9, 2019 at 8:47 PM Stephen Augustus <Ste...@agst.us> wrote:
Yeah, I think I'd prefer one more list, if that helps prevent autocomplete snafus.

-- Stephen

Stephen Augustus

unread,
Aug 13, 2019, 7:11:11 PM8/13/19
to kubernetes-se...@googlegroups.com, release-...@kubernetes.io, Kubernetes Release Managers (Private), kubernetes-sig-release, Kubernetes Release Team, release-mana...@kubernetes.io, Tim Pepper, Caleb Miles

Before granting release-mana...@kubernetes.io GCP access, I'd like to clean up membership leaving only Release Managers. 

IIUC, the PSC was only on this list to manage membership, which is no longer required.

Any issues with removing the following people?

Additionally, can we create security...@kubernetes.io, which would contain:

Per @tallclair's suggestion on the thread, this (and unnesting release-managers-private@ from security@) would allow us to minimize traffic to release-managers-private@ to be only release-focused.

Setting Tuesday, 8/13 at 5pm ET as the lazy consensus time.

-- Stephen

Stephen Augustus

unread,
Aug 13, 2019, 7:11:11 PM8/13/19
to Jordan Liggitt, kubernetes-se...@googlegroups.com, release-...@kubernetes.io, Kubernetes Release Managers (Private), kubernetes-sig-release, Kubernetes Release Team, release-mana...@kubernetes.io, Tim Pepper, Caleb Miles
Yeah, I think I'd prefer one more list, if that helps prevent autocomplete snafus.

-- Stephen

On Fri, Aug 9, 2019 at 8:35 PM Jordan Liggitt <lig...@google.com> wrote:

Brandon Philips

unread,
Aug 19, 2019, 6:57:36 PM8/19/19
to Stephen Augustus, kubernetes-se...@googlegroups.com, steering, release-...@kubernetes.io, release-mana...@kubernetes.io, kubernetes-sig-release, Kubernetes Release Team, Tim Pepper, Caleb Miles, Jordan Liggitt
Stephen-

Do you want security-r...@kubernetes.io or security...@kubernetes.io? I am going to do -team because that was in the original email.

Brandon


You received this message because you are subscribed to the Google Groups "steering" group.
To unsubscribe from this group and stop receiving emails from it, send an email to steering+u...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/steering/CAFQm5yReUMStfgs%2BT_JnPooE9yDptfQp1dW0wmEEXaF6xu1HFA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages