Pre-proposal: a git repo to hold "policies"

[Tim Hockin]

Hi all,

In the context of https://github.com/kubernetes/enhancements/pull/5039/ (KEP 5040), we have a Validating Admission Policy (VAP) that we suggest kube users install into their clusters.

We think it would be useful to store such things somewhere that users could discover (rather than buried in the text of a KEP).  As a starting point, we're proposing a new github repo.  Since it is likely to cross SIGs, we are coming to sig-arch first.

What we would like input on is:

* Do you think this is a bad idea, if so why?

* What should be the bounding scope of it?  Policies?  Just security-oriented policies?  Just CVE-mitigating policies?

* Name?

* Thoughts on internal structure.  Ideally it would be the sort of thing that doesn't get re-organized too often, so permalinks into it could exist.

Tim

[Benjamin Elder]

> * Do you think this is a bad idea, if so why?

Great idea :-) Having somewhere linkable like `git.k8s.io/${policies_repo}/${policy}.yaml` seems like a win.

> * What should be the bounding scope of it?  Policies?  Just security-oriented policies?  Just CVE-mitigating policies?

"security" or "best practices", I don't think it should _just_ be CVEs, we have some difficult-to-change defaults where offering a MAP makes sense?

> * Name?

"kubernetes-sigs/admission-policies" ?
(IMHO we should restrict this to only core built-in admission policy types and the name should reflect that, we don't want to become a magnet for any sort of "policy" format)

> * Thoughts on internal structure.  Ideally it would be the sort of thing that doesn't get re-organized too often, so permalinks into it could exist.

policies/$short-descriptive-name/{README.md,policy.yaml}`?

Ideally the URLs are descriptive but not terribly long, but we'll probably want some other top level folders for tests/linting/...

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewbkEKHVj2DxGxszDEGC9iBeZPfA4VPf7%3D6M8O6iHij5jg%40mail.gmail.com.

[Brendan Burns]

I think that we should be careful here.

The lesson of Helm 2 -> Helm 3 was that having a central repository of any sort of recipe is challenging from a maintenance standpoint. It starts looking like all the challenges we had with kubernetes-contrib.

People are happy to land things like charts and policies in a central repo, but they're much less likely to update them if there are bugs or changes necessary.

Moving to a federated model with people explicitly opting into different repositories of [policy/chart/etc] is much more scalable in the limit.

So if we are going to go forward with this, I think having some pretty strict rules (e.g. "only policies related to KEPs" or "only policies that are useful to prepare for features that are deprecating") should be in place.

Something like "generally accepted security policies" should probably be a separate CNCF project that's not part of the kubernetes org, or at the very least it should be a dedicated repo with it's own working group or SIG affiliation to make sure that we lifecycle it properly.

--brendan

---

From: 'Tim Hockin' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Friday, January 31, 2025 10:24 AM
To: kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; Vinayak Goyal <vin...@google.com>
Subject: [EXTERNAL] Pre-proposal: a git repo to hold "policies"

 

--

[Benjamin Elder]

> The lesson of Helm 2 -> Helm 3 was that having a central repository of any sort of recipe is challenging from a maintenance standpoint. It starts looking like all the challenges we had with kubernetes-contrib.

If there are only admission policies, the scope of maintenance is _way_ smaller than kubernetes-contrib.

> People are happy to land things like charts and policies in a central repo, but they're much less likely to update them if there are bugs or changes necessary.

That should be pretty rare for admission policies?

> Something like "generally accepted security policies" should probably be a separate CNCF project that's not part of the kubernetes org, or at the very least it should be a dedicated repo with it's own working group or SIG affiliation to make sure that we lifecycle it properly.

IMHO that SIG exists:
+kubernetes-sig-security

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/PH7PR21MB3261E6AD28A23B8F71311B85DBE82%40PH7PR21MB3261.namprd21.prod.outlook.com.

[Brendan Burns]

If you limit the scope to "make sure that when gitRepo deprecates you don't get broken" I think that you are right that the number of changes will be small because the policies are targeted.

If you say "secure Kubernetes best practices" the the notion of what is/is not secure is going to change, and even things like APIs moving from beta to v1 will cause the policies to break (possibly silently, which is even worse).

It will be even worse if you say "HIPAA compliant Kubernetes" or "resiliant app kubernetes policies" where opinions are going to be even more varied.

Clearly if some sig wants to create and maintain a project (with governance, ownership, etc) for a targetted set of policies, that's goodness.

But any repo needs to be focused and needs to have clear ownership and people who are committted to maintaining it, otherwise it will turn into a dumpster fire of drive-by commits and poorly maintained "best practices" over time. That's just the reality of anything like this.

--brendan

---

From: Benjamin Elder <benth...@google.com>
Sent: Friday, January 31, 2025 11:14 AM
To: Brendan Burns <bbu...@microsoft.com>
Cc: kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>; Vinayak Goyal <vin...@google.com>; Tim Hockin <tho...@google.com>; kubernetes-...@googlegroups.com <kubernetes-...@googlegroups.com>
Subject: Re: [EXTERNAL] Pre-proposal: a git repo to hold "policies"

 

[Brendan Burns]

Fwiw, placing this repo in the context of kubeadm could also work, since the kubeadm team is already kind of curating the set of "best practices" for how to deploy a kubernetes cluster.

--brendan

---

From: 'Brendan Burns' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Friday, January 31, 2025 11:19 AM
To: Benjamin Elder <benth...@google.com>

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/PH7PR21MB3261F24501322636232B5699DBE82%40PH7PR21MB3261.namprd21.prod.outlook.com.

[Jack Francis]

+1 on kubeadm

Putting this into a non-k/k repo signals "best practice" (which signals "optional" in terms of real world outcomes). The VAP defined in KEP-5040 could be installed and tested in the context of the standard kubeadm operations (init/upgrade). Here's an example kubeadm test run that demonstrates equivalent behavior validation to something like "verify that gitRepo volumes are not allowed":

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-1-32

Jack

[Benjamin Elder]

> If you say "secure Kubernetes best practices" the the notion of what is/is not secure is going to change, and even things like APIs moving from beta to v1 will cause the policies to break (possibly silently, which is even worse).

Sure, and we'll update the examples. gitRepo is itself an example of this.
This problem isn't really any different from the examples in the kubernetes docs.

> It will be even worse if you say "HIPAA compliant Kubernetes" or "resiliant app kubernetes policies" where opinions are going to be even more varied.

No one has suggested "HIPPA compliance", and I certainly wouldn't, but certainly it needs to be well-scoped.
The repo should exclude compliance, the project should not attempt to assert any particular compliance.

> since the kubeadm team is already kind of curating the set of "best practices" for how to deploy a kubernetes cluster.

Er, SIG Cluster Lifecycle has many approaches, with much respect to the kubeadm team, I don't think there's a broad agreement to that statement in the project.

IMHO, this falls under "cross-cutting security documentation" which is straight from the SIG Security charter. 

> Putting this into a non-k/k repo signals "best practice" (which signals "optional" in terms of real world outcomes). The VAP defined in KEP-5040 could be installed and tested in the context of the standard kubeadm operations (init/upgrade). Here's an example kubeadm test run that demonstrates equivalent behavior validation to something like "verify that gitRepo volumes are not allowed":

kubeadm is optional, but also the VAP needs to be optional in the immediate future (see the KEP).

I don't think kubeadm makes sense here, these policies need to be communicated to non-kubeadm users as well, and ideally they shouldn't be tightly coupled to Kubernetes releases either.

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/c9320379-6e9e-4ffc-9515-8a1d39382981n%40googlegroups.com.

[Brendan Burns]

There's a big difference between "examples" and "you should apply all of these policies"

People understand that examples may be incomplete and may be out of date. It's a problem, but it is at least an understood problem that doesn't violate the principal of least surprise. 

If you say "these are the SIG-Security (or whatever SIG)" sponsored best practices, people are going to assume they are up to date and complete. You'll need to have unit tests to make sure that they stay correct, you'll need to have some group of people who arbitrate what should and shouldn't be in that repo (and explain that to people who want their thing in the repo why it doesn't get to go in that repo).

We should identify the people who are volunteering to take on that responsibility before we identify and create such a repo.

As I said, if some SIG wants to sign up for that responsibility, great, they should do that, and they should own that repo, we already have processes for creating those repos owned by SIGs.

But if this is creating a repo without a team that is signing up for maintenance, it's going to be a dumping ground, that's just reality.

--brendan

---

From: 'Benjamin Elder' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Friday, January 31, 2025 12:33 PM
To: Jack Francis <jackf...@gmail.com>
Cc: kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAOZRXm_9dU9%2BgzJLZWpQMFXqHK5iDfTyjNnh76U1vnArwzXTYQ%40mail.gmail.com.

[Tim Hockin]

Thanks for the replies.  I do not think this would go so far as HIPAA compliance.

Security is the most obvious domain for this, and I am OK limiting it to that, but we have other things that we tell people they "can do" but don't always show them how.  For example, the ancient Docker Service Links functionality occasionally blows up some Redis user.  Would we allow an MAP which disables that?  It's not "security", but it is something we would probably change the default for in core/v2.

Another example could be a VAP which implements a piece of API validation that was missed (e.g. using the same port name on 2 containers came up again just last week).  We know it's wrong, but all we can do is issue a warning.  Would we allow VAPs for those?

> "kubernetes-sigs/admission-policies" ?

SGTM

> (IMHO we should restrict this to only core built-in admission policy types and the name should reflect that, we don't want to become a magnet for any sort of "policy" format)

agree, and I think it should be limited to things that fix our own problems.  Brendan cited HIPAA, and I think it is a great counter-example.  Not being HIPAA compliant is not a bug in Kubernetes :)

> But if this is creating a repo without a team that is signing up for maintenance, it's going to be a dumping ground, that's just reality.

There's a reason I am email sig-arch.  It's not clearly sig-security's problem (though maybe we constrain it to that).  I nominate dims. :)

Tim

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAOZRXm_9dU9%2BgzJLZWpQMFXqHK5iDfTyjNnh76U1vnArwzXTYQ%40mail.gmail.com.

[Vinayak Goyal]

> kubernetes-sigs/admission-policies 
I think this would be very helpful and I'd like to help maintain it!

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewY7S1BF5qb9yMQoSDz-v31LQGLiG1cezgOyOgdyRQKWmA%40mail.gmail.com.

[Cici Huang]

Hi,

Thank you for the suggestion! I definitely feel it would be beneficial for people who are using or gonna use VAP. Just wanna point out that the ecosystem already started a similar work and add some links for reference:

1. The OSS contributors went ahead and created a VAP library for some general security policies: https://github.com/vap-library/vap-library

2. Kubescape team also mentained a VAP library here: https://github.com/kubescape/cel-admission-library

I don't think we should spread the message like "you should apply all of these policies". But feel it would be valuable to have the examples added inside Kubernetes org to provide guidance or examples on common compliance or even suggestions like adding missed API validation. 

Best,

Cici

[Brendan Burns]

fwiw, this kind of 'who gets to be the arbiter of the official policy repo' kinds of questions are exactly why a federated model is a better approach and why Helm moved away from a central chart repo.

---

From: 'Cici Huang' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Friday, January 31, 2025 1:37:39 PM
To: kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/9b8622be-d15e-4697-8f4b-82dbcc61d85an%40googlegroups.com.

[Tim Hockin]

It's all "federated" by default (though I am not sure that's the right word) - nobody is forced to use these.

To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/PH7PR21MB3261962310FD3E669475E968DBE82%40PH7PR21MB3261.namprd21.prod.outlook.com.

[Brendan Burns]

Despite the fact it's optional and anyone can use policies from anywhere, anything that is put in k/k will automatically be seen as more official.

Federated means that all repos are seen as equal. Pragmatically if there's a k/k repo that won't be true.

---

From: Tim Hockin <tho...@google.com>
Sent: Friday, January 31, 2025 2:03:16 PM
To: Brendan Burns <bbu...@microsoft.com>
Cc: Cici Huang <ci...@google.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>

[Matthias Bertschy]

Hi everyone,

Regarding this discussion, I wanted to highlight that the Kubescape project already offers a solution for this.

Kubescape has a repository with Validating Admission Policy (VAP) policies available here: https://github.com/kubescape/cel-admission-library.

Additionally, Kubescape CLI includes a dedicated flag for deploying the CEL admission policy library: https://kubescape.io/docs/frameworks-and-controls/validating-admission-policy/#deploying-the-cel-admission-policy-library.

As a CNCF incubating project, Kubescape is committed to providing comprehensive Kubernetes security solutions and would be a perfect home for such an initiative. It offers posture and vulnerability management, automatic hardening policies, and eBPF-based threat detection. Kubescape is deeply integrated with the CNCF ecosystem, leveraging eBPF (via Inspektor Gadget) for runtime observability and Open Policy Agent (OPA) for configuration scanning. It integrates with tools like ArgoCD, Prometheus, and Headlamp.

I hope this information is helpful.

Best regards,

Matthias

[Tim Bannister]

I can see the value of incorporating these policies into Kubescape, but there are going to be some kinds of policy that make sense across a wide range of Kubernetes implementations and platforms.

If you're a vendor who offers your own thing that competes with Kubescape (or with ARMO, the organization who brought it into being), your users might still want to benefit from strongly recommended policies.

For example, we could make:

• a ValidatingAdmissionPolicy that prevents deletion of the kube-system and kube-node-lease system namespaces
• a NetworkPolicy that denies all traffic from Pods in kube-node-lease (what are those Pods doing there‽)
• a MutatingAdmissionPolicy that labels Pods with their OS if the Pod .spec declares one
• an optional but ready-to-use "contrib" ValidatingAdmissionPolicy that ensures all new Namespaces enforce at least the Baseline Pod security standard (using PSA)
• an optional, recommended ValidatingAdmissionPolicy that constrains app.kubernetes.io/name and app.kubernetes.io/part-of to all-lowercase

The idea is that whatever we pick, these are a core set of policies that are reasonable for every cluster to use, and that the Kubernetes project recommends using almost unconditionally. If people want to apply these policies using Kubescape, Kyverno, etc—go for it. We'd be really happy, as Kubernetes, to see those policies adopted.

The ask is to define where we put them.