The meaning and ownership of namespace labels

[Tim Hockin]

Context: https://github.com/kubernetes/kubernetes/issues/88253

We have at least 2 places in k8s that use label-selectors against
namespaces, NetworkPolicy and ValidatingWebhookConfiguration. In both
places, we use labels to decide who to apply policy to.

It seems that some (many?) installations allow users to label their
own namespaces. In the case of NetworkPolicy, this more or less
neutralizes the utility of this part of the API ("allow traffic from
namespaces labelled foo=bar") since any namespace owner could apply
that label.

In the bug above I argued "don't allow that", but I'll be first to
acknowledge this isn't super helpful. The proposal is to add another
option to the API - a list of Namespace names - which users could use
instead. If we added it in NetworkPolicy, I can't help but feel we'd
also want it in ValidatingWebhookConfiguration.

So I thought it was worth bringing up with a larger group with lots of
breadth. How are you handling it in your respective installations or
products?

Do you allow namespace owners to label their own Namespaces?

Do you impose restrictions on those labels (e.g. only unprefixed
labels or anything but "mycompany.com/*")?

How do you implement those restrictions (e.g. custom admission
control, Gatekeeper, ...)?

Do you provide a way for namespace owners to "request" labels and then
handle the application of those (either manually or through a portal
or controller)?

It seems to me like we should probably provide better guidance and
examples of how to manage this. Aside from that, should we add
list-of-namespace-names top these APIs?

Looking forward to discussion

Tim

[Tim Allclair]

+kubernete...@googlegroups.com

We had a related discussion in SIG auth last month about using labels to control general policy decisions (e.g. OPA Gatekeeper constraints include a label selector). I argued that although we're pushing most policy enforcement to 3rd party admission controllers, label policy might be the one case where a built-in policy API & enforcement makes sense. We have an old design that was never implemented (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/metadata-policy.md), but I wouldn't recommend moving forward with that proposal.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewZVcKZV8L%2B9hRO3X_QHBCy7AxLe7v%2Bxh3%2BYAQAXreuPVg%40mail.gmail.com.

[Dan Winship]

NetworkPolicy NamespaceSelectors are problematic if you _don't_ let users label their own Namespaces too, since in that case there may not be any label that selects the Namespace you want, and no way to add it.

It seems to me like the problem is that LabelSelectors are a good API for selecting your own objects (ie, when the person writing the LabelSelector also controls the labels) or for selecting objects where the owner of the objects is explicitly labeling them with the intention of other users consuming those labels, and the users only care about the distinctions that the owner makes (eg NodeSelector on Pod). But they don't work well if you have to select against labels created by a user or users you don't trust (the "users can label their own namespaces" NetworkPolicy problem) or if you have to select against labels created by a user who doesn't care about your use case (the "only cluster-admins can label namespaces" NetworkPolicy problem).

This is why everyone keeps asking about extending NetworkPolicy to allow selecting Namespaces by name, but no one ever suggests we need to extend it to allow selecting Pods by name; LabelSelectors work fine for the latter case, just not for the former.

Answering your specific questions for OpenShift:

On Friday, September 18, 2020 at 7:15:31 PM UTC-4 tho...@google.com wrote:

Do you allow namespace owners to label their own Namespaces?

no

 

Do you impose restrictions on those labels (e.g. only unprefixed
labels or anything but "mycompany.com/*")?

N/A

 

How do you implement those restrictions (e.g. custom admission
control, Gatekeeper, ...)?

N/A

 

Do you provide a way for namespace owners to "request" labels and then
handle the application of those (either manually or through a portal
or controller)?

no

-- Dan

[Clayton Coleman]

On Sep 19, 2020, at 6:27 PM, Dan Winship <dwin...@redhat.com> wrote:



NetworkPolicy NamespaceSelectors are problematic if you _don't_ let users label their own Namespaces too, since in that case there may not be any label that selects the Namespace you want, and no way to add it.

It seems to me like the problem is that LabelSelectors are a good API for selecting your own objects (ie, when the person writing the LabelSelector also controls the labels) or for selecting objects where the owner of the objects is explicitly labeling them with the intention of other users consuming those labels, and the users only care about the distinctions that the owner makes (eg NodeSelector on Pod). But they don't work well if you have to select against labels created by a user or users you don't trust (the "users can label their own namespaces" NetworkPolicy problem) or if you have to select against labels created by a user who doesn't care about your use case (the "only cluster-admins can label namespaces" NetworkPolicy problem).

This is why everyone keeps asking about extending NetworkPolicy to allow selecting Namespaces by name, but no one ever suggests we need to extend it to allow selecting Pods by name; LabelSelectors work fine for the latter case, just not for the former.

Answering your specific questions for OpenShift:

On Friday, September 18, 2020 at 7:15:31 PM UTC-4 tho...@google.com wrote:

Do you allow namespace owners to label their own Namespaces?

no

 

Do you impose restrictions on those labels (e.g. only unprefixed
labels or anything but "mycompany.com/*")?

N/A

Actually we came very, very, very close to implementing a prefix like this.  Jordan and I had talked about various aspects of creating an explicit namespace label prefix for end users to create and control - a similar discussion to the rules that ended up being implemented on node labels and topology and which the Kubelet could set.  Note the mechanism for control of this would have been the virtual project resource (which controls the fields that can be set on namespace by making an explicit mapping to a real namespace object)

We never came back to it because no one was beating down the door for the use case - most large scale tenancy did not involve large multinamespace apps at the time.

 

How do you implement those restrictions (e.g. custom admission
control, Gatekeeper, ...)?

N/A

 

Do you provide a way for namespace owners to "request" labels and then
handle the application of those (either manually or through a portal
or controller)?

no

-- Dan

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/dbbf6bce-e58a-4bde-8773-c5c64a0b32e4n%40googlegroups.com.

[Daniel Smith]

I think labels are at their best when the producer is also the consumer, and are at their worst when the producer and consumer don't trust each other. You can kind of make it work with webhooks, but it's awkward.

Namespaces display this problem more clearly than most resources, because there's basically only (in a stock cluster) two permissions levels: too little, and too much.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAH16Sh%2Bs1ie256Uc8sMFyThntnHvR3osLrX%2BQybrDA_XJToj3A%40mail.gmail.com.

[Brendan Burns]

In general as I've seen similar systems evolve over time there develops a need for RBAC'd labels.

Because labels (and annotations, but that's a separate topic) have evolved to have the ability to not just identify, but actually activate behaviors, it becomes important to be able to say who has permission to activate that behavior.

I think there are three forms of Labels that need to be supported in the fullness of time to make these use cases work:

1) Regular free-form labels as we know them today.

2) Write-once labels which can be written by anyone, but once written are immutable. The use case for this is something like a "created-by" identifier for a Pod. Since the Pod can never be re-created, that label should be immutable. (there are other examples related to billing, etc)

3) RBAC labels, where the permission to read/update/write the labels follows the well established RBAC rules.

(note that 2 is a subclass of 3, but we probably want to implement them separately to incremental adoption)

I think that the right thing to do is probably to introduce a "Label" or "Label Class" resource and then RBAC that and enforce the RBAC on resource admission.

All of this will take time to do (and a KEP obviously)

--brendan

---

From: 'Daniel Smith' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Monday, September 21, 2020 9:24 AM
To: Clayton Coleman <ccol...@redhat.com>
Cc: Dan Winship <dwin...@redhat.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: [EXTERNAL] Re: The meaning and ownership of namespace labels

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3baAAYLy9dVx07zYdm-n-_o1aTPJw2Cot2CtWReE6JiyWA%40mail.gmail.com.

[Daniel Smith]

I think much of the difficulty is not breaking existing users of labels.

The implementation is also challenging because it's basically a security feature, we have to be sure we've got every possible mutation point.

I think there's a debate to be had re: a) RBAC-checking (potentially many) individual labels during the course of editing an object, vs b) extracting "labels" to a separate object.

a) is authz fan-out, and we don't do much of that currently, I'm pretty sure nothing in the system has a non-bounded number of RBAC checks right now.

b) involves a join on reads if we want to present the externally-stored labels as part of the objects. This is read-time fan-out that we don't ever do right now, and would make our already-bad O(N) lists even worse-- O(N*M). But would we need to do this join? Not pretending to be the existing labels at all means we can't break existing users, and maybe it's OK if new users adjust what they do.

I agree this probably will take time to get right, and a KEP. I think we need champions from SIG Auth and SIG API Machinery. I don't have time to drive this right now, sadly.

[Brendan Burns]

I don't think the RBAC check is much of an issue because in practice the number of RBAC'd labels will be small.

You could keep a hash-table in memory in the API server and only do the RBAC check if the hash matches.

In other contexts (e.g. the Azure control plane) we do this "linked-access check" e.g. someone can only set a VM in a network if they have RBAC on that network, and it works ok.

--brendan

---

From: Daniel Smith <dbs...@google.com>
Sent: Monday, September 21, 2020 10:22 AM
To: Brendan Burns <bbu...@microsoft.com>
Cc: Clayton Coleman <ccol...@redhat.com>; Dan Winship <dwin...@redhat.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: Re: [EXTERNAL] Re: The meaning and ownership of namespace labels

 

[Tim Hockin]

On Fri, Sep 18, 2020 at 4:31 PM Tim Allclair <timal...@gmail.com> wrote:

> We had a related discussion in SIG auth last month about using labels to control general policy decisions (e.g. OPA Gatekeeper constraints include a label selector). I argued that although we're pushing most policy enforcement to 3rd party admission controllers, label policy might be the one case where a built-in policy API & enforcement makes sense.

I think I agree on this. It seems pretty fundamental.

[Tim Hockin]

On Sat, Sep 19, 2020 at 3:27 PM Dan Winship <dwin...@redhat.com> wrote:
>
> NetworkPolicy NamespaceSelectors are problematic if you _don't_ let users label their own Namespaces too, since in that case there may not be any label that selects the Namespace you want, and no way to add it.

Abstractly, there are three sets of label names:
1) Labels I can not set (but which may be set for me)
2) Labels only I can set
3) Labels anyone can set

Given #1 and #3 and the usual label semantics, I can arrange
compound-selectors that approximate #2. We don't really have #1 nor
any clear way to implement it (assuming they would want to be set on
create, admission control is awkward or impossible, I think).

> It seems to me like the problem is that LabelSelectors are a good API for selecting your own objects (ie, when the person writing the LabelSelector also controls the labels) or for selecting objects where the owner of the objects is explicitly labeling them with the intention of other users consuming those labels, and the users only care about the distinctions that the owner makes (eg NodeSelector on Pod). But they don't work well if you have to select against labels created by a user or users you don't trust (the "users can label their own namespaces" NetworkPolicy problem) or if you have to select against labels created by a user who doesn't care about your use case (the "only cluster-admins can label namespaces" NetworkPolicy problem).

If every Namespace had an "owner" label that could not be unset, you
could reduce the overall problem to "selecting your own objects"
again.

> Answering your specific questions for OpenShift:
>
> On Friday, September 18, 2020 at 7:15:31 PM UTC-4 tho...@google.com wrote:
>>
>> Do you allow namespace owners to label their own Namespaces?
>
>
> no
>
>>
>> Do you impose restrictions on those labels (e.g. only unprefixed
>> labels or anything but "mycompany.com/*")?
>
>
> N/A
>
>>
>> How do you implement those restrictions (e.g. custom admission
>> control, Gatekeeper, ...)?
>
>
> N/A
>
>>
>> Do you provide a way for namespace owners to "request" labels and then
>> handle the application of those (either manually or through a portal
>> or controller)?
>
>
> no
>

Do you pre-set labels on Namespaces that users can not change? If so,
what do they "mean" (e.g. is one an "owner" or "tenant") ?

[Tim Hockin]

My concern is that there are more such monsters under the bed, and
over-training on labels may be wrong. Having RBAC apply to a
sub-scope of fields also feels awkward and kind of wrong (and I have
argued against it in other API proposals).

My mind goes to something coarser - like a top-level group, akin to
metadata (or under metadata, since we again did not reserve names in
top-level), that carries a different RBAC scope. Similar to how I can
RBAC on "pods" (which ignores writes to pod.status) and "pods/status"
distinctly.

For example, ".metadata.superSpecial". As a user I can read but not
write that. I'm not sure how to work that into selectors (merge with
precedence?). I am not sure how to set those on create - is it
possible to disallow it from a user but allow it from admission?

[Tim Allclair]

Setting implementation aside, and thinking about the use cases:

Generally, what we're talking about is some authority wants to designate a group or set of things, e.g:

- "security team has designated a set of exceptional workloads as privileged", (and as a developer, I want to request a policy exception)

- "privacy team has flagged a set of workloads as dealing with PII"

- "app dev team has allowlisted a set of client applications"

Most of these use cases could happen out of band (i.e. not on create, or maybe even before create), but we also want to allow for self-service in some cases (e.g. this namespace is granted privileged status, but only a subset of the pods in the namespace actually need it)

As a strawperson, what about something like this:

- a "resource set" object, with both a namespaced & cluster scoped version (namespaced version can only contain objects in the same namespace)

- resource sets can be nested (this enables the self-service piece)

- resource sets can apply "hierarchically", meaning I should be able to add a deployment to the set, and in doing so include all the replica sets and pods indirectly owned by the deployment

- where a label selector is used, a resource set list may also be supplied

Concerns with this approach:

- multi-step process to create a new object & add it to the set (although maybe this is desireable in the cases where the indirection of the resource set is really required?)

- scalability of resource sets as they grow large (large objects)

- manageability of resource sets (who cleans up deleted objects?)

- performance / scalability of nested sets & ownership hierarchy

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewaXFYdnQWAaLcuV%2B%3DmOegByc06csmRF9iFjXjR%2BV0fYtg%40mail.gmail.com.

[Tim Hockin]

This sounds a lot like nested namespaces ;)

[Wojciech Tyczynski]

On Mon, Sep 21, 2020 at 7:22 PM 'Daniel Smith' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com> wrote:

I think much of the difficulty is not breaking existing users of labels.

The implementation is also challenging because it's basically a security feature, we have to be sure we've got every possible mutation point.

I think there's a debate to be had re: a) RBAC-checking (potentially many) individual labels during the course of editing an object, vs b) extracting "labels" to a separate object.

a) is authz fan-out, and we don't do much of that currently, I'm pretty sure nothing in the system has a non-bounded number of RBAC checks right now.

b) involves a join on reads if we want to present the externally-stored labels as part of the objects. This is read-time fan-out that we don't ever do right now, and would make our already-bad O(N) lists even worse-- O(N*M). But would we need to do this join? Not pretending to be the existing labels at all means we can't break existing users, and maybe it's OK if new users adjust what they do.

I'm not sure lists would have to be O(N*M) - assuming there would be 1-1 correspondence between those two objects (I don't yet see a good reason for not having 1-1), we could do lists linearly in O(N+M) (well, N=M in this case).

Watch is more tricky in this case - we would have to support a single watch on disjoint sets of keys in etcd (though it sounds doable).

[Well, I think I'm actually against this option, but I wanted to point that it may not necessary be that bad as you wrote.]

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bZyGb_ZRJLKYYnh6QHi4gtuJ5cL%2BnXFJxFqWJsZntQaeA%40mail.gmail.com.

[David Eads]

We do make use of namespace labels and annotations in OpenShift to control security aspects of the platform.  The default stance for kubernetes is to disallow namespace admin mutation of namespaces for these kinds of use-cases https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go#L274-L348 .

We did share a project API a while back to allow ACL filtered listing of namespaces (which namespaces can I see) and in principle the update of a secondary resource could more easily limited by creating a coarse separation (the change would go here).  Rather than modify existing API behavior, it may make more sense to create a new type of resource to represent the flexibility you want.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAPgfqyUFz4jvftrok96nUXJ%2BfzN2xr4XfKvqOgR8ieh-9Ms-TQ%40mail.gmail.com.

[Clayton Coleman]

On Mon, Sep 21, 2020 at 3:34 PM Tim Hockin <tho...@google.com> wrote:

My concern is that there are more such monsters under the bed, and
over-training on labels may be wrong.  Having RBAC apply to a
sub-scope of fields also feels awkward and kind of wrong (and I have
argued against it in other API proposals).

Labels definitely have a lot of limitations - when I did the rings+fences proposal I hit a bunch of places where a label selector doesn't really let you represent subdivision the way a human might want.  I.e. you want to have the concept of "everything stays within my current security set, unless I give positive affirmation I want to grow it", which means defaulting labels on things you create.  Or you cross a boundary where escalation isn't obvious (deployment -> secret use) but the label doesn't clearly indicate it. (https://docs.google.com/document/d/1dKa0eJlagu_3Tgulh8BSH5xnfbpnW6gn4VzNR0aTGAU/edit but don't read unless you have an hour or so - strictly including because it plays with how labels could be used practically)

To Tim Allclair's point, I think we aren't trying to model groups - we're trying to model a capability tree (which I still think is the gold standard for reasoning about these things), and a "group" is really just one of the branches.  I really would prefer to have something orthogonal that makes both RBAC and labels and containers (namespaces) *better* - something that can do the heavy lifting alongside those models so that most people don't have to learn something new - they are just constrained by their capabilities and you can reason about how that interacts.

[Tim Hockin]

I like the idea of capability tree, but our "tree" only has one level,
unless we embrace NS hierarchy or some other grouping (call it
projects or tenants or folders or ...)

[Derek Carr]

>Rather than modify existing API behavior, it may make more sense to create a new type of resource to represent the flexibility you want.

I tend to agree with David.

I also liked the proposal from Tim on "virtual labels" that allow selection by name (and preferably uid).

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewZpLESwn-FfRoxgUyLcYkTNuRn%3D4aqdQVr-GNzWXqk12w%40mail.gmail.com.

[Max Smythe]

Two questions I have about splitting labels out into a separate object:

How does that complicate other projects/controllers that are interacting with labels? Will they need to set up multiple watches and collate things themselves, or will the K8s client API hide that complexity?

One benefit of having labels embedded in the object is that updates are atomic. Splitting them out means that the object could change before the labels do or vice versa. At the most extreme, eventual consistency could mean labels are applied to a completely different object than the one they were intended to. Is this a concern?

[Brendan Burns]

My suggestion was to split the Label out for the purposes of RBAC, but still include the label in the metadata of the object. (effectively the metadata entry becomes a reference or pointer to the object)

eg

metadata:

  labels:

      "labels.kubernetes.io/apis/labels/v1/my-cool-label/": "my-awesome-value"

Then when the object is created, the apiserver will do the RBAC look-up for the referenced label resource (a "linked access check") to validate that the user creating the object with the label has permissions to use that label. 

The good thing about this is that it actually is backwards compatible since the RBAC version of a label is namespaced into a domain that no one should have been using, so it won't break existing label usage, but it will facilitate RBAC'd labels.

--brendan

---

From: 'Max Smythe' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Wednesday, September 23, 2020 1:18 PM
To: kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>

Subject: Re: [EXTERNAL] Re: The meaning and ownership of namespace labels

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/d087ce78-c27d-4f78-bb2b-025a1fdf2838n%40googlegroups.com.

[Max Smythe]

Yeah, the proposals for label prefixing/namespacing and adding a new ACLd metadata subresource make sense to me from an operational perspective.

I interpreted tallclair's idea as adding a separate label resource that is stored separately. That made me wonder how Gatekeeper would handle things like:

• Caching copies of the resource (including external labels), so that users can write policies against them
• Implement whatever new labelSelector logic this implies for its validating webhook

both of which could be trivial if the client API abstracts the fact that labels are actually a separate entity.

From a config perspective, the possibility of drift between objects and the labels applied to that object could potentially be concerning (e.g. temporarily routing services to the wrong Pod).

-Max

[Tim Allclair]

Couldn't gatekeeper just watch (and cache) the separate "resource set" objects? We still might need to denormalize them for other reasons. But I'm not sure it's worth taking this idea ("resource sets") too seriously.

One thing to note, I was only considering restricted labels being used to grant additional privileges, in which case eventually consistency is less of an issue. For restricting privileges with restricted labels (e.g. via a write-only label applied during admission, or namespace label applied during setup), I think separating the labels becomes a bigger issue.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAFjGRXXYabwFZiURu%3DcHKuccKvtJ%2BxDPPxYKoRmm-kx%3DH5npOg%40mail.gmail.com.

[Max Smythe]

Gatekeeper could certainly watch and cache these other objects, it's more a usability question than a technical blocker.

WRT "additional privileges" vs "restricting privileges", I think the existence of DELETE means you can't have one without the other. If you are adding additional privileges with labels, then deleting those labels is effectively restricting privileges. I think it's fair to question whether the time scale of the duration of the drift is worth worrying about, but it does add the possibility for drift where none previously existed.

-Max

[Brendan Burns]

This is broadening the scope of the discussion, so feel free to ignore, but I think we're going to need to implement general "linked-access" checks in the API server no matter what.

Labels are specific instance of this need, but as we build higher-level abstractions, the need for linked access checks is only going to grow. Even now, people would probably like to be able to restrict which users can attach a specific persistent volume or a secret to a Pod, which Ingress objects can use a particular Secret, etc. This stuff needs to be in the API server because eventual consistency opens the door to timing attacks. Basically any time any resource references another resource, we're going to need the ability to apply RBAC to validate that accessing that resource is allowed. Many of our async patterns mean that doing this in an eventually consistent way is insufficient since the actor actually accessing the resource (the operator, the kubelet, etc) is acting with it's own identity, not the identity of the user that initiated the access.

So, I think we should focus this discussion on two mechanisms:

1) Representing labels (or some new type of label like thing) as an external object that can be referenced from a resource's metadata.

2) Building out "linked access" checks as a fundamental API server capability, that can be used to restrict access to 1) but is generally useful for restricting access to all sorts of references in the API.

--brendan

---

From: Max Smythe <smy...@google.com>
Sent: Wednesday, September 23, 2020 5:01 PM
To: Tim Allclair <timal...@gmail.com>
Cc: Brendan Burns <bbu...@microsoft.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>

[Brian Grant]

I'm going to pick up on the "some new type of label like thing" idea.

In the beginning, we just had labels.

https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/

Labels were intended to reflect identifying attributes for use by the resource creators/managers, to facilitate organization into sets declaratively, as opposed to imperative add/remove operations, and in a more flexible manner than strict hierarchy would. Like resource names, labels on namespaced resources are conceptually namespaced, and most/all selectors in Kubernetes select within the same namespace as the selector, or require specifying applicable namespaces. Cluster-level resources, such as Nodes, are considered admin-owned resources and their names and labels come out of a global pool. Labels are generally set at resource creation time by the creator, such as a user invoking "kubectl apply". In order to keep them readable, they have size and character-set restrictions. They are not ACLed and frequently keys are not prefixed. 

Then we added annotations in v1beta3.

https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

Annotations were created to checkpoint additional information or attributes, primarily to be consumed by tools, controllers, and extensions. They have more permissive character sets, particularly for values, which facilitates storing structured and other arbitrary data. We recommend that keys should be prefixed in order to avoid conflicts between ecosystem tools or other automation. More often than labels, they are written dynamically. They have frequently been used as extended resource fields.

Not conflating these 2 use cases into a single key/value metadata concept improved usability for the 2 distinct use cases. 

The use case under discussion is significantly different from these two. I think of them as "admin-assigned attributes", since they wouldn't be attached by resource creators/managers, and typical use cases are administrative ones, such as policy enforcement and billing, both of which were mentioned earlier in this thread.

How would they be assigned then? Sometimes asynchronously, but typically they need to be inherited somehow in order to be attached at creation time. Otherwise there's a chicken and egg problem: how would one select which resources to apply which attributes to?

Mixing inherited, ACLed attributes with labels would be confusing for users and problematic for clients. How would a user understand which labels were ACLed and which weren't? Kubectl apply can merge attributes provided by multiple sources, but not all tools and clients do that properly. If an ACLed label were applied to a resource by a privileged client and a client without write access PUT a label map not including that label, presumably the PUT would fail. We've seen how mutations made by admission controllers can interfere with controllers, such as in the case of modification of ReplicaSets created by the Deployment controller.

Clouds have such a concept, called "tags" by at least AWS and Azure:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/tag-resources

I recommend creating a new kind of ACLed metadata, called tags, for consistency.

That would also give us the freedom to choose size and character-set limitations more aligned with cloud-provider tags, for easier interoperation, such as in the case of cloud service Operators, and for internationalization.

Tags could be inherited using the hierarchical namespace controller:

https://kubernetes.io/blog/2020/08/14/introducing-hierarchical-namespaces/

Such a map could be prototyped as a single annotation, with the map in the value. Usually selection isn't actually needed because policy engines and accounting systems look at every resource.

In the meantime, in the case of NetworkPolicy, allowing specification of Namespace names, or even Namespace name prefixes, would be a pragmatic incremental fix for multi-tenant clusters. Creating Namespaces is generally a privileged operation, since otherwise Namespace-scoped policies could be circumvented by creating a new Namespace.

Some related issues:

• support label selectors in RBAC: https://github.com/kubernetes/kubernetes/issues/44703
• bulk namespace control: https://github.com/kubernetes/kubernetes/issues/40403
• protected attributes: https://github.com/kubernetes/kubernetes/pull/27330
• policies on labels and selectors: https://github.com/kubernetes/kubernetes/issues/2211
• required, permitted, prohibited labels: https://github.com/kubernetes/kubernetes/issues/15390
• metadata policy (network policy use case): https://github.com/kubernetes/kubernetes/issues/65340

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/BY5PR21MB1444D312DA3057498767D0F1DB390%40BY5PR21MB1444.namprd21.prod.outlook.com.

[Brendan Burns]

I'm definitely ok with a new type of metadata. I'd vote for "system-tags" rather than just "tags" since I think that it helps users understand that the usage of those tags are restricted by some outside system vs labels.

(I understand that in many cases these won't be used by the actual "core" system, but rather by operators, etc, but I think those things are "system enough" that it will help clarify things.)

--brendan

---

From: 'Brian Grant' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Thursday, September 24, 2020 8:46 AM
To: Brendan Burns <bbu...@microsoft.com>
Cc: Max Smythe <smy...@google.com>; Tim Allclair <timal...@gmail.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKCBhs5Oeb7tcjyormjmsrEbJHqMiH3GrqVB4CnzJrswtBN-fg%40mail.gmail.com.

[Brian Grant]

On Thu, Sep 24, 2020 at 8:55 AM Brendan Burns <bbu...@microsoft.com> wrote:

I'm definitely ok with a new type of metadata. I'd vote for "system-tags" rather than just "tags" since I think that it helps users understand that the usage of those tags are restricted by some outside system vs labels.

(I understand that in many cases these won't be used by the actual "core" system, but rather by operators, etc, but I think those things are "system enough" that it will help clarify things.)

Do you have examples of "system" use cases along the lines of kube-system? 

The use cases I'm aware of are human-driven -- operators or admins.

Also, I think there's value to maximizing consistency with industry terminology if we can.

[Brendan Burns]

Humans definitely will be making the access decisions (e.g. "who should have access to foo")

But the use-cases that I can think of are all about "system" agents that are acting (e.g. NetworkPolicy enforcing things, kubelet mounting secrets, operators adding applications to a namespace, etc)

I don't feel super strongly, but I think that there will be lots of confusion about when to use labels and when to use tags since the names are semantically quite similar, even if we intend them for very different use-cases. Especially since tags in Azure and AWS are effectively a combination of labels and annotations.

--brendan

---

From: Brian Grant <brian...@google.com>
Sent: Thursday, September 24, 2020 8:59 AM

[khn...@gmail.com]

The situation with labels spans more than just namespaces. For example  nanny style pods (running with higher privileges) running in some user namespaces. Every time access (or a scarce resource) is given because a selector matches it puts us in a situation where "a malicious or unknowing user can set label name and its corresponding value". I have seen this in every situation where labels are used for that. Controlling Access to specific properties looks a lot like ABAC (sidebar: it is ABAC at least on target side). It is a faux pas for most. I share equal dread. But if we decided to go that route then I expect every bigCo out there to have requirements around extending it to other properties. With valid use cases (monsters under the bed?). For example: setting FQDN/public IPs usually locked to some personal.

Having "linked tags/ sys-tags" sounds interesting if (and only if) it is implemented on ObjectMeta. However i think it moves the problem to a different object. Maybe i want user to create "linked sys-tags" objects but i don't want them to use tag "some-tag-that-gives-you-scarce-resource: foo-bar"

my 2cents

[Brian Grant]

On Thu, Sep 24, 2020 at 9:46 AM khn...@gmail.com <khn...@gmail.com> wrote:

The situation with labels spans more than just namespaces. For example  nanny style pods (running with higher privileges) running in some user namespaces. Every time access (or a scarce resource) is given because a selector matches it puts us in a situation where "a malicious or unknowing user can set label name and its corresponding value". I have seen this in every situation where labels are used for that. Controlling Access to specific properties looks a lot like ABAC (sidebar: it is ABAC at least on target side). It is a faux pas for most. I share equal dread. But if we decided to go that route then I expect every bigCo out there to have requirements around extending it to other properties. With valid use cases (monsters under the bed?). For example: setting FQDN/public IPs usually locked to some personal.

For well defined attributes, such as resource fields, state-based policy constraints along the lines of PodSecurityPolicy or Gatekeeper in admission control are typically preferable. They enable controlling the WHAT rather than the WHO or HOW.  

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/ba2ab59e-f9b4-4653-8d2c-810ba8be5872n%40googlegroups.com.

[Adrian Ludwin]

Even now, people would probably like to be able to restrict which users can attach a specific persistent volume or a secret to a Pod, which Ingress objects can use a particular Secret, etc.

This is already somewhat enforced by limiting usage of Secrets and PVCs to be within a namespace, correct? So we only need stronger access controls if we want Pods with differentiated security requirements to share the same namespace. Would you agree that for this case, having better ways to allow sharing across namespaces could be an alternative  to allowing us to create better barriers within namespaces?

[Adrian Ludwin]

Tags could be inherited using the hierarchical namespace controller:

https://kubernetes.io/blog/2020/08/14/introducing-hierarchical-namespaces/

...

In the meantime, in the case of NetworkPolicy, allowing specification of Namespace names, or even Namespace name prefixes, would be a pragmatic incremental fix for multi-tenant clusters. Creating Namespaces is generally a privileged operation, since otherwise Namespace-scoped policies could be circumvented by creating a new Namespace.

Note that in the special case of "allowing specification of Namespace names," HNC supports this by introducing trusted labels that reflect the hierarchical structure of the namespaces. At a bare minimum, every [1] namespace has at least one tree label with its own name, which were specifically design to allow hierarchical Network Policies. But it also works with degenerate hierarchies of single namespaces.

Does this sound like a useful interim solution to this problem?

Cheers, A

[1] We currently exclude certain namespaces like kube-system simply since we didn't want to touch critical parts of the cluster without a good reason, but I think adding these labels to all namespaces would be a reasonable and safe step to take.

[Adrian Ludwin]

This sounds a lot like nested namespaces ;)

Indeed

 

On Mon, Sep 21, 2020, 5:33 PM Tim Allclair <timal...@gmail.com> wrote:

Setting implementation aside, and thinking about the use cases:

Generally, what we're talking about is some authority wants to designate a group or set of things, e.g:

- "security team has designated a set of exceptional workloads as privileged", (and as a developer, I want to request a policy exception)

We're currently implementing HNC exceptions (label selectors within hierarchies for individual objects).

As a strawperson, what about something like this:

- a "resource set" object, with both a namespaced & cluster scoped version (namespaced version can only contain objects in the same namespace)

- resource sets can be nested (this enables the self-service piece)

- resource sets can apply "hierarchically", meaning I should be able to add a deployment to the set, and in doing so include all the replica sets and pods indirectly owned by the deployment

- where a label selector is used, a resource set list may also be supplied

As I mentioned in another reply, why not continue to use namespaces as the resource set boundary here?

Thanks, A

[Tim Hockin]

Lots of interesting points made all around! There seems to be general
buy-in that some sort of access-controlled metadata is necessary.

Quoting Brian: "allowing specification of Namespace names ... would be

a pragmatic incremental fix for multi-tenant clusters"

While I admit I was hoping for something more elegant (the discussion
of virtual-labels for metadata.name and metadata.uid may still bear
fruit) I agree it's distinct from "tags" as described here.

Tim

On Thu, Sep 24, 2020 at 11:02 AM 'Adrian Ludwin' via
kubernetes-sig-architecture

> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/eda3c1ba-de67-4bed-9e30-4dc959abcd58n%40googlegroups.com.

[Gurleen Grewal]

Hi everyone,

I’m a member of one of Google’s security teams and have been working with Tim AllClair on brainstorming ideas for a label policy. Given that this idea has come up a few times in prior discussions, we thought that it’s best if we start by listing some of the proposals in a doc.

Once people have had chance to review, it would be great if we could discuss this in a future sig-architecture meeting and even better if we can agree on some high level approaches.

Thanks,

Gurleen

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.

[Brendan Burns]

Access control on that doc prevents comments as far as I can tell.

Thanks!

---

From: 'Gurleen Grewal' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Wednesday, October 7, 2020 10:19 AM

To: kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: Re: [EXTERNAL] Re: The meaning and ownership of namespace labels

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/7c23364c-7b18-4d48-837e-c91318e8faf1o%40googlegroups.com.

[Gurleen Grewal]

My apologies. Please try again: link.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/7c23364c-7b18-4d48-837e-c91318e8faf1o%40googlegroups.com.