Multi Tenancy when Using Persistent Volumes

[krma...@gmail.com]

My team is currently trying to enable Stateful Apps for our internal customers. One requirement that keeps coming up is how to isolate PV's of one internal customer from PV's of another internal customer. 

I see the following isolation mechanisms:-

- A PV when bound to a PVC(inside namespace A) cannot be bound to another PVC(inside namespace B) unless the unbind happens and hence are exclusive. 

- When using StorageClass, A PV of certain class can only be bound to PVC of the same class. So that means PVC(of class A) can only be bound to PV(of class A). This allows a PV allocated to one customer to not accidently get allocated to another customer)

While the above isolation is good, its not enough(as i understand it). In a multi -tenant environment we want mechanisms which can guarantee that a volume allocated to one customer can never be accidentally allocated/mounted/accessed by another customer. 

What is Kubernetes recommendation , on how to achieve this isolation ?

Few more questions:-

- Why are Persistent Volumes not namespaced ?

- Is one or more  StorageClass'es per Customer a good multi tenancy model ? What other recommendations we have ?

Would love to hear the general thinking and around this from both developers and community

-Mayank

[David Oppenheimer]

Regarding the concern of reusing a PV from one tenant to another, can you just use the "delete" reclaim policy? That means the underlying volume will not be reused (of course you can't *really* guarantee anything about what happens under the covers for network storage when it tells you it has been deleted.) For PV types that do recycling, you could use a custom recycler, to ensure the data is really deleted.

If your set of tenants is very static, i guess you could have one StorageClass per tenant and only use "recycle" reclaim policy (which seems to be what you're advocating). But this seems pretty inefficient from a utilization standpoint, as you'd end up accumulating the max number of PVs used by each tenant.

What kind of volume type are you interested in?

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/3076b2ce-68f8-4324-b353-5644891b850b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Clayton Coleman]

For hard tenancy we use the delete policy extensively in combination with dynamic provisioning.  The goal is simply to ensure we never put a new volume into the pool without it coming from a clean disk.  Reclaim is going to be a best effort, but it's also more work to use and isn't as flexible as your own provisioner.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeszSrJYA530sX%3DUtua8uVWgUkKVkx%2BZryxGMFM1xM_VQ%40mail.gmail.com.

[Mike Danese]

I don't think this is possible today but if we supported disk
encryption (e.g. LUKS) on a subset of storage drivers, a "recycle" to
a tenant that didn't have access to the secret to unlock the disk is
essentially a cryptographic erase. This would allow a single tenant to
best case recycle a disk without consequence if the disk is reclaimed
by another tenant and without the inefficiencies of maintaining
multiple storage pools.

>> email to kubernetes-de...@googlegroups.com.
>> To post to this group, send email to kuberne...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-dev/3076b2ce-68f8-4324-b353-5644891b850b%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes developer/contributor discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-de...@googlegroups.com.
> To post to this group, send email to kuberne...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeszSrJYA530sX%3DUtua8uVWgUkKVkx%2BZryxGMFM1xM_VQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes developer/contributor discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-de...@googlegroups.com.
> To post to this group, send email to kuberne...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-dev/-2498848104924029026%40unknownmsgid.

[Tim Hockin]

Hard multi-year (whatever that means) is not currently a feature of kubernetes in a first class way.  There are many things you can do to approximate it, but it is not pervasive or consistently designed.

Re why PV is not namespaced, before provisioning was available, PVs were a cluster-owned thing.  Now with provisioning, we could maybe simplify, but we still need to be compatible, so there's that challenge.

--

You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

[Jeff Vance]

"... guarantee that a volume allocated to one customer can never be accidentally allocated/mounted/accessed by another customer"

I don't see "delete" as  the answer here since this deletes the actual data in the volume. What if the customer has legacy data or important data that lives beyond the pod(s) accessing it? Perhaps an "exclusive" accessMode might help? FSGroup IDs can control access and there's been talk about  adding ACLs to PVs. Or, maybe I've misunderstood the question?

jeff

[krma...@gmail.com]

Thanks all. The suggestion to use the reclaimPolicy of delete would not work.  As Jeff points out, in case of Stateful apps, we would like to not delete the volumes immediately  but keep them around for some time and garbage collect them later.

@David Could you elaborate on the inefficient utilization when you mention the following  ?"If your set of tenants is very static, i guess you could have one StorageClass per tenant and only use "recycle" reclaim policy (which seems to be what you're advocating). But this seems pretty inefficient from a utilization standpoint, as you'd end up accumulating the max number of PVs used by each tenant."

@David we are interested in RBD to start with. We would also be doing EBS Volumes as well. 

@Mike yes looks like encryption seems like a reasonable solution as long as we can separate the encryption key per tenant and only the tenant has access to its own encryption keys. EBS volumes are the only ones supporting encryption. So for RBD , we are out of luck. Is there a general pattern of doing encryption with external provisioning  ? What about incorporating in in-tree plugins ?

@Clayton Doesn't OpenShift have  use cases for keeping the volumes around after the pods are gone ? Does OpenShift do any kind of multi tenancy on top ?

@Tim thanks. Not advocating that PV should or should not be namespaced but just trying to understand the rationale for it. One thinking was if PV's were dynamically provisioned in the namespace of the customer, that might further limit their access.

@Jeff i think your understanding of the problem is correct. But its possible, i am solving the wrong problem and thats why i am seeking out for help.

Overall i want a multi tenant model, where:-

-- accidentally its not possible for one tenant to mount a volume created by another tenant

-- its more secure and attacks /compromise limit the surface area of attacks and access to volumes

In the absence of encryption if there is a Kubernetes native way of ACL'ing the PV's that doesnt rely on the underlying storage implementation would be great. 

[krma...@gmail.com]

Adding Kubernetes-Users group to gather thinking from the community.

[Mayank]

Pinging again. I would like to learn how others using Kubernetes are separating different tenants using dynamic provisioning. Specifically if people are using RBD provisioning , are they allocating a StorageClass and hence a Ceph pool per customer ? What advantages or disadvantages do you see with this approach ?

What isolation mechanisms you use to make sure a PV of one customer is never seen/used by another customer ?

-Mayank

[krma...@gmail.com]

opened the following to track the discussion https://github.com/kubernetes/kubernetes/issues/47326