Re: [kubernetes/kubernetes] Multi Tenancy for Persistent Volumes (#47326)
Mike Danese
yes looks like encryption seems like a reasonable solution as long as we can separate the encryption key per tenant and only the tenant has access to its own encryption keys. EBS volumes are the only ones supporting encryption. So for RBD , we are out of luck. Is there a general pattern of doing encryption with external provisioning ? What about incorporating tenancy in in-tree plugins ?
Any volume plugin that is backed by a block device can probably support LUKS. @kubernetes/sig-storage-feature-requests have we ever discussed LUKS encryption layers for block device volumes?
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Huamin Chen
Why does AttachDisk on volumes always happen on the node using kubelet ? The reason i ask is ,in some networked storage, for e.g. EBS and RBD, this can be dome from the master nodes. Doing this from kubelet means exposing more permissions to users then is necessary.
AttachDisk happens on Kubernetes master for cloud block storage (EBS, PD, Cinder, Azure). Kubelet doesn't have to have privileged credentials.
krmayankk
Interesting @rootfs @msau42 so in RBD, why do we need the user secret in user namespace or it could be my wrong understanding. Putting it other way, in RBD we need two secrets admin and user. My understanding is that user secrets must be in the same namespace as the PVC and they are used for AttachDisk. If AttachDisk is happening on master, we should allow the user secret to be in any namespace or accept a namespace field for it.
Huamin Chen
Interesting @rootfs @msau42 so in RBD, why do we need the user secret in user namespace or it could be my wrong understanding. Putting it other way, in RBD we need two secrets admin and user. My understanding is that user secrets must be in the same namespace as the PVC and they are used for AttachDisk. If AttachDisk is happening on master, we should allow the user secret to be in any namespace or accept a namespace field for it.
rbd doesn't support 3rd party attach, so rbd map has to happen on kubelet.
rbd admin and user keyrings are for different purposes: admin keyring for rbd image provisioning (admin privilege), while user keyring for rbd map (non admin privilege). Pods that use rbd image don't have admin keyrings.
krmayankk
Michelle Au
The attach operation is only performed by the attach/detach controller. And that controller is only enabled to run in the master node by default. There is a kubelet option to turn on attach/detach on the node, but it's going to be removed.
Jan Šafránek
Overall i want a multi tenant model, where:-
-- accidentally its not possible for one tenant to mount a volume created by another tenant
That's already implemented. If a PV gets bound to a PVC the PV can never get bound to another PVC. Only the pods in the same namespace as the PVC can use it.
When the PVC is deleted, the PV gets Released. Based on persistentVolumeReclaimPolicy of the PV, the PV is either deleted or recycled (data on the PV are discarded in both cases) or remains Released forever and thus nobody can bind to it. Only admin can manually access data on the PV or forcefully bind the PV to another PVC.
Deyuan Deng
I think one key point from what @krmayankk describes is the identity of a PV (e.g. PV created by a tenant, PV of an internal customer). After binding, PV is attached the identity of the PVC, but the binding process itself doesn't seem to take that into consideration. From what I know, it looks into selector, storageclass, accessmode, etc. As of now, it seems the best way is to mimic identity information using selector and storageclass.
@krmayankk why do you want to reserve PV for a tenant?
krmayankk
@jsafrane @ddysher in case of dynamic provisioning, the reclaim policy is always delete. But enterprises might want to still keep the PV and not delete them immediately for safety reasons. So we cant make use of the Released phased for preventing the binding.
The identity of the PV is important because, in no circumstance we want the binding process to accidentally bind a PV of one customer to a PVC of different customer. There is nothing that prevents it. Agreed that until a PVC is bind to PV, it cannot bind to another PV, but if accidentally we have some unbound PV's (due to bug or whatever) from customer A, we would want them to not get bound to customer B's PVC. The only way to prevent that today i can think of is assign per customer storage classes.
Michelle Au
@krmayankk once the PV is deleted, then there is no PV object that exists anymore, so you can't accidentally bind to it. You would have to recreate the PV, either statically or through dynamic provisioning.
Is what you really want the ability to specify Retain policy for dynamically provisioned PVs, so that you have the chance to clean up the data before it gets put back into the provisioning pool?
krmayankk
@msau42 Since dynamic provisioning doesnt support reclaim policy and always deletes. What we are doing is when deleting statefulsets, we are not deleting the PVC's. That way we explicitly GC the PVC at a later time and hence the PV. While the PVC and hence the PV is in GC phase, i am worried that the PVC could get unbound due to bugs and hence the PV will become available for further binding by other tenants. Do you think this is possible ? If somehow the PVC/PV get unbound for a dynamically provisioned PV, will the phase of PV be Release or Available ?
Yes the ability to specify Retain policy would really give us a chance to clean up the data although some tenants may not want that and they would want new PV not recycled ones
Deyuan Deng
If a PV is bound to a PVC, then there are two pieces of information about the two-way binding:
- In PVC,
spec.volumeNametells which PV the PVC is bound to - In PV,
spec.claimReftells which PVC is bound with the PV
pvc.spec.volumeName can not be edited if already set. pv.spec.claimRef can be removed entirely and if so, pv.status.phase will become Available. However, since pvc.spec.volumeName is non-empty and point to the pv, pv_controller will try to bind the PV and PVC again.
My suspect is that In between, if another PVC is also pending, then it's possible that the other PVC will bind to the PV, and the original PVC will just stay as is (and pv_controller keeps rebinding and keeps failing). If my understanding is correct, I think the likelihood that PV is bound by other tenants is pretty low, if not impossible.
If somehow the PVC/PV get unbound for a dynamically provisioned PV, will the phase of PV be Release or Available ?
—
Michelle Au
Assuming no recycle policy, once the PV is unbound, it always goes to Released or Retain, or the PV object is deleted entirely. The same PV object cannot go to Available state. So the same PV object is never recycled. It's always a new PV object.
Now, whether or not the data on the underlying backing volume gets cleaned up before being put back into the storage pool (used for dynamic provisioning) by the storage provider is a different story, and it's going to depend on each provider. For example, for GCE PD, when the disk gets deleted, it is guaranteed that the content gets cleaned up in the underlying volume before it could be reused for a new disk. For local storage, the provided external provisioner will cleanup the data when the PV is released. For other volume plugins, that may not be the case, and I believe that is why @krmayankk wants the Retain policy to be able to manually cleanup the data on the volume.
I still think if we had the ability to set reclaim policy to Retain on dynamically provisioned volumes, then it should be able to address your concern about being able to clean up volumes before being used again by other tenants.
Jeff Vance
See also issue 38192
krmayankk
Assuming no recycle policy, once the PV is unbound, it always goes to Released or Retain, or the PV object is deleted entirely. The same PV object cannot go to Available state. So the same PV object is never recycled. It's always a new PV object.
I just tested by manually editing the PV's claimRef of an already Bound PV. I made the claimRef empty string, and soon the PV became Available and remaining available for ever. Is this a bug . Again all my tests are on 1.5.3
Michelle Au
@krmayankk A user is not going to "accidentally" delete the PV claimRef. An admin would have to intentionally do that. Is that a scenario you really want to protect from? I thought the scenario you are worried about is if a user accidentally deletes their PVC.
krmayankk
@msau42 i think the probability of a PV getting unbound from PVC is low(lets say there could be bugs in pv controller related to that). What would be best is that once a PV is dynamically provisioned, there is no way for it to become Available ever again in the realm of certain reclaimPolicy. Available is a phase which is not admin editable, even if an admin intentionally removes the binding, the PV should never go to Available. May be allow this through a reclaim policy which supports this. Maybe allow only in recycle for the PV to go to available, but never in delete, even if some admin accidentally unbinds it.
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale
fejta-bot
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
Maxim Ivanov
/remove-lifecycle stale
Calvin Hartwell
@krmayankk sorry to respond to this necro post, but did you come to any conclusion on this? I assume you can create a LimitRange which is really small for namespaces which should be restricted from using certain storageclasses, correct?
Have CSI plugins in k8s 1.10 and the new storage improvements in 1.11 sorted out any issues?
https://kubernetes.io/docs/tasks/administer-cluster/limit-storage-consumption/
Restricting storage access with Ceph is quite easy, not sure about NetApp using Trident or other mechanisms though right now.
Thanks!
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
—
krmayankk
/remove-lifecycle stale
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Kubernetes Prow Robot
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Kubernetes Prow Robot
Closed #47326.
krmayankk
/remove-lifecycle rotten
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
krmayankk
/reopen
krmayankk
/lifecycle frozen
Kubernetes Prow Robot
Reopened #47326.
Kubernetes Prow Robot
@krmayankk: Reopened this issue.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—