CSI inline volumes and secrets

71 views
Skip to first unread message

George Costea

unread,
Aug 19, 2019, 11:39:39 AM8/19/19
to container-storage-interface-community
Hi all,

Reading through the CSI inline volume proposal (https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md), there's a Secret Reference section that states the following:

The secret reference declared in an ephemeral inline volume can only be used with namespaces from pods where it is referenced.

This would imply that the secret needs to be visible to the user that is creating the pod.  The user would be able to read any secret within the namespace the pod is created in after all.  What if the admin doesn't necessarily want to provide the secret to the user?  Can we enhance the proposal to also require the secret namespace?  This way the admin can create the secret in kube-system (for example) and just provide a reference to it to the user creating the pod.

Thanks,
George

Saad Ali

unread,
Aug 19, 2019, 2:01:09 PM8/19/19
to George Costea, container-storage-interface-community
George,

The NodePublishSecretRef is intended for "per CSI operation" or "per volume" which make sense to limit to the users namespace.

If a CSI Driver (ephemeral or persistent) requires a secret for a backend (a service account, for example), and this secret is required at the "per driver" granularity (not "per CSI operation" or "per volume"), then the secret can (and should) be injected directly in to CSI driver pods via standard Kubernetes secret distribution mechanisms during deployment. So, you could have a secret in some admin controlled namespace, and when you deploy the CSI driver it can have the secret injected as an env variable or file (you just need to ensure the driver is deployed with the correct permissions to read that secret).  

Regards,

Saad Ali


--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/00ca8f90-6564-485c-828e-74791d237bb1%40googlegroups.com.

George Costea

unread,
Aug 19, 2019, 2:22:47 PM8/19/19
to container-storage-interface-community
The downside here is that we have to redeploy our bits every time the secret changes.  Or worse yet, if we have multiple arrays with different secrets we have to pass all of them in during startup.  And we have to "know" which one to use each time.  That's not so dynamic.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

Saad Ali

unread,
Aug 19, 2019, 2:36:37 PM8/19/19
to George Costea, container-storage-interface-community
Inline.

On Mon, Aug 19, 2019 at 11:22 AM George Costea <rgco...@gmail.com> wrote:
The downside here is that we have to redeploy our bits every time the secret changes.

You shouldn't have to redeploy your CSI driver to pick up new secrets. As long as you design you driver to handle secret updates. For example, if you inject the secret as a file, k8s will automatically update the content of the file when the secret changes, if your driver can handle this, it doesn't need a redeployment for credential change.
 
Or worse yet, if we have multiple arrays with different secrets we have to pass all of them in during startup.

If you have multiple storage arrays, I recommend injecting the credential for each array the CSI driver will support at driver deployment time, and then in the StorageClasses for the driver, cluster admin can pick which array to use. Based on the selected array, the driver should use the correct injected secret.
 
And we have to "know" which one to use each time.  That's not so dynamic.

What does dynamic mean here? Does it mean automatic discovery of a new storage array? Does it mean automatic provisioning of a new storage array? 


On Monday, August 19, 2019 at 2:01:09 PM UTC-4, Saad Ali wrote:
George,

The NodePublishSecretRef is intended for "per CSI operation" or "per volume" which make sense to limit to the users namespace.

If a CSI Driver (ephemeral or persistent) requires a secret for a backend (a service account, for example), and this secret is required at the "per driver" granularity (not "per CSI operation" or "per volume"), then the secret can (and should) be injected directly in to CSI driver pods via standard Kubernetes secret distribution mechanisms during deployment. So, you could have a secret in some admin controlled namespace, and when you deploy the CSI driver it can have the secret injected as an env variable or file (you just need to ensure the driver is deployed with the correct permissions to read that secret).  

Regards,

Saad Ali


On Mon, Aug 19, 2019 at 8:39 AM George Costea <rgco...@gmail.com> wrote:
Hi all,

Reading through the CSI inline volume proposal (https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md), there's a Secret Reference section that states the following:

The secret reference declared in an ephemeral inline volume can only be used with namespaces from pods where it is referenced.

This would imply that the secret needs to be visible to the user that is creating the pod.  The user would be able to read any secret within the namespace the pod is created in after all.  What if the admin doesn't necessarily want to provide the secret to the user?  Can we enhance the proposal to also require the secret namespace?  This way the admin can create the secret in kube-system (for example) and just provide a reference to it to the user creating the pod.

Thanks,
George

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/dfb8bf2c-a383-4350-a5a2-c623f260a6c1%40googlegroups.com.

George Costea

unread,
Aug 19, 2019, 3:23:28 PM8/19/19
to container-storage-interface-community
So what if we add a second array later?  Won't we have to redeploy our CSI driver to be aware of a second secret?
 
If you have multiple storage arrays, I recommend injecting the credential for each array the CSI driver will support at driver deployment time, and then in the StorageClasses for the driver, cluster admin can pick which array to use. Based on the selected array, the driver should use the correct injected secret.
This works for normal PVC creation clearly.  But not so much with the inline volume design.  If we inject the credentials for each array, we still need to specify which credentials to use for an inline volume.  Are you suggesting we specify that in the volume attributes of the pod specification?

What does dynamic mean here? Does it mean automatic discovery of a new storage array? Does it mean automatic provisioning of a new storage array? 
It means adding a second array. 

On Monday, August 19, 2019 at 2:36:37 PM UTC-4, Saad Ali wrote:
Inline.

On Mon, Aug 19, 2019 at 11:22 AM George Costea <rgco...@gmail.com> wrote:
The downside here is that we have to redeploy our bits every time the secret changes.

You shouldn't have to redeploy your CSI driver to pick up new secrets. As long as you design you driver to handle secret updates. For example, if you inject the secret as a file, k8s will automatically update the content of the file when the secret changes, if your driver can handle this, it doesn't need a redeployment for credential change.
 
Or worse yet, if we have multiple arrays with different secrets we have to pass all of them in during startup.

If you have multiple storage arrays, I recommend injecting the credential for each array the CSI driver will support at driver deployment time, and then in the StorageClasses for the driver, cluster admin can pick which array to use. Based on the selected array, the driver should use the correct injected secret.
 
And we have to "know" which one to use each time.  That's not so dynamic.

What does dynamic mean here? Does it mean automatic discovery of a new storage array? Does it mean automatic provisioning of a new storage array? 


On Monday, August 19, 2019 at 2:01:09 PM UTC-4, Saad Ali wrote:
George,

The NodePublishSecretRef is intended for "per CSI operation" or "per volume" which make sense to limit to the users namespace.

If a CSI Driver (ephemeral or persistent) requires a secret for a backend (a service account, for example), and this secret is required at the "per driver" granularity (not "per CSI operation" or "per volume"), then the secret can (and should) be injected directly in to CSI driver pods via standard Kubernetes secret distribution mechanisms during deployment. So, you could have a secret in some admin controlled namespace, and when you deploy the CSI driver it can have the secret injected as an env variable or file (you just need to ensure the driver is deployed with the correct permissions to read that secret).  

Regards,

Saad Ali


On Mon, Aug 19, 2019 at 8:39 AM George Costea <rgco...@gmail.com> wrote:
Hi all,

Reading through the CSI inline volume proposal (https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md), there's a Secret Reference section that states the following:

The secret reference declared in an ephemeral inline volume can only be used with namespaces from pods where it is referenced.

This would imply that the secret needs to be visible to the user that is creating the pod.  The user would be able to read any secret within the namespace the pod is created in after all.  What if the admin doesn't necessarily want to provide the secret to the user?  Can we enhance the proposal to also require the secret namespace?  This way the admin can create the secret in kube-system (for example) and just provide a reference to it to the user creating the pod.

Thanks,
George

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

Saad Ali

unread,
Aug 19, 2019, 5:16:12 PM8/19/19
to George Costea, container-storage-interface-community
Inline.

On Mon, Aug 19, 2019 at 12:23 PM George Costea <rgco...@gmail.com> wrote:
So what if we add a second array later?  Won't we have to redeploy our CSI driver to be aware of a second secret?

It depends on how the CSI Driver is written and how it discovers storage arrays. If the discovery logic only runs at deployment time and depends on the person deploying, then yes it will require redeployment. If redeployment is unacceptable, the CSI driver must be desgined to handle automatic storage array discovery.

 
If you have multiple storage arrays, I recommend injecting the credential for each array the CSI driver will support at driver deployment time, and then in the StorageClasses for the driver, cluster admin can pick which array to use. Based on the selected array, the driver should use the correct injected secret.
This works for normal PVC creation clearly.  But not so much with the inline volume design.  If we inject the credentials for each array, we still need to specify which credentials to use for an inline volume.  Are you suggesting we specify that in the volume attributes of the pod specification?

Yes, the user would select the storage array they want for an ephemeral volume via inline volumeAttributes.
 
Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/909d2783-1b4a-48ed-9755-784dbe915d9f%40googlegroups.com.

George Costea

unread,
Aug 19, 2019, 6:11:49 PM8/19/19
to container-storage-interface-community
Yes, the user would select the storage array they want for an ephemeral volume via inline volumeAttributes.
This is essentially what we're looking for without all of the added complexity.  The secret is a great place to keep the storage array details.  When you deploy a new array, you can add a new secret, configure some new storage classes, then start using them without having to build a "discovery" mechanism into the CSI driver.  The csi provisioner, attacher, and snapshotter are all conducive to this approach.  It would be a nice improvement for inline volumes to behave the same way.
Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

Saad Ali

unread,
Aug 19, 2019, 6:45:33 PM8/19/19
to George Costea, container-storage-interface-community
Is the volume you are creating actually ephemeral (i.e. volume is created when pod is created, and is deleted when pod is deleted)? Or is the functionality you want "pod inline persistent volumes"? 

On Mon, Aug 19, 2019 at 3:11 PM George Costea <rgco...@gmail.com> wrote:
Yes, the user would select the storage array they want for an ephemeral volume via inline volumeAttributes.
This is essentially what we're looking for without all of the added complexity.  The secret is a great place to keep the storage array details.  When you deploy a new array, you can add a new secret, configure some new storage classes, then start using them without having to build a "discovery" mechanism into the CSI driver.

Storing configuration storage array config info in a secret seems wrong. It sounds like you do not want the end user to have to select the storage array (via inline volumeAttributes)? But you don't want the CSI driver to implement discovery logic in the StorageArray either because that would be complicated? Instead you want a cluster admin to set it somewhere "global"? And StorageClass would be a good place to specify it, but ephemeral volumes don't have StorageClass, so a "namespaced secret" for ephemeral inline volumes sounds like it would get you that? Is that accurate?
 
Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/28aba844-13a1-489b-aa8f-c6a3517cc1a8%40googlegroups.com.

George Costea

unread,
Aug 19, 2019, 7:02:28 PM8/19/19
to container-storage-interface-community
Yes, the volume follows the life of the pod.  It is created when the pod is created and deleted when the pod is deleted.

Storing configuration storage array config info in a secret seems wrong. 
That seems to be the purpose of the secrets.  The spec states "Secrets required by the plugin to complete volume create/delete/etc. operation".  Why wouldn't we store username and password here?

 It sounds like you do not want the end user to have to select the storage array (via inline volumeAttributes)? 
We do.  We want the user to select the storage array indirectly by selecting the secret.  This is one option.  I'm sure there are others.  We just noticed that NodePublishSecretRef is part of the Volume definition for the inline volume.  We are just looking for a way to allow secrets configured by an admin to allow volume creation rather than having to expose the secret details to the user.

Instead you want a cluster admin to set it somewhere "global"? And StorageClass would be a good place to specify it, but ephemeral volumes don't have StorageClass, so a "namespaced secret" for ephemeral inline volumes sounds like it would get you that? Is that accurate?
Ya.  More or less.  Again, this is in the spec.  Secrets are required by the plugin to complete volume create/delete operation.  But this particular use case requires a secret that is 'visible" to the user.  We really don't want the user to have access to the username and password of the storage array.
Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

Saad Ali

unread,
Aug 19, 2019, 7:32:49 PM8/19/19
to George Costea, container-storage-interface-community
On Mon, Aug 19, 2019 at 4:02 PM George Costea <rgco...@gmail.com> wrote:
Yes, the volume follows the life of the pod.  It is created when the pod is created and deleted when the pod is deleted.

Storing configuration storage array config info in a secret seems wrong. 
That seems to be the purpose of the secrets.  The spec states "Secrets required by the plugin to complete volume create/delete/etc. operation".  Why wouldn't we store username and password here?

You said "the secret is a great place to keep the storage array details" which I misunderstood to mean you wanted to put other configuration information in the secret. It's perfectly ok to store username/password/oauth tokens/SSH keys in a secret. So it sounds like the username/password determines which storage array gets used? Which is not an issue.


 It sounds like you do not want the end user to have to select the storage array (via inline volumeAttributes)? 
We do.  We want the user to select the storage array indirectly by selecting the secret.  This is one option.  I'm sure there are others.  We just noticed that NodePublishSecretRef is part of the Volume definition for the inline volume.  We are just looking for a way to allow secrets configured by an admin to allow volume creation rather than having to expose the secret details to the user.

If I understand correctly, you are asking for the ability for an admin to create one or more secret in a namespace (e.g. kube-system) that is different from the users namespace. And you want the user to be able to point to one of those secrets via NodePublishSecretRef (even though it is not in there namespace)?

The logical result of that is a user could point to any secret, in any namespace, and have that secret passed to a CSI Driver.

Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/fb51d942-e029-46a8-b82d-90886443c14e%40googlegroups.com.

George Costea

unread,
Aug 19, 2019, 7:56:20 PM8/19/19
to container-storage-interface-community
The logical result of that is a user could point to any secret, in any namespace, and have that secret passed to a CSI Driver.
Right.  But the same is true for a storage class referencing an "unknown" CSI provisioner.  I think the CSI driver would just have to report that the secret didn't work or adhere to what was expected.
Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

Saad Ali

unread,
Aug 19, 2019, 8:05:23 PM8/19/19
to George Costea, container-storage-interface-community
On Mon, Aug 19, 2019 at 4:56 PM George Costea <rgco...@gmail.com> wrote:
The logical result of that is a user could point to any secret, in any namespace, and have that secret passed to a CSI Driver.
Right.  But the same is true for a storage class referencing an "unknown" CSI provisioner.  I think the CSI driver would just have to report that the secret didn't work or adhere to what was expected.

StorageClass is different because it is a cluster-wide non-namespaced object. In a properly configured k8s cluster, only cluster admins should be able to create non-namespaced objects. StorageClass and PVs are considered cluster admin objects, where as Pods and PVCs are considered user objects.

Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/3775241d-af61-4cb4-ae54-6cfebb5047e8%40googlegroups.com.

George Costea

unread,
Aug 19, 2019, 8:49:23 PM8/19/19
to container-storage-interface-community
Ok.  So then how would you propose we provide the secret necessary to the CSI driver without inadvertently granting access to it to the user?

One thought we had was to just present the secret namespace in the volume attributes if it can't be part of the secret reference.
Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.

Saad Ali

unread,
Aug 20, 2019, 1:49:57 PM8/20/19
to George Costea, container-storage-interface-community
What it boils down to is your requirement that "the admin doesn't necessarily want to provide the secret to the user" -- this means the user objects (pods and PVCs) can not reference the secret, the secret must be set up by cluster admin and passed directly to the driver.
I recommend the solution we discussed earlier: deploy the CSI driver with the secrets for all known storage arrays. If automatic discovery of new storage arrays is important and redeployment is unacceptable, the CSI driver must handle automatic storage array discovery (this can be as simple as having the driver periodically check an admin created config map mounted as a file, for example). This way, the user simply selects the storage array they want via a volumeAttributes parameter in their pod definition.

On Mon, Aug 19, 2019 at 5:49 PM George Costea <rgco...@gmail.com> wrote:
Ok.  So then how would you propose we provide the secret necessary to the CSI driver without inadvertently granting access to it to the user?

One thought we had was to just present the secret namespace in the volume attributes if it can't be part of the secret reference.

Inline volumeAttributes means in the user controlled pod definition which we want to avoid.
 
Inline.

Inline.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/456a82b8-1862-4309-a781-ac02e51e07de%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages