Copy data from one PVC to another

[ta...@redhat.com]

Hi, 

I would like to copy the data behind one pvc to a newly-bound pvc in order to have a copy of that data available to my pod. I would appreciate suggestions/ideas how to do that.

Thanks,

Tzvika

[Michelle Au]

Can you start up a pod with both PVCs?

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-st...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/707f25ca-7120-4b26-a9ed-6441ed422967%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kangarlou, Ardalan]

Tzvika, there is currently no mechanism in Kubernetes to specify a copy/clone operation for a PVC. As Michelle suggested, you can do a manual copy, but this can take a long time for large volumes and can double the storage space usage.

 

With Trident (NetApp’s plugin for Kubernetes), we’re using a custom PVC annotation to enable the efficient copy/clone of a PVC based on the capabilities of the underlying NetApp backends: https://netapp.io/2017/12/14/trident-18-01-beta-1-introducing-volume-cloning-kubernetes/

 

I think the community would benefit if there is a standard way to specify the PVC copy operation (e.g., a standard annotation) and a way to plumb such annotation through the CSI plugins.

 

Thanks,

-Ardalan

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CAAG1FwjZ9qa3Lbx%2BVXeEsii1qXwbWu5pBg9fb8NbbtM3dJfSrg%40mail.gmail.com.

[gma...@kasten.io]

Hi Tzvika, 

As others have mentioned, there is no Kubernetes native mechanism for PVC copying. Depending on your broader use case, if you are looking to do some of this in the context of an application (high fidelity copy of the application including underlying data), you can take a look at Kasten's K10 platform (https://kasten.io). It does integrate with the underlying storage provider to be able to perform such operations efficiently and takes care of all the coordination with K8s to make this happen. Feel free to reach out if you'd like to know more. 

Thanks, 

-Georgi

[dna...@gmail.com]

Hi Tzvika,

Although there is no way to do this right now I believe the kubernetes snapshot project is working towards this. Portworx has built upon that to support creating snapshots from PVCs and then create PVCs from those snapshots through a project called STORK.

Instructions to use this snapshot/clone mechanism can be found here: https://docs.portworx.com/scheduler/kubernetes/snaps-local.html

Thanks

Dinesh

On Tuesday, June 5, 2018 at 11:11:50 PM UTC-7, ta...@redhat.com wrote:

[Adam Litke]

On Wed, Jun 6, 2018 at 10:11 AM, 'Michelle Au' via kubernetes-sig-storage <kubernetes-...@googlegroups.com> wrote:

Can you start up a pod with both PVCs?

This is a good idea and what I have been thinking also.  The only limitation I see is it would not work to clone across namespaces.  I know there had been some discussion on cross namespace operations in the past relating to the snapshots feature.  Is it feasible?

 

On Tue, Jun 5, 2018, 23:11 <ta...@redhat.com> wrote:

Hi, 

I would like to copy the data behind one pvc to a newly-bound pvc in order to have a copy of that data available to my pod. I would appreciate suggestions/ideas how to do that.

Thanks,

Tzvika

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-storage+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-storage@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/707f25ca-7120-4b26-a9ed-6441ed422967%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-storage+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-storage@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CAAG1FwjZ9qa3Lbx%2BVXeEsii1qXwbWu5pBg9fb8NbbtM3dJfSrg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam Litke

[John Griffith]

​As Dinesh pointed out it does seem that Snapshots might be a good fit here (without knowing much detail about what you need).  Maybe there are some details around your use case that this doesn't cover? 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-st...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/cc639833-5512-4257-862b-31163ae9e562%40googlegroups.com.

[Adam Litke]

On Thu, Jun 7, 2018 at 9:27 AM, John Griffith <john.gr...@gmail.com> wrote:

​As Dinesh pointed out it does seem that Snapshots might be a good fit here (without knowing much detail about what you need).  Maybe there are some details around your use case that this doesn't cover? 

It's true that you should be able to achieve a clone by: 1) creating a shapshot, 2) creating a new PVC based on the snapshot, and 3) deleting the snapshot.  For storage which can clone directly, the above is less efficient and more difficult to manage than a direct clone.  I think it makes sense to have both facilities.  Also, a host-assisted cloning mechanism could bring basic cloning support in a manner completely agnostic to the underlying storage.

 

On Wed, Jun 6, 2018 at 3:49 PM <dna...@gmail.com> wrote:

Hi Tzvika,

Although there is no way to do this right now I believe the kubernetes snapshot project is working towards this. Portworx has built upon that to support creating snapshots from PVCs and then create PVCs from those snapshots through a project called STORK.

Instructions to use this snapshot/clone mechanism can be found here: https://docs.portworx.com/scheduler/kubernetes/snaps-local.html

Thanks

Dinesh

On Tuesday, June 5, 2018 at 11:11:50 PM UTC-7, ta...@redhat.com wrote:

Hi, 

I would like to copy the data behind one pvc to a newly-bound pvc in order to have a copy of that data available to my pod. I would appreciate suggestions/ideas how to do that.

Thanks,

Tzvika

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-storage+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-storage@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/cc639833-5512-4257-862b-31163ae9e562%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-storage+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-storage@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CAPWkaSUqptFuTGwvsoSXEUJ6pwCS7c%2BvRM1ysxn4Z_VHDWZ1mg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam Litke

[ta...@redhat.com]

Thank you everyone for all your answers, it seems that at the moment there are many different ways to accomplish this (storage array clone, host-assisted copy, etc.) but there's nothing really designed for this purpose yet and no standard. Do you think it would make sense to start coordinating on a design for a controller/service that can orchestrate the flow for this use-case? E.g. use storage array clone when available, fallback to host-assisted copy when cross storage-classes, etc. Who is interesting in participating in this?