Using Velero CSI plugin with PVC

[Suren Sethumadhavan]

Hello

I am trying to use Velero CSI plugin to backup a PVC.  Backup fails with an error (from the logs)

"level=error msg="Error backing up item" 

backup=velero/surens-csi-cluster-backup error="error executing custom action (groupResource=persistentvolumeclaims, namespace=default, name=csi-pvc): 

rpc error: code = Unknown 

desc = failed to get volumesnapshotclass for storageclass standard-rwo: failed to get volumesnapshotclass for provisioner pd.csi.storage.gke.io, 

 ensure that the desired volumesnapshot class has the velero.io/csi-volumesnapshot-class label" logSource="pkg/backup/backup.go:455" name=web-server"

Not sure how I would "volumesnapshot class has the velero.io/csi-volumesnapshot-class label". How do I go about debugging this? 

Thanks in advance for any insights.

Suren

[Swanand Shende]

Hi Suren,

       Thank you for the log dump. We can see:

• Name of your backup is surens-csi-cluster-backup.
• You have CSI snapshot feature gate enabled for your Velero installation.
• You have a PVC with name csi-pvc in namespace default. The provisioner for this PVC is pd.csi.storage.gke.io
• You are using Velero CSI plugin version 0.1.2

       Root Cause Analysis:

• Either there is no VolumeSnapshotClass object for provisioner pd.csi.storage.gke.io found, or, no VolumeSnapshotClass object for the same provisioner with label velero.io/csi-volumesnapshot-class found
  

       In this version of Velero CSI plugin, whenever a PVC (backed by a CSI volume) is found, a VolumeSnapshotClass is searched for. Two conditions need to be fulfilled by the VolumeSnapshotClass to allow Velero to execute a CSI snapshot of that PVC:

1. The VolumeSnapshotClass must have a matching provisioner: pd.csi.storage.gke.io
2. The VolumeSnapshotClass must have a label with key: velero.io/csi-volumesnapshot-class
3. It would be good to have the deletionPolicy set to "Retain" in order to preserve the storage snapshot when required during restore.

     Possible Solutions:

     1. Check if you have a VolumeSnapshotClass object created. If not, create one by copying the following content<Regard the indentations> to a YAML file and deploy it using kubectl:

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapshot-class # <Replace with whichever name you wish>

  labels:
   velero.io/csi-volumesnapshot-class: "true"

driver: pd.csi.storage.gke.io #< Replace with the name of CSI provisioner>
deletionPolicy: Retain

   2. If you already have a VolumeSnapshotClass present for provisioner pd.csi.storage.gke.io, Simply add a label to it.

       You could add a label to the exisiting VolumeSnapshotClass either by editing the resource object using kubectl edit volumesnapshotclass.snapshot.storage.k8s.io/<NAME of Volumesnapshotclass> and adding the label as shown in the example above, or execute the following command: kubectl label volumesnapshotclass <NAME of Volumesnapshotclass> velero.io/csi-volumesnapshot-class: "true"

   Try executing the backup again with these suggested changes. Let us know how it went...

    Reference: https://velero.io/docs/v1.5/csi/#implementation-choices

Regards,

Swanand

Catalogic Software

[Suren Sethumadhavan]

Hi Swanand

Thank you for the quick response.  I am still working my way through this, but it looks like I need to also create a StorageClass. I created one (see below) but that seems to mess up something in Velero (which was already installed). .Velero commands started complaining about BackupStorageLocation being not defined. I probably messed up my setup. So I am going tp clean it up and start with a new cluster.  Will update again after that.

Is there an particular order in which these things need to be done? 

1. Create cluster and set EnableCSI=true

2. Create StorageClass

3. Create VolumeSnapshotClass 

4. Create PVC and pods

5. Install Velero with gcp and csi plugin

Does this look right?

Thanks

Suren

[Swanand Shende]

Hi Suren,

      the only point to note for a successful CSI backup using Velero is that the PVC(which uses the CSI provisioner) must be in a "BOUND" state, Velero Server needs to be installed with the CSI flag enabled and Velero's BackupStorageLocation defined.

      So, the sequence could be:

1. Create a Kubernetes cluster

2. Create StorageClass for the CSI provisioner that is available. Before that, ensure the CSI drivers(CSI Provisioner) are deployed and Snapshotter CRD's (VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotclass) are available in the cluster.

3. Create the necessary PVC and pods. Ensure the POD, which is using the PVC, is in a running state.

4. Create VolumeSnapshotClass for the same CSI provisioner

5. Install Velero server with EnableCSI=true flag set. Add GCP and CSI plugins either during Velero server install, or later, but before initiating the backup. Configure the default backup storage location during install or create one later, again, before initiating the backup. (This step can also be done after Step 2 mentioned above)

6. Define/Create the necessary backup.

      Hope this helps.

Regards,

Swanand

https://cloudcasa.io/

[Suren Sethumadhavan]

HI Swanand

Thanks for all your help with this.  With your tips, I am now able to successfully take a backup of a CVI PV with Velero.  Restore also works.  I was also able to create snapshots manually.  I think the 2 things that helped were

1. creating a custom StorageClass for provisioner: pd.csi.storage.gke.io and making it the default. I also added an annotation <storageclass.kubernetes.io/is-default-class: 'true'> . I did this before installing Velero

2. create a VolumeSnapshotClass for driver: pd.csi.storage.gke.io.

The steps you outlined below are spot on.  One thing to note is that when creating the cluster on GKE, the master version must be 1.17+ . Just for the record I used version 1.17.13-gke.2600.

Thank you again.

Suren

[Swanand Shende]

Cheers!

--
You received this message because you are subscribed to a topic in the Google Groups "Project Velero" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/projectvelero/H2-YLV_Fkog/unsubscribe.
To unsubscribe from this group and all its topics, send an email to projectveler...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/projectvelero/52f4880d-9c2c-4f6b-a152-2688e80c92f4n%40googlegroups.com.

[Shibily Shukoor]

@swanand For me everything works fine until I try to restore. I have opened a ticket in GitHub. Any help would be appreciated.