Bug with restic backups on Kubernetes 1.21 when using restic by default

[Bridget McErlean]

Hi everyone,

Last week, we discovered an issue in Velero which affects restic backups on Kubernetes 1.21 clusters.

TL;DR: If you are using Velero on Kubernetes 1.21 and are backing up all volumes using restic by default, these backups cannot be restored from. Please switch to using the "opt-in" method for configuring restic and do not include any volumes which have a projected source such as Service Account volumes with the prefix "kube-api-access". Please upgrade to Velero 1.6.1 when it becomes available.

---

Kubernetes 1.21 introduced a change to use projected volumes for Service Accounts rather than secret based volumes. Usually, Velero will not attempt to backup volumes that have a secret or config map as the source as these are backed up separately. With the change to use volume projection, Velero will attempt to back these volumes up and upon restore they trigger an issue in restic which will cause the restore to fail.

This affects Velero 1.5 and Velero 1.6 which include the "--default-volumes-to-restic" feature which will backup all volumes using restic by default. Until we get a patch release out which skips the projected volume type, please switch to using the "opt-in" method for configuring restic and do not include any volumes which have a projected source such as Service Account volumes with the prefix "kube-api-access". Please note that this will only fix new backups and restores of older backups will not be possible until the patch release is available.

We already have one fix merged which will skip projected volumes during backup. We are also working on a fix to skip restore of projected volumes. We hope to have a 1.6.1 release available this week and will send another update here when it is ready.

Thanks,

Bridget

[Bridget McErlean]

Hi everyone,

To follow up on the previous message, Velero 1.6.1 is now available which includes a fix for the previously described issue when using restic by default on Kubernetes 1.21.

If you are running Velero on Kubernetes 1.21 and using restic, please upgrade to ensure that you can successfully restore your backups.

Please check the release notes for more details and see our documentation for details on upgrading to this version.

Thanks,

Bridget

---

From: Bridget McErlean
Sent: 15 June 2021 13:39
To: projec...@googlegroups.com <projec...@googlegroups.com>
Subject: Bug with restic backups on Kubernetes 1.21 when using restic by default