[lxc-users] backup strategies for lxd + zfs

[Spike]

Hi,

I'm on Ubuntu Xenial running LXD from PPA + a ZFS storage backend.

I've been reading everything I could find on backup strategies, but I've not found anything "official" or that even looked like a best practice so I thought I'd ask the list.

First thing first, I think everybody agrees that you need /var/lib/lxd with lxd's database + the containers' fs.

For /var/lib/lxd + db it seems that a dump of sqlite + rsync should be sufficient, without the need to stop LXD during the process. Is this true or are there possible race conditions that require LXD to be stopped before the db can be dumped?

For the FS, given ZFS, it seems that a snapshot + send|receive should do, however I'm unclear if one can simply snap and move the entire pool or if there are reasons why images should be snap'ed individually and sent over.

The other approach I found is entirely different and 100% lxd based in which case you run LXD on the backup host too, snap containers, and then publish an image from the snaps to the backup host. However, unless I'm missing something, this process includes downtime because lxc snap will stop the container, but maybe I was doing something wrong when I tested (it makes sense in a way tho, it's trying to ensure consistency).

Besides the downtime, what I'm wondering about with this second approach is if it requires to still dumping/rsyincing the LXD's database or if the image will carry with itself all the configuration too.

Third, I found people using the image export command to generate tarballs which are then copied over to the backup host. Similarly to the publish mechanism, I'm unclear if this is going to carry over everything to restart the container configured as it was or not.

can anybody provide some recommendations from experience running lxc in a production environment with zfs?

thanks,

Spike

[Fajar A. Nugraha]

On Wed, Mar 8, 2017 at 12:08 AM, Spike <sp...@drba.org> wrote:

Hi,

I'm on Ubuntu Xenial running LXD from PPA + a ZFS storage backend.

I've been reading everything I could find on backup strategies, but I've not found anything "official" or that even looked like a best practice so I thought I'd ask the list.

 

First thing first, I think everybody agrees that you need /var/lib/lxd with lxd's database + the containers' fs.

Yes, but they have different requirements. 

An extreme case, if you have a fixed, never changing amount of containers, you only need to backup /var/lib/lxd once. FWIW, I also put /var/lib/lxd on zfs (I usually use the dataset name 'data/lib/lxd', with mountpoint set manually to /var/lib/lxd)

 

For the FS, given ZFS, it seems that a snapshot + send|receive should do, however I'm unclear if one can simply snap and move the entire pool or if there are reasons why images should be snap'ed individually and sent over.

Both might work. Depending on how well your send/receive mechanism handles new/deleted/renamed dataset.

The other approach I found is entirely different and 100% lxd based in which case you run LXD on the backup host too, snap containers, and then publish an image from the snaps to the backup host.

Too much additional cpu and disk usage ...

Third, I found people using the image export command to generate tarballs which are then copied over to the backup host.

... and unnecessary additional complexity.

can anybody provide some recommendations from experience running lxc in a production environment with zfs?

I use sanoid + syncoid (https://github.com/jimsalterjrs/sanoid). Works well, with the catch that (for syncoid's case) I need to manually specify which datasets I want to backup.

-- 

Fajar

[Spike]

hadn't seen sanoid, very cool, my zfs tooling is still pretty bare and I'm generally using obnam for linux backups. I was planning on finding something for automatic zfs snapshots and I guess sanoid answers that question, thank you. Also very nice trick to put lib/lxd under zfs too, hadn't thought of it.

thanks again,

Spike

_______________________________________________
lxc-users mailing list
lxc-...@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[Luis Felipe Marzagao]

Swap is ok on the host:

manager@andromeda:~$ sudo swapon --show
NAME       TYPE      SIZE USED PRIO
/dev/zram0 partition   2G 256K    5
/dev/zram1 partition   2G 256K    5
/dev/zram2 partition   2G 256K    5
/dev/zram3 partition   2G   0B    5
/swapfile  file        2G   0B   -2

manager@andromeda:~$ free -h
              total        used        free      shared buff/cache  
available
Mem:            15G         11G        809M         53M 3.0G        3.5G
Swap:          9.8G        768K        9.8G

But there is no swap inside containers:

manager@andromeda:~$ lxc exec pbx -- free -h
              total        used        free      shared buff/cache  
available
Mem:            15G        546M         14G         50M 187M         14G
Swap:            0B          0B          0B

manager@andromeda:~$ lxc exec sombrero -- free -h
              total        used        free      shared buff/cache  
available
Mem:            15G        630M         14G        164K 102M         14G
Swap:            0B          0B          0B

I didn´t change any settings regarding swap or memory. This is a pretty
standard ubuntu + lxd installation.

manager@andromeda:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

manager@andromeda:~$ snap list
Name    Version    Rev    Tracking       Publisher   Notes
core    16-2.45.2  9665   latest/stable  canonical✓  core
core18  20200724   1885   latest/stable  canonical✓  base
lxd     4.4        16530  latest/stable  canonical✓  -

Any pointers?

Thanks  a lot.

[Kees Bos]

On Sat, 2020-08-08 at 12:20 -0300, Luis Felipe Marzagao wrote:
<snip />
>
> Any pointers?
>

https://discuss.linuxcontainers.org/t/invalid-swaptotal-in-proc-meminfo-swaptotal-0/8231/17