[lxc-users] cluster node with different storage config

0 views
Skip to first unread message

Aleksandar Ivanisevic

unread,
Oct 4, 2020, 3:19:11 AM10/4/20
to LXC users mailing-list
Hi,

I’m trying to join a node to a lxd cluster that has different storage configuration. My cluster uses storage pool ‘local’ in zfs pool ‘ee’ and the new node uses ‘rpool/lxd’. Although the documentation (https://linuxcontainers.org/lxd/docs/master/clustering.html) suggests that should be possible, no matter what I put in the preseed yaml it always either tries to import the zfs pool ‘ee’ or create an already existing pool ‘local'

Does anyone know what is the correct syntax or is it possible at all?

$ lxd init --preseed < /tmp/cluster.yaml
$ cat /tmp/cluster.yaml
cluster:
enabled: true
server_name: ${HOSTNAME%%.*}
server_address: ${HOSTNAME%%.*}:8443
cluster_address: lxd1:8443
cluster_certificate: "-----BEGIN CERTIFICATE——
...
-----END CERTIFICATE-----
"
cluster_password: “..."
member_config:

# no matter what i put after this line it is always the same errors
# Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to create storage pool 'local': Storage pool directory "/var/snap/lxd/common/lxd/storage-pools/local" already exists
# or, if the new member’s local pool is not called ‘local’ then it complains about zpool import ‘ee’ failed.


- entity: storage-pool
name: cluster
key: source
value: "rpool/lxd"
key: zfs.pool_name
value: "rpool/lxd"

Aleksandar Ivanisevic

unread,
Oct 6, 2020, 8:56:32 AM10/6/20
to LXC users mailing-list
Hi,

at the end this worked for me:

:; cat<<EOF|lxd init --preseed
cluster:
  enabled: true
  server_name: ${HOSTNAME%%.*}
  server_address: ${HOSTNAME%%.*}:8443
  cluster_address: cluster:8443
  cluster_certificate: "$(sed ':a;N;$!ba;s/\n/\n\n/g' /tmp/cluster.crt)
"
  cluster_password: “password"
  member_config:
  - entity: storage-pool
    name: your-storage-pool-name
    key: source
    value: yourzfspool/yourzfsfs
EOF

the trick is that you have to first migrate or delete all the existing containers on the joining node and destroy the storage pool if it has the same name as the cluster pool. New node can only join the cluster completely empty, I don’t know why I was somehow expecting that the existing containers will be merged with the ones in the cluster. It is a nice feature request though ;)

Stéphane Graber

unread,
Oct 16, 2020, 12:07:31 PM10/16/20
to LXC users mailing-list
Joining nodes should be empty, they should not have any existing
network or storage pool configured in LXD.

> _______________________________________________
> lxc-users mailing list
> lxc-...@lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users

--
Stéphane

Reply all
Reply to author
Forward
0 new messages