Shared Disk between VMs

[Kariste]

Hi,

    I need to create few vms in ganeti with a second common local disk attached. This is a feature that vmware[1] and ovirt[2] already support it since it is required from several cluster applications.

My setup is consisted of a few ganeti nodes with drbd and nfs vms. I tried to use the adopt disk feature but it didn't work with sharedifle or drdb. So the only solution that I found was to add a new sharedfile disk to every VM that need to have the shared storage, then edit config data and change the disk path to point to the same nfs file. Surprisingly this seems to work. The question is, is there a better way to achieve this in ganeti? Should I trust the solution with the same disk over nfs?

thank you

[1] https://theitbros.com/share-disk-between-vms-on-vmware-esxi/

[2] https://ovirt.org/develop/release-management/features/storage/sharedrawdisk/

[Darius Spitznagel]

Hello Kariste,

I don't think this will work until you use a cluster aware filesystem like OCFS2 on this virtual drive and configure the VMs which use it as OCFS2 Nodes.

What I mean is...

Did you check if you create a file from one VM on this shared disk with "touch /mnt/shared_disk/file1" you can see it on the other VMs? I'm pretty sure NO - you will damage your data on this disk pretty fast.

This is why you need a cluster aware filesystem like OCFS2.

Link 1 says:

In some cases (as a rule in clustering scenarios) it may be necessary to share the same disk between 2 (or more) virtual machines (VMs).

This VMtrash (sorry VMware) link does only talk about howto share a disk with multiple VMs.

Link 2 says:

Detailed Description

The shared disk feature should provide the ability to attach a disk to multiple VMs. It is the user's responsibility to make sure that the VMs do not corrupt disk data.

This means you have to install cluster aware FS.

And last but not least YES this works!!!!

I did this some months/years ago with three ganeti VMs with my "Shared LVM" based ext provider... similar to you.

https://github.com/ganeti/ganeti/wiki/External-Storage-Providers (look at the bottom).

1. I did install 3 VMs and OCFS2 on them.

2. Added a second disk to all VMs with the same size.

3. Configured all VMs as OCFS2 nodes.

4. Created Linux partion on 1st VMs second disk (shared disk) and formatted with OCFS2.

5. shutdown VM2 and VM3.

6. stopped ganeti on master.

7. took a backup from config.data

8. changed 2nd disk of VM2 and VM3 in config.data to the one from VM1.

9. Manually deleted 2nd disk from VM2 and VM3.

10. started ganeti on master > gnt-cluster verify > OK.

11. started VM2 and VM3.

12. "tail -f /var/log/syslog" to see VM2 and VM3 join the cluster.

This should also work with sharedfile as you did with nfs.

NOW the bad thing...

As long as you don't forget that VM1, VM2 and VM3 share the same disk all is cool.

BUT when you accidentally remove VM1, VM2 or VM3 from ganeti cluster then ganeti WILL REMOVE THE SHARED DISK WITHOUT WARNING.

Ganeti needs a feature to recognize that THIS disk is "shared" and simply check if it not used by any other instances before it can be deleted.

for example....

...

  Disks: 

    - disk/0: ext, size 8.0G

      access mode: rw,shared

      logical_id: ['lvm', '1a6d3ac0-2177-4ea0-8be5-5a1d27f1fde3.ext.disk0']

      on primary: /dev/rktstorage2/1a6d3ac0-2177-4ea0-8be5-5a1d27f1fde3.ext.disk0 (254:35)

      name: None

      UUID: 050baaea-277f-4e75-b426-b8f1d76afff5

...

Regards

Darius

Hinweis zur DSGVO (Datenschutzgrundverordnung):
Wir verarbeiten personenbezogene Daten zu geschäftlichen Zwecken.
Weitere Informationen finden Sie unter https://www.rkt-online.com/Datenschutzerklärung oder senden wir Ihnen auf Anfrage an in...@rkt-online.com gerne zu.

[candlerb]

> Should I trust the solution with the same disk over nfs?

Erm... I can't think why not, but the idea of cluster filesystems on top of disk image files stored over NFS with unknown levels of NFS caching gives me the willies.

What you want for this application is a SAN.  I would be inclined to create a drbd VM, add a disk, share that disk using iscsid or nbd, and then attach to it from within the other instances.

[sascha...@web.de]

Hi,

On Wedi, 25 Jul 2018 13:05:47 +0200 Kariste wrote:

> This is a feature that vmware[1] and ovirt[2] already support it

with the disk templates drbd, sharedfile, plain, file ganeti does not
support a shared disk between VMs. With disk template blockdev it will
work[1] (by accident). Therefore you will need something like iSCSI, FC ...
real blockdevs.

However, one may write/adopt an ESI[2], i.e. for NFS, like "Shared Filer" from
GRNET, to accomplish shared disks between VMs (with userspace access it
should be possible to skip losetup).

> use the adopt disk feature but it didn't work with sharedifle or drdb. So the
> only solution that I found was to add a new sharedfile disk to every VM that
> need to have the shared storage, then edit config data and change the disk
> path to point to the same nfs file. Surprisingly this seems to work. The
> question is, is there a better way to achieve this in ganeti? Should I trust
> the solution with the same disk over nfs?

This sounds valid. An other possebility instead of editing the config is to
alter the filsystem layout on the sharedfile dir, to link the shared disks
togher (hardlink on the NFS server?)

I see no reason, why you shouldn't trust this setup. VMware with datastore
on NFS does the same.

Thanks, Sascha.

[1] https://github.com/ganeti/ganeti/issues/1132
[2] https://github.com/ganeti/ganeti/wiki/External-Storage-Providers