DRBD 8.4 and ubuntu 20.04?
826 views
Skip to first unread message
Daniel Howard
Jan 25, 2021, 2:58:13 PM1/25/21
to Ganeti Users list
Hello,
I'm trying out Apollon's new ganeti-3.0 PPA on Ubuntu 20.04 LTS.
Is there anywhere online that 3.0 docs are browsable? I understand there is a problem with getting http://www.ganeti.org/ and http://docs.ganeti.org/ updated .. :/
https://github.com/ganeti/ganeti/blob/master/doc/install.rst says supported DRBD is DRBD 8.0-8.4
Unfortunately, it looks like drbd8 has been dropped for Ubuntu 20.04: https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=drbd
Has anyone found a working combination?
Thanks,
-danny
Daniel Howard
Jan 25, 2021, 7:40:04 PM1/25/21
to Ganeti Users list
So far ... ganeti-3.0 seems to work just fine with DRBD9 ... I can create an instance, replace the secondary node, migrate ... is anyone else running with DRBD9? Are there any suggested checks I shuod make?
My present big limitation is synnefo appears to want a python2 library and ubuntu20 is pretty opposed to getting python2 libraries. I guess tomorrow I'll figure out how to make python2 happy or ... find an alternative way to bootstrap ubuntu VMs on ganeti. (All we really need is something to set up a disk, static IP, ssh, and a single user who can log in and sudo.)
My present big limitation is synnefo appears to want a python2 library and ubuntu20 is pretty opposed to getting python2 libraries. I guess tomorrow I'll figure out how to make python2 happy or ... find an alternative way to bootstrap ubuntu VMs on ganeti. (All we really need is something to set up a disk, static IP, ssh, and a single user who can log in and sudo.)
-danny
Phil Regnauld
Jan 26, 2021, 3:35:50 AM1/26/21
to gan...@googlegroups.com
Daniel Howard (dannyman) writes:
> So far ... ganeti-3.0 seems to work just fine with DRBD9 ... I can create
> an instance, replace the secondary node, migrate ... is anyone else running
> with DRBD9? Are there any suggested checks I shuod make?
That's interesting - is drbd9 just backwards compatible ? I thought there
> So far ... ganeti-3.0 seems to work just fine with DRBD9 ... I can create
> an instance, replace the secondary node, migrate ... is anyone else running
> with DRBD9? Are there any suggested checks I shuod make?
were enough changes that it didn't just work. I wonder if things like
protocol / resync speed settings work.
> My present big limitation is synnefo appears to want a python2 library and
> ubuntu20 is pretty opposed to getting python2 libraries. I guess tomorrow
> I'll figure out how to make python2 happy or ... find an alternative way to
> bootstrap ubuntu VMs on ganeti. (All we really need is something to set up
> a disk, static IP, ssh, and a single user who can log in and sudo.)
suggested recently:
https://github.com/pdxcat/ganeti_cloud-init
https://github.com/neicnordic/ganeti-os-nocloud
Sascha Lucas
Jan 26, 2021, 8:27:26 AM1/26/21
to gan...@googlegroups.com
Hi,
Phil Regnauld writes:
> Daniel Howard (dannyman) writes:
> > So far ... ganeti-3.0 seems to work just fine with DRBD9 ... I can create
> > an instance, replace the secondary node, migrate ... is anyone else running
> > with DRBD9? Are there any suggested checks I shuod make?
>
> That's interesting - is drbd9 just backwards compatible ? I thought
> there
> were enough changes that it didn't just work. I wonder if things like
> protocol / resync speed settings work.
please do not mix drbd-utils and drbd-kernel-module. The first is compatible
with drbd8 and 9, while the in kernel module still is 8.4. see also
/proc/drbd or "modinfo drbd". the drbd-kernel-module-9 is not in mainline
kernel (maybe as a dkms-package in debian/ubuntu).
HTH, Sascha.
Phil Regnauld writes:
> Daniel Howard (dannyman) writes:
> > So far ... ganeti-3.0 seems to work just fine with DRBD9 ... I can create
> > an instance, replace the secondary node, migrate ... is anyone else running
> > with DRBD9? Are there any suggested checks I shuod make?
>
> That's interesting - is drbd9 just backwards compatible ? I thought
> there
> were enough changes that it didn't just work. I wonder if things like
> protocol / resync speed settings work.
with drbd8 and 9, while the in kernel module still is 8.4. see also
/proc/drbd or "modinfo drbd". the drbd-kernel-module-9 is not in mainline
kernel (maybe as a dkms-package in debian/ubuntu).
HTH, Sascha.
Daniel Howard
Jan 26, 2021, 12:42:38 PM1/26/21
to Ganeti Users list
On Tue, Jan 26, 2021 at 12:35 AM Phil Regnauld <regn...@gmail.com> wrote:
https://github.com/pdxcat/ganeti_cloud-init
https://github.com/neicnordic/ganeti-os-nocloud
I am going to explore nocloud today. If I get it to work I reckon to submit a PR to make the instructions more accessible to simple folks like me. :)
Daniel Howard
Jan 26, 2021, 12:56:33 PM1/26/21
to Ganeti Users list
On Tue, Jan 26, 2021 at 5:27 AM Sascha Lucas <sascha...@gisa.de> wrote:
please do not mix drbd-utils and drbd-kernel-module. The first is compatible
with drbd8 and 9, while the in kernel module still is 8.4. see also
/proc/drbd or "modinfo drbd". the drbd-kernel-module-9 is not in mainline
kernel (maybe as a dkms-package in debian/ubuntu).
Oh, Sascha, thank you for bringing this up. It looks like ubuntu 20.04 has the drbd 8 kernel module and the 9drbd 9 utilities.
And the system has man pages for 8.3.2, 8.4.0 ...
Cheers,
-danny
Daniel Howard
Jan 27, 2021, 1:27:14 PM1/27/21
to Ganeti Users list
On Tue, Jan 26, 2021 at 5:27 AM Sascha Lucas <sascha...@gisa.de> wrote:
please do not mix drbd-utils and drbd-kernel-module. The first is compatible
with drbd8 and 9, while the in kernel module still is 8.4. see also
/proc/drbd or "modinfo drbd". the drbd-kernel-module-9 is not in mainline
kernel (maybe as a dkms-package in debian/ubuntu).
Update:
Out of the box, Ubuntu 20.04 wants to give you drbd 8 in the kernel, and drbd 9 utils.
I tried that out, and it "works" but after moving an instance's disks around a bit I get errors that the old LVM can't be removed because someone is using it. Huh?
So, I went to https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack and installed drbd9-dkms and drbd9-utils and now all the drbd matches up with itself and ganeti yells at me:
Wed Jan 27 10:20:11 2021 - WARNING: Device creation failed
Failure: command execution error:
Can't create block device <DRBD8(hosts=62555880-2775-4210-a9ef-c5d876d93801/0-ee61ba88-b67b-4f0c-9e98-c4e0f628c987/0, port=11001, backend=<LogicalVolume(/dev/xenvg/bee9203e-64ae-4060-bebf-31d8411b36b7.disk0_data, not visible, size=16384m)>, metadev=<LogicalVolume(/dev/xenvg/bee9203e-64ae-4060-bebf-31d8411b36b7.disk0_meta, not visible, size=128m)>, visible as /dev/disk/0, size=16384m)> on node dr64-booby.las.quantifind.com for instance gnt18.las.quantifind.com: Can't create block device: Mismatch in DRBD kernel version and requested ganeti usage: kernel is 9.0, ganeti wants 8.x
So ... ganeti will tolerate drbd9 utilities but the kernel version is a hard 8.
I am going to REMOVE the drbd9 stuff and see if I can, say, install the ubuntu 18.04 drbd8-utils package.
Out of the box, Ubuntu 20.04 wants to give you drbd 8 in the kernel, and drbd 9 utils.
I tried that out, and it "works" but after moving an instance's disks around a bit I get errors that the old LVM can't be removed because someone is using it. Huh?
So, I went to https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack and installed drbd9-dkms and drbd9-utils and now all the drbd matches up with itself and ganeti yells at me:
Wed Jan 27 10:20:11 2021 - WARNING: Device creation failed
Failure: command execution error:
Can't create block device <DRBD8(hosts=62555880-2775-4210-a9ef-c5d876d93801/0-ee61ba88-b67b-4f0c-9e98-c4e0f628c987/0, port=11001, backend=<LogicalVolume(/dev/xenvg/bee9203e-64ae-4060-bebf-31d8411b36b7.disk0_data, not visible, size=16384m)>, metadev=<LogicalVolume(/dev/xenvg/bee9203e-64ae-4060-bebf-31d8411b36b7.disk0_meta, not visible, size=128m)>, visible as /dev/disk/0, size=16384m)> on node dr64-booby.las.quantifind.com for instance gnt18.las.quantifind.com: Can't create block device: Mismatch in DRBD kernel version and requested ganeti usage: kernel is 9.0, ganeti wants 8.x
So ... ganeti will tolerate drbd9 utilities but the kernel version is a hard 8.
I am going to REMOVE the drbd9 stuff and see if I can, say, install the ubuntu 18.04 drbd8-utils package.
-danny
Sascha Lucas
Jan 28, 2021, 11:10:29 AM1/28/21
to Ganeti Users list
Hi Daniel,
On Wed, 27 Jan 2021, Daniel Howard wrote:
> Out of the box, Ubuntu 20.04 wants to give you drbd 8 in the kernel, and
> drbd 9 utils.
>
> I tried that out, and it "works" but after moving an instance's disks
> around a bit I get errors that the old LVM can't be removed because someone
> is using it. Huh?
There is really no need to change drbd-utils version. drbd-utils-9.x should
be fine for 8.4-in-kernel-module. As far as I remember, it was written
with full 8.4 compatibility in mind (both sides: the kernel module and
consumers like Ganeti).
It's more likely that DRBD upstream has not yet pushed DRBD-9/10 into
mainline kernel, which could be a difficult process. DRBD-8 has long
history outside mainline kernel and the main reason for including DRBD-8
was, that everyone is using it, so it can be included.
The reason for some LVs can not be closed must be somewhere else. Maybe
stacked LVM (LVM inside Ganeti instances), without supplying a lvm filter,
or active partition mappings (i.e. from instance-debootstrap). Please try:
# on effected LV
$ lsof /dev/mapper/vgX_lvX
$ lsof /dev/dm-X # with X where symlink in /dev/mapper points to
or more likely
$ dmsetup ls
to find out who/what is holding things open.
HTH, Sascha.
On Wed, 27 Jan 2021, Daniel Howard wrote:
> Out of the box, Ubuntu 20.04 wants to give you drbd 8 in the kernel, and
> drbd 9 utils.
>
> I tried that out, and it "works" but after moving an instance's disks
> around a bit I get errors that the old LVM can't be removed because someone
> is using it. Huh?
be fine for 8.4-in-kernel-module. As far as I remember, it was written
with full 8.4 compatibility in mind (both sides: the kernel module and
consumers like Ganeti).
It's more likely that DRBD upstream has not yet pushed DRBD-9/10 into
mainline kernel, which could be a difficult process. DRBD-8 has long
history outside mainline kernel and the main reason for including DRBD-8
was, that everyone is using it, so it can be included.
The reason for some LVs can not be closed must be somewhere else. Maybe
stacked LVM (LVM inside Ganeti instances), without supplying a lvm filter,
or active partition mappings (i.e. from instance-debootstrap). Please try:
# on effected LV
$ lsof /dev/mapper/vgX_lvX
$ lsof /dev/dm-X # with X where symlink in /dev/mapper points to
or more likely
$ dmsetup ls
to find out who/what is holding things open.
HTH, Sascha.
Daniel Howard
Jan 28, 2021, 5:38:54 PM1/28/21
to Ganeti Users list
I posted all I have learned this week here: https://groups.google.com/g/ganeti/c/wgsc5lpkjrI
I'll just "close the loop" here, given the chance that you are looking at this thread based on a Google search. ;)
I'll just "close the loop" here, given the chance that you are looking at this thread based on a Google search. ;)
On Thu, Jan 28, 2021 at 8:10 AM Sascha Lucas <sascha...@web.de> wrote:
There is really no need to change drbd-utils version. drbd-utils-9.x should
be fine for 8.4-in-kernel-module. As far as I remember, it was written
with full 8.4 compatibility in mind (both sides: the kernel module and
consumers like Ganeti).
When I was trying it out, the drbd9-utils seemed to work just fine with the drbd8 kernel. I figured people writing disk utilities would be conscientious about supporting compatibility across versions. I did find that I could install drbd8-utils built for Ubuntu 18, so I am rolling with that for now. Had I not encountered (unrelated) issues I would probably be running with the drbd9-utils.
It's more likely that DRBD upstream has not yet pushed DRBD-9/10 into
mainline kernel, which could be a difficult process. DRBD-8 has long
history outside mainline kernel and the main reason for including DRBD-8
was, that everyone is using it, so it can be included.
Linbit supplies a PPA that lets you install drbd9-dkms and drbd9-utils. It seems very nice. When I tried the drbd9 kernel, Ganeti said "no no no we don't do DRBD9" so ... I took that out.
I am a little worried that the kernel will move to DRBD9 before Ganeti will accept it. I really really really hope Canonical doesn't "surprise" us in our LTS kernel. :)
The reason for some LVs can not be closed must be somewhere else.
It was hard to track this down, but the big win for me came to "disable multipathd" which for some reason gets activated on Ubuntu 20.04 and it really messes with DRBD.
-danny
Brian Candler
Feb 5, 2021, 9:52:25 AM2/5/21
to ganeti
On Thursday, 28 January 2021 at 22:38:54 UTC Daniel Howard wrote:
Linbit supplies a PPA that lets you install drbd9-dkms and drbd9-utils. It seems very nice. When I tried the drbd9 kernel, Ganeti said "no no no we don't do DRBD9" so ... I took that out.
DRBD9 is quite different in fundamental ways: it allow multiples replicas (up to 32), and remote access from diskless clients. Linbit have used this to great effect in "Linstor", which also uses DRBD-over-LVM replication as pioneered by Ganeti.
I've written up some experiments with Linstor here:
https://brian-candler.medium.com/linstor-networked-storage-without-the-complexity-c3178960ce6b
https://brian-candler.medium.com/linstor-networked-storage-without-the-complexity-c3178960ce6b
A Ganeti extstorage plugin for Linstor looks like it would be straightforward to implement. But also, using Linstor with Proxmox VE (which is already integrated) could be a usable alternative to Ganeti.
Cheers,
Brian.
Reply all
Reply to author
Forward
0 new messages