NOTES: ganeti-3.0 on Ubuntu 20.04 using DRBD

[Daniel Howard]

Hello,

Thanks a lot to Apollon and Petter and really everyone who makes Ganeti happen, I have a very nice ganeti-3.0 cluster running on Ubuntu 20.04. There are a few tricks I thought worth sharing for posterity.

1) Obtain drbd8-utils. On Ubuntu 20.04, the kernel comes with drbd8, but the OS only supplies drbd-utils for version 9. Ganeti will refuse to operate with a drbd9 kernel. So, I installed drbd8-utils from the previous LTS:

https://mirrors.edge.kernel.org/ubuntu/pool/main/d/drbd-utils/

Specifically:

sudo dpkg -i https://mirrors.edge.kernel.org/ubuntu/pool/main/d/drbd-utils/drbd-utils_8.9.10-2ubuntu0.1_amd64.deb

(The ubuntu0.1 version has an important patch for newer kernel versions.)

2) Disable multipathd. I was getting hella errors from DRBD that disks were left open, but I could not track down how. After two days of frustration, I noticed some log messages from multipathd. When I turned it off, things were pretty great! I don't speak fluent systemd, but here's my Ansible:

- name: Disable multipathd
  when: ansible_distribution == 'Ubuntu' and ansible_distribution_version|float >= 20
  systemd:
    name: multipathd
    enabled: no
    state: stopped

I don't know what multipathd is or why it is there. Ubuntu 18.04 does not launch this service. I assume this is part of a scheme from systemd to Make Linux Great Again.

3) Use ganeti-os-nocloud. We have used snf-image for years, but it does not support Ubuntu-20.04 as a guest, and it needs to build a Python 2.7 package, and Ubuntu and I guess Python are pretty hostile to that. So I dropped snf-image for nocloud. This was a little confusing at first but on the whole, a better solution. One advantage is that Ubuntu supplies base images that are updated daily, so you can periodically download fresh base images and when you spin up an instance it wastes less time running security patches.

https://github.com/neicnordic/ganeti-os-nocloud

I just added an example user-data on my fork of the repo:

https://github.com/dannyman/ganeti-os-nocloud/tree/master/examples/ubuntu/user-data

4) Make DRBD faster!

There is an outstanding issue to tune the default values supplied by Ganeti. These values are applied to DRBDs upon creation, so it is good to get something in when you first build the cluster. The issue and various solutions are offered here:

https://github.com/ganeti/ganeti/issues/1229

Brian Candler has a similar environment to ours, so I cribbed his config and I am very pleased with the results:

gnt-cluster modify -D drbd:resync-rate=204800,disk-custom='--c-plan-ahead 0',net-custom='--max-buffers 16384 --max-epoch-size 16384'

5) Share what you learn with the community!

As far as I know, we may be the first shop running ganeti-3.0 on Ubuntu 20.04. Some of the tricks here were so frustrating that I was close to giving up on this project a few times. I could not have gotten things running without the support of the community and all the folks who have shared their software, documentation, and troubleshooting through the years. I hope these notes help the next folks who want to get Ganeti running on Ubuntu skip some frustrations and achieve a working cluster quickly!

Thanks Again!

-danny

--

http://dannyman.toldme.com

[Phil Regnauld]

Daniel Howard (dannyman) writes:
> *2) Disable multipathd.* I was getting hella errors from DRBD that disks

> were left open, but I could not track down how. After two days of
> frustration, I noticed some log messages from multipathd. When I turned it
> off, things were pretty great! I don't speak fluent systemd, but here's my
> Ansible:

Multipathd might have been pulled in by drbd9 ?

Try 'aptitude why multipathd --show-summary'

> https://github.com/neicnordic/ganeti-os-nocloud
>
> I just added an example user-data on my fork of the repo:
>
> https://github.com/dannyman/ganeti-os-nocloud/tree/master/examples/ubuntu/user-data

That's a nice one, cool!

Thanks for the updates/feedback.

Cheers,
Phil

[Phil Regnauld]

Phil Regnauld (regnauld) writes:
>
> Try 'aptitude why multipathd --show-summary'

Actually 'aptitude why multipath-tools' -- although I can't
really see anything that requires it on the various 20.04
platforms I'm running.

[Apollon Oikonomopoulos]

Hi danny!

Daniel Howard <dann...@toldme.com> writes:

> Hello,
>
> Thanks a lot to Apollon and Petter and really everyone who makes Ganeti
> happen, I have a very nice ganeti-3.0 cluster running on Ubuntu 20.04.
> There are a few tricks I thought worth sharing for posterity.

Thanks for the notes!

>
> *1) Obtain drbd8-utils.* On Ubuntu 20.04, the kernel comes with drbd8, but

> the OS only supplies drbd-utils for version 9. Ganeti will refuse to
> operate with a drbd9 kernel. So, I installed drbd8-utils from the previous
> LTS:
>
> https://mirrors.edge.kernel.org/ubuntu/pool/main/d/drbd-utils/
>
> Specifically:
>
> sudo dpkg -i
> https://mirrors.edge.kernel.org/ubuntu/pool/main/d/drbd-utils/drbd-utils_8.9.10-2ubuntu0.1_amd64.deb
>
> (The ubuntu0.1 version has an important patch for newer kernel
> versions.)

The Linux kernel still has the DRBD 8.4 *kernel* module and drbd-utils
9.x supports DRBD 8.4 the same way drbd8-utils did (despite that
misleading "9" in the drbd-utils version). In fact we have been testing
Ganeti with drbd-utils 9.x for quite some time now without issues, so
there should be no need to fetch older drbd-utils packages.

>
> *2) Disable multipathd.* I was getting hella errors from DRBD that disks

> were left open, but I could not track down how. After two days of
> frustration, I noticed some log messages from multipathd. When I turned it
> off, things were pretty great! I don't speak fluent systemd, but here's my
> Ansible:
>
> - name: Disable multipathd
> when: ansible_distribution == 'Ubuntu' and
> ansible_distribution_version|float >= 20
> systemd:
> name: multipathd
> enabled: no
> state: stopped
>
> I don't know what multipathd is or why it is there. Ubuntu 18.04 does not
> launch this service. I assume this is part of a scheme from systemd to Make
> Linux Great Again.

multipathd is there to facilitate complex storage networks with multiple
paths to the same disk. This is something that applies mostly to SANs
using FC/FCoE. It's an old piece of software, much older than systemd
itself. If you are not sure you need it, then you probably can remove
the multipath-tools package.

>
> *3) Use ganeti-os-nocloud.* We have used snf-image for years, but it does

> not support Ubuntu-20.04 as a guest, and it needs to build a Python 2.7
> package, and Ubuntu and I guess Python are pretty hostile to that. So I
> dropped snf-image for nocloud. This was a little confusing at first but on
> the whole, a better solution. One advantage is that Ubuntu supplies base
> images that are updated daily, so you can periodically download fresh base
> images and when you spin up an instance it wastes less time running
> security patches.
>
> https://github.com/neicnordic/ganeti-os-nocloud
>
> I just added an example user-data on my fork of the repo:
>
> https://github.com/dannyman/ganeti-os-nocloud/tree/master/examples/ubuntu/user-data
>

> *4) Make DRBD faster!*

>
> There is an outstanding issue to tune the default values supplied by
> Ganeti. These values are applied to DRBDs upon creation, so it is good to
> get something in when you first build the cluster. The issue and various
> solutions are offered here:
>
> https://github.com/ganeti/ganeti/issues/1229
>
> Brian Candler has a similar environment to ours, so I cribbed his config
> and I am very pleased with the results:
>
> gnt-cluster modify -D drbd:resync-rate=204800,disk-custom='--c-plan-ahead
> 0',net-custom='--max-buffers 16384 --max-epoch-size 16384'

We should really make these default, as discussed in
https://github.com/ganeti/ganeti/issues/1545.
>
> *5) Share what you learn with the community!*

>
> As far as I know, we may be the first shop running ganeti-3.0 on Ubuntu
> 20.04. Some of the tricks here were so frustrating that I was close to
> giving up on this project a few times. I could not have gotten things
> running without the support of the community and all the folks who have
> shared their software, documentation, and troubleshooting through the
> years. I hope these notes help the next folks who want to get Ganeti
> running on Ubuntu skip some frustrations and achieve a working cluster
> quickly!
>
> Thanks Again!
>
> -danny
>
> --
> http://dannyman.toldme.com
>

> --
> You received this message because you are subscribed to the Google Groups "ganeti" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ganeti+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti/CAKU%3DtE8%3DBWDaiNqeu%2BXF73QDKABKzoqwtHANkAPmALTcFSHt3g%40mail.gmail.com.

[Daniel Howard]

djh@dr64-dingo ~> aptitude why multipath-tools
i   ubuntu-server Depends multipath-tools

So, with Ubuntu 20.04 LTS, Canonical put multipathd in the base OS and pulled out ifconfig.

--

http://dannyman.toldme.com