Re: Problem while attaching existing disk to instance

[Sascha Lucas]

Hi,

thanks a lot for the discussion. I think Brian made clear what
completing the design doc regarding disks would mean. From my point of
view the "big can of worms" are only caused by DRBD-8.4's nature for
primary and secondary. All other (redundant) disk templates, which
Ganeti calls "externally mirrored" do not have such limitations.

It might be considered to not allow de-/attach for DRBD on the first
run, or even at all (same for file and plain?). Maybe the use case for
de-/attach itself is somewhat special. Completing the disk design doc,
would open the way for mixing disk types, again limited for the DRBD
case, but not for externally mirrored.

I also agree with Valentin, that solving this types of problems with
the external storage interface instead of inside Ganeti is not the most
valuable variant. Besides that, it can be observed, that this
"functions as executables" interfaces, namely OS and storage, tend to
everyone inventing their own wheel. Probably a subclassing and more
integrated approach would be better?

Valentin offered to contribute towards completing the disk design doc,
which would be a great improvement for the Ganeti project. Personally I
like to support this from my end. But this must been seen more from a
point of a longtime user, than a Ganeti developer (my programming
capabilities are enough for smaller fixes, but easily lost inside
Haskell). So I'm cc-ing to ganeti-devel here, probably to catch some
other's thoughts as well. The full discussion can be found at [1].

Thanks, Sascha.

[1] https://groups.google.com/g/ganeti/c/FwMZhBP9Ru0

[Valentin]

Hi,

thanks for posting this to ganeti-devel, i was planning to do so myself.

On 15/02/2021 23:41, Sascha Lucas wrote:
> thanks a lot for the discussion. I think Brian made clear what
> completing the design doc regarding disks would mean. From my point of
> view the "big can of worms" are only caused by DRBD-8.4's nature for
> primary and secondary. All other (redundant) disk templates, which
> Ganeti calls "externally mirrored" do not have such limitations.
>
> It might be considered to not allow de-/attach for DRBD on the first
> run, or even at all (same for file and plain?). Maybe the use case for
> de-/attach itself is somewhat special. Completing the disk design doc,
> would open the way for mixing disk types, again limited for the DRBD
> case, but not for externally mirrored.

I think if the design doc is implemented nicely all of this can be
solved by doing the right checks on the drbd volumes. With mixed disk
types all instance modification commands impacting the storage need to
check attached disks and their states anyway instead of just checking
for the disk templates.
If we do this it should only be a small step to checking a drbd disks
hosts and suggest an action to move the drbd disk to primary/secondary
hosts that correspond to the desired instance.

> Valentin offered to contribute towards completing the disk design doc,
> which would be a great improvement for the Ganeti project. Personally I
> like to support this from my end. But this must been seen more from a
> point of a longtime user, than a Ganeti developer (my programming
> capabilities are enough for smaller fixes, but easily lost inside
> Haskell). So I'm cc-ing to ganeti-devel here, probably to catch some
> other's thoughts as well.

I was starting to come up with a plan for implementation together with
my colleagues.
As this is our main use case we would start with changes to the cli to
be able to manage detached disks and the extStorage Interface part of
it, but we try to keep the bigger picture in mind to make changes
suitable for widespread use with other disk types.
Our proposed changes would consist of an extension of the gnt-storage
cli to manage disks and appropriate changes to instance modification
commands and the model to be able to handle mixed disk templates.

gnt-storage cli:
* list -- show all disks (maybe allow selector/filtering)
* add -- add a disk to config (and create if needed)
* remove -- remove a disk from config
* parameter:keep -- optionally preserve data on disk (for all types
of storage, should be easy enough)
* compatibility with current gnt-instance commands should be maintained
(but maybe not extended)
* special commands that might be useful:
* drbd -- switch primary/secondary
* lvm/plain -- copy data between storage/nodes
* convert -- disk type conversion
* mount -- allow introspection of disk in node (only for disks that
aren't in use; possible issues may arise from storage location)

attach/detach of disks should be an instance operations. (reasoning: on
all hypervisors disk operations are part of the instance management;
compatibility with current command line interfaces)

storage configuration for instances:
* disk templates
* use disk templates in disk configs only
* allow disks to get attached to any instance (ganeti has to deduce
from all disk templates how a machine might get migrated or started;
also see below)
* disks need thorough checking of the availability of the disk for a
(running) instance
* diskless template needs to be dropped, checks for it should instead
test if no disk is attached
* instance debootstrap needs to be modified to not use disk templates
(cfgupgrade as well)
* all instance modification commands that impact storage need to check
attached disks and their state(s) (availability, ...) instead of just
checking for the instance disk template
* operations that operate on the instance and extract the disk template
e.g. for creation of a new disk will require an additional parameter for
the disk template. Several commands already provide an optional
parameter to override the instance setting, this will become required.

In addition we want to implement for ESI (ExtStorage Interface):
* add adopt option
* fix the issues with exclusive/shared mode outlined in [1]

I would love to hear some feedback from you on those plans.

In the meantime we have opened some smaller issues and PRs on github.
[2][3][4]

Thanks,
Valentin


[1] https://github.com/ganeti/ganeti/issues/1581
[2] https://github.com/ganeti/ganeti/issues/1583
[3] https://github.com/ganeti/ganeti/pull/1582
[4] https://github.com/ganeti/instance-debootstrap/pull/12

[Valentin Kleibel]

I just realized that the web-view of google groups defeated some of my structuring effort.

If you can please view the mail in plain text format, otherwise here are the lists formatted differently:

gnt-storage cli:
* list -- show all disks (maybe allow selector/filtering)
* add -- add a disk to config (and create if needed)
* remove -- remove a disk from config

** parameter:keep -- optionally preserve data on disk (for all types

of storage, should be easy enough)
* compatibility with current gnt-instance commands should be maintained
(but maybe not extended)
* special commands that might be useful:

** drbd -- switch primary/secondary
** lvm/plain -- copy data between storage/nodes
** convert -- disk type conversion
** mount -- allow introspection of disk in node (only for disks that

aren't in use; possible issues may arise from storage location)

attach/detach of disks should be an instance operations. (reasoning: on
all hypervisors disk operations are part of the instance management;
compatibility with current command line interfaces)

storage configuration for instances:
* disk templates

** use disk templates in disk configs only
** allow disks to get attached to any instance (ganeti has to deduce

from all disk templates how a machine might get migrated or started;
also see below)
* disks need thorough checking of the availability of the disk for a
(running) instance
* diskless template needs to be dropped, checks for it should instead
test if no disk is attached
* instance debootstrap needs to be modified to not use disk templates
(cfgupgrade as well)
* all instance modification commands that impact storage need to check
attached disks and their state(s) (availability, ...) instead of just
checking for the instance disk template
* operations that operate on the instance and extract the disk template
e.g. for creation of a new disk will require an additional parameter for
the disk template. Several commands already provide an optional
parameter to override the instance setting, this will become required.

Sorry for the inconvenience,
Valentin

[Phil Regnauld]

Sascha Lucas (sascha_lucas) writes:
>
> It might be considered to not allow de-/attach for DRBD on the first
> run, or even at all (same for file and plain?). Maybe the use case for
> de-/attach itself is somewhat special. Completing the disk design doc,
> would open the way for mixing disk types, again limited for the DRBD
> case, but not for externally mirrored.

Since nothing is known about the "migratability" of a given
external disk template, it is a bit all-or-nothing with DRBD, yes.

But, we would love to be able to have system disks on DRBD SSD storage,
and have the second/data disks on RBD (CEPH), so we'd be able to migrate.

Maybe having external disk template provide some sort of flag/feature
"can_migrate" or similar ?

Cheers,
Phil