Bug#1001710: multipath-tools: booting with lvm over multipath it does not setup devices for lvm

[Carlos Barros]

Package: multipath-tools
Version: 0.8.5-2
Severity: important
X-Debbugs-Cc: carlos...@econocom.com

Dear Maintainer,

A server was setup with some disks configured in multipath.
The root filesystem is on LVM on internal disk, not configured for multipath.

Then build another lvm configuration on those devices (/dev/mapper/...)

Now, every time it starts, lvm takes the /dev/sd.. before multipath sets /dev/mapper/... up.
Also, the booting proccess delays for 2+ minutes, mostly because of systemd-udev-settle.service:
root@panblando:~# systemd-analyze blame
2min 22ms systemd-udev-settle.service
1min 105ms NetworkManager-wait-online.service
9.983s smartmontools.service
4.415s dev-mapper-dl380g8\x2d\x2dvg\x2droot.device

But this service is called from multipath:
[Unit]
Description=Device-Mapper Multipath Device Controller
Wants=systemd-udev-trigger.service systemd-udev-settle.service
Before=iscsi.service iscsid.service lvm2-activation-early.service
Before=local-fs-pre.target blk-availability.service
After=multipathd.socket systemd-udev-trigger.service systemd-udev-settle.service
DefaultDependencies=no
Conflicts=shutdown.target
ConditionKernelCommandLine=!nompath
ConditionKernelCommandLine=!multipath=off

According to systemd-udev-settle.service(8) it is not recommended, as that service can fail (in fact,
in the server it fails as systemd says)

I think that the problem is related to multipath, because it did not setup the devices before lvm,
and maybe the problem is inside the initrd.

Then I did install multipath-tools-boot but the problem is the same.
The only way is to avoid mounting the filesystems in the LVM (or umounting them) vgchange -an VG
and do the multipath and mount it, all by hand

The disk list in multipath is short (8 disks with only 1 path each by now)

Best regards,
Carlos Barros.

-- Package-specific info:
Contents of /etc/multipath.conf:
defaults {
polling_interval 10
user_friendly_names yes
}
blacklist {
device {
vendor "HP"
product "LOGICAL VOLUME" #HPSA raid internal
}
}



-- System Information:
Debian Release: 11.1
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-9-amd64 (SMP w/32 CPU threads)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=es_ES.UTF-8, LC_CTYPE=es_ES.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages multipath-tools depends on:
ii kpartx 0.8.5-2
ii libaio1 0.3.112-9
ii libc6 2.31-13+deb11u2
ii libdevmapper1.02.1 2:1.02.175-2.1
ii libjson-c5 0.15-2
ii libreadline8 8.1-1
ii libsystemd0 247.3-6
ii libudev1 247.3-6
ii liburcu6 0.12.2-1
ii lsb-base 11.1.0
ii sg3-utils-udev 1.45-1
ii udev 247.3-6

multipath-tools recommends no packages.

Versions of packages multipath-tools suggests:
ii multipath-tools-boot 0.8.5-2

-- no debconf information

[Ritesh Raj Sarraf]

Hello Carlos,

On Tue, 2021-12-14 at 19:04 +0100, Carlos Barros wrote:
> A server was setup with some disks configured in multipath.
> The root filesystem is on LVM on internal disk, not configured for
> multipath.
>
> Then build another lvm configuration on those devices
> (/dev/mapper/...)
>
> Now, every time it starts, lvm takes the /dev/sd.. before multipath
> sets /dev/mapper/...  up.

This combination of the setup, where you have Multipath + LVM, is a
known working config.

There are some special tweaks needed though. I don't recollect the
exact specifics but basically there might be 2 things.

1. You need to ensure that you run pvcreate on top of the multipathed
devices and NOT on the bare SCSI devices.

2. THere's also a filter in /etc/lvm/ where you define the list of SCSI
devices that should be blacklisted from LVM.

> Also, the booting proccess delays for 2+  minutes, mostly because of
> systemd-udev-settle.service:
>         root@panblando:~# systemd-analyze blame
>          2min 22ms systemd-udev-settle.service
>         1min 105ms NetworkManager-wait-online.service
>             9.983s smartmontools.service
>             4.415s dev-mapper-dl380g8\x2d\x2dvg\x2droot.device
>
> But this service is called from multipath:
>         [Unit]
>         Description=Device-Mapper Multipath Device Controller
>         Wants=systemd-udev-trigger.service systemd-udev-
> settle.service
>         Before=iscsi.service iscsid.service lvm2-activation-
> early.service
>         Before=local-fs-pre.target blk-availability.service
>         After=multipathd.socket systemd-udev-trigger.service systemd-
> udev-settle.service
>         DefaultDependencies=no
>         Conflicts=shutdown.target
>         ConditionKernelCommandLine=!nompath
>         ConditionKernelCommandLine=!multipath=off
>
> According to systemd-udev-settle.service(8) it is not recommended, as
> that service can fail (in fact,
> in the server it fails as systemd says)
>

I don't recollect the specifics but iirc that service is required for a
reason. Remember that, typically, multipath-tools is used in
combination with SAN devices which are remote.

> I think that the problem is related to multipath, because it did not
> setup the devices before lvm,
> and maybe the problem is inside the initrd.
>

Please explore the suggestions I mentioned above. I am fairly sure that
it may just be a case of proper blacklisting.

> Then I did install multipath-tools-boot but the problem is the same.
> The only way is to avoid mounting the filesystems in the LVM (or
> umounting them) vgchange -an VG
> and do the multipath and mount it, all by hand
>

Yes. THis is expected behavior. mutliapth can only acquire the device
when it is free.



--
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System

[Ritesh Raj Sarraf]

Control: tag -1 +pending

Thanks for confirming Carlos. The following has been added and will be
part of the next upload.


commit 00b79442b9af0c5275880b73b9bae5c68150e2c5 (HEAD -> master)
Author: Ritesh Raj Sarraf <r...@debian.org>
Date: Mon Dec 20 15:29:43 2021 +0530

Add some documentation about LVM + DM-Multipath setup

Thanks: Carlos Barros
Closes: 1001710

diff --git a/debian/multipath-tools.README.Debian b/debian/multipath-tools.README.Debian
index 257eebdb..ec1dee40 100644
--- a/debian/multipath-tools.README.Debian
+++ b/debian/multipath-tools.README.Debian
@@ -1,6 +1,24 @@
Additional information for users of multipath-tools from Debian.


+LVM over DM-Multipath
+=====================
+
+Debigu Bug #1001710
+
+On a setup that involves both, LVM and DM-Multipath, it is important to set the
+right filter in lvm.conf file to ensure that LVM does not acquire the bare SCSI devices
+before DM-Multipath can.
+
+On such combined setups, the standard recommendation is to pvcreate devices on the
+enumerated multipathed devices. And also ensure that the bare SCSI devices are filtered out
+from LVM's list
+
+ # filter = [ "a|/dev/mapper/|", "r|/block/|", "r|/disk/|", "r|/sd[b-z].*|", "a|.*|" ]
+
+In the above example snippet, it is ensured that LVM filters out any SCSI devices of name type
+sd[b-z].*
+



On Fri, 2021-12-17 at 21:59 +0000, BARROS FERNANDEZ, Carlos wrote:
> Hello Ritesh.
>
> Thanks for your answer.
>
> Tried serveral things to make it work, but only a filter in lvm.conf
> did work. (the pv's were created under multipath, and also did
> several vgcfgbackup, the pv's did not have any /dev/sd?? written in
> their sectors)
> The filter that I put is:
> filter = [ "a|/dev/mapper/|", "r|/block/|", "r|/disk/|", "r|/sd[b-
> z].*|", "a|.*|" ]
>
> I had to put sd[b-z].* because the system in installed in lvm too, so
> I had to exclude that device. Maybe there is a better filter, but I
> do not know how to write it.
>
> The server uses a RAID controller for internal disks and hba for the
> external disks
>
> Right now, it seems stable between reboots, I hope that the raid
> controller will always setup before the hba, otherwise the filter
> won't work. 
>
> I think that this info can be of help to other users, could you put a
> README.debian including this filter? (or modify the lvm.conf to add
> the note)
> Like:
> To setup LVM over multipath, and allow other LVM over a non
> multpathed device, if you encounter that the pv-devices are not
> under 
> multipath control, try this filter line in lvm.conf, update-initrd
> and reboot. to check it that it works ok.
>
> filter = [ "a|/dev/mapper/|", "r|/block/|", "r|/disk/|",
> "r|/sd[b-z].*|", "a|.*|" ]
>   sd[b-z].* makes all /dev/sd* be skipped but /dev/sda.
> Adjust for your hardware configuration.

[Ritesh Raj Sarraf]

Hello Carlos,


The thing about these documentations is that there is a lot of
combinations. As you must be aware, Linux Storage is stacked up of
multiple independent components, tied tightly together to work in
unison. This often need a good understanding of each involved stack, as
well as the overall desired solution.


Documenting everything is beyond the scope, especially for a general
purpose universal distribution like Debian.

I have, in the past, personally relied on the Red Hat Storage Admin
Guide. They do a good job of documenting all the combinations and
ensuring that the documents are valid and up-to-date.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/index




On Mon, 2021-12-20 at 16:03 +0000, BARROS FERNANDEZ, Carlos wrote:
> Hi Ritessh.
>
> On another server I have a stranger setup :) :
> 1 LVM over multipath
> +
> 1 LVM over mdadm-device and the mdadm-device over multipaths
>
> Did had problems because the mdadm-device did not setup the
> multipath, just the /dev/sdX devices
> In this case, mdadm was called from udev as soon as a block device
> was setup, so it always started the software-raid on /dev/sdX
> What I did was tell mdadm to not setup any block devices, only
> devices belonging to /dev/disk/by-id/dm-name-* because of the
> 'friendly-names' used un multipath.
> I spent some time to figure it out. 
> The file /etc/mdadm/mdadm.conf should have a 'DEVICES' line that
> tells mdadm to only read those devices as follows:
> DEVICE /dev/disk/by-id/dm-name-*
>
> # definitions of existing MD arrays
> ARRAY /dev/md/0 metadata=1.2
> UUID=d7285198:51ed20e4:7d1e82e6:b01569dd devices=/dev/disk/by-id/dm-
> name-* name=pandora:0
>
> In this case, it is simpler, but it is very different from LVM as to
> where it must be set.(I find it clearer too)
>
> Also, the filter in LVM only takes care of the /dev/sd[b-z) disks,
> but it is common to have several paths( and disks) like  /dev/sda[a-
> z] /dev/sdb[a-z] and so on.
> That filter would not filter out /dev/sda[a-z]*, it will use them.
> So I did change the rule in LVM to include those cases, one for
> /dev/sda[a-z] and one for /dev/sd[b-z] that covers the extra devices.

> filter = [ "a|/dev/mapper/|", "r|/block/|", "r|/disk/|", "r|/sd[b-

> z].*|", "r|/sda[a-z].*|", "a|.*|" ]
>
>
> Would you modify the info to reflect the 'modified' filter rule? Also
> a note on mdadm with that info would be nicer!
>
>
> Regards.
> INFORMACIÓN BÁSICA SOBRE PROTECCIÓN DE DATOS CONFORME EL NUEVO
> REGLAMENTO EUROPEO (RGPD)
>
> RESPONSABLE: Grupo Econocom en España (Econocom SA, Econocom
> Servicios S.A., Econocom Products & Solutions S.A.U.) sitas en
> Cardenal Marcelo Spínola, 4, 28016 Madrid
> FINALIDAD PRINCIPAL: Mantener relaciones profesionales y/o
> comerciales con la empresa en la que usted trabaja o de la que es
> representante.
> LEGITIMACIÓN: Consentimiento del interesado.
> DESTINATARIOS: No se cederán datos a terceros, salvo autorización
> expresa u obligación legal.
> DERECHOS: Acceder, rectificar y suprimir los datos, portabilidad de
> los datos, limitación u oposición a su tratamiento, transparencia y
> derecho a no ser objeto de decisiones automatizadas.
> INFORMACIÓN ADICIONAL: Puede consultar la información adicional y
> detallada sobre nuestra política de privacidad
> DELEGADO DE PROTECCIÓN DE DATOS (DPD): Miguel Angel Muñoz +34 93 470
> 30 00 DPO_eco...@econocom.com
> CONFIDENCIALIDAD: Si Ud. no es el destinatario y recibe este mail por
> error, rogamos se ponga en contacto con nosotros y borre de inmediato
> el mail por error recibido con todos sus documentos adjuntos sin
> leerlos ni hacer ningún uso de los datos que en ellos figuren,
> ateniéndose a las consecuencias que de un uso indebido de dichos
> datos puedan derivarse.
>
>
> BASIC INFORMATION ON DATA PROTECTION ACCORDING TO THE NEW EUROPEAN
> REGULATION (GDPR)
>
> CONTROLLER: Grupo Econocom in Spain. (Econocom SA, Econocom Servicios
> S.A., Econocom Products & Solutions S.A.U.) registered at Cardenal
> Marcelo Spínola, 4, 28016 Madrid
> PURPOSE: Maintain professional and / or commercial relationship with
> the company which you represent or work for
> LEGITIMATION: Data subject consent.
> RECIPIENTS: data will not be transferred to third parties, unless
> explicit consent or legal obligation.
> RIGHTS: Right to Access, right to rectification and erasure, data
> portability, right to restriction of processing, right to object and
> not been subject to a decision based solely on automated processing,
> and to be informed in a clearly and transparent way of how their
> personal data are processing.
> ADDITIONAL INFORMATION: you can consult the additional and detailed
> information about our privacy policy
> DATA PROTECTION OFFICER (DPO): Miguel Angel Muñoz +34 93 470 30
> 00 DPO_eco...@econocom.com 
> CONFIDENTIALITY: If you received this email in error, please
> immediately contact us and destroy the material in its entirety,
> whether in electronic or hard copy format. If you are not the
> intended recipient, you are hereby notified that any disclosure,
> copying, distribution, or use of the information contained herein
> (including any reliance thereon) is strictly prohibited, on his/her
> own responsibility.