Poor quality of multipath-tools
John Goerzen
I'm quite concerned about the poor quality of multipath-tools. This is
an absolutely vital application for many, required to even get a machine
to *boot*.
There are a lot of outstanding bugs:
* Fails to run kpartx when it should. This would have been detectable
with a trivial amount of testing. #376161
* Wrong path to scsi_id. #358985, 99 days old.
* PID file handling is broken. #294066, 1y 145d old, patch available.
* Doesn't integrate into *ANY* initrd system. Means that systems that
need multipath can't even *BOOT*. initrd patch available for 1y 193d.
I will be packaging up my own multipath-tools-initramfs package so that
the last problem can at least be solved for those of us that need to be
able to boot a multipath system.
I am gravely concerned, though, about the lack of attention this package
is receiving. Does anyone intend to give it some TLC anytime soon?
For those that have a fibre channel setup where multipath-tools is
required, this is probably one of the most critical packages on the
system.
Thanks,
-- John
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Petter Reinholdtsen
[John Goerzen]
> I'm quite concerned about the poor quality of multipath-tools. This
> is an absolutely vital application for many, required to even get a
> machine to *boot*.
Well, I do not even know what multipath _is_, nor why it is important.
If that is representative, I suspect the people interested in
multipath have some work to do to raise the awareness of the problem.
This email is a very good start, but it seem to assume that everyone
know what multipart is and why it is important. Is multipath machines
as common as the ppc64 machines, or is the problem affecting a lot of
users?
> I am gravely concerned, though, about the lack of attention this
> package is receiving. Does anyone intend to give it some TLC
> anytime soon?
Perhaps you could give it some tender loving care, and talk to the
people maintaining the affected packages using IRC and email, and
hopefully get them to realize why they should fix it in time for
etch. :)
I suspect you might wait in wane if you expect someone else to do
it. :)
Friendly,
--
Petter Reinholdtsen
Steinar H. Gunderson
> Well, I do not even know what multipath _is_, nor why it is important.
> If that is representative, I suspect the people interested in
> multipath have some work to do to raise the awareness of the problem.
Multipath is for when you have multiple paths from your machine to the same
data -- say, redundant FC controllers.
/* Steinar */
--
Homepage: http://www.sesse.net/
John Goerzen
> If that is representative, I suspect the people interested in
> multipath have some work to do to raise the awareness of the problem.
>
> This email is a very good start, but it seem to assume that everyone
> know what multipart is and why it is important. Is multipath machines
> as common as the ppc64 machines, or is the problem affecting a lot of
> users?
Let me give a brief explanation of multipath.
Let's say you want a bunch of disk space. A whole lot -- maybe
terabytes worth. So you buy a SAN, which is a device that might have
dozens or hundreds of disks in it. And you can hook multiple servers to
this SAN. So you have a SAN controller, a bunch of disks in whatever
RAID configurations you like hooked to it, a fibre channel switch, and
each server hooked to the FC switch.
Suddenly you have a lot of really important single points of failure
that could take down not just one but many servers -- the FC switch, the
SAN controller, the FC cables, etc.
So the solution is to build two distinct I/O paths for any server to
reach the disks. The SAN will have two controllers (each with access to
disk enclosures). You'll have two FC switches, one controller cabled to
each. And each server will have two FC links, one to each switch.
Now, when you bring up this system, Linux will assign *two* /dev/sdx
devices for each RAID LUN (basically looks like a disk). At any given
time, exactly one will be readable and useful. That is, the disk can be
probed on both controllers, but only one path will support I/O at any
given time.
Adding to the complexity, which one to use can vary while the system
runs. For instance, if a SAN controller dies, everybody switches over
to the backup path.
The multipath-tools package is the userland support necessary to make
all this work in a sane fashion. It uses the dm-multipath kernel module
to do that.
But it's got some problems:
1) It doesn't properly scan partition tables in multipath devices
2) It doesn't integrate with initramfs, so it's not possible to boot
off a multipath device unless more work is done
3) Some other general bugs and issues
BTW, multipath is often called MPIO (MultiPath I/O)
> > I am gravely concerned, though, about the lack of attention this
> > package is receiving. Does anyone intend to give it some TLC
> > anytime soon?
>
> Perhaps you could give it some tender loving care, and talk to the
> people maintaining the affected packages using IRC and email, and
> hopefully get them to realize why they should fix it in time for
> etch. :)
That's what I intend to do. It's maintained by the LVM folks, though,
and seems to be tied reasonably closely to that somehow. I'm not as
familiar with all this as they are. But it seems like the package is
not really being looked after, given its bug reports.
I have already uploaded multipath-tools-initramfs to Incoming, which
simply installs initramfs hooks and scripts to make it possible to boot
from multipath. We are successfully using it with these scripts at our
site.
> I suspect you might wait in wane if you expect someone else to do
> it. :)
I understand. I'm just trying to figure out if there are interested
parties out here to pitch in, if the LVM folks have plans for it, etc.
I'm brand-new at this and wouldn't be at all surprised if someone else
was more capable at it than I am.
-- John
Dave Holland
> Now, when you bring up this system, Linux will assign *two* /dev/sdx
> devices for each RAID LUN (basically looks like a disk). At any given
> time, exactly one will be readable and useful. That is, the disk can be
> probed on both controllers, but only one path will support I/O at any
> given time.
To add further complexity: there may be more than two paths to a LUN,
which means there will be more /dev/sdX devices - typically four in a
dual-controller/dual-SAN-fabric setup. And some disk systems will accept
I/O on either controller. Then it becomes sensible to try to
load-balance across the controllers for maximum throughput.
> I understand. I'm just trying to figure out if there are interested
> parties out here to pitch in, if the LVM folks have plans for it, etc.
We used the multipath tools at work (sanger.ac.uk) and I'd like to see
the package maintained well. Let me know if I can help.
Dave
Pasi Kärkkäinen
Not always true. Both paths can be active at the same time.. if supported by
the SAN array. Then you do also load balancing between the paths..
I'm currently using multipath with iSCSI SAN, using two active paths with
load balancing and failover.
So I'm also interested in this stuff..
- Pasi Kärkkäinen
John Goerzen
> Not always true. Both paths can be active at the same time.. if supported by
> the SAN array. Then you do also load balancing between the paths..
Quite true, though my impression is that this is much more rare. Our
controller (HP MSA1500cs) seems to have added active/active controllers
as a recent option.
I'm not really sure if multipath-tools supports active/active
controllers, though. Do you know?
-- John
Stephen Frost
> On Thu, Jul 06, 2006 at 02:39:16PM +0300, Pasi Kärkkäinen wrote:
> > Not always true. Both paths can be active at the same time.. if supported by
> > the SAN array. Then you do also load balancing between the paths..
>
> Quite true, though my impression is that this is much more rare. Our
> controller (HP MSA1500cs) seems to have added active/active controllers
> as a recent option.
Just another 'me too', I've got a nice IBM SAN at work (DS4300) and we
use the multipath tools also. They could certainly use some
improvement. My main beef has been with multipathd... I've never
actually seen it work correctly.
> I'm not really sure if multipath-tools supports active/active
> controllers, though. Do you know?
I havn't got full active/active controllers, but my controllers each
have two ports, which go to each of my fibre switches, and then I've got
both controllers in the host hooked up to each switch, so in the end
there's 4 paths, 2 through each controller, so I load-balance across the
two paths through the same controller and that has worked fine in
general (once I could convince multipath to set the paths up correctly
in the kernel, that is).
Thanks,
Stephen
Dave Holland
> Quite true, though my impression is that this is much more rare. Our
> controller (HP MSA1500cs) seems to have added active/active controllers
> as a recent option.
That's worth knowing.
> I'm not really sure if multipath-tools supports active/active
> controllers, though. Do you know?
Certainly does: this is a LUN on a HP EVA8000 (dual HSV210 controllers)
via dual fabric:
sptsrv2:~# multipath -ll
3600508b40010400f0000f000007e0000
[size=10 GB][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=4][enabled]
\_ 0:0:0:9 sde 8:64 [active][ready]
\_ 0:0:1:9 sdf 8:80 [active][ready]
\_ 1:0:0:9 sdg 8:96 [active][ready]
\_ 1:0:1:9 sdh 8:112 [active][ready]
and using iostat shows traffic across all four paths.
Dave
Pasi Kärkkäinen
Yes, active/active is working fine:
$ multipath -ll
system (30690a018c032d43ece43440c0000d044)
[size=18 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
\_ 0:0:1:0 sda 8:0 [active][ready]
\_ 1:0:1:0 sdc 8:32 [active][ready]
that's LUN from Equallogic iSCSI SAN array.. loadbalanced with round-robin.
Box is even booted from the SAN :) took some time to create the initrd
images to set up multipathing for boot/system volume..
-- Pasi Kärkkäinen
^
. .
Linux
/ - \
Choice.of.the
.Next.Generation.