Q: multipath not recovering after device was offline
Ulrich Windl
This is not an exact open-iscsi question, but tighlty related:
On a SAN using FibreChannel I had a 4-way multipath device. The basic configuration (without aliases for devices) is:
devices {
device {
vendor "HP"
product "HSV2.*"
path_grouping_policy "group_by_prio"
path_checker "tur"
prio "alua"
failback "immediate"
#polling_interval 30
no_path_retry 1000
features "1 queue_if_no_path"
}
}
I did a "LUN shrink" on the storage system (because I had created the LUN too large by mistake). During that the LUN went offline, and multipath did detect that fine. However after the LUN shrink was complete (and the LUN was supposed to be online again), multipath still blocked any access to the device (like fdisk -l). Reboot did fix it, but I'm hoping for a better solution.
I'm asking theis question here, because I expect that with iSCSI the problem would have been much the same.
So:
1) Did I misconfigure multipath?
2) Could I do something to make multipath rediscover the online-again paths?
Lengthy Syslog messages (only a part):
May 11 16:07:06 hostname multipathd: 65:224: mark as failed
May 11 16:07:06 hostname multipathd: DISK-E1: remaining active paths: 3
May 11 16:07:06 hostname kernel: [ 8064.784883] sd 3:0:2:4: [sdae] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:06 hostname kernel: [ 8064.784889] sd 3:0:2:4: [sdae] Sense Key : Illegal Request [current]
May 11 16:07:06 hostname kernel: [ 8064.784894] sd 3:0:2:4: [sdae] Add. Sense: Logical block address out of range
May 11 16:07:06 hostname kernel: [ 8064.784899] sd 3:0:2:4: [sdae] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:06 hostname kernel: [ 8064.784906] end_request: I/O error, dev sdae, sector 1048575992
May 11 16:07:06 hostname kernel: [ 8064.784911] device-mapper: multipath: Failing path 65:224.
May 11 16:07:06 hostname kernel: [ 8064.787969] sd 2:0:0:4: [sdac] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:06 hostname kernel: [ 8064.787973] sd 2:0:0:4: [sdac] Sense Key : Illegal Request [current]
May 11 16:07:06 hostname kernel: [ 8064.787976] sd 2:0:0:4: [sdac] Add. Sense: Logical block address out of range
May 11 16:07:06 hostname kernel: [ 8064.787980] sd 2:0:0:4: [sdac] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:06 hostname kernel: [ 8064.787987] end_request: I/O error, dev sdac, sector 1048575992
May 11 16:07:06 hostname kernel: [ 8064.787992] device-mapper: multipath: Failing path 65:192.
May 11 16:07:06 hostname kernel: [ 8064.792011] sd 2:0:3:4: [sdad] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:06 hostname kernel: [ 8064.792015] sd 2:0:3:4: [sdad] Sense Key : Illegal Request [current]
May 11 16:07:06 hostname kernel: [ 8064.792019] sd 2:0:3:4: [sdad] Add. Sense: Logical block address out of range
May 11 16:07:06 hostname kernel: [ 8064.792023] sd 2:0:3:4: [sdad] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:06 hostname kernel: [ 8064.792031] end_request: I/O error, dev sdad, sector 1048575992
May 11 16:07:06 hostname kernel: [ 8064.792037] device-mapper: multipath: Failing path 65:208.
May 11 16:07:06 hostname kernel: [ 8064.793499] sd 3:0:3:4: [sdaf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:06 hostname kernel: [ 8064.793502] sd 3:0:3:4: [sdaf] Sense Key : Illegal Request [current]
May 11 16:07:06 hostname kernel: [ 8064.793506] sd 3:0:3:4: [sdaf] Add. Sense: Logical block address out of range
May 11 16:07:06 hostname kernel: [ 8064.793509] sd 3:0:3:4: [sdaf] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:06 hostname kernel: [ 8064.793519] end_request: I/O error, dev sdaf, sector 1048575992
May 11 16:07:06 hostname kernel: [ 8064.793524] sd 3:0:3:4: [sdaf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:06 hostname kernel: [ 8064.793529] device-mapper: multipath: Failing path 65:240.
May 11 16:07:06 hostname kernel: [ 8064.793536] sd 3:0:3:4: [sdaf] Sense Key : Illegal Request [current]
May 11 16:07:06 hostname kernel: [ 8064.793541] sd 3:0:3:4: [sdaf] Add. Sense: Logical block address out of range
May 11 16:07:06 hostname kernel: [ 8064.793545] sd 3:0:3:4: [sdaf] CDB: Read(10): 28 00 3e 7f ff 80 00 00 08 00
May 11 16:07:06 hostname kernel: [ 8064.793553] end_request: I/O error, dev sdaf, sector 1048575872
May 11 16:07:07 hostname multipathd: 65:192: mark as failed
May 11 16:07:07 hostname multipathd: DISK-E1: remaining active paths: 2
May 11 16:07:07 hostname multipathd: 65:208: mark as failed
May 11 16:07:07 hostname multipathd: DISK-E1: remaining active paths: 1
May 11 16:07:07 hostname multipathd: 65:240: mark as failed
May 11 16:07:07 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:07 hostname multipathd: DISK-E1: remaining active paths: 0
May 11 16:07:07 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:08 hostname multipathd: DISK-E1: sdac - tur checker reports path is up
May 11 16:07:08 hostname multipathd: 65:192: reinstated
May 11 16:07:08 hostname multipathd: DISK-E1: queue_if_no_path enabled
May 11 16:07:08 hostname multipathd: DISK-E1: Recovered to normal mode
May 11 16:07:08 hostname multipathd: DISK-E1: remaining active paths: 1
May 11 16:07:08 hostname kernel: [ 8066.175407] sd 2:0:0:4: [sdac]
May 11 16:07:08 hostname kernel: [ 8066.175413] sd 2:0:0:4: [sdac] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:08 hostname kernel: [ 8066.175422] sd 2:0:0:4: [sdac] Sense Key : Illegal Request [current]
May 11 16:07:08 hostname kernel: [ 8066.175431] sd 2:0:0:4: [sdac] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:08 hostname kernel: [ 8066.175438] Add. Sense: Logical block address out of range
May 11 16:07:08 hostname kernel: [ 8066.175444] sd 2:0:0:4: [sdac] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:08 hostname kernel: [ 8066.175463] end_request: I/O error, dev sdac, sector 1048575992
May 11 16:07:08 hostname kernel: [ 8066.175469] sd 2:0:0:4: [sdac] Sense Key : Illegal Request
May 11 16:07:08 hostname kernel: [ 8066.175473] device-mapper: multipath: Failing path 65:192.
May 11 16:07:08 hostname kernel: [ 8066.175478] [current]
May 11 16:07:08 hostname kernel: [ 8066.175482] sd 2:0:0:4: [sdac] Add. Sense: Logical block address out of range
May 11 16:07:08 hostname kernel: [ 8066.175487] sd 2:0:0:4: [sdac] CDB: Read(10): 28 00 3e 7f ff 80 00 00 08 00
May 11 16:07:08 hostname kernel: [ 8066.175495] end_request: I/O error, dev sdac, sector 1048575872
May 11 16:07:08 hostname multipathd: 65:192: mark as failed
May 11 16:07:08 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:08 hostname multipathd: DISK-E1: remaining active paths: 0
May 11 16:07:08 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:09 hostname multipathd: DISK-E1: sdae - tur checker reportspath is up
May 11 16:07:09 hostname multipathd: 65:224: reinstated
May 11 16:07:09 hostname multipathd: DISK-E1: queue_if_no_path enabled
May 11 16:07:09 hostname multipathd: DISK-E1: Recovered to normal mode
May 11 16:07:09 hostname multipathd: DISK-E1: remaining active paths: 1
May 11 16:07:09 hostname multipathd: DISK-E1: sdaf - tur checker reportspath is up
May 11 16:07:09 hostname multipathd: 65:240: reinstated
May 11 16:07:09 hostname multipathd: DISK-E1: remaining active paths: 2
May 11 16:07:09 hostname kernel: [ 8067.190846] sd 3:0:2:4: [sdae]
May 11 16:07:09 hostname kernel: [ 8067.190852] sd 3:0:2:4: [sdae] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:09 hostname kernel: [ 8067.190860] sd 3:0:2:4: [sdae] Sense Key : Illegal Request [current]
May 11 16:07:09 hostname kernel: [ 8067.190869] sd 3:0:2:4: [sdae] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:09 hostname kernel: [ 8067.190876] Add. Sense: Logical block address out of range
May 11 16:07:09 hostname kernel: [ 8067.190882] sd 3:0:2:4: [sdae]
May 11 16:07:09 hostname kernel: [ 8067.190885] sd 3:0:2:4: [sdae] CDB: Sense Key : Illegal Request [current]
May 11 16:07:09 hostname kernel: [ 8067.190894] sd 3:0:2:4: [sdae] Add. Sense: Logical block address out of rangeRead(10)
May 11 16:07:09 hostname kernel: [ 8067.190903] sd 3:0:2:4: [sdae] CDB: : 28 003e 7f ff f8 00 00 08 00
May 11 16:07:09 hostname kernel: [ 8067.190919] end_request: I/O error, dev sdae, sector 1048575992
May 11 16:07:09 hostname kernel: [ 8067.190925] Read(10): 28 00 3e 7f ff 80 00 00 08 00
May 11 16:07:09 hostname kernel: [ 8067.190932] end_request: I/O error, dev sdae, sector 1048575872
May 11 16:07:09 hostname kernel: [ 8067.190937] device-mapper: multipath: Failing path 65:224.
May 11 16:07:09 hostname kernel: [ 8067.195326] sd 3:0:3:4: [sdaf]
May 11 16:07:09 hostname kernel: [ 8067.195330] sd 3:0:3:4: [sdaf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:09 hostname kernel: [ 8067.195338] sd 3:0:3:4: [sdaf] Sense Key : Illegal Request [current]
May 11 16:07:09 hostname kernel: [ 8067.195347] sd 3:0:3:4: [sdaf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:09 hostname kernel: [ 8067.195354] Add. Sense: Logical block address out of range
May 11 16:07:09 hostname kernel: [ 8067.195360] sd 3:0:3:4: [sdaf] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:09 hostname kernel: [ 8067.195380] end_request: I/O error, dev sdaf, sector 1048575992
May 11 16:07:09 hostname kernel: [ 8067.195385] sd 3:0:3:4: [sdaf] Sense Key : Illegal Request
May 11 16:07:09 hostname kernel: [ 8067.195390] device-mapper: multipath: Failing path 65:240.
May 11 16:07:09 hostname kernel: [ 8067.195395] [current]
May 11 16:07:09 hostname kernel: [ 8067.195397] sd 3:0:3:4: [sdaf] Add. Sense: Logical block address out of range
May 11 16:07:09 hostname kernel: [ 8067.195401] sd 3:0:3:4: [sdaf] CDB: Read(10): 28 00 3e 7f ff 80 00 00 08 00
May 11 16:07:09 hostname kernel: [ 8067.195409] end_request: I/O error, dev sdaf, sector 1048575872
May 11 16:07:09 hostname multipathd: 65:224: mark as failed
May 11 16:07:09 hostname multipathd: DISK-E1: remaining active paths: 1
May 11 16:07:09 hostname multipathd: 65:240: mark as failed
May 11 16:07:09 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:09 hostname multipathd: DISK-E1: remaining active paths: 0
May 11 16:07:09 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:10 hostname multipathd: DISK-E1: sdad - tur checker reportspath is up
May 11 16:07:10 hostname multipathd: 65:208: reinstated
May 11 16:07:10 hostname multipathd: DISK-E1: queue_if_no_path enabled
May 11 16:07:10 hostname multipathd: DISK-E1: Recovered to normal mode
May 11 16:07:10 hostname multipathd: DISK-E1: remaining active paths: 1
May 11 16:07:10 hostname multipathd: DISK-E1: switch to path group #2
May 11 16:07:10 hostname multipathd: DISK-E1: switch to path group #2
May 11 16:07:10 hostname kernel: [ 8068.198241] sd 2:0:3:4: [sdad] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:10 hostname kernel: [ 8068.198245] sd 2:0:3:4: [sdad] Sense Key : Illegal Request [current]
May 11 16:07:10 hostname kernel: [ 8068.198249] sd 2:0:3:4: [sdad] Add. Sense: Logical block address out of range
May 11 16:07:10 hostname kernel: [ 8068.198253] sd 2:0:3:4: [sdad] CDB: Read(10): 28 00 3e 7f ff 80 00 00 08 00
May 11 16:07:10 hostname kernel: [ 8068.198260] end_request: I/O error, dev sdad, sector 1048575872
May 11 16:07:10 hostname kernel: [ 8068.198266] device-mapper: multipath: Failing path 65:208.
May 11 16:07:10 hostname kernel: [ 8068.198270] sd 2:0:3:4: [sdad] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:10 hostname kernel: [ 8068.198274] sd 2:0:3:4: [sdad] Sense Key : Illegal Request [current]
May 11 16:07:10 hostname kernel: [ 8068.198278] sd 2:0:3:4: [sdad] Add. Sense: Logical block address out of range
May 11 16:07:10 hostname kernel: [ 8068.198282] sd 2:0:3:4: [sdad] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:10 hostname kernel: [ 8068.198290] end_request: I/O error, dev sdad, sector 1048575992
May 11 16:07:10 hostname multipathd: 65:208: mark as failed
May 11 16:07:10 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:10 hostname multipathd: DISK-E1: remaining active paths: 0
May 11 16:07:10 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:13 hostname multipathd: DISK-E1: sdac - tur checker reportspath is up
May 11 16:07:13 hostname multipathd: 65:192: reinstated
May 11 16:07:13 hostname multipathd: DISK-E1: queue_if_no_path enabled
May 11 16:07:13 hostname multipathd: DISK-E1: Recovered to normal mode
May 11 16:07:13 hostname multipathd: DISK-E1: remaining active paths: 1
May 11 16:07:13 hostname multipathd: DISK-E1: switch to path group #1
May 11 16:07:13 hostname multipathd: DISK-E1: switch to path group #1
May 11 16:07:13 hostname kernel: [ 8071.222307] sd 2:0:0:4: [sdac]
May 11 16:07:13 hostname kernel: [ 8071.222313] sd 2:0:0:4: [sdac] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:13 hostname kernel: [ 8071.222323] sd 2:0:0:4: [sdac] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:13 hostname kernel: [ 8071.222330] sd 2:0:0:4: [sdac] Sense Key : Illegal Request [current]
May 11 16:07:13 hostname kernel: [ 8071.222333] sd 2:0:0:4: [sdac] Add. Sense: Logical block address out of range
May 11 16:07:13 hostname kernel: [ 8071.222335] sd 2:0:0:4: [sdac] CDB: Sense Key : Illegal Request [current]
May 11 16:07:13 hostname kernel: [ 8071.222344] sd 2:0:0:4: [sdac] Add. Sense: Logical block address out of rangeRead(10): 28 00 3e 7f ff 80 00 00 08 00
May 11 16:07:13 hostname kernel: [ 8071.222364]
May 11 16:07:13 hostname kernel: [ 8071.222366] end_request: I/O error, dev sdac, sector 1048575872
May 11 16:07:13 hostname kernel: [ 8071.222370] sd 2:0:0:4: [sdac] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:13 hostname kernel: [ 8071.222378] end_request: I/O error, dev sdac, sector 1048575992
May 11 16:07:13 hostname kernel: [ 8071.222379] device-mapper: multipath: Failing path 65:192.
May 11 16:07:13 hostname multipathd: 65:192: mark as failed
May 11 16:07:13 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:13 hostname multipathd: DISK-E1: remaining active paths: 0
May 11 16:07:13 hostname multipathd: DISK-E1: Entering recovery mode: max_retries=1000
May 11 16:07:14 hostname multipathd: DISK-E1: sdae - tur checker reportspath is up
May 11 16:07:14 hostname multipathd: 65:224: reinstated
May 11 16:07:14 hostname multipathd: DISK-E1: queue_if_no_path enabled
May 11 16:07:14 hostname multipathd: DISK-E1: Recovered to normal mode
May 11 16:07:14 hostname multipathd: DISK-E1: remaining active paths: 1
May 11 16:07:14 hostname multipathd: DISK-E1: sdaf - tur checker reportspath is up
May 11 16:07:14 hostname multipathd: 65:240: reinstated
May 11 16:07:14 hostname multipathd: DISK-E1: remaining active paths: 2
May 11 16:07:14 hostname kernel: [ 8072.237208] sd 3:0:2:4: [sdae]
May 11 16:07:14 hostname kernel: [ 8072.237216] sd 3:0:2:4: [sdae] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:14 hostname kernel: [ 8072.237221] sd 3:0:2:4: [sdae] Sense Key : Illegal Request [current]
May 11 16:07:14 hostname kernel: [ 8072.237227] sd 3:0:2:4: [sdae] Add. Sense: Logical block address out of range
May 11 16:07:14 hostname kernel: [ 8072.237231] sd 3:0:2:4: [sdae] CDB: Read(10): 28 00 3e 7f ff 80 00 00 08Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:14 hostname kernel: [ 8072.237238] 00
May 11 16:07:14 hostname kernel: [ 8072.237241] end_request: I/O error, dev sdae, sector 1048575872
May 11 16:07:14 hostname kernel: [ 8072.237245] sd 3:0:2:4: [sdae] Sense Key : Illegal Request [current]
May 11 16:07:14 hostname kernel: [ 8072.237249] sd 3:0:2:4: [sdae]
May 11 16:07:14 hostname kernel: [ 8072.237252] device-mapper: multipath: Failing path 65:224.
May 11 16:07:14 hostname kernel: [ 8072.237255] Add. Sense: Logical block address out of range
May 11 16:07:14 hostname kernel: [ 8072.237261] sd 3:0:2:4: [sdae] CDB: Read(10): 28 00 3e 7f ff f8 00 00 08 00
May 11 16:07:14 hostname kernel: [ 8072.237275] end_request: I/O error, dev sdae, sector 1048575992
May 11 16:07:14 hostname kernel: [ 8072.241846] sd 3:0:3:4: [sdaf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:14 hostname kernel: [ 8072.241853] sd 3:0:3:4: [sdaf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 11 16:07:14 hostname kernel: [ 8072.241861] sd 3:0:3:4: [sdaf] Sense Key : llegal Request [current]
May 11 16:07:14 hostname kernel: [ 8072.241871] sd 3:0:3:4: [sdaf] Add. Sense: Logical block address out of range
May 11 16:07:14 hostname kernel: [ 8072.241878] sd 3:0:3:4: [sdaf] CDB:
May 11 16:07:14 hostname kernel: [ 8072.241879] sd 3:0:3:4: [sdaf] Sense Key : Illegal Request [current]
May 11 16:07:14 hostname kernel: [ 8072.241889] sd 3:0:3:4: [sdaf] Add. Sense: Logical block address out of rangeRead(10): 28 00 3e 7f ff 80 00 00 08 00
May 11 16:07:14 hostname kernel: [ 8072.241913] end_request: I/O error, dev sdaf, sector 1048575872
May 11 16:07:14 hostname kernel: [ 8072.241917]
May 11 16:07:14 hostname kernel: [ 8072.241919] sd 3:0:3:4: [sdaf] CDB: Read(10): 28 00 3e 7f ff f8 00
[...]
Regards,
Ulrich
Mike Christie
> Hi!
>
> This is not an exact open-iscsi question, but tighlty related:
> On a SAN using FibreChannel I had a 4-way multipath device. The basic configuration (without aliases for devices) is:
> devices {
> device {
> vendor "HP"
> product "HSV2.*"
> path_grouping_policy "group_by_prio"
> path_checker "tur"
> prio "alua"
> failback "immediate"
> #polling_interval 30
> no_path_retry 1000
> features "1 queue_if_no_path"
> }
> }
>
> I did a "LUN shrink" on the storage system (because I had created the LUN too large by mistake). During that the LUN went offline, and multipath did detect that fine. However after the LUN shrink was complete (and the LUN was supposed to be online again), multipath still blocked any access to the device (like fdisk -l). Reboot did fix it, but I'm hoping for a better solution.
>
> I'm asking theis question here, because I expect that with iSCSI the problem would have been much the same.
>
> So:
> 1) Did I misconfigure multipath?
> 2) Could I do something to make multipath rediscover the online-again paths?
>
I think you need to rescan the devices at the scsi layer level (like
doing a echo 1 > /sys/block/sdX/device/rescan) then run some multipath
to command, then run some FS and LVM commands if needed.
Here is some info for the multipath command from red hat
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch21s14s03.html
Rene
> I think you need to rescan the devices at the scsi layer level (like
> doing a echo 1 > /sys/block/sdX/device/rescan) then run some multipath
> to command, then run some FS and LVM commands if needed.
Hi,
I'm having a similar problem and stumbled over this thread. For me just running
"multipath -v0" was enough. My question is: Shouldn't multipathd be able to do
that on its own? Whats the point of having 2 paths if it doesn't automatically
recover by itself?
Mike Christie
Ask the multipath list not the iscsi list.
It's probably due to multipath being geared to handling path failures.
The errors we see in Ulrich's log are device failures. We see them on
all paths since they all point to the same device. multipath knows how
to recover from a path failure, but it does not know enough about the
problem to do something about the error we see in this case. It could be
made to do so, but for "Illegal Request - Logical block address out of
range" we normally need changes in the entire kernel. For example you
need to rescan the scsi device manually as it is now. multipath should
not have to do that. The scsi layer or some other handler should, so it
works for non multipath cases.