Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present
126 views
Skip to first unread message
Andrew Patterson
Jun 14, 2016, 7:40:02 PM6/14/16
to
Package: multipath-tools
Version: 0.6.1-3
Multipath discovery fails to find some paths when a large number of
paths are used. I used the following to test this:
1. Present 64 FC LUNs to the OS with 4 paths each (256 paths total).
2. Run multipath -F to make sure we don't have any intermediate results.
3. Run the multipath command, e.g., multipath -v2. In the error case, you
will see "io_setup failed" for some devices.
4. Run multipath -l, checking each mulipath device to make sure
all paths are present (a random number will be missing in the error case).
You can run "multipath -l | grep -E '\|-|`- ' | grep active | wc -l)" to
count the active paths.
Example:
Testing on an amd64 system running sid with
multipath-tools/kpartx_0.6.1-3 from experimental. We hav 64 FC LUNs
with 4 paths each (total 256 paths used mutlipath). The kernel version
is 4.6.0-1-amd64.
With the following /etc/multipath.conf file:
defaults {
user_friendly_names "yes"
}
blacklist {
device {
vendor "HP"
product "LOGICAL VOLUME"
}
Here are the devices used (the boot device is blacklisted):
# lsscsi
[0:0:0:0] storage HP P244br 2.52 -
[0:1:0:0] disk HP LOGICAL VOLUME 2.52 /dev/sda
[1:0:0:0] disk 3PARdata VV 3212 /dev/sdj
[1:0:0:1] disk 3PARdata VV 3212 /dev/sdl
[1:0:0:2] disk 3PARdata VV 3212 /dev/sdn
[1:0:0:3] disk 3PARdata VV 3212 /dev/sdq
[1:0:0:4] disk 3PARdata VV 3212 /dev/sdr
[1:0:0:5] disk 3PARdata VV 3212 /dev/sdt
[1:0:0:6] disk 3PARdata VV 3212 /dev/sdv
[1:0:0:7] disk 3PARdata VV 3212 /dev/sdx
[1:0:0:8] disk 3PARdata VV 3212 /dev/sdah
[1:0:0:9] disk 3PARdata VV 3212 /dev/sdaj
[1:0:0:10] disk 3PARdata VV 3212 /dev/sdal
[1:0:0:11] disk 3PARdata VV 3212 /dev/sdan
[1:0:0:12] disk 3PARdata VV 3212 /dev/sdap
[1:0:0:13] disk 3PARdata VV 3212 /dev/sdar
[1:0:0:14] disk 3PARdata VV 3212 /dev/sdat
[1:0:0:15] disk 3PARdata VV 3212 /dev/sdav
[1:0:0:16] disk 3PARdata VV 3212 /dev/sday
[1:0:0:17] disk 3PARdata VV 3212 /dev/sdbs
[1:0:0:18] disk 3PARdata VV 3212 /dev/sdbu
[1:0:0:19] disk 3PARdata VV 3212 /dev/sdbw
[1:0:0:20] disk 3PARdata VV 3212 /dev/sdby
[1:0:0:21] disk 3PARdata VV 3212 /dev/sdca
[1:0:0:22] disk 3PARdata VV 3212 /dev/sdcc
[1:0:0:23] disk 3PARdata VV 3212 /dev/sdce
[1:0:0:24] disk 3PARdata VV 3212 /dev/sdcg
[1:0:0:25] disk 3PARdata VV 3212 /dev/sdci
[1:0:0:26] disk 3PARdata VV 3212 /dev/sdck
[1:0:0:27] disk 3PARdata VV 3212 /dev/sdcm
[1:0:0:28] disk 3PARdata VV 3212 /dev/sdco
[1:0:0:29] disk 3PARdata VV 3212 /dev/sdcq
[1:0:0:30] disk 3PARdata VV 3212 /dev/sdcs
[1:0:0:31] disk 3PARdata VV 3212 /dev/sdcu
[1:0:0:32] disk 3PARdata VV 3212 /dev/sdea
[1:0:0:33] disk 3PARdata VV 3212 /dev/sdeb
[1:0:0:34] disk 3PARdata VV 3212 /dev/sded
[1:0:0:35] disk 3PARdata VV 3212 /dev/sdef
[1:0:0:36] disk 3PARdata VV 3212 /dev/sdeh
[1:0:0:37] disk 3PARdata VV 3212 /dev/sdej
[1:0:0:38] disk 3PARdata VV 3212 /dev/sdel
[1:0:0:39] disk 3PARdata VV 3212 /dev/sden
[1:0:0:40] disk 3PARdata VV 3212 /dev/sdep
[1:0:0:41] disk 3PARdata VV 3212 /dev/sder
[1:0:0:42] disk 3PARdata VV 3212 /dev/sdet
[1:0:0:43] disk 3PARdata VV 3212 /dev/sdev
[1:0:0:44] disk 3PARdata VV 3212 /dev/sdex
[1:0:0:45] disk 3PARdata VV 3212 /dev/sdez
[1:0:0:46] disk 3PARdata VV 3212 /dev/sdfb
[1:0:0:47] disk 3PARdata VV 3212 /dev/sdfc
[1:0:0:48] disk 3PARdata VV 3212 /dev/sdff
[1:0:0:49] disk 3PARdata VV 3212 /dev/sdfh
[1:0:0:50] disk 3PARdata VV 3212 /dev/sdfj
[1:0:0:51] disk 3PARdata VV 3212 /dev/sdfl
[1:0:0:52] disk 3PARdata VV 3212 /dev/sdfn
[1:0:0:53] disk 3PARdata VV 3212 /dev/sdfp
[1:0:0:54] disk 3PARdata VV 3212 /dev/sdfr
[1:0:0:55] disk 3PARdata VV 3212 /dev/sdft
[1:0:0:56] disk 3PARdata VV 3212 /dev/sdfv
[1:0:0:57] disk 3PARdata VV 3212 /dev/sdfw
[1:0:0:58] disk 3PARdata VV 3212 /dev/sdfz
[1:0:0:59] disk 3PARdata VV 3212 /dev/sdgb
[1:0:0:60] disk 3PARdata VV 3212 /dev/sdgd
[1:0:0:61] disk 3PARdata VV 3212 /dev/sdgf
[1:0:0:62] disk 3PARdata VV 3212 /dev/sdgh
[1:0:0:63] disk 3PARdata VV 3212 /dev/sdgj
[1:0:0:254] enclosu 3PARdata SES 3212 -
[1:0:1:0] disk 3PARdata VV 3212 /dev/sdz
[1:0:1:1] disk 3PARdata VV 3212 /dev/sdaa
[1:0:1:2] disk 3PARdata VV 3212 /dev/sdab
[1:0:1:3] disk 3PARdata VV 3212 /dev/sdac
[1:0:1:4] disk 3PARdata VV 3212 /dev/sdad
[1:0:1:5] disk 3PARdata VV 3212 /dev/sdae
[1:0:1:6] disk 3PARdata VV 3212 /dev/sdaf
[1:0:1:7] disk 3PARdata VV 3212 /dev/sdag
[1:0:1:8] disk 3PARdata VV 3212 /dev/sdba
[1:0:1:9] disk 3PARdata VV 3212 /dev/sdbb
[1:0:1:10] disk 3PARdata VV 3212 /dev/sdbe
[1:0:1:11] disk 3PARdata VV 3212 /dev/sdbf
[1:0:1:12] disk 3PARdata VV 3212 /dev/sdbh
[1:0:1:13] disk 3PARdata VV 3212 /dev/sdbj
[1:0:1:14] disk 3PARdata VV 3212 /dev/sdbk
[1:0:1:15] disk 3PARdata VV 3212 /dev/sdbm
[1:0:1:16] disk 3PARdata VV 3212 /dev/sdbo
[1:0:1:17] disk 3PARdata VV 3212 /dev/sdcv
[1:0:1:18] disk 3PARdata VV 3212 /dev/sdcy
[1:0:1:19] disk 3PARdata VV 3212 /dev/sdcz
[1:0:1:20] disk 3PARdata VV 3212 /dev/sddc
[1:0:1:21] disk 3PARdata VV 3212 /dev/sdde
[1:0:1:22] disk 3PARdata VV 3212 /dev/sddg
[1:0:1:23] disk 3PARdata VV 3212 /dev/sddi
[1:0:1:24] disk 3PARdata VV 3212 /dev/sddk
[1:0:1:25] disk 3PARdata VV 3212 /dev/sddm
[1:0:1:26] disk 3PARdata VV 3212 /dev/sddo
[1:0:1:27] disk 3PARdata VV 3212 /dev/sddq
[1:0:1:28] disk 3PARdata VV 3212 /dev/sddt
[1:0:1:29] disk 3PARdata VV 3212 /dev/sddv
[1:0:1:30] disk 3PARdata VV 3212 /dev/sddx
[1:0:1:31] disk 3PARdata VV 3212 /dev/sddy
[1:0:1:32] disk 3PARdata VV 3212 /dev/sdgl
[1:0:1:33] disk 3PARdata VV 3212 /dev/sdgn
[1:0:1:34] disk 3PARdata VV 3212 /dev/sdgo
[1:0:1:35] disk 3PARdata VV 3212 /dev/sdgq
[1:0:1:36] disk 3PARdata VV 3212 /dev/sdgs
[1:0:1:37] disk 3PARdata VV 3212 /dev/sdgu
[1:0:1:38] disk 3PARdata VV 3212 /dev/sdgw
[1:0:1:39] disk 3PARdata VV 3212 /dev/sdgy
[1:0:1:40] disk 3PARdata VV 3212 /dev/sdha
[1:0:1:41] disk 3PARdata VV 3212 /dev/sdhd
[1:0:1:42] disk 3PARdata VV 3212 /dev/sdhg
[1:0:1:43] disk 3PARdata VV 3212 /dev/sdhi
[1:0:1:44] disk 3PARdata VV 3212 /dev/sdhk
[1:0:1:45] disk 3PARdata VV 3212 /dev/sdhm
[1:0:1:46] disk 3PARdata VV 3212 /dev/sdho
[1:0:1:47] disk 3PARdata VV 3212 /dev/sdhq
[1:0:1:48] disk 3PARdata VV 3212 /dev/sdhr
[1:0:1:49] disk 3PARdata VV 3212 /dev/sdht
[1:0:1:50] disk 3PARdata VV 3212 /dev/sdhv
[1:0:1:51] disk 3PARdata VV 3212 /dev/sdhx
[1:0:1:52] disk 3PARdata VV 3212 /dev/sdhz
[1:0:1:53] disk 3PARdata VV 3212 /dev/sdib
[1:0:1:54] disk 3PARdata VV 3212 /dev/sdid
[1:0:1:55] disk 3PARdata VV 3212 /dev/sdif
[1:0:1:56] disk 3PARdata VV 3212 /dev/sdih
[1:0:1:57] disk 3PARdata VV 3212 /dev/sdik
[1:0:1:58] disk 3PARdata VV 3212 /dev/sdil
[1:0:1:59] disk 3PARdata VV 3212 /dev/sdin
[1:0:1:60] disk 3PARdata VV 3212 /dev/sdip
[1:0:1:61] disk 3PARdata VV 3212 /dev/sdir
[1:0:1:62] disk 3PARdata VV 3212 /dev/sdit
[1:0:1:63] disk 3PARdata VV 3212 /dev/sdiw
[1:0:1:254] enclosu 3PARdata SES 3212 -
[2:0:0:0] disk 3PARdata VV 3212 /dev/sdb
[2:0:0:1] disk 3PARdata VV 3212 /dev/sdc
[2:0:0:2] disk 3PARdata VV 3212 /dev/sdd
[2:0:0:3] disk 3PARdata VV 3212 /dev/sde
[2:0:0:4] disk 3PARdata VV 3212 /dev/sdf
[2:0:0:5] disk 3PARdata VV 3212 /dev/sdg
[2:0:0:6] disk 3PARdata VV 3212 /dev/sdh
[2:0:0:7] disk 3PARdata VV 3212 /dev/sdi
[2:0:0:8] disk 3PARdata VV 3212 /dev/sdai
[2:0:0:9] disk 3PARdata VV 3212 /dev/sdak
[2:0:0:10] disk 3PARdata VV 3212 /dev/sdam
[2:0:0:11] disk 3PARdata VV 3212 /dev/sdao
[2:0:0:12] disk 3PARdata VV 3212 /dev/sdaq
[2:0:0:13] disk 3PARdata VV 3212 /dev/sdas
[2:0:0:14] disk 3PARdata VV 3212 /dev/sdau
[2:0:0:15] disk 3PARdata VV 3212 /dev/sdaw
[2:0:0:16] disk 3PARdata VV 3212 /dev/sdax
[2:0:0:17] disk 3PARdata VV 3212 /dev/sdbr
[2:0:0:18] disk 3PARdata VV 3212 /dev/sdbt
[2:0:0:19] disk 3PARdata VV 3212 /dev/sdbv
[2:0:0:20] disk 3PARdata VV 3212 /dev/sdbx
[2:0:0:21] disk 3PARdata VV 3212 /dev/sdbz
[2:0:0:22] disk 3PARdata VV 3212 /dev/sdcb
[2:0:0:23] disk 3PARdata VV 3212 /dev/sdcd
[2:0:0:24] disk 3PARdata VV 3212 /dev/sdcf
[2:0:0:25] disk 3PARdata VV 3212 /dev/sdch
[2:0:0:26] disk 3PARdata VV 3212 /dev/sdcj
[2:0:0:27] disk 3PARdata VV 3212 /dev/sdcl
[2:0:0:28] disk 3PARdata VV 3212 /dev/sdcn
[2:0:0:29] disk 3PARdata VV 3212 /dev/sdcp
[2:0:0:30] disk 3PARdata VV 3212 /dev/sdcr
[2:0:0:31] disk 3PARdata VV 3212 /dev/sdct
[2:0:0:32] disk 3PARdata VV 3212 /dev/sddz
[2:0:0:33] disk 3PARdata VV 3212 /dev/sdec
[2:0:0:34] disk 3PARdata VV 3212 /dev/sdee
[2:0:0:35] disk 3PARdata VV 3212 /dev/sdeg
[2:0:0:36] disk 3PARdata VV 3212 /dev/sdei
[2:0:0:37] disk 3PARdata VV 3212 /dev/sdek
[2:0:0:38] disk 3PARdata VV 3212 /dev/sdem
[2:0:0:39] disk 3PARdata VV 3212 /dev/sdeo
[2:0:0:40] disk 3PARdata VV 3212 /dev/sdeq
[2:0:0:41] disk 3PARdata VV 3212 /dev/sdes
[2:0:0:42] disk 3PARdata VV 3212 /dev/sdeu
[2:0:0:43] disk 3PARdata VV 3212 /dev/sdew
[2:0:0:44] disk 3PARdata VV 3212 /dev/sdey
[2:0:0:45] disk 3PARdata VV 3212 /dev/sdfa
[2:0:0:46] disk 3PARdata VV 3212 /dev/sdfd
[2:0:0:47] disk 3PARdata VV 3212 /dev/sdfe
[2:0:0:48] disk 3PARdata VV 3212 /dev/sdfg
[2:0:0:49] disk 3PARdata VV 3212 /dev/sdfi
[2:0:0:50] disk 3PARdata VV 3212 /dev/sdfk
[2:0:0:51] disk 3PARdata VV 3212 /dev/sdfm
[2:0:0:52] disk 3PARdata VV 3212 /dev/sdfo
[2:0:0:53] disk 3PARdata VV 3212 /dev/sdfq
[2:0:0:54] disk 3PARdata VV 3212 /dev/sdfs
[2:0:0:55] disk 3PARdata VV 3212 /dev/sdfu
[2:0:0:56] disk 3PARdata VV 3212 /dev/sdfx
[2:0:0:57] disk 3PARdata VV 3212 /dev/sdfy
[2:0:0:58] disk 3PARdata VV 3212 /dev/sdga
[2:0:0:59] disk 3PARdata VV 3212 /dev/sdgc
[2:0:0:60] disk 3PARdata VV 3212 /dev/sdge
[2:0:0:61] disk 3PARdata VV 3212 /dev/sdgg
[2:0:0:62] disk 3PARdata VV 3212 /dev/sdgi
[2:0:0:63] disk 3PARdata VV 3212 /dev/sdgk
[2:0:0:254] enclosu 3PARdata SES 3212 -
[2:0:1:0] disk 3PARdata VV 3212 /dev/sdk
[2:0:1:1] disk 3PARdata VV 3212 /dev/sdm
[2:0:1:2] disk 3PARdata VV 3212 /dev/sdo
[2:0:1:3] disk 3PARdata VV 3212 /dev/sdp
[2:0:1:4] disk 3PARdata VV 3212 /dev/sds
[2:0:1:5] disk 3PARdata VV 3212 /dev/sdu
[2:0:1:6] disk 3PARdata VV 3212 /dev/sdw
[2:0:1:7] disk 3PARdata VV 3212 /dev/sdy
[2:0:1:8] disk 3PARdata VV 3212 /dev/sdaz
[2:0:1:9] disk 3PARdata VV 3212 /dev/sdbc
[2:0:1:10] disk 3PARdata VV 3212 /dev/sdbd
[2:0:1:11] disk 3PARdata VV 3212 /dev/sdbg
[2:0:1:12] disk 3PARdata VV 3212 /dev/sdbi
[2:0:1:13] disk 3PARdata VV 3212 /dev/sdbl
[2:0:1:14] disk 3PARdata VV 3212 /dev/sdbn
[2:0:1:15] disk 3PARdata VV 3212 /dev/sdbp
[2:0:1:16] disk 3PARdata VV 3212 /dev/sdbq
[2:0:1:17] disk 3PARdata VV 3212 /dev/sdcw
[2:0:1:18] disk 3PARdata VV 3212 /dev/sdcx
[2:0:1:19] disk 3PARdata VV 3212 /dev/sdda
[2:0:1:20] disk 3PARdata VV 3212 /dev/sddb
[2:0:1:21] disk 3PARdata VV 3212 /dev/sddd
[2:0:1:22] disk 3PARdata VV 3212 /dev/sddf
[2:0:1:23] disk 3PARdata VV 3212 /dev/sddh
[2:0:1:24] disk 3PARdata VV 3212 /dev/sddj
[2:0:1:25] disk 3PARdata VV 3212 /dev/sddl
[2:0:1:26] disk 3PARdata VV 3212 /dev/sddn
[2:0:1:27] disk 3PARdata VV 3212 /dev/sddp
[2:0:1:28] disk 3PARdata VV 3212 /dev/sddr
[2:0:1:29] disk 3PARdata VV 3212 /dev/sdds
[2:0:1:30] disk 3PARdata VV 3212 /dev/sddu
[2:0:1:31] disk 3PARdata VV 3212 /dev/sddw
[2:0:1:32] disk 3PARdata VV 3212 /dev/sdgm
[2:0:1:33] disk 3PARdata VV 3212 /dev/sdgp
[2:0:1:34] disk 3PARdata VV 3212 /dev/sdgr
[2:0:1:35] disk 3PARdata VV 3212 /dev/sdgt
[2:0:1:36] disk 3PARdata VV 3212 /dev/sdgv
[2:0:1:37] disk 3PARdata VV 3212 /dev/sdgx
[2:0:1:38] disk 3PARdata VV 3212 /dev/sdgz
[2:0:1:39] disk 3PARdata VV 3212 /dev/sdhb
[2:0:1:40] disk 3PARdata VV 3212 /dev/sdhc
[2:0:1:41] disk 3PARdata VV 3212 /dev/sdhe
[2:0:1:42] disk 3PARdata VV 3212 /dev/sdhf
[2:0:1:43] disk 3PARdata VV 3212 /dev/sdhh
[2:0:1:44] disk 3PARdata VV 3212 /dev/sdhj
[2:0:1:45] disk 3PARdata VV 3212 /dev/sdhl
[2:0:1:46] disk 3PARdata VV 3212 /dev/sdhn
[2:0:1:47] disk 3PARdata VV 3212 /dev/sdhp
[2:0:1:48] disk 3PARdata VV 3212 /dev/sdhs
[2:0:1:49] disk 3PARdata VV 3212 /dev/sdhu
[2:0:1:50] disk 3PARdata VV 3212 /dev/sdhw
[2:0:1:51] disk 3PARdata VV 3212 /dev/sdhy
[2:0:1:52] disk 3PARdata VV 3212 /dev/sdia
[2:0:1:53] disk 3PARdata VV 3212 /dev/sdic
[2:0:1:54] disk 3PARdata VV 3212 /dev/sdie
[2:0:1:55] disk 3PARdata VV 3212 /dev/sdig
[2:0:1:56] disk 3PARdata VV 3212 /dev/sdii
[2:0:1:57] disk 3PARdata VV 3212 /dev/sdij
[2:0:1:58] disk 3PARdata VV 3212 /dev/sdim
[2:0:1:59] disk 3PARdata VV 3212 /dev/sdio
[2:0:1:60] disk 3PARdata VV 3212 /dev/sdiq
[2:0:1:61] disk 3PARdata VV 3212 /dev/sdis
[2:0:1:62] disk 3PARdata VV 3212 /dev/sdiu
[2:0:1:63] disk 3PARdata VV 3212 /dev/sdiv
[2:0:1:254] enclosu 3PARdata SES 3212 -
Here is the output of multipath -v3. I have only shown the output for
one succeeding and one failing case.
Jun 14 16:41:26 | sddn: udev property ID_WWN whitelisted
Jun 14 16:41:26 | sddn: mask = 0x3f
Jun 14 16:41:26 | sddn: dev_t = 71:80
Jun 14 16:41:26 | sddn: size = 2097152
Jun 14 16:41:26 | sddn: vendor = 3PARdata
Jun 14 16:41:26 | sddn: product = VV
Jun 14 16:41:26 | sddn: rev = 3212
Jun 14 16:41:26 | sddn: h:b:t:l = 2:0:1:26
Jun 14 16:41:26 | SCSI target 2:0:1 -> FC rport 2:0-4
Jun 14 16:41:26 | sddn: tgt_node_name = 0x2ff70002ac0028be
Jun 14 16:41:26 | sddn: path state = running
Jun 14 16:41:26 | sddn: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 14 16:41:26 | sddn: serial = 1410430
Jun 14 16:41:26 | sddn: get_state
Jun 14 16:41:26 | sddn: path_checker = directio (controller setting)
Jun 14 16:41:26 | sddn: checker timeout = 30 ms (internal default)
Jun 14 16:41:26 | directio: starting new request
Jun 14 16:41:26 | directio: io finished 4096/0
Jun 14 16:41:26 | sddn: state = up
Jun 14 16:41:26 | sddn: uid_attribute = ID_SERIAL (config file default)
Jun 14 16:41:26 | sddn: uid = 360002ac00000000000001997000028be (udev)
Jun 14 16:41:26 | sddn: detect_prio = no (config file default)
Jun 14 16:41:26 | sddn: prio = const controller setting)
Jun 14 16:41:26 | sddn: prio args = "" controller setting)
Jun 14 16:41:26 | sddn: const prio = 1
Jun 14 16:41:26 | sddp: udev property ID_WWN whitelisted
Jun 14 16:41:26 | sddp: mask = 0x3f
Jun 14 16:41:26 | sddp: dev_t = 71:112
Jun 14 16:41:26 | sddp: size = 2097152
Jun 14 16:41:26 | sddp: vendor = 3PARdata
Jun 14 16:41:26 | sddp: product = VV
Jun 14 16:41:26 | sddp: rev = 3212
Jun 14 16:41:26 | sddp: h:b:t:l = 2:0:1:27
Jun 14 16:41:26 | SCSI target 2:0:1 -> FC rport 2:0-4
Jun 14 16:41:26 | sddp: tgt_node_name = 0x2ff70002ac0028be
Jun 14 16:41:26 | sddp: path state = running
Jun 14 16:41:26 | sddp: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 14 16:41:26 | sddp: serial = 1410430
Jun 14 16:41:26 | sddp: get_state
Jun 14 16:41:26 | sddp: path_checker = directio (controller setting)
Jun 14 16:41:26 | sddp: checker timeout = 30 ms (internal default)
Jun 14 16:41:26 | io_setup failed
Jun 14 16:41:26 | sddp: checker init failed
There is no output in dmesg.
Changing /proc/sys/fs/aio-max-nr from the default 65536 to 1048576
seems to help. With this change, I can go up to 256 LUNs (1024 paths)
with no problems.
I have seen similar issues in earlier versions of multipath-tools.
--
Andrew Patterson
Hewlett-Packard Enterprise
Version: 0.6.1-3
Multipath discovery fails to find some paths when a large number of
paths are used. I used the following to test this:
1. Present 64 FC LUNs to the OS with 4 paths each (256 paths total).
2. Run multipath -F to make sure we don't have any intermediate results.
3. Run the multipath command, e.g., multipath -v2. In the error case, you
will see "io_setup failed" for some devices.
4. Run multipath -l, checking each mulipath device to make sure
all paths are present (a random number will be missing in the error case).
You can run "multipath -l | grep -E '\|-|`- ' | grep active | wc -l)" to
count the active paths.
Example:
Testing on an amd64 system running sid with
multipath-tools/kpartx_0.6.1-3 from experimental. We hav 64 FC LUNs
with 4 paths each (total 256 paths used mutlipath). The kernel version
is 4.6.0-1-amd64.
With the following /etc/multipath.conf file:
defaults {
user_friendly_names "yes"
}
blacklist {
device {
vendor "HP"
product "LOGICAL VOLUME"
}
Here are the devices used (the boot device is blacklisted):
# lsscsi
[0:0:0:0] storage HP P244br 2.52 -
[0:1:0:0] disk HP LOGICAL VOLUME 2.52 /dev/sda
[1:0:0:0] disk 3PARdata VV 3212 /dev/sdj
[1:0:0:1] disk 3PARdata VV 3212 /dev/sdl
[1:0:0:2] disk 3PARdata VV 3212 /dev/sdn
[1:0:0:3] disk 3PARdata VV 3212 /dev/sdq
[1:0:0:4] disk 3PARdata VV 3212 /dev/sdr
[1:0:0:5] disk 3PARdata VV 3212 /dev/sdt
[1:0:0:6] disk 3PARdata VV 3212 /dev/sdv
[1:0:0:7] disk 3PARdata VV 3212 /dev/sdx
[1:0:0:8] disk 3PARdata VV 3212 /dev/sdah
[1:0:0:9] disk 3PARdata VV 3212 /dev/sdaj
[1:0:0:10] disk 3PARdata VV 3212 /dev/sdal
[1:0:0:11] disk 3PARdata VV 3212 /dev/sdan
[1:0:0:12] disk 3PARdata VV 3212 /dev/sdap
[1:0:0:13] disk 3PARdata VV 3212 /dev/sdar
[1:0:0:14] disk 3PARdata VV 3212 /dev/sdat
[1:0:0:15] disk 3PARdata VV 3212 /dev/sdav
[1:0:0:16] disk 3PARdata VV 3212 /dev/sday
[1:0:0:17] disk 3PARdata VV 3212 /dev/sdbs
[1:0:0:18] disk 3PARdata VV 3212 /dev/sdbu
[1:0:0:19] disk 3PARdata VV 3212 /dev/sdbw
[1:0:0:20] disk 3PARdata VV 3212 /dev/sdby
[1:0:0:21] disk 3PARdata VV 3212 /dev/sdca
[1:0:0:22] disk 3PARdata VV 3212 /dev/sdcc
[1:0:0:23] disk 3PARdata VV 3212 /dev/sdce
[1:0:0:24] disk 3PARdata VV 3212 /dev/sdcg
[1:0:0:25] disk 3PARdata VV 3212 /dev/sdci
[1:0:0:26] disk 3PARdata VV 3212 /dev/sdck
[1:0:0:27] disk 3PARdata VV 3212 /dev/sdcm
[1:0:0:28] disk 3PARdata VV 3212 /dev/sdco
[1:0:0:29] disk 3PARdata VV 3212 /dev/sdcq
[1:0:0:30] disk 3PARdata VV 3212 /dev/sdcs
[1:0:0:31] disk 3PARdata VV 3212 /dev/sdcu
[1:0:0:32] disk 3PARdata VV 3212 /dev/sdea
[1:0:0:33] disk 3PARdata VV 3212 /dev/sdeb
[1:0:0:34] disk 3PARdata VV 3212 /dev/sded
[1:0:0:35] disk 3PARdata VV 3212 /dev/sdef
[1:0:0:36] disk 3PARdata VV 3212 /dev/sdeh
[1:0:0:37] disk 3PARdata VV 3212 /dev/sdej
[1:0:0:38] disk 3PARdata VV 3212 /dev/sdel
[1:0:0:39] disk 3PARdata VV 3212 /dev/sden
[1:0:0:40] disk 3PARdata VV 3212 /dev/sdep
[1:0:0:41] disk 3PARdata VV 3212 /dev/sder
[1:0:0:42] disk 3PARdata VV 3212 /dev/sdet
[1:0:0:43] disk 3PARdata VV 3212 /dev/sdev
[1:0:0:44] disk 3PARdata VV 3212 /dev/sdex
[1:0:0:45] disk 3PARdata VV 3212 /dev/sdez
[1:0:0:46] disk 3PARdata VV 3212 /dev/sdfb
[1:0:0:47] disk 3PARdata VV 3212 /dev/sdfc
[1:0:0:48] disk 3PARdata VV 3212 /dev/sdff
[1:0:0:49] disk 3PARdata VV 3212 /dev/sdfh
[1:0:0:50] disk 3PARdata VV 3212 /dev/sdfj
[1:0:0:51] disk 3PARdata VV 3212 /dev/sdfl
[1:0:0:52] disk 3PARdata VV 3212 /dev/sdfn
[1:0:0:53] disk 3PARdata VV 3212 /dev/sdfp
[1:0:0:54] disk 3PARdata VV 3212 /dev/sdfr
[1:0:0:55] disk 3PARdata VV 3212 /dev/sdft
[1:0:0:56] disk 3PARdata VV 3212 /dev/sdfv
[1:0:0:57] disk 3PARdata VV 3212 /dev/sdfw
[1:0:0:58] disk 3PARdata VV 3212 /dev/sdfz
[1:0:0:59] disk 3PARdata VV 3212 /dev/sdgb
[1:0:0:60] disk 3PARdata VV 3212 /dev/sdgd
[1:0:0:61] disk 3PARdata VV 3212 /dev/sdgf
[1:0:0:62] disk 3PARdata VV 3212 /dev/sdgh
[1:0:0:63] disk 3PARdata VV 3212 /dev/sdgj
[1:0:0:254] enclosu 3PARdata SES 3212 -
[1:0:1:0] disk 3PARdata VV 3212 /dev/sdz
[1:0:1:1] disk 3PARdata VV 3212 /dev/sdaa
[1:0:1:2] disk 3PARdata VV 3212 /dev/sdab
[1:0:1:3] disk 3PARdata VV 3212 /dev/sdac
[1:0:1:4] disk 3PARdata VV 3212 /dev/sdad
[1:0:1:5] disk 3PARdata VV 3212 /dev/sdae
[1:0:1:6] disk 3PARdata VV 3212 /dev/sdaf
[1:0:1:7] disk 3PARdata VV 3212 /dev/sdag
[1:0:1:8] disk 3PARdata VV 3212 /dev/sdba
[1:0:1:9] disk 3PARdata VV 3212 /dev/sdbb
[1:0:1:10] disk 3PARdata VV 3212 /dev/sdbe
[1:0:1:11] disk 3PARdata VV 3212 /dev/sdbf
[1:0:1:12] disk 3PARdata VV 3212 /dev/sdbh
[1:0:1:13] disk 3PARdata VV 3212 /dev/sdbj
[1:0:1:14] disk 3PARdata VV 3212 /dev/sdbk
[1:0:1:15] disk 3PARdata VV 3212 /dev/sdbm
[1:0:1:16] disk 3PARdata VV 3212 /dev/sdbo
[1:0:1:17] disk 3PARdata VV 3212 /dev/sdcv
[1:0:1:18] disk 3PARdata VV 3212 /dev/sdcy
[1:0:1:19] disk 3PARdata VV 3212 /dev/sdcz
[1:0:1:20] disk 3PARdata VV 3212 /dev/sddc
[1:0:1:21] disk 3PARdata VV 3212 /dev/sdde
[1:0:1:22] disk 3PARdata VV 3212 /dev/sddg
[1:0:1:23] disk 3PARdata VV 3212 /dev/sddi
[1:0:1:24] disk 3PARdata VV 3212 /dev/sddk
[1:0:1:25] disk 3PARdata VV 3212 /dev/sddm
[1:0:1:26] disk 3PARdata VV 3212 /dev/sddo
[1:0:1:27] disk 3PARdata VV 3212 /dev/sddq
[1:0:1:28] disk 3PARdata VV 3212 /dev/sddt
[1:0:1:29] disk 3PARdata VV 3212 /dev/sddv
[1:0:1:30] disk 3PARdata VV 3212 /dev/sddx
[1:0:1:31] disk 3PARdata VV 3212 /dev/sddy
[1:0:1:32] disk 3PARdata VV 3212 /dev/sdgl
[1:0:1:33] disk 3PARdata VV 3212 /dev/sdgn
[1:0:1:34] disk 3PARdata VV 3212 /dev/sdgo
[1:0:1:35] disk 3PARdata VV 3212 /dev/sdgq
[1:0:1:36] disk 3PARdata VV 3212 /dev/sdgs
[1:0:1:37] disk 3PARdata VV 3212 /dev/sdgu
[1:0:1:38] disk 3PARdata VV 3212 /dev/sdgw
[1:0:1:39] disk 3PARdata VV 3212 /dev/sdgy
[1:0:1:40] disk 3PARdata VV 3212 /dev/sdha
[1:0:1:41] disk 3PARdata VV 3212 /dev/sdhd
[1:0:1:42] disk 3PARdata VV 3212 /dev/sdhg
[1:0:1:43] disk 3PARdata VV 3212 /dev/sdhi
[1:0:1:44] disk 3PARdata VV 3212 /dev/sdhk
[1:0:1:45] disk 3PARdata VV 3212 /dev/sdhm
[1:0:1:46] disk 3PARdata VV 3212 /dev/sdho
[1:0:1:47] disk 3PARdata VV 3212 /dev/sdhq
[1:0:1:48] disk 3PARdata VV 3212 /dev/sdhr
[1:0:1:49] disk 3PARdata VV 3212 /dev/sdht
[1:0:1:50] disk 3PARdata VV 3212 /dev/sdhv
[1:0:1:51] disk 3PARdata VV 3212 /dev/sdhx
[1:0:1:52] disk 3PARdata VV 3212 /dev/sdhz
[1:0:1:53] disk 3PARdata VV 3212 /dev/sdib
[1:0:1:54] disk 3PARdata VV 3212 /dev/sdid
[1:0:1:55] disk 3PARdata VV 3212 /dev/sdif
[1:0:1:56] disk 3PARdata VV 3212 /dev/sdih
[1:0:1:57] disk 3PARdata VV 3212 /dev/sdik
[1:0:1:58] disk 3PARdata VV 3212 /dev/sdil
[1:0:1:59] disk 3PARdata VV 3212 /dev/sdin
[1:0:1:60] disk 3PARdata VV 3212 /dev/sdip
[1:0:1:61] disk 3PARdata VV 3212 /dev/sdir
[1:0:1:62] disk 3PARdata VV 3212 /dev/sdit
[1:0:1:63] disk 3PARdata VV 3212 /dev/sdiw
[1:0:1:254] enclosu 3PARdata SES 3212 -
[2:0:0:0] disk 3PARdata VV 3212 /dev/sdb
[2:0:0:1] disk 3PARdata VV 3212 /dev/sdc
[2:0:0:2] disk 3PARdata VV 3212 /dev/sdd
[2:0:0:3] disk 3PARdata VV 3212 /dev/sde
[2:0:0:4] disk 3PARdata VV 3212 /dev/sdf
[2:0:0:5] disk 3PARdata VV 3212 /dev/sdg
[2:0:0:6] disk 3PARdata VV 3212 /dev/sdh
[2:0:0:7] disk 3PARdata VV 3212 /dev/sdi
[2:0:0:8] disk 3PARdata VV 3212 /dev/sdai
[2:0:0:9] disk 3PARdata VV 3212 /dev/sdak
[2:0:0:10] disk 3PARdata VV 3212 /dev/sdam
[2:0:0:11] disk 3PARdata VV 3212 /dev/sdao
[2:0:0:12] disk 3PARdata VV 3212 /dev/sdaq
[2:0:0:13] disk 3PARdata VV 3212 /dev/sdas
[2:0:0:14] disk 3PARdata VV 3212 /dev/sdau
[2:0:0:15] disk 3PARdata VV 3212 /dev/sdaw
[2:0:0:16] disk 3PARdata VV 3212 /dev/sdax
[2:0:0:17] disk 3PARdata VV 3212 /dev/sdbr
[2:0:0:18] disk 3PARdata VV 3212 /dev/sdbt
[2:0:0:19] disk 3PARdata VV 3212 /dev/sdbv
[2:0:0:20] disk 3PARdata VV 3212 /dev/sdbx
[2:0:0:21] disk 3PARdata VV 3212 /dev/sdbz
[2:0:0:22] disk 3PARdata VV 3212 /dev/sdcb
[2:0:0:23] disk 3PARdata VV 3212 /dev/sdcd
[2:0:0:24] disk 3PARdata VV 3212 /dev/sdcf
[2:0:0:25] disk 3PARdata VV 3212 /dev/sdch
[2:0:0:26] disk 3PARdata VV 3212 /dev/sdcj
[2:0:0:27] disk 3PARdata VV 3212 /dev/sdcl
[2:0:0:28] disk 3PARdata VV 3212 /dev/sdcn
[2:0:0:29] disk 3PARdata VV 3212 /dev/sdcp
[2:0:0:30] disk 3PARdata VV 3212 /dev/sdcr
[2:0:0:31] disk 3PARdata VV 3212 /dev/sdct
[2:0:0:32] disk 3PARdata VV 3212 /dev/sddz
[2:0:0:33] disk 3PARdata VV 3212 /dev/sdec
[2:0:0:34] disk 3PARdata VV 3212 /dev/sdee
[2:0:0:35] disk 3PARdata VV 3212 /dev/sdeg
[2:0:0:36] disk 3PARdata VV 3212 /dev/sdei
[2:0:0:37] disk 3PARdata VV 3212 /dev/sdek
[2:0:0:38] disk 3PARdata VV 3212 /dev/sdem
[2:0:0:39] disk 3PARdata VV 3212 /dev/sdeo
[2:0:0:40] disk 3PARdata VV 3212 /dev/sdeq
[2:0:0:41] disk 3PARdata VV 3212 /dev/sdes
[2:0:0:42] disk 3PARdata VV 3212 /dev/sdeu
[2:0:0:43] disk 3PARdata VV 3212 /dev/sdew
[2:0:0:44] disk 3PARdata VV 3212 /dev/sdey
[2:0:0:45] disk 3PARdata VV 3212 /dev/sdfa
[2:0:0:46] disk 3PARdata VV 3212 /dev/sdfd
[2:0:0:47] disk 3PARdata VV 3212 /dev/sdfe
[2:0:0:48] disk 3PARdata VV 3212 /dev/sdfg
[2:0:0:49] disk 3PARdata VV 3212 /dev/sdfi
[2:0:0:50] disk 3PARdata VV 3212 /dev/sdfk
[2:0:0:51] disk 3PARdata VV 3212 /dev/sdfm
[2:0:0:52] disk 3PARdata VV 3212 /dev/sdfo
[2:0:0:53] disk 3PARdata VV 3212 /dev/sdfq
[2:0:0:54] disk 3PARdata VV 3212 /dev/sdfs
[2:0:0:55] disk 3PARdata VV 3212 /dev/sdfu
[2:0:0:56] disk 3PARdata VV 3212 /dev/sdfx
[2:0:0:57] disk 3PARdata VV 3212 /dev/sdfy
[2:0:0:58] disk 3PARdata VV 3212 /dev/sdga
[2:0:0:59] disk 3PARdata VV 3212 /dev/sdgc
[2:0:0:60] disk 3PARdata VV 3212 /dev/sdge
[2:0:0:61] disk 3PARdata VV 3212 /dev/sdgg
[2:0:0:62] disk 3PARdata VV 3212 /dev/sdgi
[2:0:0:63] disk 3PARdata VV 3212 /dev/sdgk
[2:0:0:254] enclosu 3PARdata SES 3212 -
[2:0:1:0] disk 3PARdata VV 3212 /dev/sdk
[2:0:1:1] disk 3PARdata VV 3212 /dev/sdm
[2:0:1:2] disk 3PARdata VV 3212 /dev/sdo
[2:0:1:3] disk 3PARdata VV 3212 /dev/sdp
[2:0:1:4] disk 3PARdata VV 3212 /dev/sds
[2:0:1:5] disk 3PARdata VV 3212 /dev/sdu
[2:0:1:6] disk 3PARdata VV 3212 /dev/sdw
[2:0:1:7] disk 3PARdata VV 3212 /dev/sdy
[2:0:1:8] disk 3PARdata VV 3212 /dev/sdaz
[2:0:1:9] disk 3PARdata VV 3212 /dev/sdbc
[2:0:1:10] disk 3PARdata VV 3212 /dev/sdbd
[2:0:1:11] disk 3PARdata VV 3212 /dev/sdbg
[2:0:1:12] disk 3PARdata VV 3212 /dev/sdbi
[2:0:1:13] disk 3PARdata VV 3212 /dev/sdbl
[2:0:1:14] disk 3PARdata VV 3212 /dev/sdbn
[2:0:1:15] disk 3PARdata VV 3212 /dev/sdbp
[2:0:1:16] disk 3PARdata VV 3212 /dev/sdbq
[2:0:1:17] disk 3PARdata VV 3212 /dev/sdcw
[2:0:1:18] disk 3PARdata VV 3212 /dev/sdcx
[2:0:1:19] disk 3PARdata VV 3212 /dev/sdda
[2:0:1:20] disk 3PARdata VV 3212 /dev/sddb
[2:0:1:21] disk 3PARdata VV 3212 /dev/sddd
[2:0:1:22] disk 3PARdata VV 3212 /dev/sddf
[2:0:1:23] disk 3PARdata VV 3212 /dev/sddh
[2:0:1:24] disk 3PARdata VV 3212 /dev/sddj
[2:0:1:25] disk 3PARdata VV 3212 /dev/sddl
[2:0:1:26] disk 3PARdata VV 3212 /dev/sddn
[2:0:1:27] disk 3PARdata VV 3212 /dev/sddp
[2:0:1:28] disk 3PARdata VV 3212 /dev/sddr
[2:0:1:29] disk 3PARdata VV 3212 /dev/sdds
[2:0:1:30] disk 3PARdata VV 3212 /dev/sddu
[2:0:1:31] disk 3PARdata VV 3212 /dev/sddw
[2:0:1:32] disk 3PARdata VV 3212 /dev/sdgm
[2:0:1:33] disk 3PARdata VV 3212 /dev/sdgp
[2:0:1:34] disk 3PARdata VV 3212 /dev/sdgr
[2:0:1:35] disk 3PARdata VV 3212 /dev/sdgt
[2:0:1:36] disk 3PARdata VV 3212 /dev/sdgv
[2:0:1:37] disk 3PARdata VV 3212 /dev/sdgx
[2:0:1:38] disk 3PARdata VV 3212 /dev/sdgz
[2:0:1:39] disk 3PARdata VV 3212 /dev/sdhb
[2:0:1:40] disk 3PARdata VV 3212 /dev/sdhc
[2:0:1:41] disk 3PARdata VV 3212 /dev/sdhe
[2:0:1:42] disk 3PARdata VV 3212 /dev/sdhf
[2:0:1:43] disk 3PARdata VV 3212 /dev/sdhh
[2:0:1:44] disk 3PARdata VV 3212 /dev/sdhj
[2:0:1:45] disk 3PARdata VV 3212 /dev/sdhl
[2:0:1:46] disk 3PARdata VV 3212 /dev/sdhn
[2:0:1:47] disk 3PARdata VV 3212 /dev/sdhp
[2:0:1:48] disk 3PARdata VV 3212 /dev/sdhs
[2:0:1:49] disk 3PARdata VV 3212 /dev/sdhu
[2:0:1:50] disk 3PARdata VV 3212 /dev/sdhw
[2:0:1:51] disk 3PARdata VV 3212 /dev/sdhy
[2:0:1:52] disk 3PARdata VV 3212 /dev/sdia
[2:0:1:53] disk 3PARdata VV 3212 /dev/sdic
[2:0:1:54] disk 3PARdata VV 3212 /dev/sdie
[2:0:1:55] disk 3PARdata VV 3212 /dev/sdig
[2:0:1:56] disk 3PARdata VV 3212 /dev/sdii
[2:0:1:57] disk 3PARdata VV 3212 /dev/sdij
[2:0:1:58] disk 3PARdata VV 3212 /dev/sdim
[2:0:1:59] disk 3PARdata VV 3212 /dev/sdio
[2:0:1:60] disk 3PARdata VV 3212 /dev/sdiq
[2:0:1:61] disk 3PARdata VV 3212 /dev/sdis
[2:0:1:62] disk 3PARdata VV 3212 /dev/sdiu
[2:0:1:63] disk 3PARdata VV 3212 /dev/sdiv
[2:0:1:254] enclosu 3PARdata SES 3212 -
Here is the output of multipath -v3. I have only shown the output for
one succeeding and one failing case.
Jun 14 16:41:26 | sddn: udev property ID_WWN whitelisted
Jun 14 16:41:26 | sddn: mask = 0x3f
Jun 14 16:41:26 | sddn: dev_t = 71:80
Jun 14 16:41:26 | sddn: size = 2097152
Jun 14 16:41:26 | sddn: vendor = 3PARdata
Jun 14 16:41:26 | sddn: product = VV
Jun 14 16:41:26 | sddn: rev = 3212
Jun 14 16:41:26 | sddn: h:b:t:l = 2:0:1:26
Jun 14 16:41:26 | SCSI target 2:0:1 -> FC rport 2:0-4
Jun 14 16:41:26 | sddn: tgt_node_name = 0x2ff70002ac0028be
Jun 14 16:41:26 | sddn: path state = running
Jun 14 16:41:26 | sddn: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 14 16:41:26 | sddn: serial = 1410430
Jun 14 16:41:26 | sddn: get_state
Jun 14 16:41:26 | sddn: path_checker = directio (controller setting)
Jun 14 16:41:26 | sddn: checker timeout = 30 ms (internal default)
Jun 14 16:41:26 | directio: starting new request
Jun 14 16:41:26 | directio: io finished 4096/0
Jun 14 16:41:26 | sddn: state = up
Jun 14 16:41:26 | sddn: uid_attribute = ID_SERIAL (config file default)
Jun 14 16:41:26 | sddn: uid = 360002ac00000000000001997000028be (udev)
Jun 14 16:41:26 | sddn: detect_prio = no (config file default)
Jun 14 16:41:26 | sddn: prio = const controller setting)
Jun 14 16:41:26 | sddn: prio args = "" controller setting)
Jun 14 16:41:26 | sddn: const prio = 1
Jun 14 16:41:26 | sddp: udev property ID_WWN whitelisted
Jun 14 16:41:26 | sddp: mask = 0x3f
Jun 14 16:41:26 | sddp: dev_t = 71:112
Jun 14 16:41:26 | sddp: size = 2097152
Jun 14 16:41:26 | sddp: vendor = 3PARdata
Jun 14 16:41:26 | sddp: product = VV
Jun 14 16:41:26 | sddp: rev = 3212
Jun 14 16:41:26 | sddp: h:b:t:l = 2:0:1:27
Jun 14 16:41:26 | SCSI target 2:0:1 -> FC rport 2:0-4
Jun 14 16:41:26 | sddp: tgt_node_name = 0x2ff70002ac0028be
Jun 14 16:41:26 | sddp: path state = running
Jun 14 16:41:26 | sddp: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 14 16:41:26 | sddp: serial = 1410430
Jun 14 16:41:26 | sddp: get_state
Jun 14 16:41:26 | sddp: path_checker = directio (controller setting)
Jun 14 16:41:26 | sddp: checker timeout = 30 ms (internal default)
Jun 14 16:41:26 | io_setup failed
Jun 14 16:41:26 | sddp: checker init failed
There is no output in dmesg.
Changing /proc/sys/fs/aio-max-nr from the default 65536 to 1048576
seems to help. With this change, I can go up to 256 LUNs (1024 paths)
with no problems.
I have seen similar issues in earlier versions of multipath-tools.
--
Andrew Patterson
Hewlett-Packard Enterprise
Ritesh Raj Sarraf
Jun 15, 2016, 10:50:03 AM6/15/16
to
Control: tag -1 moreinfo
On Tue, 2016-06-14 at 17:30 -0600, Andrew Patterson wrote:
> There is no output in dmesg.
>
> Changing /proc/sys/fs/aio-max-nr from the default 65536 to 1048576
> seems to help. With this change, I can go up to 256 LUNs (1024 paths)
> with no problems.
>
> I have seen similar issues in earlier versions of multipath-tools.
Thank you for bringing this up, and the pointer. In multipath.conf there was a
directive added for something similar.
max_fds Specify the maximum number of file descriptors that
can be opened by multipath and multipathd. This is
equivalent to ulimit -n. A value of max will set this
to the system limit from /proc/sys/fs/nr_open. If this
is not set, the maximum number of open fds is taken
from the calling process. It is usually 1024. To be
safe, this should be set to the maximum number of
paths plus 32, if that number is greated than 1024.
It seems to be inline with what aio-max-nr documentation says.
==============================================================
aio-nr & aio-max-nr:
aio-nr is the running total of the number of events specified on the
io_setup system call for all currently active aio contexts. If aio-nr
reaches aio-max-nr then io_setup will fail with EAGAIN. Note that
raising aio-max-nr does not result in the pre-allocation or re-sizing
of any kernel data structures.
==============================================================
Have you tried it with the "max_fds" option set ?
--
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Andrew Patterson
Jun 15, 2016, 1:30:03 PM6/15/16
to
# cat /proc/sys/fs/nr_open
1048576
and
# multipath -t | grep max_fds
max_fds "max"
So this should already be set to 1048576. I tried the test again after
setting max_fd 1048576 in /etc/multipath.conf and got the same
behavior.
Ritesh Raj Sarraf
Jun 15, 2016, 3:00:03 PM6/15/16
to
On Wed, 2016-06-15 at 11:12 -0600, Andrew Patterson wrote:
>
> Thanks for reminding me about max_fds. At boot:
>
> # cat /proc/sys/fs/nr_open
> 1048576
>
> and
>
> # multipath -t | grep max_fds
> max_fds "max"
>
> So this should already be set to 1048576. I tried the test again after
> setting max_fd 1048576 in /etc/multipath.conf and got the same
> behavior.
This dates way back. 2009 is a memorable year.
>
> Thanks for reminding me about max_fds. At boot:
>
> # cat /proc/sys/fs/nr_open
> 1048576
>
> and
>
> # multipath -t | grep max_fds
> max_fds "max"
>
> So this should already be set to 1048576. I tried the test again after
> setting max_fd 1048576 in /etc/multipath.conf and got the same
> behavior.
https://www.redhat.com/archives/dm-devel/2009-July/msg00020.html
Quoting:
+
+4. I see a lot of "io_setup failed" message using the directio checker
+======================================================================
+
+The directio path checker makes use of the asynchronous I/O API (aio) provided
+by modern Linux systems. Asynchronous I/O allows an application to submit I/O
+requests asynchronously and be notified later of their completion status. To
+allow this, we must allocate an asynchronous I/O context (an object of type
+aio_context_t) and this task is handled by the io_setup system call.
+
+A system wide limit on the number of AIO contexts that may be active
+simultaneously is imposed via the aio-max-nr sysctl parameter.
+
+Once this limit has been reached further calls to io_setup will fail with the
+error number EAGAIN leading to the "io_setup failed" messages seen for e.g.
when
+running "multipath -ll".
+
+To avoid this problem the number of available aio contexts should be increased
+by setting the aio-max-nr parameter. This can be set on a one-time basis via
the
+/proc file system, for e.g.:
+
+ # echo 131072 > /proc/sys/fs/aio-max-nr
+
+Doubles the number of available contexts from the default value of 65536.
+
+To make this setting persistent a line may be added to /etc/sysctl.conf:
+
+ fs.aio-max-nr = 131072
+
+Consult appropriate application and operating system tuning recommendations for
+guidance on appropriate values for this parameter.
Andrew Patterson
Jun 15, 2016, 3:20:03 PM6/15/16
to
On Thu, 16 Jun 2016 00:15:37 +0530 Ritesh Raj Sarraf <r...@debian.org> wrote:
> On Wed, 2016-06-15 at 11:12 -0600, Andrew Patterson wrote:
> >
> > Thanks for reminding me about max_fds. At boot:
> >
> > # cat /proc/sys/fs/nr_open
> > 1048576
> >
> > and
> >
> > # multipath -t | grep max_fds
> > Â Â Â Â Â Â Â max_fds "max"
> On Wed, 2016-06-15 at 11:12 -0600, Andrew Patterson wrote:
> >
> > Thanks for reminding me about max_fds. At boot:
> >
> > # cat /proc/sys/fs/nr_open
> > 1048576
> >
> > and
> >
> > # multipath -t | grep max_fds
> +
> +Doubles the number of available contexts from the default value of 65536.
> +
> +To make this setting persistent a line may be added to /etc/sysctl.conf:
> +
> +Â Â Â Â Â Â Â Â fs.aio-max-nr = 131072
> +Doubles the number of available contexts from the default value of 65536.
> +
> +To make this setting persistent a line may be added to /etc/sysctl.conf:
> +
> +
> +Consult appropriate application and operating system tuning recommendations for
> +guidance on appropriate values for this parameter.
>
>
> --
> Ritesh Raj Sarraf | http://people.debian.org/~rrs
For most users this is not an issue. How many systems are going to use
> +Consult appropriate application and operating system tuning recommendations for
> +guidance on appropriate values for this parameter.
>
>
> --
> Ritesh Raj Sarraf | http://people.debian.org/~rrs
that many LUNs?
One counter-example is using one or more FC LUNs per VM guest on a
host. In such a case, one can just modify /etc/sysctl.conf to increase
the setting. We can run into problems when using
multipath-tools-boot. In this case, we use the default kernel settings
for aio-max-nr since multipath is run from the initramfs. I have
worked around this by modifying the script in
/usr/share/initramfs/scripts/local-top/multipath to temporarilly set
aio-max-nr to 1048576 while doing discovery.
Ritesh Raj Sarraf
Jun 16, 2016, 6:00:03 AM6/16/16
to
Control: tag -1 +pending
On Wed, 2016-06-15 at 13:03 -0600, Andrew Patterson wrote:
> For most users this is not an issue. How many systems are going to use
> that many LUNs?
>
I agree.
> One counter-example is using one or more FC LUNs per VM guest on a
> host. In such a case, one can just modify /etc/sysctl.conf to increase
> the setting.
Yes. Most hypervisors may want this. But again, like you said, most users do not
map that many LUNs. And recommended practice is to have a VM backed by a file
backend mostly.
> We can run into problems when using
> multipath-tools-boot. In this case, we use the default kernel settings
> for aio-max-nr since multipath is run from the initramfs. I have
> worked around this by modifying the script in
> /usr/share/initramfs/scripts/local-top/multipath to temporarilly set
> aio-max-nr to 1048576 while doing discovery.
So, I'll just add this to README.Debian. There's not much that we can do beyond
documenting such behavior.
On Wed, 2016-06-15 at 13:03 -0600, Andrew Patterson wrote:
> For most users this is not an issue. How many systems are going to use
> that many LUNs?
>
> One counter-example is using one or more FC LUNs per VM guest on a
> host. In such a case, one can just modify /etc/sysctl.conf to increase
> the setting.
map that many LUNs. And recommended practice is to have a VM backed by a file
backend mostly.
> We can run into problems when using
> multipath-tools-boot. In this case, we use the default kernel settings
> for aio-max-nr since multipath is run from the initramfs. I have
> worked around this by modifying the script in
> /usr/share/initramfs/scripts/local-top/multipath to temporarilly set
> aio-max-nr to 1048576 while doing discovery.
documenting such behavior.
Andrew Patterson
Jun 20, 2016, 6:50:03 PM6/20/16
to
On Thu, 16 Jun 2016 15:19:22 +0530 Ritesh Raj Sarraf <r...@debian.org> wrote:
> Control: tag -1 +pending
>
> On Wed, 2016-06-15 at 13:03 -0600, Andrew Patterson wrote:
> > For most users this is not an issue. How many systems are going to use
> > that many LUNs?
> >
>
> I agree.
>
> > One counter-example is using one or more FC LUNs per VM guest on a
> > host. In such a case, one can just modify /etc/sysctl.conf to increase
> > the setting.
>
> Yes. Most hypervisors may want this. But again, like you said, most users do not
> map that many LUNs. And recommended practice is to have a VM backed by a file
> backend mostly.
OpenStack/cinder can (and is) be uses to provision VMs with networked
> Control: tag -1 +pending
>
> On Wed, 2016-06-15 at 13:03 -0600, Andrew Patterson wrote:
> > For most users this is not an issue. How many systems are going to use
> > that many LUNs?
> >
>
> I agree.
>
> > One counter-example is using one or more FC LUNs per VM guest on a
> > host. In such a case, one can just modify /etc/sysctl.conf to increase
> > the setting.
>
> Yes. Most hypervisors may want this. But again, like you said, most users do not
> map that many LUNs. And recommended practice is to have a VM backed by a file
> backend mostly.
drives. It is pretty common for cloud applications. I don't know what
a practical limit for the number of volume per VMs per host should be.
>
> > We can run into problems when using
> > multipath-tools-boot. In this case, we use the default kernel settings
> > for aio-max-nr since multipath is run from the initramfs. I have
> > worked around this by modifying the script in
> > /usr/share/initramfs/scripts/local-top/multipath to temporarilly set
> > aio-max-nr to 1048576 while doing discovery.
>
>
> So, I'll just add this to README.Debian. There's not much that we can do beyond
> documenting such behavior.
in the initramfs. You can then decide if it is worth it to apply it.
In the meantime, the README.Debian change sounds fine to me.
Andrew Patterson
Jun 23, 2016, 7:00:03 PM6/23/16
to
initramfs. There is more to do to solve the problem with booting with
many LUNs, but this is a prerequisite for completely solving that
issue. This patch assumes that the patch in:
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=827407;filename=multipath-tools-827407.patch;msg=30
has been applied. I can rework it if that one is not accepted.
Andrew
Ritesh Raj Sarraf
Jun 25, 2016, 4:30:02 AM6/25/16
to
On Thu, 2016-06-23 at 16:50 -0600, Andrew Patterson wrote:
> +# Save boot setting for aio-max-nr.
> +BOOT_AIO_MAX_NR=$(cat /proc/sys/fs/aio-max-nr)
> +
> if [ -f /etc/multipath-tools-boot.conf ]; then
> . /etc/multipath-tools-boot.conf
> else
> MULTIPATH_TOOLS_MODULES="dm-mod dm-multipath"
> + MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$BOOT_AIO_MAX_NR
> fi
>
I'd let the user define the AIO_MAX value explicitly in /etc/default/multipath-
tools. And use the sysctl/procfs option as a fallback.
What do you say ?
> +# Save boot setting for aio-max-nr.
> +BOOT_AIO_MAX_NR=$(cat /proc/sys/fs/aio-max-nr)
> +
> if [ -f /etc/multipath-tools-boot.conf ]; then
> . /etc/multipath-tools-boot.conf
> else
> MULTIPATH_TOOLS_MODULES="dm-mod dm-multipath"
> + MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$BOOT_AIO_MAX_NR
> fi
>
I'd let the user define the AIO_MAX value explicitly in /etc/default/multipath-
tools. And use the sysctl/procfs option as a fallback.
What do you say ?
Andrew Patterson
Jun 25, 2016, 9:50:02 AM6/25/16
to
On 06/25/2016 02:16 AM, Ritesh Raj Sarraf wrote:
> On Thu, 2016-06-23 at 16:50 -0600, Andrew Patterson wrote: >> +# Save boot setting for aio-max-nr. >> +BOOT_AIO_MAX_NR=$(cat /proc/sys/fs/aio-max-nr) >> + >> if [ -f /etc/multipath-tools-boot.conf ]; then >> . /etc/multipath-tools-boot.conf >> else >> MULTIPATH_TOOLS_MODULES="dm-mod dm-multipath" >> + MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$BOOT_AIO_MAX_NR >> fi >> > > I'd let the user define the AIO_MAX value explicitly in /etc/default/multipath- > tools. And use the sysctl/procfs option as a fallback. > > What do you say ? > >
> On Thu, 2016-06-23 at 16:50 -0600, Andrew Patterson wrote: >> +# Save boot setting for aio-max-nr. >> +BOOT_AIO_MAX_NR=$(cat /proc/sys/fs/aio-max-nr) >> + >> if [ -f /etc/multipath-tools-boot.conf ]; then >> . /etc/multipath-tools-boot.conf >> else >> MULTIPATH_TOOLS_MODULES="dm-mod dm-multipath" >> + MULTIPATH_TOOLS_MIN_AIO_MAX_NR=$BOOT_AIO_MAX_NR >> fi >> > > I'd let the user define the AIO_MAX value explicitly in /etc/default/multipath- > tools. And use the sysctl/procfs option as a fallback. > > What do you say ? > >
That was where I went initially, but I thought that means that the
user now needs to configure aio-max-nr in two places, once in
sysctl.conf and once in /etc/default/multipath-tools (for when running
multipath tools by hand or in scripts). I thought this solution
resulted in less overall system configuration. The
/etc/default/multipath-tools is more explicit, so it does have
advantages too.
user now needs to configure aio-max-nr in two places, once in
sysctl.conf and once in /etc/default/multipath-tools (for when running
multipath tools by hand or in scripts). I thought this solution
resulted in less overall system configuration. The
/etc/default/multipath-tools is more explicit, so it does have
advantages too.
0 new messages