Proper ALL_TG_PT handling?
Dave
registrations using ALL_TG_PT. From what I can see in the source, I
don't think it does.
Specifically, SPC4 states that "with the ALL_TG_PT bit set to one, it
shall create the specified registration on all target ports in the
SCSI target device known to the device server (i.e., as if the same
registration request had been received individually through each
target port)."
That language indicates to me that multiple registrations should be
created (one for each target port), rather than a single registration
that has a flag saying "this applies to all target ports", which
appears to be how lio-target implements it.
Consider this scenario:
1. Initiator A connects to Device X through Port A and registers using
ALL_TG_PT (I-T Nexus 1)
2. Initiator A connects to Device X through Port B (I-T Nexus 2)
According to SPC4, each I-T Nexus should be considered registered at
this point. If we wanted to issue the RESERVE command from I-T Nexus
2, we should be able to.
However, based on my reading of the code, I don't think that will
work.
Specifically:
core_scsi3_pro_reserve calls core_scsi3_locate_pr_reg to locate the
registration for the node_acl. However the pr_reg_nacl for the
registration is held by I-T Nexus 1 connected on Port A, so I-T Nexus
2 on Port B isn't considered registered.
Am I reading that correctly?
Nicholas A. Bellinger
> I'm trying to determine if lio-target properly handles PR
> registrations using ALL_TG_PT. From what I can see in the source, I
> don't think it does.
Sure it does, the code is just not that obvious..
>
> Specifically, SPC4 states that "with the ALL_TG_PT bit set to one, it
> shall create the specified registration on all target ports in the
> SCSI target device known to the device server (i.e., as if the same
> registration request had been received individually through each
> target port)."
>
Yes, so this means that for a single I_T Nexus, or a list of I_T
Nexus'es provided in TransportID form using SPEC_I_PT=1, that the
registration can be carried over for the *same* I_T Nexus for the *same*
backstore across different fabric module target Port LUNs.
> That language indicates to me that multiple registrations should be
> created (one for each target port), rather than a single registration
> that has a flag saying "this applies to all target ports", which
> appears to be how lio-target implements it.
>
Wrong. There is no reason why ALL_TG_PT=1 needs to contain multiple
registrations for each target port for each I_T Nexus. I think you are
misunderstanding the purpose of ALL_TG_PT=1.
The best method for TCM v3.x turned out to be to simply match the
explict or dynamic NodeACL for the SCSI Initiator Port, and do the
explict check to match sure the LUNs match with an explict check of
ALL_TG_PT=0 in core_scsi3_pro_reserve():
/*
* For a given ALL_TG_PT=0 PR registration, a recevied PR reserve must
* be on the same matching se_portal_group_t + se_lun_t.
*/
if (!(pr_reg->pr_reg_all_tg_pt) &&
(pr_reg->pr_reg_tg_pt_lun != se_lun)) {
printk(KERN_ERR "SPC-3 PR: Unable to handle RESERVE because"
" ALL_TG_PT=0 and RESERVE was not received on same"
" target port as REGISTER\n");
core_scsi3_put_pr_reg(pr_reg);
return PYX_TRANSPORT_RESERVATION_CONFLICT;
}
> Consider this scenario:
> 1. Initiator A connects to Device X through Port A and registers using
> ALL_TG_PT (I-T Nexus 1)
> 2. Initiator A connects to Device X through Port B (I-T Nexus 2)
> According to SPC4, each I-T Nexus should be considered registered at
> this point. If we wanted to issue the RESERVE command from I-T Nexus
> 2, we should be able to.
> However, based on my reading of the code, I don't think that will
> work.
The existing code most certainly does work for all defined SPC-4
ALL_TG_PT=1 cases. Here is how that simple test looks for a single
TCM/IBLOCK storage object across two LIO-Target Port/LUNs:
root@ubuntu:~# lsscsi --transport
<SNIP>
[3:0:0:0] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1 /dev/sde
[4:0:0:0] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1 /dev/sdf
root@ubuntu:~# multipath -ll
<SNIP>
mpath4 (36001405a97e4ce21c0711de829b000c2) dm-2 LIO-ORG,IBLOCK
[size=1.5G][features=0][hwhandler=1 alua]
\_ round-robin 0 [prio=50][enabled]
\_ 3:0:0:0 sde 8:64 [active][ready]
\_ round-robin 0 [prio=10][active]
\_ 4:0:0:0 sdf 8:80 [active][ready]
*) Do a PR REGISTER with ALL_TG_PT=1 on the first iSCSI Target Port
at /dev/sde
root@ubuntu:~# sg_persist -vvv --out --device /dev/sde --register -Y --param-rk=0 --param-sark=0x1234ffff
open /dev/sde with flags=0x800
inquiry cdb: 12 00 00 00 24 00
duration=0 ms
LIO-ORG IBLOCK 3.1
Peripheral device type: disk
open /dev/sde with flags=0x802
Persistent Reservation Out cmd: 5f 00 00 00 00 00 00 00 18 00
Persistent Reservation Out parameters:
00 00 00 00 00 00 00 00 00 00 00 00 00 12 34 ff ff .............4..
10 00 00 00 00 04 00 00 00 ........
duration=0 ms
PR out: command (Register) successful
*) Do a PR RESERVE from the second iSCSi Target port at /dev/sdf
root@ubuntu:~# sg_persist -vvv --out --device /dev/sdf --reserve -d /dev/sde --param-rk=0x1234ffff -T 5
open /dev/sde with flags=0x800
inquiry cdb: 12 00 00 00 24 00
duration=0 ms
LIO-ORG IBLOCK 3.1
Peripheral device type: disk
open /dev/sde with flags=0x802
Persistent Reservation Out cmd: 5f 01 05 00 00 00 00 00 18 00
Persistent Reservation Out parameters:
00 00 00 00 00 12 34 ff ff 00 00 00 00 00 00 00 00 .....4..........
10 00 00 00 00 01 00 00 00 ........
duration=4 ms
PR out: command (Reserve) successful
*) Do a READ_FULL_STATUS
root@ubuntu:~# sg_persist -v -s -d /dev/sde
inquiry cdb: 12 00 00 00 24 00
LIO-ORG IBLOCK 3.1
Peripheral device type: disk
Persistent Reservation In cmd: 5e 03 00 00 00 00 00 20 00 00
PR generation=0x5
Key=0x1234ffff
All target ports bit set
<< Reservation holder >>
scope: LU_SCOPE, type: Write Exclusive, registrants only
Transport Id of initiator:
iSCSI world wide unique port id: iqn.1993-08.org.debian:01:2dadf92d0ef
*) TCM kernel ring buffer output
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for ALL TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000001234ffff PRgeneration: 0x00000004 APTPL: 0
SPC-3 PR: Set APTPL Bit Deactivated for REGISTER
SPC-3 PR [iSCSI] Service Action: RESERVE created new reservation holder TYPE: Write Exclusive Access, Registrants Only ALL_TG_PT: 1
SPC-3 PR [iSCSI] RESERVE Node: iqn.1993-08.org.debian:01:2dadf92d0ef
> Specifically:
> core_scsi3_pro_reserve calls core_scsi3_locate_pr_reg to locate the
> registration for the node_acl. However the pr_reg_nacl for the
> registration is held by I-T Nexus 1 connected on Port A, so I-T Nexus
> 2 on Port B isn't considered registered.
>
I believe that is where you are misunderstanding the logic defined by
ALL_TG_PT=1. The PR registration only applies to the *single* Initiator
Port, the Initiator Port that the REGISTER was received on with
SPEC_I_PT=0) across all Target Ports, and not ALL initiator ports across
ALL target ports.
This also means that if you want to do multiple Initiator Port
registrations potentially across multiple fabrics, SPC-4 defines that
fabric specific TransportIDs to be passed with REGISTER via SPEC_I_PT=1
mode for doing specific SCSI Initiator Port PR registrations that also
may contain the ALL_TG_PT=1 bit. That logic can be found in TCM PR code
at core_scsi3_decode_spec_i_port().
So the main point here is that ALL_TG_PT=1 is intended for allowing
"this single specific SCSI Initiator Port's PR registration should carry
across all the target ports for this fabric module", and not to "create
multiple SCSI Initiator Port registrations for all target ports".
Explictly enabling SCSI Initiator Port registrations via REGISTER is
what TransportID processing with SPEC_I_PT=1 logic was designed to
achieve.
Best,
--nab
Nicholas A. Bellinger
Ugh, I just noticed the extra '-d /dev/sde' in the above example for
REGISTER, which unfortuately means my test was bogus.. :-(
So I stand corrected wrt to your observations. I should point out that
my original testing with ALL_TG_PT=1 was using a single backstore across
multiple target ports in the same I_T Nexus (which does work as
expected), but as you pointed things now appear to be broken for
ALL_TG_PT=1 across different I_T Nexuses.
This is because of core_scsi3_locate_pr_reg() checking the se_node_acl_t
pointer (which is going to be I_T Nexus dependent), and not on the
actual SCSI Initiator Port Name that we would expect to be the same
InitiatorName for the iSCSI fabric case.
My apologies for the confusion. I will go ahead and produce a patch +
proper test case to resolve ALL_TG_PT=1 operation across multiple I_T
Nexuses with the same matching SCSI Initiator Port.
Thanks for pointing this out!
--nab
Dave
to what I was expecting.
Can you post the target configuration used in the above example?
The reason I ask is that it looks like you've configured two targets,
each with one TPGT, that have a LUN pointing to the same backing store
(device).
The configuration I was expecting was one target with two TPGT, each
with a LUN pointing to the same backing store. Will the patch work for
that case as well?
Assuming it does, I think it's fine, although I am curious about lio-
target allowing you to expose the same device via multiple targets
(not just multiple TPGT).
The reason I say that is because I didn't think SAM3 defines the
behavior for an LU (which is really a backing store/device in TCM)
exposed through multiple SCSI Target Devices. It simply allows a
single SCSI Target Device to have multiple SCSI Target Ports, which
can expose the same LUs (but aren't required to).
Mapping SAM3 to ISCSI, Target Device = ISCSI Target, Target Port =
TPGT, would mean that you can expose the same device through multiple
TPGTs under a Target, but not necessarily multiple targets.
I'm not sure it really matters as long as you allow multiple TPGT
access to the same device under a single target as well, but was just
curious whether you considered the current behavior consistent with
SAM or simply added flexibility.
-Dave
On Mar 23, 12:50 am, "Nicholas A. Bellinger" <n...@linux-iscsi.org>
Nicholas A. Bellinger
> Thanks Nicholas. I took a quick look at the patch and it's much closer
> to what I was expecting.
>
> Can you post the target configuration used in the above example?
>
Hi Dave,
Here is the TCM config via tcm_dump --s
modprobe target_core_mod
#### ALUA Logical Unit Groups
#### Parameters for TCM subsystem plugin storage object reference
tcm_node --establishdev iblock_0/lvm_test0 /dev/lio-test/test0
tcm_node --setunitserialwithmd iblock_0/lvm_test0 a97e4ce21c0711de829b000c2943d57b
#### ALUA Target Port Groups
tcm_node --addaluatpgwithmd iblock_0/lvm_test0 lio_alua_east 2
echo 3 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_east/alua_access_type
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_east/preferred
echo 100 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_east/nonop_delay_msecs
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_east/trans_delay_msecs
tcm_node --addaluatpgwithmd iblock_0/lvm_test0 lio_alua_west 1
echo 3 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_west/alua_access_type
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_west/preferred
echo 100 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_west/nonop_delay_msecs
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/lio_alua_west/trans_delay_msecs
tcm_node --addaluatpgwithmd iblock_0/lvm_test0 default_tg_pt_gp 0
echo 3 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/default_tg_pt_gp/alua_access_type
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/default_tg_pt_gp/preferred
echo 100 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/default_tg_pt_gp/nonop_delay_msecs
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/alua/default_tg_pt_gp/trans_delay_msecs
#### Attributes for /sys/kernel/config/target/core/iblock_0/lvm_test0
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/attrib/task_timeout
echo 32 > /sys/kernel/config/target/core/iblock_0/lvm_test0/attrib/queue_depth
echo 32 > /sys/kernel/config/target/core/iblock_0/lvm_test0/attrib/max_sectors
echo 512 > /sys/kernel/config/target/core/iblock_0/lvm_test0/attrib/block_size
echo 1 > /sys/kernel/config/target/core/iblock_0/lvm_test0/attrib/emulate_tas
echo 0 > /sys/kernel/config/target/core/iblock_0/lvm_test0/attrib/emulate_ua_intlck_ctrl
and the abridged LIO Target config from lio_dump --s:
mkdir /sys/kernel/config/target/iscsi
#### iSCSI Discovery authentication information
<SNIP>
#### Network portals for iSCSI Target Portal Group
mkdir -p /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/np/172.16.201.128:3260
#### iSCSI Target Ports
mkdir -p /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/lun/lun_0
ln -s /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/lun/lun_0/../../../../../../target/core/iblock_0/lvm_test0 /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/lun/lun_0/lio_west_port
#### ALUA Target Port Group
echo lio_alua_east > /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/
lun/lun_0/alua_tg_pt_gp
lio_node --aluasecmd iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd 1 0
#### iSCSI Initiator ACLs for iSCSI Target Portal Group
mkdir -p /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef
echo 16 > /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/cmdsn_depth
#### iSCSI Initiator ACL authentication information
#### iSCSI Initiator ACL TPG attributes
<SNIP>
#### iSCSI Initiator LUN ACLs for iSCSI Target Portal Group
mkdir -p /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0
ln -s /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0/../../../../../../../target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/lun/lun_0 /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0/lun_0
echo 0 > /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0/write_protect
#### Network portals for iSCSI Target Portal Group
mkdir -p /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/np/172.16.201.128:3260
#### iSCSI Initiator ACLs for iSCSI Target Portal Group
mkdir -p /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef
echo 16 > /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/cmdsn_depth
#### iSCSI Initiator ACL authentication information
#### iSCSI Initiator ACL TPG attributes
<SNIP>
#### iSCSI Initiator LUN ACLs for iSCSI Target Portal Group
mkdir -p /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0
ln -s /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0/../../../../../../../target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/lun/lun_0 /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0/lun_0
echo 0 > /sys/kernel/config/target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0/tpgt_1/acls/iqn.1993-08.org.debian:01:2dadf92d0ef/lun_0/write_protect
> The reason I ask is that it looks like you've configured two targets,
> each with one TPGT, that have a LUN pointing to the same backing store
> (device).
This is correct, I am using multiple iSCSI TargetNames with the same
TargetPortalGroupTag=1 for the example in the ALL_TG_PT=1 patch commit.
>
> The configuration I was expecting was one target with two TPGT, each
> with a LUN pointing to the same backing store. Will the patch work for
> that case as well?
>
Yes, the patch should be working for all cases including the single
Target WWN (iSCSI TargetName) and multiple Target Ports (iSCSI
TargetPortalGroupTags) with:
/sys/kernel/config/target/iscsi/$TARGETNAME/tpgt_[1-65535]
> Assuming it does, I think it's fine, although I am curious about lio-
> target allowing you to expose the same device via multiple targets
> (not just multiple TPGT).
>
> The reason I say that is because I didn't think SAM3 defines the
> behavior for an LU (which is really a backing store/device in TCM)
> exposed through multiple SCSI Target Devices. It simply allows a
> single SCSI Target Device to have multiple SCSI Target Ports, which
> can expose the same LUs (but aren't required to).
>
> Mapping SAM3 to ISCSI, Target Device = ISCSI Target, Target Port =
> TPGT, would mean that you can expose the same device through multiple
> TPGTs under a Target, but not necessarily multiple targets.
>
> I'm not sure it really matters as long as you allow multiple TPGT
> access to the same device under a single target as well, but was just
> curious whether you considered the current behavior consistent with
> SAM or simply added flexibility.
The ability to run a backstore from multiple iSCSI TargetNames is
flexibility originally implemented in the standalone LIO-Target v2.x
code, which did not explictly restrict individual backstores to a single
TargetName / multiple TargetPortalGroupTag setup.
I had extended this particular ability into the LIO-Target iSCSI fabric
module for TCM/ConfigFS v3.x, and is also allowed for FCoE Target ports
in the tcm_fc fabric module. This ability is also present in the
current WIP code for a generic TCM v4.0 configfs fabric layout in:
I have not seen any specific language that a single backstore across
multiple TargetNames is illegal in iSCSI, but I think you make a good
point here about LU behaviour being undefined in SAM-3. The current
logic in the LIO v3.x fabric module does not treat the multiple
Targetname case any differently, so I would be inclined to agree with
your statement above that as long as the backstore can still be shared
on a single TargetName across multiple TargetPortalGroupTags, that LU
behaviour is going to identically work as expected for both cases.
Best,
--nab
Dave
I'd like to dig into two other possible issues with the ALL_TG_PT
handling.
First is how unregistration of a reservation key is handled with
ALL_TG_PT.
Let's say I register a reservation key using ALL_TG_PT on a target
with two ports. Two registrations are now created. I only ever connect
on one of the ports.
Now, I unregister using PR_OUT REGISTER, SARK=0.
Are one or both of the registrations removed?
Does the ALL_TG_PT flag on the unregistration request have an effect
on that?
Unfortunately I find SPC4 is not clear about how to handle
unregistration of ALL_TG_PT reservation keys.
There are three possibilities I see:
1. ALL_TG_PT is ignored when unregistering, and the registration is
only removed on the I_T nexus that made the request.
2. ALL_TG_PT is ignored when unregistering, and the registration is
removed from all TGPTs
3. ALL_TG_PT is used when unregistering, and the behavior is #1 or #2
depending on how it is set.
From what I can see around line 2072 in target_core_pr.c, TCM does
#1.
That would seem to orphan the registration that was made on the other
port. The only way to remove it would be to connect via that port and
unregister, or use a PREEMPT or CLEAR to remove the registration.
#3 would make the most sense to me, as it would appear to be the most
flexible, but again, I see nothing in SPC4 that specifies the
behavior. Other targets I've looked at (e.g. Comstar) appear to do #2,
but I see other issues in their implementation that make me think this
is a side effect more than their plan.
One problem with #2 or #3 is that it could lead to unregistering an
I_T nexus other than the one making the request, something that does
not appear to be addressed in SPC4 outside a PREEMPT or CLEAR request
(for example, should an attention condition be raised on the other I_T
nexuses whose registrations are being removed?)
My second question has to do with how ALL_TG_PT matches the initiator
port on the other target ports. SPC4 table 48 says that for ALL_TG_PT,
"The port's name or identifier" should be used when creating the
registrations on all of the target ports. However, it appears you only
match nacl->initiatorname.
The ISCSI RFC (as well as parts of SPC4) state that the port name is
actually "iSCSI Initiator Name + ',i,'+ ISID" - the ISID needs to be
the same for the port to be considered the same.
That means that ALL_TG_PT should NOT match initiator ports whose ISID
is different than the I_T nexus making the request, even if the
intiatorname matches.
With that restriction in place, I'm not even sure how the multipathing
example you used is supposed to work. I believe multipath uses a
different ISID for each session, but I'm not sure (it doesn't appear
to show the ISID in any of the output you included).
If multipath were connecting to the same target via the same tgpt (but
perhaps via a different network portal) it would certainly need to use
different ISIDs, otherwise there would be two identical I_T nexuses,
so my guess is that it uses a different ISID all the time... in which
case the registration should not apply to the second connection in
your multipath example.
fwiw, TransportIDs in SPEC_I_PT registrations appear to be handled
somewhat differently (at least for ISCSI). There are two address
formats (7.5.4.6), one of which includes only the initiator name (and
is optional), and the other of which includes the ISID and is required
to support.
However from my reading, ALL_TG_PT registrations should ALWAYS match
the full initiator port (initiator name + ISID) in order to apply.
-Dave
Nicholas A. Bellinger
> Thanks Nicholas,
> I'd like to dig into two other possible issues with the ALL_TG_PT
> handling.
>
Hi Dave,
Apologies for the slight delay, my comments are below..
> First is how unregistration of a reservation key is handled with
> ALL_TG_PT.
> Let's say I register a reservation key using ALL_TG_PT on a target
> with two ports. Two registrations are now created. I only ever connect
> on one of the ports.
> Now, I unregister using PR_OUT REGISTER, SARK=0.
> Are one or both of the registrations removed?
> Does the ALL_TG_PT flag on the unregistration request have an effect
> on that?
>
> Unfortunately I find SPC4 is not clear about how to handle
> unregistration of ALL_TG_PT reservation keys.
> There are three possibilities I see:
> 1. ALL_TG_PT is ignored when unregistering, and the registration is
> only removed on the I_T nexus that made the request.
> 2. ALL_TG_PT is ignored when unregistering, and the registration is
> removed from all TGPTs
> 3. ALL_TG_PT is used when unregistering, and the behavior is #1 or #2
> depending on how it is set.
> >From what I can see around line 2072 in target_core_pr.c, TCM does
> #1.
This is indeed what TCM/PR logic is currently doing.
> That would seem to orphan the registration that was made on the other
> port. The only way to remove it would be to connect via that port and
> unregister, or use a PREEMPT or CLEAR to remove the registration.
> #3 would make the most sense to me, as it would appear to be the most
> flexible, but again, I see nothing in SPC4 that specifies the
> behavior. Other targets I've looked at (e.g. Comstar) appear to do #2,
> but I see other issues in their implementation that make me think this
> is a side effect more than their plan.
>
Hmmm, I do see your point here and agree that SPC-4 does not mention any
specifics about what happens for 'orphaned registrations' created by
ALL_TG_PT=1 when the original registration goes away.
> One problem with #2 or #3 is that it could lead to unregistering an
> I_T nexus other than the one making the request, something that does
> not appear to be addressed in SPC4 outside a PREEMPT or CLEAR request
> (for example, should an attention condition be raised on the other I_T
> nexuses whose registrations are being removed?)
>
Yes, establishing a UA for this type of event affecting other I_T
Nexuses would be expected. These are the ASC 0x2A codes defined in
include/target/target_core_ua.h from SPC-4 that are PR releated
conditions established based on events happening on other I_T Nexues:
#define ASCQ_2AH_RESERVATIONS_PREEMPTED 0x03
#define ASCQ_2AH_RESERVATIONS_RELEASED 0x04
#define ASCQ_2AH_REGISTRATIONS_PREEMPTED 0x05
Unfortuately there is no 'REGISTRATIONS RELEASED' equivilent defined for
the type of UA that would be 'nice to have' for implementing #2 or #3
mentioned above.
Implementing #2 and #3 is straight forward enough on my end, but I would
be great to get a bit more input from T10 folks, or see what other
target implementations (or more importantly how PR consumers react when
this happens) before I code this up.
Is this going to be something specific that your PR consumer setup
requires..?
>
> My second question has to do with how ALL_TG_PT matches the initiator
> port on the other target ports. SPC4 table 48 says that for ALL_TG_PT,
> "The port's name or identifier" should be used when creating the
> registrations on all of the target ports. However, it appears you only
> match nacl->initiatorname.
> The ISCSI RFC (as well as parts of SPC4) state that the port name is
> actually "iSCSI Initiator Name + ',i,'+ ISID" - the ISID needs to be
> the same for the port to be considered the same.
> That means that ALL_TG_PT should NOT match initiator ports whose ISID
> is different than the I_T nexus making the request, even if the
> intiatorname matches.
My interpretation of ALL_TG_PT=1 with the particular language of 'The
port's name or identifier' means that any fabric defined local
TransportID representing the SCSI Initiator Port of the I_T Nexus that
the PROUT REGISTER w/ ALL_TG_PT=1 was received may be used when matching
other SCSI Initiator Ports.
As you mention below, iSCSI is the only fabric to have two TransportIDs
for defining a iSCSI Initiator Port. From SPC-4 7.5.4.6 TransportID for
initiator ports using SCSI over iSCSI:
00b Initiator port is identified using the world wide unique SCSI device name of the iSCSI initiator
device containing the initiator port (see table 389). (eg: the iSCSI InitiatorName)
01b Initiator port is identified using the world wide unique initiator port identifier (see table 390).
(eg: the iSCSI InitiatorName+i+0x$ISID
>
> With that restriction in place, I'm not even sure how the multipathing
> example you used is supposed to work. I believe multipath uses a
> different ISID for each session, but I'm not sure (it doesn't appear
> to show the ISID in any of the output you included).
> If multipath were connecting to the same target via the same tgpt (but
> perhaps via a different network portal) it would certainly need to use
> different ISIDs, otherwise there would be two identical I_T nexuses,
> so my guess is that it uses a different ISID all the time... in which
> case the registration should not apply to the second connection in
> your multipath example.
There are a couple of reasons why I have found using the above 01b
transportID (containing an InitiatorName+ISID) to represent an iSCSI
Initiator port can be problematic in the context of SCSI persistent
reservations.
The first one being that most iSCSI initiators in deployment keep the
iSCSI InitiatorName persistent (as required by RFC-3720), but do not
keep ISIDs persistent across power loss, reboots, etc.
The other is that (to my knowledge) the majority of software initiators
are using a ISID that (partialy) contains a 16-bit or 24-bit counter
that is incremented as new sessions and/or targets appear. Having a
ISID that is not persistent across Initiator power loss or even across
iSCSI Initiator defined SCSI Hosts for a TargetName somewhat defeats the
purpose of defining a iSCSI Initiator Port (and not the individual I_T
Nexus) that can be referenced for SCSI persistent reservations purposes.
As all other SCSI fabrics aside from iSCSI define a single TransportID
to reference their SCSI Initiator Ports, and none of them AFAICT depend
upon a I_T Nexus specific value such as an ISID. So using an iSCSI
specific I_T Nexus information in the context of ALL_TG_PT=1 did not
make a whole lot of sense to me at the time..
>
> fwiw, TransportIDs in SPEC_I_PT registrations appear to be handled
> somewhat differently (at least for ISCSI). There are two address
> formats (7.5.4.6), one of which includes only the initiator name (and
> is optional), and the other of which includes the ISID and is required
> to support.
You will notice that aside from the ALL_TG_PT=1 SCSI Initiator Port
matching case, that target_core_pr.c does support the iSCSI 01b
Transport ID containing InitiatorName+ISID to reference an active I_T
Nexus (iSCSI session) for SPEC_I_PT=1 and REGISTER_AND_MOVE cases. This
is done by the fabric module setting the return **port_nexus_ptr in
struct target_core_fabric_ops->tpg_parse_pr_out_transport_id(), and the
iSCSI specific TransportID processing can be found here:
drivers/target/lio-target/iscsi_target_tpg.c:lio_tpg_parse_pr_out_transport_id()
> However from my reading, ALL_TG_PT registrations should ALWAYS match
> the full initiator port (initiator name + ISID) in order to apply.
>
So to summarize, I do not believe there is a strict requirement by SPC-4
or RFC-3720 to match the iSCSI 01b TransportID containing an ISID value
for ALL_TG_PT=1, or any iSCSI PR operation. My intrepreation is that
any valid SPC-4 defined fabric TransportID may be used to match other
SCSI Initiator Ports for the additional ALL_TG_PT=1 registrations, using
local SCSI Initiator Port TransportID equivilent as the reference that
the REGISTER w/ ALL_TG_PT=1 was received from.
I would be happy to be proved wrong on this, but I am pretty sure this
is left up to the implementation. This would actually be a really good
question for Julio on the IETF IPS list..!
Best,
--nab
Nicholas A. Bellinger
I did find one additional reference in SPC-4 section '7.5.4.6
TransportID for initiator ports using SCSI over iSCSI' to backup my
interpretation:
"If a iSCSI TransportID with the format code set to 00b appears in a
PERSISTENT RESERVE OUT parameter list (see 6.14.3), all initiator ports
known to the device server with an iSCSI node name matching the one in the
TransportID shall be registered."
This does not directly mention ALL_TG_PT=1 operation, but I think the
wording does imply that using iSCSI InitiatorName w/o the ISID (00b
TransportID) is allowed to reference SCSI Initiator Ports within SPC-4
defined persistent reservations logic.
Best,
--nab
Dave
> Implementing #2 and #3 is straight forward enough on my end, but I would
> be great to get a bit more input from T10 folks, or see what other
> target implementations (or more importantly how PR consumers react when
> this happens) before I code this up.
I went back and looked at Comstar, which I've found to be one of the
better PR implementations, and they do a variation on #2 -
specifically, they REQUIRE ALL_TG_PT be set on the unregister request
if it was set on the original registration request, and they remove
all the registrations (if it's not set, they return an error - no way
to orphan a registration).
>
> Is this going to be something specific that your PR consumer setup
> requires..?
Well, as it stands now I probably won't even attempt to use ALL_TG_PT,
given the wide differences in implementations I've seen. I was just
hoping to get some clarity, if for no other reason than to nudge the
various target providers into a consistent behavior.
> The first one being that most iSCSI initiators in deployment keep the
> iSCSI InitiatorName persistent (as required by RFC-3720), but do not
> keep ISIDs persistent across power loss, reboots, etc.
I hear what you're saying about the ISID's not being persistent, which
obviously defeats the purpose of PR. I did a quick check of other PR
target implementations and it looks like they match the reservation
based on the initator name alone, not the ISID.
That does raise an interesting question about how multiple sessions
from the same initiator name (different ISIDs) to the same target+tpgt
are handled with PR (this is without any involvement of ALL_TG_PT).
For example, in your multi-pathing example, what if the two initiator
sessions were connected to the same target name / port (probably
through different portals)? That would seem to be a valid
configuration - the I_T nexues are unique using the initator port with
ISID, however from the perspective of PR, it would seem that both
sessions would match the same registration.
From my view of the TCM code, it looks like both sessions would be
considered a match to the registration, AND both would be considered
the "holder" of any reservation on that registration (since their
nacls would be the same).
That behavior seems to be in direct contradiction to SPC4 which states
that only one I_T nexus can be the "holder" of a reservation.
This again is a place where implementations appear to diverge.
istgt uses the full initiator port format (name,i,0xisid) to match
registrations.
Comstar uses just the initiator name, however it ensures that only one
session can match a given registration (using a "first in wins"
approach).
tscsitgtd (solaris user mode target) appears to be the same as TCM, it
just matches the initiator name and duplicate sessions from the same
iname are considered holders of the registration key / reservation.
> I would be happy to be proved wrong on this, but I am pretty sure this
> is left up to the implementation. This would actually be a really good
> question for Julio on the IETF IPS list..!
I'll look into posting there. I think the last issue (matching based
on just iname, or iname+isid) concerns me the most, because it doesn't
even deal with the somewhat obscure ALL_TG_PT flag.
-Dave
Nicholas A. Bellinger
> > Implementing #2 and #3 is straight forward enough on my end, but I would
> > be great to get a bit more input from T10 folks, or see what other
> > target implementations (or more importantly how PR consumers react when
> > this happens) before I code this up.
>
> I went back and looked at Comstar, which I've found to be one of the
> better PR implementations, and they do a variation on #2 -
> specifically, they REQUIRE ALL_TG_PT be set on the unregister request
> if it was set on the original registration request, and they remove
> all the registrations (if it's not set, they return an error - no way
> to orphan a registration).
>
Ok, I think doing this is going to make the most sense for ALL_TG_PT=1
unregister logic, and will have look at coding this up..
Btw, would you mind figuring out which UA ASCQ (if any) is generated by
Comstar for this case..? I would be leaning in the direction of
ASCQ_2AH_REGISTRATIONS_PREEMPTED, but would be interested in seeing what
they actually do for this.
> >
> > Is this going to be something specific that your PR consumer setup
> > requires..?
>
> Well, as it stands now I probably won't even attempt to use ALL_TG_PT,
> given the wide differences in implementations I've seen. I was just
> hoping to get some clarity, if for no other reason than to nudge the
> various target providers into a consistent behavior.
>
> > The first one being that most iSCSI initiators in deployment keep the
> > iSCSI InitiatorName persistent (as required by RFC-3720), but do not
> > keep ISIDs persistent across power loss, reboots, etc.
>
> I hear what you're saying about the ISID's not being persistent, which
> obviously defeats the purpose of PR. I did a quick check of other PR
> target implementations and it looks like they match the reservation
> based on the initator name alone, not the ISID.
>
> That does raise an interesting question about how multiple sessions
> from the same initiator name (different ISIDs) to the same target+tpgt
> are handled with PR (this is without any involvement of ALL_TG_PT).
>
> For example, in your multi-pathing example, what if the two initiator
> sessions were connected to the same target name / port (probably
> through different portals)? That would seem to be a valid
> configuration
This is indeed a valid configuration for allowing a single InitiatorName
access to the same iSCSI TargetName+TargetPortalGroupTag endpoint.
> - the I_T nexues are unique using the initator port with
> ISID, however from the perspective of PR, it would seem that both
> sessions would match the same registration.
> >From my view of the TCM code, it looks like both sessions would be
> considered a match to the registration, AND both would be considered
> the "holder" of any reservation on that registration (since their
> nacls would be the same).
There does appear to be a problem specfic to the iSCSI fabric module
here..
> That behavior seems to be in direct contradiction to SPC4 which states
> that only one I_T nexus can be the "holder" of a reservation.
Indeed, having a se_node_acl_t referenced for multiple iSCSI sessions
for an individual iSCSI InitiatorName and multiple ISIDs is going to be
the main issue here..
> This again is a place where implementations appear to diverge.
> istgt uses the full initiator port format (name,i,0xisid) to match
> registrations.
>
I am still wondering a bit about how this the 01b TransportID mode for
iSCSI Initiator Port would be effecting existing iSCSI Initiators as PR
consumers..
Hmmmm, I may just end up just adding an iSCSI TPG attribute (that would
be disabled by default) with some PR I_T Nexus specific logic using
existing TFO API calls to optionally support this iSCSI specific case
for completeness sake with iSCSI 01b TransportID already being supported
with SPEC_I_PT=1 and REGISTER_AND_MOVE. A patch to TCM PR code to
handle the other I_T Nexus specific cases is easy enough..
> Comstar uses just the initiator name, however it ensures that only one
> session can match a given registration (using a "first in wins"
> approach).
>
Ok, I think this is the way it should be be done for SCSI Initiator Port
PR cases that contain I_T Nexus metadata.
> tscsitgtd (solaris user mode target) appears to be the same as TCM, it
> just matches the initiator name and duplicate sessions from the same
> iname are considered holders of the registration key / reservation.
>
> > I would be happy to be proved wrong on this, but I am pretty sure this
> > is left up to the implementation. This would actually be a really good
> > question for Julio on the IETF IPS list..!
>
> I'll look into posting there. I think the last issue (matching based
> on just iname, or iname+isid) concerns me the most, because it doesn't
> even deal with the somewhat obscure ALL_TG_PT flag.
>
I will have a closer look and see what should be done in order to follow
the "first in wins" approach mentioned above and how TCM se_node_acl_t
refrence is handled for the single InitiatorName + multiple ISID case in
LIO-Target.
Thanks for your input Dave!
Best,
--nab
Dave
>
> Btw, would you mind figuring out which UA ASCQ (if any) is generated by
> Comstar for this case..? I would be leaning in the direction of
> ASCQ_2AH_REGISTRATIONS_PREEMPTED, but would be interested in seeing what
> they actually do for this.
Actually it appears to use RESERVATIONS_RELEASED, but only if the key
was for a registration holder (and again, they only allow a single
reservation holder per registration.. first-in). Otherwise no UA is
sent.
If ALL_TG_PT is not set on the release request, an
INVALID_FIELD_IN_CDB sense code is returned.
For reference, see sbd_pgr.c sbd_pgr_out_register()
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/comstar/lu/stmf_sbd/sbd_pgr.c
> > This again is a place where implementations appear to diverge.
> > istgt uses the full initiator port format (name,i,0xisid) to match
> > registrations.
>
> I am still wondering a bit about how this the 01b TransportID mode for
> iSCSI Initiator Port would be effecting existing iSCSI Initiators as PR
> consumers..
Yes.. I suspect this may be a case where the "right" implementation
isn't what works in practice, due to the ISID issues on the initiator
you mentioned.
>
> I will have a closer look and see what should be done in order to follow
> the "first in wins" approach mentioned above and how TCM se_node_acl_t
> refrence is handled for the single InitiatorName + multiple ISID case in
> LIO-Target.
It looks like handling of PR in this case is really a pain. The main
reason for multiple ISID sessions would be the multipath example, but
a simplistic multipath implementation wouldn't handle PR at all - the
reservation would get created on one of the I_T nexus, but commands
sent across the other link would get a reservation conflict.
Based on this thread on the IPS list, it sounds like different
multipath implementations handle PR differently:
http://www.ietf.org/mail-archive/web/ips/current/msg02170.html
It would actually be nice if the current implementation was allowed
(registration/reservation held by all I_T nexuses from the same
initiator).. that would make multipath implementation simple, however
that violates SPC4 and it doesn't appear than any of the multipath
implementations expect that behavior.
For my $0.02 I think the PR feature was poorly designed :) I would
have allowed sharing of a reservation based on shared reservation
keys, rather than tying a reservation to a single I_T nexus and having
special "all registrants" reservation types to allow sharing of
reservations. It almost seems like that's the way someone intended it
originally, but it got twisted around at some point before the spec
was issued.
-Dave
Nicholas A. Bellinger
> >
> > Btw, would you mind figuring out which UA ASCQ (if any) is generated by
> > Comstar for this case..? I would be leaning in the direction of
> > ASCQ_2AH_REGISTRATIONS_PREEMPTED, but would be interested in seeing what
> > they actually do for this.
>
> Actually it appears to use RESERVATIONS_RELEASED, but only if the key
> was for a registration holder (and again, they only allow a single
> reservation holder per registration.. first-in). Otherwise no UA is
> sent.
> If ALL_TG_PT is not set on the release request, an
> INVALID_FIELD_IN_CDB sense code is returned.
> For reference, see sbd_pgr.c sbd_pgr_out_register()
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/comstar/lu/stmf_sbd/sbd_pgr.c
Ok, I am clear on the ALL_TG_PT=1 UNREGISTER case, thanks for the code
reference.
>
> > > This again is a place where implementations appear to diverge.
> > > istgt uses the full initiator port format (name,i,0xisid) to match
> > > registrations.
> >
> > I am still wondering a bit about how this the 01b TransportID mode for
> > iSCSI Initiator Port would be effecting existing iSCSI Initiators as PR
> > consumers..
>
> Yes.. I suspect this may be a case where the "right" implementation
> isn't what works in practice, due to the ISID issues on the initiator
> you mentioned.
I would agree that this is the case here.
>
> >
> > I will have a closer look and see what should be done in order to follow
> > the "first in wins" approach mentioned above and how TCM se_node_acl_t
> > refrence is handled for the single InitiatorName + multiple ISID case in
> > LIO-Target.
>
> It looks like handling of PR in this case is really a pain. The main
> reason for multiple ISID sessions would be the multipath example, but
> a simplistic multipath implementation wouldn't handle PR at all - the
> reservation would get created on one of the I_T nexus, but commands
> sent across the other link would get a reservation conflict.
> Based on this thread on the IPS list, it sounds like different
> multipath implementations handle PR differently:
> http://www.ietf.org/mail-archive/web/ips/current/msg02170.html
Yes, I believe that the single InitiatorName + multiple ISID sessions to
a single TargetName+TargetPortalGroupTag endpoint is primarily intended
for the legacy symmetric multipath case.
However using modern NAA WWN naming and ALUA multipath across multiple
TargetName+TargetPortalGroupTag endpoints makes the ISID slightly less
important today for multipath than the pre iSCSI RFC days..
>
> It would actually be nice if the current implementation was allowed
> (registration/reservation held by all I_T nexuses from the same
> initiator).. that would make multipath implementation simple, however
> that violates SPC4 and it doesn't appear than any of the multipath
> implementations expect that behavior.
Yeah, I don't think allowing more than one I_T Nexus to be the
reservation holder (aside from all valid registrations in the 'all
registrants' case you mention below) is ever going to make sense.
>
> For my $0.02 I think the PR feature was poorly designed :) I would
> have allowed sharing of a reservation based on shared reservation
> keys, rather than tying a reservation to a single I_T nexus and having
> special "all registrants" reservation types to allow sharing of
> reservations. It almost seems like that's the way someone intended it
> originally, but it got twisted around at some point before the spec
> was issued.
The 'all registrants' reservation type caused me a bit of confusion
originally, but I came to believe that it's primary use was intended for
shared disk cluster filesystems or other active/active designs. These
designs are going to expect multiple I_T Nexus'es from different SCSI
Initiator ports (eg: different nodes or VMs) to multiple target
Port/LUNs across potentially different SCSI fabrics for a single storage
backstore.
Also wrt to the PR feature itself, having to design something that works
across existing SCSI fabrics for all possible usage cases at the time
was not an easy job for T10. Especially for the existing SPI cases
which do not contain any WWN values within their SCSI Initiator Port
TransportID or SCSI Target Port Identifiers, which means that some of
the PR described in this thread not possible with legacy SPI or even SPI
CDB level emulation.
--nab
Nicholas A. Bellinger
> On Wed, 2010-03-24 at 11:38 -0700, Dave wrote:
> > >
> > > Btw, would you mind figuring out which UA ASCQ (if any) is generated by
> > > Comstar for this case..? I would be leaning in the direction of
> > > ASCQ_2AH_REGISTRATIONS_PREEMPTED, but would be interested in seeing what
> > > they actually do for this.
> >
> > Actually it appears to use RESERVATIONS_RELEASED, but only if the key
> > was for a registration holder (and again, they only allow a single
> > reservation holder per registration.. first-in). Otherwise no UA is
> > sent.
> > If ALL_TG_PT is not set on the release request, an
> > INVALID_FIELD_IN_CDB sense code is returned.
> > For reference, see sbd_pgr.c sbd_pgr_out_register()
> > http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/comstar/lu/stmf_sbd/sbd_pgr.c
>
> Ok, I am clear on the ALL_TG_PT=1 UNREGISTER case, thanks for the code
> reference.
Ok, please have a look at the ALL_TG_PT=1 UNREGISTER patch at:
I am still looking at how this particular case for iSCSI should best be
handled.
Best,
--nab
Dave
two questions to the IPS IETF list, although the message hasn't shown
up yet - not sure if it's stuck in moderation somewhere or if they've
turned down that list.
-Dave
On Mar 25, 2:34 am, "Nicholas A. Bellinger" <n...@linux-iscsi.org>
wrote:
> On Wed, 2010-03-24 at 12:54 -0700, Nicholas A. Bellinger wrote:
> > On Wed, 2010-03-24 at 11:38 -0700, Dave wrote:
>
> > > > Btw, would you mind figuring out which UA ASCQ (if any) is generated by
> > > > Comstar for this case..? I would be leaning in the direction of
> > > > ASCQ_2AH_REGISTRATIONS_PREEMPTED, but would be interested in seeing what
> > > > they actually do for this.
>
> > > Actually it appears to use RESERVATIONS_RELEASED, but only if the key
> > > was for a registration holder (and again, they only allow a single
> > > reservation holder per registration.. first-in). Otherwise no UA is
> > > sent.
> > > If ALL_TG_PT is not set on the release request, an
> > > INVALID_FIELD_IN_CDB sense code is returned.
> > > For reference, see sbd_pgr.c sbd_pgr_out_register()
> > >http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/com...
>
> > Ok, I am clear on the ALL_TG_PT=1 UNREGISTER case, thanks for the code
> > reference.
>
> Ok, please have a look at the ALL_TG_PT=1 UNREGISTER patch at:
>
> http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=comm...