Disk Disappears From System When open-iscsi Re-connects To COMSTAR HA Target

[Preston Connors]

Good day,

I am utilizing a Ubuntu 9.10 amd64 open-iscsi 2.0-870 initiator
connecting to a highly available OpenSolaris snv_134 64bit COMSTAR
target. The HA target consists of one active and one passive identical
OpenSolaris servers utilizing the same disk. When the active target
fails the passive target comes on-line using the same I.P. address,
target information, and serves up the same disk.

I have verified that the OpenSolaris servers are failing over
properly. The whole fail over process takes between 30-45 seconds.

The problem lies after the active target goes off-line and the passive
target becomes the active target. During fail over I can see the
initiator iSCSI disk going into block state and the initiator session
going into FAIL state. When the target fail over is complete I can see
the initiator iSCSI disk is in state running and the initiator session
state is LOGGED_IN. Even though the output from iscsiadm states this
disk is running and the iSCSI session is established I can no longer
access this iSCSI disk via the file system and the disk no longer
shows up in fdisk nor am I able to mount it. If I log out of the
target and log back in the disk becomes available and usable.
Unfortunately I do not want to log out and log back in to the target
because there will be active KVM VMs whose file system is based off of
this iSCSI connection and the KVM VMs will cease to work if a logout/
login happens.

Another oddity I find is that when I fail back to the original state
(the original passive node is no longer active and the off-line active
node is now back on-line and active) the disk in question becomes
available and usable by the system.

Here is output from some relevant processes while the initiator is
successfully connected to the HA target.

Any help, insight, or ideas would be greatly appreciated! I can
provide any other output and/or settings needed to provide more in-
depth information.

Thank you in advance,
Preston Connors

FDISK OUPUT BEFORE FAIL OVER:

root@host:~# fdisk -l /dev/sdd

Disk /dev/sdd: 5368 MB, 5368709120 bytes
166 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 10292 * 512 = 5269504 bytes
Disk identifier: 0x508cb22f

Device Boot Start End Blocks Id System
/dev/sdd1 1 1018 5238597 83 Linux


ISCSIADM OUTPUT BEFORE FAIL OVER:

root@host:~# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
iscsiadm version 2.0-870

Target: iqn.1986-03.com.sun:mirror:iscsi-failover-test
Current Portal: 192.168.1.1:3260,2
Persistent Portal: 192.168.1.1:3260,2
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:bb82d0f5e87f
Iface IPaddress: 192.168.1.2
Iface HWaddress: default
Iface Netdev: default
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 131072
MaxXmitDataSegmentLength: 32768
FirstBurstLength: 65536
MaxBurstLength: 524288
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 6 State: running
scsi6 Channel 00 Id 0 Lun: 0
Attached scsi disk sdd State: running

ISCSIADM OUTPUT DURING FAIL OVER:

root@host:~# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
iscsiadm version 2.0-870

Target: iqn.1986-03.com.sun:mirror:iscsi-failover-test
Current Portal: 192.168.1.1:3260,2
Persistent Portal: 192.168.1.1:3260,2
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:bb82d0f5e87f
Iface IPaddress: 192.168.1.2
Iface HWaddress: default
Iface Netdev: default
SID: 3
iSCSI Connection State: TRANSPORT WAIT
iSCSI Session State: FAILED
Internal iscsid Session State: REPOEN
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 131072
MaxXmitDataSegmentLength: 32768
FirstBurstLength: 65536
MaxBurstLength: 524288
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 6 State: running
scsi6 Channel 00 Id 0 Lun: 0
Attached scsi disk sdd State: blocked

FDISK OUTPUT AFTER FAIL OVER:
root@kvm-host-3:~# fdisk -l /dev/sdd
<no output>

ISCSIADM OUTPUT AFTER FAIL OVER:
root@host:~# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
iscsiadm version 2.0-870

Target: iqn.1986-03.com.sun:mirror:iscsi-failover-test
Current Portal: 192.168.1.1:3260,2
Persistent Portal: 192.168.1.1:3260,2
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:bb82d0f5e87f
Iface IPaddress: 192.168.1.2
Iface HWaddress: default
Iface Netdev: default
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 131072
MaxXmitDataSegmentLength: 32768
FirstBurstLength: 65536
MaxBurstLength: 524288
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 6 State: running
scsi6 Channel 00 Id 0 Lun: 0
Attached scsi disk sdd State: running

Here is my /etc/iscsi/iscsid.conf:

root@kvm-host-3:~# grep -v "#" /etc/iscsi/iscsid.conf

node.startup = automatic
node.session.timeo.replacement_timeout = 120
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 20
node.session.initial_login_retry_max = 8
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.session.iscsi.FastAbort = Yes

--
You received this message because you are subscribed to the Google Groups "open-iscsi" group.
To post to this group, send email to open-...@googlegroups.com.
To unsubscribe from this group, send email to open-iscsi+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.

[Mike Christie]

On 04/30/2010 02:07 PM, Preston Connors wrote:
> FDISK OUTPUT AFTER FAIL OVER:
> root@kvm-host-3:~# fdisk -l /dev/sdd
> <no output>
>

It is weird that the /dev/sdd link is now gone, but iscsiadm can see the
disk below. iscsiadm looks in /sys/block. It does not look at the /dev dir.

The iscsi layer does not really do anything wrt /dev population except
transport requests. I wonder what is removing the /dev links during this
time. It is not the iscsi layer or tools (we do not touch that stuff).
Are you using udev? Is it possible to rerun udev to create the /dev
links? Do you see anything in /dev/disk?

While the failover is occurring are you running IO through the FS? Do
you see IO errors in /var/log/messages during this time?

After the failover, if you try to send IO to the FS what IO errors do
you see in /var/log/messages?

If you just do IO directly to the disk, before and after the failover
(so just leave a dd if=/dev/sdd of=/dev/null running during the test) do
you see any IO errors in /var/log/messages?

[Ulrich Windl]

On 5 May 2010 at 10:51, Mike Christie wrote:

> On 04/30/2010 02:07 PM, Preston Connors wrote:
> > FDISK OUTPUT AFTER FAIL OVER:
> > root@kvm-host-3:~# fdisk -l /dev/sdd
> > <no output>
> >
>
>
> It is weird that the /dev/sdd link is now gone, but iscsiadm can see the
> disk below. iscsiadm looks in /sys/block. It does not look at the /dev dir.

I'm observing a udev/multipath race condition if too many disks and
partitions are appearing rapidly, especially on a NUMA SMP machine.
With SLES10 SP2 things improved, but it's not completely gone. Maybe
look there. (I usually add missing symlinks manually, because we reboot
rarely (and I only had it after reboot))

Maybe that helps.
[...]

Ulrich

[Preston Connors]

Runnig udevadm trigger (udevadm version 147) does not repopulate the /
dev disk entries. During the fail over IO is being sent to the disk
and there are IO errors relating to this disk during this time. I will
report back when this development environment is accessible again with
more specific details and error messages and try dd if=/dev/sdd of=/
dev/null on the disk while fail over is occurring as well.