[Iscsitarget-devel] vmware esx 4.0 and iet iscsi-target

[matth ingersoll]

I seem to have a random connectivity problem while doing some testing. Three vmware esx 4.0 hosts (esx 3.5 seems to work fine or takes a lot longer to fail) are connected to a single linux server running iscsi-target. At random points, the vmware hosts lose connectivity to the storage but can be fixed by doing a LUN reset on the vmware host. The answer I received from vmware tech support is that it sounds like a storage and/or networking (switch) issue and not vmware - so I'm posting here. Keep in mind that this setup will run for weeks without any issue. The linux iscsi target host log:

----
Nov 11 00:35:56 test01 kernel: iscsi_trgt: cmnd_abort(1144) 7e766500 1 0 42 512 0 0
Nov 11 00:35:56 test01 kernel: iscsi_trgt: Abort Task (01) issued on tid:15 lun:11 by sid:17455846606569984 (Unknown Task)
Nov 11 00:35:56 test01 kernel: iscsi_trgt: cmnd_abort(1144) 7f766500 1 0 42 512 0 0
Nov 11 00:35:56 test01 kernel: iscsi_trgt: Abort Task (01) issued on tid:15 lun:6 by sid:17455846606569984 (Unknown Task)
Nov 11 00:35:56 test01 kernel: iscsi_trgt: cmnd_abort(1144) b4db1f00 1 0 42 512 0 0
Nov 11 00:35:56 test01 kernel: iscsi_trgt: Abort Task (01) issued on tid:18 lun:105 by sid:7883498375152128 (Unknown Task)
Nov 11 00:35:56 test01 kernel: iscsi_trgt: cmnd_abort(1144) e5d6500 1 0 42 512 0 0
Nov 11 00:35:56 test01 kernel: iscsi_trgt: Abort Task (01) issued on tid:15 lun:9 by sid:18017697048363520 (Unknown Task)
Nov 11 00:35:56 test01 kernel: iscsi_trgt: cmnd_abort(1144) bfb56500 1 0 42 512 0 0
Nov 11 00:35:56 test01 kernel: iscsi_trgt: Abort Task (01) issued on tid:15 lun:4 by sid:11541573560762880 (Unknown Task)
Nov 11 00:35:56 test01 kernel: iscsi_trgt: cmnd_abort(1144) b5db1f00 1 0 42 512 0 0
Nov 11 00:35:56 test01 kernel: iscsi_trgt: Abort Task (01) issued on tid:18 lun:110 by sid:7883498375152128 (Unknown Task)
Nov 11 00:35:56 test01 kernel: iscsi_trgt: cmnd_abort(1144) 7d766500 1 0 42 512 0 0
....
# example of a reset from the vmware host
Nov 11 00:36:24 test01 kernel: iscsi_trgt: Logical Unit Reset (05) issued on tid:27 lun:2 by sid:281475064594944 (Function Complete)
----

The excerpt from a vmware host log is as follows:
----
Nov 10 16:35:56 vmhost01 vmkernel: 16:16:40:28.384 cpu4:10727)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410002125e40) to NMP device "t10.94544500000000004475D213538300000000000000000000" failed on physi
cal path "vmhba34:C2:T3:L8" H:0x5 D:0x40 P:0x0 Possible sense data: 0x2 0x3a 0x1
Nov 10 16:35:56 vmhost01 vmkernel: 16:16:40:28.384 cpu2:4242)WARNING: iscsi_vmk: iscsivmk_TaskMgmtAbortCommands: vmhba34:CH:2 T:3 L:8 : Abort task response indicates task with itt=0x97ea33 has been completed o
n the target but the task response has not arrived
Nov 10 16:35:56 vmhost01 vmkernel: 16:16:40:28.384 cpu4:10727)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "t10.94544500000000004475D213538300000000000000000000" state in doubt; requested fast pa
th state update...
Nov 10 16:35:56 vmhost01 vmkernel: 16:16:40:28.384 cpu4:10727)ScsiDeviceIO: 747: Command 0x2a to device "t10.94544500000000004475D213538300000000000000000000" failed H:0x5 D:0x40 P:0x0 Possible sense data: 0x2
0x3a 0x1.
....
Nov 10 16:36:48 vmhost01 vmkernel: 16:16:41:19.918 cpu3:4111)WARNING: ScsiDeviceIO: 1374: I/O failed due to too many reservation conflicts. t10.94544500000000004475D213538300000000000000000000 (880 0 3)
Nov 10 16:36:48 vmhost01 vmkernel: 16:16:41:20.249 cpu1:4116)WARNING: ScsiDeviceIO: 1374: I/O failed due to too many reservation conflicts. t10.94544500000000004475D213538300000000000000000000 (880 0 3)
Nov 10 16:36:48 vmhost01 vmkernel: 16:16:41:20.249 cpu1:4116)WARNING: FS3: 6509: Reservation error: SCSI reservation conflict
....
Nov 10 16:36:56 vmhost01 vmkernel: 16:16:41:27.795 cpu6:4179)Resv: 542: Executed out-of-band lun reset on t10.94544500000000004475D213538300000000000000000000
----

I have tried evaluating the sense codes but do not have enough knowledge to make sense of them yet :)

Any help will be extremely appreciated.

--
Matth

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Iscsitarget-devel mailing list
Iscsitar...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

[Juan Aristizabal]

Hi.

I just want to mention that I have experienced the same problems with
ESX 4.0. The implementation of ESX 3.5 might be different since I have
not seen issues like losing the storage when I use iSCSI enterprise target.

On ESX 4.0 I have seen my storage disappearing after I power on a VM,
but unfortunately I wan't able to recover the LUN after a LUN reset.

These are the messages I get from the vmware logs:

vmkernel: 18:19:14:45.681 cpu3:4106)ScsiDeviceIO: 2348: Could not
execute READ CAPACITY for Device
"t10.945445000000000000000000190000001900000020000000" from Plugin "NMP"
due to SCSI reservation. Using default values.

vmkernel: 18:19:14:50.872 cpu0:4106)WARNING: ScsiDeviceIO: 1374: I/O

failed due to too many reservation conflicts.

t10.945445000000000000000000190000001900000020000000 (920 0 3)

vmkernel: 18:19:15:28.566 cpu1:4106)WARNING: ScsiDeviceIO: 1265: Failed
to issue command on device
t10.945445000000000000000000190000001900000020000000: Timeout

So far I haven't looked at the messages from iSCSI target, but there is
definitely an issue with ESX 4.0 since there are a couple of threads
on vmware forums with open filer users struggling with the storage on
vSphere (ESX 4.0).

--
Juan Aristizabal
Network engineer

[Ross S. W. Walker]

The READ CAPACITY under reservation bug was fixed in 1.4.18,
we are actually just about to release 1.4.19 which I
recommend trying out as 1.4.18 had an bug in initiators.allow
processing which made initiator filtering, well... broken.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

[Juan Aristizabal]

Thanks a lot for the update Ross.

--
Juan Aristizabal
Network engineer
Datagardens