Trouble with iscsi, got CHECK_CONDITION but invalid data buffer size of 0
TC
I'm an iscsi newbie. My goal is to use an iscsi disk for a cluster.
I seem to have the target connected, but it's not working for me.
Here are some details:
cm1:~ # uname -a
Linux cm1 2.6.16.53-0.8-smp #1 SMP Fri Aug 31 13:07:27 UTC 2007 x86_64
x86_64 x86_64 GNU/Linux
cm1:~ # iscsiadm -m session -i
iscsiadm version 2.0-754
************************************
Session (sid 0) using module tcp:
************************************
TargetName: iqn.1994-04.org.netbsd.iscsi-target:target0
Portal Group Tag: 1
Network Portal: xxx.xxx.xxx.xxx:3260
iSCSI Connection State: LOGGED IN
Internal iscsid Session State: NO CHANGE
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 65536
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 262144
MaxBurstLength: 16776192
ImmediateData: Yes
InitialR2T: No
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 1 State: running
scsi1 Channel 00 Id 0 Lun: 0
Attached scsi disk sdb State: running
cm1:~ # dmesg -c > /dev/null
cm1:~ # dd if=/dev/zero of=/dev/sdb1 bs=1M count=1 oflag=direct
dd: writing `/dev/sdb1': Input/output error
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.807986 seconds, 0.0 kB/s
cm1:~ # dmesg
iscsi: Got CHECK_CONDITION but invalid data buffer size of 0
sd 1:0:0:0: SCSI error: return code = 0x00040000
end_request: I/O error, dev sdb, sector 63
connection0:0: iscsi: detected conn error (1011)
iscsi: Got CHECK_CONDITION but invalid data buffer size of 0
sd 1:0:0:0: SCSI error: return code = 0x00040000
end_request: I/O error, dev sdb, sector 1087
cm1:~ #
Using dd oflag=dsync works, and I can partition the disk and even
create an ext3 filesystem on it. But trying to create a cluster
filesystem does not:
cm1:~ # dmesg -c > /dev/null
cm1:~ # mkfs.ocfs2 -F -b 4K -C 32K -L OCFS2 /dev/sdb1
mkfs.ocfs2 1.2.3
mkfs.ocfs2: Could not write: Input/output error
cm1:~ # dmesg
iscsi: Got CHECK_CONDITION but invalid data buffer size of 0
sd 1:0:0:0: SCSI error: return code = 0x00040000
end_request: I/O error, dev sdb, sector 1087
connection0:0: iscsi: detected conn error (1011)
iscsi: Got CHECK_CONDITION but invalid data buffer size of 0
sd 1:0:0:0: SCSI error: return code = 0x00040000
end_request: I/O error, dev sdb, sector 63
cm1:~ #
Any ideas what could be wrong?
-TC
Mike Christie
> ************************************
> Session (sid 0) using module tcp:
> ************************************
> TargetName: iqn.1994-04.org.netbsd.iscsi-target:target0
Is this the netbsd software target? Did they use the intel target source
as the base?
> cm1:~ # dmesg -c > /dev/null
> cm1:~ # dd if=/dev/zero of=/dev/sdb1 bs=1M count=1 oflag=direct
> dd: writing `/dev/sdb1': Input/output error
> 1+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.807986 seconds, 0.0 kB/s
> cm1:~ # dmesg
> iscsi: Got CHECK_CONDITION but invalid data buffer size of 0
The intel target, which I thought I remember the netbsd one being based
on (maybe it was one of the other bsds through), has a bug where it
sends a check condition but no sense. I think with that target if we let
it go and let the scsi error handler request the sense it works out, but
the target should be fixed.
Why we get into this problem at all (why oflag=direct does not work) is
another problem. I have tried "dd if=/dev/zero of=/dev/sdb1 bs=1M
count=1 oflag=direct" on a couple other targets and it is working ok here.
Where did you get the kernel 2.6.16.53-0.8-smp? Is it from SUSE or
RedHat? Are you using the iscsi modules and tools from that distro or
from a open-iscsi.org release or are you mixing and matching them?
Would it be possible to try the open-iscsi tools and kernel modules from
open-iscsi.org http://www.open-iscsi.org/bits/open-iscsi-2.0-865.15.tar.gz?
If that does not work could you build those tools and modules with
make DEBUG_SCSI=1 DEBUG_TCP=1
make DEBUG_SCSI=1 DEBUG_TCP=1 install
then run the "dd if=/dev/zero of=/dev/sdb1 bs=1M count=1 oflag=direct"
and send all the log output?
TC
Thanks Mike,
I'm using FreeNAS. Your first question made me go look to verify that
it's using netbsd-iscsi. Searching for the version of that, I noticed
that FreeNAS just released a new version this month, and its changelog
contains: "Upgrade netbsd-iscsi (iscsi-target) to 20070925".
So, I did an upgrade FreeNAS from 0.685b to 0.686b1 and the problem
has disappeared. Yay! I looked (a little) to try to find out what
older version of iscsi-target FreeNAS 0.685b had in it, but didn't see
that documented.
Your explanation makes sense. I think I was indeed hitting the bug
you described.
To answer your other questions, I'm using SUSE 10 Enterprise, and my
open-iscsi is from that distribution, version 2.0.707-0.24. Do you
think I should still upgrade to 2.0-865,15?
Thanks again!
-TC
Mike Christie
> Resolved!
>
> Thanks Mike,
no problem.
>
> I'm using FreeNAS. Your first question made me go look to verify that
> it's using netbsd-iscsi. Searching for the version of that, I noticed
> that FreeNAS just released a new version this month, and its changelog
> contains: "Upgrade netbsd-iscsi (iscsi-target) to 20070925".
>
> So, I did an upgrade FreeNAS from 0.685b to 0.686b1 and the problem
> has disappeared. Yay! I looked (a little) to try to find out what
> older version of iscsi-target FreeNAS 0.685b had in it, but didn't see
> that documented.
>
> Your explanation makes sense. I think I was indeed hitting the bug
> you described.
>
> To answer your other questions, I'm using SUSE 10 Enterprise, and my
> open-iscsi is from that distribution, version 2.0.707-0.24. Do you
> think I should still upgrade to 2.0-865,15?
>
You should probably not upgrade to a open-iscsi.org release, but maybe
you want to upgrade to the current SUSE 10 Enterprise release. I have
not used SUSE in a while so I am not sure what is in there.