Ok, I am now moving I/O running with LIO-SE RAMDISK_DR in my x86_64 VM with CONFIG_SLUB=y on 2.6.24. I also have CONFIG_SLUB_DEBUG=y set, and am passing slub_debug=FZPU into the kernel command line and not seen any warnings or exceptions. So far I have used Open/iSCSI to move 10 GB of R/W traffic, and I am going to let it run for a few hours to make sure it is stable.
You should be ready to go with LIO-Target r250 and the default Ubuntu config with SLUB enabled. Please let me know if you run into any futher issues, and I will jump on them right away.
On Wed, 2008-02-13 at 14:40 +0100, Bart Van Assche wrote: > On Feb 13, 2008 1:24 PM, Nicholas A. Bellinger <n...@linux-iscsi.org> wrote: > > Sorry to keep bugging you on your progress. I am eagerly awaiting to > > hear your results now that we have been able to get back CONFIG_SLUB.
> It might help to add the following to the kernel command line: > slub_debug=FZPU. The output that I got on my setup proves that there > is a memory corruption triggered when performing target discovery on > LIO-SE. I'm not going to run any further LIO-SE tests until this is > solved:
> ------------------------------------------------------------------ > HeaderDigest: None > DataDigest: None > MaxRecvDataSegmentLength: 32768 > IFMarker: No > OFMarker: No > ------------------------------------------------------------------ > ------------------------------------------------------------------ > InitiatorName: iqn.1993-08.org.debian:01:e52cfb64aea > TargetAlias: iSBE Target > InitiatorAlias: INF010 > TargetPortalGroupTag: 0 > DefaultTime2Wait: 2 > DefaultTime2Retain: 0 > ErrorRecoveryLevel: 0 > SessionType: Discovery > ------------------------------------------------------------------ > iSCSI Login successful on CID: 0 from 192.168.102.10 to 192.168.102.12:3260,0 > Incremented iSCSI Connection count to 1 from node: > iqn.1993-08.org.debian:01:e52cfb64aea > Established iSCSI session from node: iqn.1993-08.org.debian:01:e52cfb64aea > Incremented number of active iSCSI sessions to 1 on iSCSI Target Portal Group: 0 > =========================================================================== == > Cleared np->np_login_tpg > BUG kmalloc-16: Redzone overwritten > --------------------------------------------------------------------------- --
> Decremented iSCSI connection count to 0 from node: > iqn.1993-08.org.debian:01:e52cfb64aea > Released iSCSI session from node: iqn.1993-08.org.debian:01:e52cfb64aea > Decremented number of active iSCSI Sessions on iSCSI TPG: 0 to 0
Just FYI, this LIO-Target VM w/ r250 has been up all day and has moved a few hundred GB of traffic from two different Open/iSCSI initiators. I tested PSCSI (to a USB drive) and RAMDISK_DR, and everything looks fine.
Please let me know if you have any futher questions with your setup.
On Wed, 2008-02-13 at 08:26 -0800, Nicholas A. Bellinger wrote: > Hi Bart,
> Ok, I am now moving I/O running with LIO-SE RAMDISK_DR in my x86_64 VM > with CONFIG_SLUB=y on 2.6.24. I also have CONFIG_SLUB_DEBUG=y set, and > am passing slub_debug=FZPU into the kernel command line and not seen any > warnings or exceptions. So far I have used Open/iSCSI to move 10 GB of > R/W traffic, and I am going to let it run for a few hours to make sure > it is stable.
> You should be ready to go with LIO-Target r250 and the default Ubuntu > config with SLUB enabled. Please let me know if you run into any futher > issues, and I will jump on them right away.
> --nab
> On Wed, 2008-02-13 at 14:40 +0100, Bart Van Assche wrote: > > On Feb 13, 2008 1:24 PM, Nicholas A. Bellinger <n...@linux-iscsi.org> wrote: > > > Sorry to keep bugging you on your progress. I am eagerly awaiting to > > > hear your results now that we have been able to get back CONFIG_SLUB.
> > It might help to add the following to the kernel command line: > > slub_debug=FZPU. The output that I got on my setup proves that there > > is a memory corruption triggered when performing target discovery on > > LIO-SE. I'm not going to run any further LIO-SE tests until this is > > solved:
> > ------------------------------------------------------------------ > > HeaderDigest: None > > DataDigest: None > > MaxRecvDataSegmentLength: 32768 > > IFMarker: No > > OFMarker: No > > ------------------------------------------------------------------ > > ------------------------------------------------------------------ > > InitiatorName: iqn.1993-08.org.debian:01:e52cfb64aea > > TargetAlias: iSBE Target > > InitiatorAlias: INF010 > > TargetPortalGroupTag: 0 > > DefaultTime2Wait: 2 > > DefaultTime2Retain: 0 > > ErrorRecoveryLevel: 0 > > SessionType: Discovery > > ------------------------------------------------------------------ > > iSCSI Login successful on CID: 0 from 192.168.102.10 to 192.168.102.12:3260,0 > > Incremented iSCSI Connection count to 1 from node: > > iqn.1993-08.org.debian:01:e52cfb64aea > > Established iSCSI session from node: iqn.1993-08.org.debian:01:e52cfb64aea > > Incremented number of active iSCSI sessions to 1 on iSCSI Target Portal Group: 0 > > =========================================================================== == > > Cleared np->np_login_tpg > > BUG kmalloc-16: Redzone overwritten > > --------------------------------------------------------------------------- --
> > Decremented iSCSI connection count to 0 from node: > > iqn.1993-08.org.debian:01:e52cfb64aea > > Released iSCSI session from node: iqn.1993-08.org.debian:01:e52cfb64aea > > Decremented number of active iSCSI Sessions on iSCSI TPG: 0 to 0
On Thu, Feb 14, 2008 at 7:20 AM, Nicholas A. Bellinger
<n...@linux-iscsi.org> wrote: > Please let me know if you have any futher questions with your setup.
Thanks for asking. I am trying to configure a target with LIO, but open-iscsi discovery does not report any target nodes. All that is reported by open-iscsi discovery is the following:
On Thu, 2008-02-14 at 08:42 +0100, Bart Van Assche wrote: > On Thu, Feb 14, 2008 at 7:20 AM, Nicholas A. Bellinger > <n...@linux-iscsi.org> wrote: > > Please let me know if you have any futher questions with your setup.
> Thanks for asking. I am trying to configure a target with LIO, but > open-iscsi discovery does not report any target nodes. All that is > reported by open-iscsi discovery is the following:
This is what would be expected from your setup below (ie: a single node name + single target portal group with two network portals) when you still Open/iSCSI to perform SendTarget=All with SessionType=Discovery.
>From here with Open/iSCSI (once the Portals and TargetNames have been
reported, and can be seen with iscsiadm -m node), I am using the following to establish SessionType=Normal and access iSCSI LUNs, etc:
Btw, these first three steps you really should be using '/etc/init.d/target start'.
This script will generate /etc/targetname.iscsi from iscsi-name, and load this IQN with settargetname after modprobe.
Also, you can setup and manage additional target node names (you don't need this for your test, but is handy to know) with target-ctl coreaddtiqn and target-ctl coredeltiqn.
By default when you issue any target-ctl with 'tpgt=', it will assume you are reference the _DEFAULT_ target IQN name that was set with target-ctl settargetname. You can explictly pass 'targetname=' to reference a non default target node name into any target-ctl that acceps 'tpgt='. Again, something that you do not have to worry about now, but allows the admin to take advantage of the full iSCSI addressing model as described in "What does the conceptual model of iSNS and iSCSI look like..?"
This all looks fine, and can go in your /etc/iscsi/install.target, that will be loaded from /etc/init.d/target start after modprobe and settargetname is set from /etc/targetname.iscsi.
On Thu, 2008-02-14 at 08:57 +0100, Bart Van Assche wrote: > On Thu, Feb 14, 2008 at 8:42 AM, Bart Van Assche > <bart.vanass...@gmail.com> wrote: > > # iscsi_target_ip=192.168.102.12 > > # rm -rf /etc/iscsi/nodes /etc/iscsi/send_targets > > # iscsiadm -m discovery -t sendtargets -p ${iscsi_target_ip} > > 10.100.100.12:3260,1 iqn.2007-05.com.example > > 192.168.102.12:3260,1 iqn.2007-05.com.example
> Just for clarity, I was expecting something like this: > 10.100.100.12:3260,1 iqn.2007-05.com.example:storage.disk2.sys1.xyz > 192.168.102.12:3260,1 iqn.2007-05.com.example:storage.disk2.sys1.xyz
Ok, I see what yo mean.
The SendTargets=All payload will contain identical TargetName= values for whatever you pass in via target-ctl settargetname or target-ctl coreaddtiqn.
The list of exactly what will be returned during SendTargets can be viewed with target-ctl listgninfo.
Btw, adding additional storage naming information (say if you wanted to use a portal group per storage object in your setup, as compared to say multiple-luns per target portal group) is something that I can easily append and enabled if this helps the admin.
As I mentioned, the LIO-Target implements the advanced method of TargetName+TPGT+NP mappings, and you can create any arbitary configuration using target-ctl.
Just for clarity, I was expecting something like this: 10.100.100.12:3260,1 iqn.2007-05.com.example:storage.disk2.sys1.xyz 192.168.102.12:3260,1 iqn.2007-05.com.example:storage.disk2.sys1.xyz
Sorry, I was really tired last night and I did not notice this in your config. By default, the LIO-Target requires that each exported SCSI Target port (ie: addluntodev) be explictly mapped to each SCSI Initiator Port.
This means that addnodetotpg, and then subsequent addnodetolun calls are required for each initiator to access each TPG mapped LUN. Please add the additional line here:
FYI, iscsi_lun= is the value from addluntodev, mapped_lun= is what the initiator will actually see (they can be different) and lun_access= is 0 == RO and 1 == RW.
> After this command the following is logged in the initiator kernel log: > scsi27 : iSCSI Initiator over TCP/IP
> However, no device name (/dev/sd...) appears. How can I now access > this device from the initiator ?
> This is the output I get when logging in with the same iSCSI initiator > to another iSCSI target (SCST): > scsi4 : iSCSI Initiator over TCP/IP > scsi 4:0:0:0: Direct-Access SCST_FIO vdisk0 096 PQ: 0 ANSI: 4 > sd 4:0:0:0: [sde] 4194304 512-byte hardware sectors (2147 MB) > sd 4:0:0:0: [sde] Write Protect is off > sd 4:0:0:0: [sde] Mode Sense: 6b 00 10 08 > sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, supports > DPO and FUA > sd 4:0:0:0: [sde] 4194304 512-byte hardware sectors (2147 MB) > sd 4:0:0:0: [sde] Write Protect is off > sd 4:0:0:0: [sde] Mode Sense: 6b 00 10 08 > sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, supports > DPO and FUA > sde: unknown partition table > sd 4:0:0:0: [sde] Attached SCSI disk > sd 4:0:0:0: Attached scsi generic sg5 type 0
Note that there is also the 'Demo Mode' feature, that is used by default with the LIO-VM images, and will allow (on a per Target Portal Group basis) all Initiators to have access to all SCSI Target Ports.
By default, all TYPE_DISK LUNs in DEMO mode will be RO. I do think to prevent folks using demo mode from overwriting devices without a cluster filesystem on them (which is usually the case). For folks who really know what they are doing, there is also the following attrib to enable RW with demo mode.
On Thu, 2008-02-14 at 00:09 -0800, Nicholas A. Bellinger wrote: > On Thu, 2008-02-14 at 08:57 +0100, Bart Van Assche wrote: > > On Thu, Feb 14, 2008 at 8:42 AM, Bart Van Assche > > <bart.vanass...@gmail.com> wrote: > > > # iscsi_target_ip=192.168.102.12 > > > # rm -rf /etc/iscsi/nodes /etc/iscsi/send_targets > > > # iscsiadm -m discovery -t sendtargets -p ${iscsi_target_ip} > > > 10.100.100.12:3260,1 iqn.2007-05.com.example > > > 192.168.102.12:3260,1 iqn.2007-05.com.example
> > Just for clarity, I was expecting something like this: > > 10.100.100.12:3260,1 iqn.2007-05.com.example:storage.disk2.sys1.xyz > > 192.168.102.12:3260,1 iqn.2007-05.com.example:storage.disk2.sys1.xyz
> Ok, I see what yo mean.
> The SendTargets=All payload will contain identical TargetName= values > for whatever you pass in via target-ctl settargetname or target-ctl > coreaddtiqn.
> The list of exactly what will be returned during SendTargets can be > viewed with target-ctl listgninfo.
> Btw, adding additional storage naming information (say if you wanted to > use a portal group per storage object in your setup, as compared to say > multiple-luns per target portal group) is something that I can easily > append and enabled if this helps the admin.
Btw, I was thinking about this some more, and the key point with regard to any iSCSI IQN is that they are 1) Unique, 2) Permanent.
>From the admin's point of view, the best way to describe a iSCSI Portal
Group with a human readable string is with TargetAlias. This can be set with:
Thanks for the help -- I can now log in with open-iscsi to a LIO-SE target. Performance for 1MB is as expected, but performance for 512-byte requests is abnormally low (IPoIB):
$ dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 3.4495 seconds, 304 MB/s $ dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 1000+0 records in 1000+0 records out 512000 bytes (512 kB) copied, 39.612 seconds, 12.9 kB/s
On Fri, 2008-02-15 at 11:44 +0100, Bart Van Assche wrote: > On Thu, Feb 14, 2008 at 7:22 PM, Nicholas A. Bellinger > <n...@linux-iscsi.org> wrote: > [ ... ]
> Thanks for the help -- I can now log in with open-iscsi to a LIO-SE > target. Performance for 1MB is as expected, but performance for > 512-byte requests is abnormally low (IPoIB):
> $ dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes (1.0 GB) copied, 3.4495 seconds, 304 MB/s > $ dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 > 1000+0 records in > 1000+0 records out > 512000 bytes (512 kB) copied, 39.612 seconds, 12.9 kB/s
Indeed. I have no explantation why LIO-Target would be causing this to be so slow. You can check /proc/iscsi_target/mibs/scsi_lu to see the total CDBs, MB transferred, etc. It would be interesting to see if you can notice anything strange about how quickly these numbers (espically the total CDB count) get incremented while running the 512 byte block test.
Could you try running this across the non IPoIB network portal and see if we can get a baseline for the small blocksize..? Also, what types of numbers have you previously seen using IPoIB with other targets..? Is there any way this could be related to the IPoIB performance jitters with iperf you has posted earlier..?
> On Fri, 2008-02-15 at 11:44 +0100, Bart Van Assche wrote: > > Thanks for the help -- I can now log in with open-iscsi to a LIO-SE > > target. Performance for 1MB is as expected, but performance for > > 512-byte requests is abnormally low (IPoIB):
> > $ dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 > > 1000+0 records in > > 1000+0 records out > > 1048576000 bytes (1.0 GB) copied, 3.4495 seconds, 304 MB/s > > $ dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 > > 1000+0 records in > > 1000+0 records out > > 512000 bytes (512 kB) copied, 39.612 seconds, 12.9 kB/s
The results over the Ethernet network are consistent:
# dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 9.43356 seconds, 111 MB/s # dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 1000+0 records in 1000+0 records out 512000 bytes (512 kB) copied, 39.6742 seconds, 12.9 kB/s
The data transfer rate for 512 byte blocks is really slow: even carrying over one floppy disk per minute manually from target to inititiator yields a higher transfer rate (1440 KB / 60 s = 24 KB/s). All other targets I tested (iETD, STGT, SCST) work considerably faster for 512 byte blocks (transfer rates between 4 and 12 MB/s via IPoIB). What transfer rate do you obtain for direct I/O of 512 byte blocks over Ethernet ?
On Fri, 2008-02-15 at 12:32 +0100, Bart Van Assche wrote: > On Fri, Feb 15, 2008 at 12:12 PM, Nicholas A. Bellinger > <n...@linux-iscsi.org> wrote:
> > On Fri, 2008-02-15 at 11:44 +0100, Bart Van Assche wrote: > > > Thanks for the help -- I can now log in with open-iscsi to a LIO-SE > > > target. Performance for 1MB is as expected, but performance for > > > 512-byte requests is abnormally low (IPoIB):
> > > $ dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 > > > 1000+0 records in > > > 1000+0 records out > > > 1048576000 bytes (1.0 GB) copied, 3.4495 seconds, 304 MB/s > > > $ dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 > > > 1000+0 records in > > > 1000+0 records out > > > 512000 bytes (512 kB) copied, 39.612 seconds, 12.9 kB/s
> The results over the Ethernet network are consistent:
> # dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes (1.0 GB) copied, 9.43356 seconds, 111 MB/s > # dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 > 1000+0 records in > 1000+0 records out > 512000 bytes (512 kB) copied, 39.6742 seconds, 12.9 kB/s
> The data transfer rate for 512 byte blocks is really slow: even > carrying over one floppy disk per minute manually from target to > inititiator yields a higher transfer rate (1440 KB / 60 s = 24 KB/s). > All other targets I tested (iETD, STGT, SCST) work considerably faster > for 512 byte blocks (transfer rates between 4 and 12 MB/s via IPoIB). > What transfer rate do you obtain for direct I/O of 512 byte blocks > over Ethernet ?
I am seeing the same issue with 512 and 1024 block sizes between different VMs. Once I jump up to 2048 I get ~5 MB, and everything above 2k looks fine. I will keep looking and let you know if I find anything.
On Fri, 2008-02-15 at 04:17 -0800, Nicholas A. Bellinger wrote:
> On Fri, 2008-02-15 at 12:32 +0100, Bart Van Assche wrote: > > On Fri, Feb 15, 2008 at 12:12 PM, Nicholas A. Bellinger > > <n...@linux-iscsi.org> wrote:
> > > On Fri, 2008-02-15 at 11:44 +0100, Bart Van Assche wrote: > > > > Thanks for the help -- I can now log in with open-iscsi to a LIO-SE > > > > target. Performance for 1MB is as expected, but performance for > > > > 512-byte requests is abnormally low (IPoIB):
> > > > $ dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 > > > > 1000+0 records in > > > > 1000+0 records out > > > > 1048576000 bytes (1.0 GB) copied, 3.4495 seconds, 304 MB/s > > > > $ dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 > > > > 1000+0 records in > > > > 1000+0 records out > > > > 512000 bytes (512 kB) copied, 39.612 seconds, 12.9 kB/s
> > The results over the Ethernet network are consistent:
> > # dd if=/dev/sde of=/dev/null iflag=direct bs=1M count=1000 > > 1000+0 records in > > 1000+0 records out > > 1048576000 bytes (1.0 GB) copied, 9.43356 seconds, 111 MB/s > > # dd if=/dev/sde of=/dev/null iflag=direct bs=512 count=1000 > > 1000+0 records in > > 1000+0 records out > > 512000 bytes (512 kB) copied, 39.6742 seconds, 12.9 kB/s
> > The data transfer rate for 512 byte blocks is really slow: even > > carrying over one floppy disk per minute manually from target to > > inititiator yields a higher transfer rate (1440 KB / 60 s = 24 KB/s). > > All other targets I tested (iETD, STGT, SCST) work considerably faster > > for 512 byte blocks (transfer rates between 4 and 12 MB/s via IPoIB). > > What transfer rate do you obtain for direct I/O of 512 byte blocks > > over Ethernet ?
> I am seeing the same issue with 512 and 1024 block sizes between > different VMs. Once I jump up to 2048 I get ~5 MB, and everything above > 2k looks fine. I will keep looking and let you know if I find anything.
Ok, so 512 and 1024 for WRITEs with iflag=direct is fine. Also, removing iflag=direct from 512 and 1024 for READs produces normal results.
I am still trying a couple of different configurations to see if I can locate the reason why <= 1024 w/ iflag=direct is so dead slow.
While read tests from the LIO-SE target work fine, write tests result in strange errors (block size 2KB).
From the initiator kernel log:
iscsi: cmd 0x2a is not queued (8) sd 4:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK end_request: I/O error, dev sde, sector 2161664
From the target kernel log:
Received iSCSI login request from 192.168.102.12 on TCP Network Portal 192.168.102.10:3260 Located Storage Object: iqn.2003-01.org.linux-iscsi.INF010.x86_64:sn.856bce71e53c Located Portal Group Object: 1 Set np->np_login_tpg to ffff8101588c8200 iscsi_handle_login_thread_timeout:665: ***ERROR*** iSCSI Login timeout on Network Portal 192.168.102.10:3260
On Fri, 2008-02-15 at 15:28 +0100, Bart Van Assche wrote: > Hello Nicholas,
> While read tests from the LIO-SE target work fine,
I am still doing some profiling here on the issue with <= 1024 READs. I have not found anything just yet..
> write tests result > in strange errors (block size 2KB).
> >From the initiator kernel log:
> iscsi: cmd 0x2a is not queued (8) > sd 4:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT > driverbyte=DRIVER_OK,SUGGEST_OK > end_request: I/O error, dev sde, sector 2161664
It looks like Open/iSCSI failed the outstanding struct scsi_cmnd back up to the SCSI ML.
> >From the target kernel log:
> Received iSCSI login request from 192.168.102.12 on TCP Network Portal > 192.168.102.10:3260 > Located Storage Object: > iqn.2003-01.org.linux-iscsi.INF010.x86_64:sn.856bce71e53c > Located Portal Group Object: 1 > Set np->np_login_tpg to ffff8101588c8200 > iscsi_handle_login_thread_timeout:665: ***ERROR*** iSCSI Login timeout > on Network Portal 192.168.102.10:3260
Here, the initiator did not complete the login phase before successfully moving to full feature phase, and hence the login timeout handler fired. (This can be set with target-ctl settpgattrib login_timeout btw).
Are you still able to --logout and --login with Open/iSCSI..?
Are you able to /etc/init.d/target restart properly..?
<n...@linux-iscsi.org> wrote: > > >From the initiator kernel log:
> > iscsi: cmd 0x2a is not queued (8) > > sd 4:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT > > driverbyte=DRIVER_OK,SUGGEST_OK > > end_request: I/O error, dev sde, sector 2161664
> It looks like Open/iSCSI failed the outstanding struct scsi_cmnd back up > to the SCSI ML.
> > >From the target kernel log:
> > Received iSCSI login request from 192.168.102.12 on TCP Network Portal > > 192.168.102.10:3260 > > Located Storage Object: > > iqn.2003-01.org.linux-iscsi.INF010.x86_64:sn.856bce71e53c > > Located Portal Group Object: 1 > > Set np->np_login_tpg to ffff8101588c8200 > > iscsi_handle_login_thread_timeout:665: ***ERROR*** iSCSI Login timeout > > on Network Portal 192.168.102.10:3260
> Here, the initiator did not complete the login phase before successfully > moving to full feature phase, and hence the login timeout handler fired. > (This can be set with target-ctl settpgattrib login_timeout btw).
> Are you still able to --logout and --login with Open/iSCSI..?
> Are you able to /etc/init.d/target restart properly..?
The above happened during transferring data, not during a normal login. So it is very suspicious that open-iscsi had to login again -- this is either a bug in open-iscsi or in LIO-SE.
After the above happened, LIO-SE on the target system seems to be locked up: * Discovery from the initiator results in timeouts, even after having rebooted the initiator system. * The following output from netstat shows that LIO-SE does no longer read the data that arrived at its sockets (Recv-Q > 0): $ netstat -aen|grep -Ew '^Active|^Proto|3260' Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode tcp 0 0 192.168.102.10:3260 0.0.0.0:* LISTEN 0 11347 tcp 0 0 10.100.100.10:3260 0.0.0.0:* LISTEN 0 11337 tcp 525 0 192.168.102.10:3260 192.168.102.12:59518 CLOSE_WAIT 0 0 tcp 525 0 192.168.102.10:3260 192.168.102.12:59520 CLOSE_WAIT 0 0 tcp 97 0 192.168.102.10:3260 192.168.102.12:33320 CLOSE_WAIT 0 11373 tcp 1 0 192.168.102.10:3260 192.168.102.12:59515 CLOSE_WAIT 0 11374 tcp 525 0 192.168.102.10:3260 192.168.102.12:59517 CLOSE_WAIT 0 0 tcp 525 0 192.168.102.10:3260 192.168.102.12:59521 CLOSE_WAIT 0 0 tcp 525 0 192.168.102.10:3260 192.168.102.12:59519 CLOSE_WAIT 0 0 tcp 525 0 192.168.102.10:3260 192.168.102.12:59516 CLOSE_WAIT 0 0
On Fri, 2008-02-15 at 16:19 +0100, Bart Van Assche wrote: > On Fri, Feb 15, 2008 at 3:55 PM, Nicholas A. Bellinger > <n...@linux-iscsi.org> wrote: > > > >From the initiator kernel log:
> > It looks like Open/iSCSI failed the outstanding struct scsi_cmnd back up > > to the SCSI ML.
> > > >From the target kernel log:
> > > Received iSCSI login request from 192.168.102.12 on TCP Network Portal > > > 192.168.102.10:3260 > > > Located Storage Object: > > > iqn.2003-01.org.linux-iscsi.INF010.x86_64:sn.856bce71e53c > > > Located Portal Group Object: 1 > > > Set np->np_login_tpg to ffff8101588c8200 > > > iscsi_handle_login_thread_timeout:665: ***ERROR*** iSCSI Login timeout > > > on Network Portal 192.168.102.10:3260
> > Here, the initiator did not complete the login phase before successfully > > moving to full feature phase, and hence the login timeout handler fired. > > (This can be set with target-ctl settpgattrib login_timeout btw).
> > Are you still able to --logout and --login with Open/iSCSI..?
> > Are you able to /etc/init.d/target restart properly..?
> The above happened during transferring data, not during a normal > login. So it is very suspicious that open-iscsi had to login again -- > this is either a bug in open-iscsi or in LIO-SE.
> After the above happened, LIO-SE on the target system seems to be locked up: > * Discovery from the initiator results in timeouts, even after having > rebooted the initiator system. > * The following output from netstat shows that LIO-SE does no longer > read the data that arrived at its sockets (Recv-Q > 0): > $ netstat -aen|grep -Ew '^Active|^Proto|3260' > Active Internet connections (servers and established) > Proto Recv-Q Send-Q Local Address Foreign Address > State User Inode > tcp 0 0 192.168.102.10:3260 0.0.0.0:* > LISTEN 0 11347 > tcp 0 0 10.100.100.10:3260 0.0.0.0:* > LISTEN 0 11337 > tcp 525 0 192.168.102.10:3260 192.168.102.12:59518 > CLOSE_WAIT 0 0 > tcp 525 0 192.168.102.10:3260 192.168.102.12:59520 > CLOSE_WAIT 0 0 > tcp 97 0 192.168.102.10:3260 192.168.102.12:33320 > CLOSE_WAIT 0 11373 > tcp 1 0 192.168.102.10:3260 192.168.102.12:59515 > CLOSE_WAIT 0 11374 > tcp 525 0 192.168.102.10:3260 192.168.102.12:59517 > CLOSE_WAIT 0 0 > tcp 525 0 192.168.102.10:3260 192.168.102.12:59521 > CLOSE_WAIT 0 0 > tcp 525 0 192.168.102.10:3260 192.168.102.12:59519 > CLOSE_WAIT 0 0 > tcp 525 0 192.168.102.10:3260 192.168.102.12:59516 > CLOSE_WAIT 0 0
Ok, I am thinking that this must have something to do with IPoIB, or the interaction between LIO and IPoIB. The next question is:
Can this be reproduced every time running over the IPoIB network portal..? Can this be reproduced running over the 1 Gb/sec network portal..? I am guessing that things will be stable on the latter case.
I do not believe this is an issue with LIO-Target and LIO-SE scaling to the typical IPoIB numbers you are seeing (~300 MB/sec). Over the years on 10 Gb/sec Ethernet, going ~600 MB/sec to real (SAS) storage has been typical, and using RAMDISK_DR, ~1200 MB/sec with multiple initiators is also not a problem.
On Fri, Feb 15, 2008 at 4:38 PM, Nicholas A. Bellinger
<n...@linux-iscsi.org> wrote: > Ok, I am thinking that this must have something to do with IPoIB, or the > interaction between LIO and IPoIB. The next question is:
You are making an assumption here, this is not something you have verified.
> Can this be reproduced every time running over the IPoIB network > portal..? Can this be reproduced running over the 1 Gb/sec network > portal..? I am guessing that things will be stable on the latter case.
Perfectly reproducible via IPoIB. The IPoIB stack works perfectly with iETD / STGT and SCST. So this issue is probably a LIO-SE issue. I will repeat the test via the Gigabit Ethernet network.
On Fri, 2008-02-15 at 17:17 +0100, Bart Van Assche wrote: > On Fri, Feb 15, 2008 at 4:38 PM, Nicholas A. Bellinger > <n...@linux-iscsi.org> wrote: > > Ok, I am thinking that this must have something to do with IPoIB, or the > > interaction between LIO and IPoIB. The next question is:
> You are making an assumption here, this is not something you have verified.
Yes, I am making this assumption because I don't have IB hardware handy at the moment. I am also making this assumption because there are plenty of nodes have been using LIO-SE in production on 1 Gb/sec and 10 Gb/sec ports (for years in some cases) and do not report any problems.
> > Can this be reproduced every time running over the IPoIB network > > portal..? Can this be reproduced running over the 1 Gb/sec network > > portal..? I am guessing that things will be stable on the latter case.
> Perfectly reproducible via IPoIB. The IPoIB stack works perfectly with > iETD / STGT and SCST. So this issue is probably a LIO-SE issue. I will > repeat the test via the Gigabit Ethernet network.
They are working for your simple tests. As I do not believe you have conducted any long running stress tests or simulations on any of the above implementations, I think that 'works perfectly' is an exaggeration.
If you are really interested in algorithmically stress testing iSCSI Target implementations, you should look at Core-iSCSI-dv. This will give you a much better idea of the maturity of different iSCSI targets than simply running dd.
Being able to pass Core-iSCSI-dv using a tool that will write known patterns across the face of the media and then read-back and compare will demonstrate the maturity much better than your tests.
Hey Nick, has the linux kernel ever been published and distributed with malicious code? What is the likelyhood that some kind of backdoor/malware could make it into the kernel in the future?
On Fri, 2008-02-15 at 18:33 -0800, Thomas Visser wrote: > Hey Nick, has the linux kernel ever been published and distributed > with malicious code? What is the likelyhood that some kind of > backdoor/malware could make it into the kernel in the future?
Not to my knowledge, no. I think these chances are very slim, considering the number of eyes that are going over patches that are posted to LKML and elsewhere these days. Either someone who maintains a particular tree would have to overlook the malicious code (and then include it in their tree) and then have their tree pulled by upstream maintainers, who would also have to miss it during peer review.
> On Fri, 2008-02-15 at 17:17 +0100, Bart Van Assche wrote:
> > On Fri, Feb 15, 2008 at 4:38 PM, Nicholas A. Bellinger
> > <n...@linux-iscsi.org> wrote:
> > > Ok, I am thinking that this must have something to do with IPoIB, or the
> > > interaction between LIO and IPoIB. The next question is:
> > You are making an assumption here, this is not something you have verified.
> Yes, I am making this assumption because I don't have IB hardware handy
> at the moment. I am also making this assumption because there are
> plenty of nodes have been using LIO-SE in production on 1 Gb/sec and 10
> Gb/sec ports (for years in some cases) and do not report any problems.
> > > Can this be reproduced every time running over the IPoIB network
> > > portal..? Can this be reproduced running over the 1 Gb/sec network
> > > portal..? I am guessing that things will be stable on the latter case.
> > Perfectly reproducible via IPoIB. The IPoIB stack works perfectly with
> > iETD / STGT and SCST. So this issue is probably a LIO-SE issue. I will
> > repeat the test via the Gigabit Ethernet network.
> They are working for your simple tests. As I do not believe you have
> conducted any long running stress tests or simulations on any of the
> above implementations, I think that 'works perfectly' is an
> exaggeration.
> If you are really interested in algorithmically stress testing iSCSI
> Target implementations, you should look at Core-iSCSI-dv. This will
> give you a much better idea of the maturity of different iSCSI targets
> than simply running dd.
> Being able to pass Core-iSCSI-dv using a tool that will write known
> patterns across the face of the media and then read-back and compare
> will demonstrate the maturity much better than your tests.
Btw, I am not saying your tests are not of worth. I am simply saying
that if you are framing them in the
context of getting kernel code accepted upstream, dd + IPoIB tests are
not the end all for the discussion
for the kernel storage crowd.
Also, if you are interested in comparing the different
implementations, you should really try to understand
where the strengths of each resides. This is something that I will be
focusing on in my discussion at LFS in a
couple of weeks. Understanding that LIO-Target (the iSCSI Target) and
LIO-SE (the storage engine that talks to
the different kernel storage subsystems) are two seperate pieces and
that I will be eventually sending them
upstream as such should help you when it comes to the discussion on
LKML.
The thing that I think is most important is not rushing a particular
iSCSI Target implementation into the kernel,
but rather, ensuring that there is a proper generic target mode
storage engine in place that will allow even non
SCSI or non transport layer based (ie: NBD) to use common code when
interacting between underlying kernel
storage subsystems.
On Feb 16, 2008 6:13 AM, n...@linux-iscsi.org <n...@linux-iscsi.org> wrote:
> The thing that I think is most important is not rushing a particular > iSCSI Target implementation into the kernel, > but rather, ensuring that there is a proper generic target mode > storage engine in place that will allow even non > SCSI or non transport layer based (ie: NBD) to use common code when > interacting between underlying kernel > storage subsystems.
Can you please repeat the argument about NBD in the thread about the integration of a kernelspace iSCSI target in the mainstream Linux kernel ?
