Maximum LUN support

[Shrey]

Dear All,

I started using/studying open-iscsi recently. I was wondering if there
is any hard-coded limit to number of targets devices that can
discovered per iSCSI target? Is there any hard-coded limit on the
number of LUNs supported per target device (assuming that somehow HBA
on iscsi target doesn't present any limit) ?

Actually I have a simple setup wherein Machine A and Machine B are
connected over a 10Gbps Ethernet, and Machine B has a HBA to a single
SCSI disk. Machine A has open-iscsi-2.0-869. Machine B is running
iscsi-target-0.4.16. I wanted to know the peak capability of the open-
iscsi initiator for certain study.

Hope I am able to put forward my doubt clearly. I myself haven't
searching for this too hard - but at least I couldn't find anything in
the docs/README.

Thanks in advance
---
Shrey

[Stefan de Konink]

On Wed, 14 May 2008, Shrey wrote:

> I started using/studying open-iscsi recently. I was wondering if there
> is any hard-coded limit to number of targets devices that can
> discovered per iSCSI target? Is there any hard-coded limit on the
> number of LUNs supported per target device (assuming that somehow HBA
> on iscsi target doesn't present any limit) ?

It is a module parameter, check modinfo.

Stefan

[Shrey]

Hi,

Thanks :) -
defined as
----- module_param_named(max_lun, iscsi_max_lun, uint, S_IRUGO);
-----
and has a default value of 512.

But is there any limit on the number of target devices that can be
discovered per iscsi target? Something like "only x targets can be
fetched using sendtarget from machine B to Machine A, each having
max_lun maximum number of LUNs"

Thanks again for the information/help.

--
Shreyansh

[Stefan de Konink]

Mmm... I wanted to know this last week. And in the SCSI module there is
written you can go up to 16k. But I suggest if you go above 4k to review
the time that is needed to rescan ;)

I don't know about targets, I only checked for luns :)

Stefan

[Shrey]

Thanks once again :) - If I manage to find/assert it, I will surely
post back.

-
Shreyansh

[Shrey]

Hi all,

Further update on this topic:

>
> Thanks once again :) - If I manage to find/assert it, I will surely
> post back.

I tried to do an experiment to confirm the maximum number of available
targets and LUNs with an initiator. I did:

On the target machine:
1. I created softlinks for a device such that they are named /dev/link
$i_$j where i varies from 1-100 and j varies from 0-10. Thus, I had
links like "link1_0, link1_1, ...,link30_0,...,
link99_0, ...link100_10".
2. Each link was a softlink of a valid SCSI device on the target. All
this was pushed into the /etc/ietd.conf file and service iscsi-target
was restarted. (Yup, I am using iscsi-target-0.4.16 on the target
side.)
3. This gave me 100 targets, each having 10 LUNs - starting from
mydisk1 to mydisk100 (as target names).

On the initiator:
1. I set the discovery on this target machine IP. and tried to log in.

result:
Discovery itself is not able to add more than 89 (twice I got 90)
devices. It simply doesn't get any information about the top 11
devices from the ietd.conf file.
On the target, the message log contained something like "Dropping key
(target ___ )" which, I found, was appearing from text_key_add()
function from the iscsi-target package.

The code is something like:
...
if (conn->rsp.datasize + len > INCOMING_BUFSIZE) {
log_warning("Dropping key (%s=%s)", key, value);
return;
}
...

Initiator doesn't show any error, and infact the node list (iscsiadm -
m node) doesn't display any device starting from mydevice1 -
mydevice11 in the list.

I tried this experiment many times by
1. changing the number of LUNs,
2. changing the device being pointed to by the soft links,
3. changing the number of target devices defined in ietd,
4. making half the links point to one device, and other half to other
5. Shuffling the order of ietd defined target names (and each time
starting 10-11, according to definition in ietd.conf, are not send by
target, whatever there name be).

Each time the result was same (atleast) - 89-90 can only be added.

I am still trying to find what could be the possible bottleneck that
is preventing me to add more devices - till then, I would really
appreciate if there is anything anyone would like to point out. Any
hints or mistakes I am making? Hopefully, using dummy (links) is not
something "not-allowed" as per target/initiator implementation.

--
Shreyansh

[Shrey]

Ok, I got something to ponder from one of the old discussions
:
http://groups.google.com/group/open-iscsi/browse_thread/thread/cc57ad9b5f571b31

(I wonder why I did not find it earlier - might be acted a little
lazily!!)

Will post back if I find anything useful/problematic

Thanks
Shreyansh

[Konrad Rzeszutek]

On Mon, May 19, 2008 at 06:53:07AM -0700, Shrey wrote:
>
> Hi all,
>
> Further update on this topic:
>
> >
> > Thanks once again :) - If I manage to find/assert it, I will surely
> > post back.
>
> I tried to do an experiment to confirm the maximum number of available
> targets and LUNs with an initiator. I did:
>
> On the target machine:
> 1. I created softlinks for a device such that they are named /dev/link
> $i_$j where i varies from 1-100 and j varies from 0-10. Thus, I had
> links like "link1_0, link1_1, ...,link30_0,...,
> link99_0, ...link100_10".
> 2. Each link was a softlink of a valid SCSI device on the target. All

You have actually 100 SCSI devices on your target?

> this was pushed into the /etc/ietd.conf file and service iscsi-target
> was restarted. (Yup, I am using iscsi-target-0.4.16 on the target
> side.)
> 3. This gave me 100 targets, each having 10 LUNs - starting from
> mydisk1 to mydisk100 (as target names).
>
> On the initiator:
> 1. I set the discovery on this target machine IP. and tried to log in.
>
> result:
> Discovery itself is not able to add more than 89 (twice I got 90)
> devices. It simply doesn't get any information about the top 11
> devices from the ietd.conf file.

Which is that the first target wouldn't show up right?

> On the target, the message log contained something like "Dropping key
> (target ___ )" which, I found, was appearing from text_key_add()
> function from the iscsi-target package.
>
> The code is something like:
> ...
> if (conn->rsp.datasize + len > INCOMING_BUFSIZE) {
> log_warning("Dropping key (%s=%s)", key, value);
> return;
> }

Interesting. That does seem like a limitation in that code. Did you
try to change the INCOMING_BUFSIZE to a higher value? Or to decrease
the length of the target's name? So: iqn.2008.com:t1, iqn.2008.com:t2,
and so on?

> ...
>
> Initiator doesn't show any error, and infact the node list (iscsiadm -
> m node) doesn't display any device starting from mydevice1 -
> mydevice11 in the list.

Which would imply that the target didn't send that IQN to the initiator.

>
> I tried this experiment many times by
> 1. changing the number of LUNs,
> 2. changing the device being pointed to by the soft links,
> 3. changing the number of target devices defined in ietd,

So you added more targets and that didn't change it, and when
you lowered the target count it would be a lower amount
of block disks showing up on the initiator side?

> 4. making half the links point to one device, and other half to other
> 5. Shuffling the order of ietd defined target names (and each time
> starting 10-11, according to definition in ietd.conf, are not send by
> target, whatever there name be).

Yeah, it probably sorts them.

>
> Each time the result was same (atleast) - 89-90 can only be added.
>
> I am still trying to find what could be the possible bottleneck that
> is preventing me to add more devices - till then, I would really

Looks like a bug in iscsi-target. You probably show post this question
on its mailing list as well.

[Mike Christie]

Shrey wrote:
> Hi all,
>
> Further update on this topic:
>
>> Thanks once again :) - If I manage to find/assert it, I will surely
>> post back.
>
> I tried to do an experiment to confirm the maximum number of available
> targets and LUNs with an initiator. I did:
>
> On the target machine:
> 1. I created softlinks for a device such that they are named /dev/link
> $i_$j where i varies from 1-100 and j varies from 0-10. Thus, I had
> links like "link1_0, link1_1, ...,link30_0,...,
> link99_0, ...link100_10".
> 2. Each link was a softlink of a valid SCSI device on the target. All
> this was pushed into the /etc/ietd.conf file and service iscsi-target
> was restarted. (Yup, I am using iscsi-target-0.4.16 on the target
> side.)
> 3. This gave me 100 targets, each having 10 LUNs - starting from
> mydisk1 to mydisk100 (as target names).
>
> On the initiator:
> 1. I set the discovery on this target machine IP. and tried to log in.
>
> result:
> Discovery itself is not able to add more than 89 (twice I got 90)
> devices. It simply doesn't get any information about the top 11
> devices from the ietd.conf file.

When you say devices, you mean targets right?

open-iscsi-2.0-869 should work. I believe iscsi-target-0.4.16 does not
support lots of targets. You shuold post to the IET list. I think there
are threads on this already, so actually you should search their list to
make sure it is not fixed in the svn tree.

[Shrey]

Hi Konrad and list,

Thanks for the reply. Please see my comments inline.

On May 19, 7:13 pm, Konrad Rzeszutek <kon...@virtualiron.com> wrote:
> On Mon, May 19, 2008 at 06:53:07AM -0700, Shrey wrote:
>
> > Hi all,
>
> > Further update on this topic:
>
> > > Thanks once again :) - If I manage to find/assert it, I will surely
> > > post back.
>
> > I tried to do an experiment to confirm the maximum number of available
> > targets and LUNs with an initiator. I did:
>
> > On the target machine:
> > 1. I created softlinks for a device such that they are named /dev/link
> > $i_$j where i varies from 1-100 and j varies from 0-10. Thus, I had
> > links like "link1_0, link1_1, ...,link30_0,...,
> > link99_0, ...link100_10".
> > 2. Each link was a softlink of a valid SCSI device on the target. All
>
> You have actually 100 SCSI devices on your target?

ohh no - I wish I had :). I have created softlinks to some valid
devices. I have couple of SCSI disks, and I have created more than 100
links, spread across all of them so that not all links point to same
device. This is just a ploy to make iscsi-target feel that 100 targets
have been attached. Best part - it takes them silently, without
complaining. :)

>
> > this was pushed into the /etc/ietd.conf file and service iscsi-target
> > was restarted. (Yup, I am using iscsi-target-0.4.16 on the target
> > side.)
> > 3. This gave me 100 targets, each having 10 LUNs - starting from
> > mydisk1 to mydisk100 (as target names).
>
> > On the initiator:
> > 1. I set the discovery on this target machine IP. and tried to log in.
>
> > result:
> > Discovery itself is not able to add more than 89 (twice I got 90)
> > devices. It simply doesn't get any information about the top 11
> > devices from the ietd.conf file.
>
> Which is that the first target wouldn't show up right?

I have added targets in ietd.conf on iscsi-target as:

Target iqn.2008-04.mydevice1.com:storage.disk1
LUN 0 /dev/link1_0
LUN 1 /dev/link1_1
Target iqn.2008-04.mydevice2.com:storage.disk2
LUN 0 /dev/link2_0
LUN 1 /dev/link2_1
...
...
Target iqn.2008-04.mydevicen.com:storage.diskn
LUN 0 /dev/linkn_0

Now, whichever devices are on top of this list are thrown out.
Probably this as a result of some kind of target sorting as you said -
but, mydevice11 should come after mydevice100 isn't it?? but it
doesn't - iscsi-target simply throws (Dropping key) of top 'x' entries
where 'x = y - 90' where y is total number of devices (targets) added
in this list

>
> > On the target, the message log contained something like "Dropping key
> > (target ___ )" which, I found, was appearing from text_key_add()
> > function from the iscsi-target package.
>
> > The code is something like:
> > ...
> > if (conn->rsp.datasize + len > INCOMING_BUFSIZE) {
> > log_warning("Dropping key (%s=%s)", key, value);
> > return;
> > }
>
> Interesting. That does seem like a limitation in that code. Did you
> try to change the INCOMING_BUFSIZE to a higher value? Or to decrease
> the length of the target's name? So: iqn.2008.com:t1, iqn.2008.com:t2,
> and so on?

Nope, I did not, though I should have. That would be my next step -
thanks.
I did try using different names, but i think that is not the problem
because if I use the same target name as above "Target iqn.
2008-04.mydevice(n).com:storage.disk(n)" - iscsi-target is able to add
mydevice100 but not mydevice1. So, I have ruled out that the target
name is doing something strange.

>
> > ...
>
> > Initiator doesn't show any error, and infact the node list (iscsiadm -
> > m node) doesn't display any device starting from mydevice1 -
> > mydevice11 in the list.
>
> Which would imply that the target didn't send that IQN to the initiator.

Yup, this is what even I interpreted. The output of 'iscsi_discovery
<my ip> ' doesn't contain the name of those devices which have been
dropped and which have already been reported as "Dropping keys .." at
the iscsi-target side.

>
>
>
> > I tried this experiment many times by
> > 1. changing the number of LUNs,
> > 2. changing the device being pointed to by the soft links,
> > 3. changing the number of target devices defined in ietd,
>
> So you added more targets and that didn't change it, and when
> you lowered the target count it would be a lower amount
> of block disks showing up on the initiator side?

Precisely, this is what happened.

>
> > 4. making half the links point to one device, and other half to other
> > 5. Shuffling the order of ietd defined target names (and each time
> > starting 10-11, according to definition in ietd.conf, are not send by
> > target, whatever there name be).
>
> Yeah, it probably sorts them.

OK, but still, i feel mydevice23 should come before mydevice4 - that
way, mydevice23 should not be added even if it is at the bottom of the
list - but that is not happening. top 10-11 devices are simply dumped.

>
>
>
> > Each time the result was same (atleast) - 89-90 can only be added.
>
> > I am still trying to find what could be the possible bottleneck that
> > is preventing me to add more devices - till then, I would really
>
> Looks like a bug in iscsi-target. You probably show post this question
> on its mailing list as well.

Ok, that sounds like a nice advice. Will do that. Right now I am
trying to figure out what exactly happens in this function to make
that function work.

Thanks for your response.

---
Shreyansh

[Mike Christie]

Shrey wrote:
> Dear All,
>
> I started using/studying open-iscsi recently. I was wondering if there
> is any hard-coded limit to number of targets devices that can
> discovered per iSCSI target? Is there any hard-coded limit on the
> number of LUNs supported per target device (assuming that somehow HBA
> on iscsi target doesn't present any limit) ?
>

As others pointed out there is a module param for the max lun for
iscsi_tcp, but that should actually be limited by a libiscsi/scsi-ml
limit. libiscsi uses a function named int_to_scsilun in
drivers/scsi/scsi_scan.c and as you can see that only implments 2
levels. libiscsi/iscsi_tcp also does not do the device scanning (the
iscsi layer only finds the targets) so we are limited by scsi_scan.c. To
get the limits for that do "modinfo scsi_mod".

There is also a limit on the number of targets. Because we allocate a
scsi_host per session, and the scsi layer uses a unsigned short for the
host number the number of targets is 2^16.

[Shrey]

Hi Mike,

Thanks for you reply. Please find my reply inline.

On May 19, 11:04 pm, Mike Christie <micha...@cs.wisc.edu> wrote:
> Shrey wrote:
> > Dear All,
>
> > I started using/studying open-iscsi recently. I was wondering if there
> > is any hard-coded limit to number of targets devices that can
> > discovered per iSCSI target? Is there any hard-coded limit on the
> > number of LUNs supported per target device (assuming that somehow HBA
> > on iscsi target doesn't present any limit) ?
>
> As others pointed out there is a module param for the max lun for
> iscsi_tcp, but that should actually be limited by a libiscsi/scsi-ml
> limit. libiscsi uses a function named int_to_scsilun in
> drivers/scsi/scsi_scan.c and as you can see that only implments 2
> levels. libiscsi/iscsi_tcp also does not do the device scanning (the
> iscsi layer only finds the targets) so we are limited by scsi_scan.c. To
> get the limits for that do "modinfo scsi_mod".

Ok, that gives me something to thing/experiment about. Thanks

>
> There is also a limit on the number of targets. Because we allocate a
> scsi_host per session, and the scsi layer uses a unsigned short for the
> host number the number of targets is 2^16.

That would mean 65536 devices. But I get stuck far below that - just
about 90 targets.
Are some parameters which I am missing which I need to configure or
take care of?
I tried tinkering "node.conn[0].iscsi.MaxRecvDataSegmentLength =
65536" but there was no difference. This actually is making me think
that target is acting as a bottleneck.

I think I will post all this on the iscsi-target list.

Thanks a lot

--
Shreyansh

[Mike Christie]

Shrey wrote:
> Hi Mike,
>
> Thanks for you reply. Please find my reply inline.
>
> On May 19, 11:04 pm, Mike Christie <micha...@cs.wisc.edu> wrote:
>> Shrey wrote:
>>> Dear All,
>>> I started using/studying open-iscsi recently. I was wondering if there
>>> is any hard-coded limit to number of targets devices that can
>>> discovered per iSCSI target? Is there any hard-coded limit on the
>>> number of LUNs supported per target device (assuming that somehow HBA
>>> on iscsi target doesn't present any limit) ?
>> As others pointed out there is a module param for the max lun for
>> iscsi_tcp, but that should actually be limited by a libiscsi/scsi-ml
>> limit. libiscsi uses a function named int_to_scsilun in
>> drivers/scsi/scsi_scan.c and as you can see that only implments 2
>> levels. libiscsi/iscsi_tcp also does not do the device scanning (the
>> iscsi layer only finds the targets) so we are limited by scsi_scan.c. To
>> get the limits for that do "modinfo scsi_mod".
>
> Ok, that gives me something to thing/experiment about. Thanks
>
>> There is also a limit on the number of targets. Because we allocate a
>> scsi_host per session, and the scsi layer uses a unsigned short for the
>> host number the number of targets is 2^16.
>
> That would mean 65536 devices. But I get stuck far below that - just
> about 90 targets.

If you are using 869.* then the problem is most likely on the IET side.
Like I said before I think you need to upgrade. I had the same problem
when I was testing my fixes for lots of targets support.

Oh yeah, I said this in some other mail, but will say it here so people
can search for it. There is another target limit and that is the number
of files a process can have open. We open a tcp socket for each session
and we do a session to each target portal. We also will open other files
so the exact number is not known. I think it is probably around 4000. I
do not have a setup that supports that many targets yet to know exactly
though. Let us know what you find out.

> Are some parameters which I am missing which I need to configure or
> take care of?
> I tried tinkering "node.conn[0].iscsi.MaxRecvDataSegmentLength =
> 65536" but there was no difference. This actually is making me think
> that target is acting as a bottleneck.
>

That is just for normal sessions. You would want to set
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength

but like I said that will not completely fix the problem.

[Shreyansh]

Hi Mike and List,

Thanks for the reply.

Hmm, that sounds quite a large value as against what I am
experiencing. I will try and find out what is happening in my case -
probably the iscsi-target is acting like some kind of bottleneck (the
max size of PDU is my initial guess) - or may be I am missing
something.

Nevetheless, thanks for the info - i really appreciate your help.

>
>> Are some parameters which I am missing which I need to configure or
>> take care of?
>> I tried tinkering "node.conn[0].iscsi.MaxRecvDataSegmentLength =
>> 65536" but there was no difference. This actually is making me think
>> that target is acting as a bottleneck.
>>
>
> That is just for normal sessions. You would want to set
> discovery.sendtargets.iscsi.MaxRecvDataSegmentLength

Oops - I should have noticed. Thanks once again. I will try tinkering with this.

>
> but like I said that will not completely fix the problem.

Well, this should give me something very good to start with. I will
post whatever my finding are (as soon as I find something ;) ). Thanks
all.

Shreyansh

[Shrey]

Hi,

After a little more experimenting, I think this iscsi-targets fault.

I agree this does not have any relevance to the open-iscsi code,
nevertheless, I though it would nice to post here as well to keep this
post updated. Do apologize if that is against the principles of this
mailing-list.

> Interesting. That does seem like a limitation in that code. Did you
> try to change the INCOMING_BUFSIZE to a higher value? Or to decrease
> the length of the target's name? So: iqn.2008.com:t1, iqn.2008.com:t2,
> and so on?

As suggested by someone on this list earlier, I tried to reduce the
target name and instantly I was able to add (made discoverable) more
of the devices at the initiator side. Initially the target names
consisted of about 44 characters (when only 89 devices were getting
transferred across to initiator). I changed the name such that it took
about 15 characters - and this time I was able to add 139 devices with
the same configuration of IETd. :)

Also, I changed a variable INCOMING_BUFFER (again, as suggested by
Konrad) to 16384 - then I was able to add (discoverable on initiator)
179 devices with target name consisting of 44 chars. :)

By add I mean, rest all the devices defined at target were being
dropped by target and not sent to initiator. I had a target config
list of about 1000 soft links to a valid device.

> On Wed, May 21, 2008 at 12:28 AM, Mike Christie <micha...@cs.wisc.edu> wrote:
> > If you are using 869.* then the problem is most likely on the IET side.
> > Like I said before I think you need to upgrade. I had the same problem
> > when I was testing my fixes for lots of targets support.

This would certainly mean that iscsi-target is playing a game with
me :D

Regards
Shreyansh