Performance issues when logging into a large number of targets

[Syed Mushtaq]

Hi,

We run a small public cloud in Canada and were testing out a VDI-per-LUN type of storage where each virtual disk is mapped to a LUN on the storage array. We have been seeing delays when an iscsi login is made. We use open-iscsi 2.0-873 in our production. I compiled that with profiling support to figure out where the delay is. I would love if someone could help me figure out the performance bottlenecks.


I currently have the following sessions:

[root@ucs-xen1 ~]# iscsiadm -m session | wc -l
798
[root@ucs-xen1 ~]#

If I attach a new LUN the time taken is about 9.3 seconds

[root@ucs-xen1 ~]# time ./iscsiadm -m node -p 172.31.255.160 -T  iqn.2010-01.com.solidfire:giwd.sr-198.3588 -l
Logging in to [iface: default, target: iqn.2010-01.com.solidfire:giwd.sr-198.3588, portal: 172.31.255.160,3260] (multiple)
Login to [iface: default, target: iqn.2010-01.com.solidfire:giwd.sr-198.3588, portal: 172.31.255.160,3260] successful.
Number of recs found 1


real    0m9.312s
user    0m0.088s
sys     0m0.120s
[root@ucs-xen1 ~]#

I am attaching the gmon output here

-Syed

[Mike Christie]

Hey,

Chris Leech has been working on this problem. He has some patches which
help, but I do not see them. Chris?

I am also attaching a hacky patch to remove another problem source with
lots of targets. Could you run that with Chris's patches when he responds?

On 06/07/2016 02:24 PM, Syed Mushtaq wrote:
>
>
> I am attaching the gmon output here
>

This was taken on iscsiadm right? Not on iscsid?

[Syed Mushtaq]

Thanks for the patch Mike. Will test it with this and let you know if I see some improvement. And yes the gmon output is for iscsiadm and not iscsid.

-Syed

--
You received this message because you are subscribed to a topic in the Google Groups "open-iscsi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/open-iscsi/XzT3IXcHlcs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to open-iscsi+...@googlegroups.com.
To post to this group, send email to open-...@googlegroups.com.
Visit this group at https://groups.google.com/group/open-iscsi.
For more options, visit https://groups.google.com/d/optout.

[Michael Christie]

Wait to test with Chris's patches. If he does not reply I will dig them up from the list. His fixes should provide the best boost and I think he was saying there are some other patches that are needed.

You received this message because you are subscribed to the Google Groups "open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+...@googlegroups.com.

[Mike Christie]

I think Chris is busy. Here is a link to his tree

https://github.com/cleech/open-iscsi

that has patches for large number of targets. Apply my patch over it and
try it out.

Do not do it on your production systems.

On 06/08/2016 05:48 PM, Michael Christie wrote:
> Wait to test with Chris's patches. If he does not reply I will dig them
> up from the list. His fixes should provide the best boost and I think he
> was saying there are some other patches that are needed.
>
> On Jun 8, 2016, at 2:05 PM, Syed Mushtaq <syed1....@gmail.com

>> <mailto:open-iscsi%2Bunsu...@googlegroups.com>.

>> To post to this group, send email to open-...@googlegroups.com

>> <mailto:open-...@googlegroups.com>.

>> Visit this group at https://groups.google.com/group/open-iscsi.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "open-iscsi" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to open-iscsi+...@googlegroups.com

>> <mailto:open-iscsi+...@googlegroups.com>.

>> To post to this group, send email to open-...@googlegroups.com

>> <mailto:open-...@googlegroups.com>.

>> Visit this group at https://groups.google.com/group/open-iscsi.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "open-iscsi" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to open-iscsi+...@googlegroups.com

> <mailto:open-iscsi+...@googlegroups.com>.

> To post to this group, send email to open-...@googlegroups.com

> <mailto:open-...@googlegroups.com>.

[Syed Mushtaq]

Thanks Mike. I'm testing all of this in my lab right now. I will apply the patches from Chris and let you know if I see performance improvements.

-Syed

To unsubscribe from this group and all its topics, send an email to open-iscsi+...@googlegroups.com.
To post to this group, send email to open-...@googlegroups.com.

[Chris Leech]

Sorry about that, last week was a busy one for me. Removing the sysfs
attr_list "cache" should be the biggest help for this that I've seen,
and that patch is in my github tree that Mike pointed to.

Looking forward to hearing if it helps in your situation.

- Chris

> To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+...@googlegroups.com.