Large Immediate and/or Unsolicted Data causes long delays on R2T responses
18 views
Skip to first unread message
ajhu...@gmail.com
May 2, 2020, 2:30:27 PM5/2/20
to open-iscsi
I am able to create a condition where the open-iscsi initiator fails to respond to an R2T request if the immediate/unsolicited data support is large ~128KB. I've seen instances where a delay on an R2T is only a few seconds and other instances where no response is received in 180 seconds.
If the host is doing a prefill operation with large writes that can be completed with immediate data alone and a large write that requires an R2T is sent, the open-iscsi initiator sometimes fails to respond to the target's R2T.
After inspecting the code, I am convinced it is caused by the lack of fairness in the libiscsi iscsi_data_xmit routine, which always favors the sending a new command over responding to R2Ts.
/** * iscsi_data_xmit - xmit any command into the scheduled connection * @conn: iscsi connection * * Notes: * The function can return -EAGAIN in which case the caller must * re-schedule it again later or recover. '0' return code means * successful xmit. **/static int iscsi_data_xmit(struct iscsi_conn *conn){... /* * process mgmt pdus like nops before commands since we should * only have one nop-out as a ping from us and targets should not * overflow us with nop-ins */ while (!list_empty(&conn->mgmtqueue)) {...
/* process pending command queue */ while (!list_empty(&conn->cmdqueue)) {... while (!list_empty(&conn->requeue)) {
Am I looking at this code correctly? I guess this order might be better for parallelization at the target by getting more commands onboard before responding to outstanding R2Ts. With immediate/unsolicited data enabled, the overhead of transmitting a new commands if higher and probably shouldn't come before responding to R2Ts.
The Lee-Man
Jun 29, 2020, 2:59:39 PM6/29/20
to open-iscsi
Do you have NOPs enabled? If so, do you see this issue with them disabled? I seriously dislike and advise against NOPs. I've never seen them actually help anything.
Have you tried playing with this code, i.e. changing the order? Without looking deeply, are the R2Ts in the command queue and not in the requeue queue?
What kind of load are you presenting to the server?
What do you mean by "the immediate/unsolicited data support is large ~128KB"? What setting(s) did you change?
Ulrich Windl
Jun 30, 2020, 2:36:18 AM6/30/20
to open-iscsi
>>> The Lee-Man <leeman...@gmail.com> schrieb am 29.06.2020 um 20:59 in
Nachricht
<4792_1593457181_5EFA3A1C_4792_1489_1_4c70b62c-467c-4860-a951-663fb88158c7o@goog
egroups.com>:
>
> Have you tried playing with this code, i.e. changing the order? Without
> looking deeply, are the R2Ts in the command queue and not in the requeue
> queue?
>
> What kind of load are you presenting to the server?
>
> What do you mean by "the immediate/unsolicited data support is large
> ~128KB"? What setting(s) did you change?
>
> --
> You received this message because you are subscribed to the Google Groups
> "open-iscsi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to open-iscsi+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/open-iscsi/4c70b62c-467c-4860-a951-663fb881
> 58c7o%40googlegroups.com.
Nachricht
<4792_1593457181_5EFA3A1C_4792_1489_1_4c70b62c-467c-4860-a951-663fb88158c7o@goog
egroups.com>:
>> I am able to create a condition where the open-iscsi initiator fails to
>> respond to an R2T request if the immediate/unsolicited data support is
>> large ~128KB. I've seen instances where a delay on an R2T is only a few
>> seconds and other instances where no response is received in 180 seconds.
>>
>> If the host is doing a prefill operation with large writes that can be
>> completed with immediate data alone and a large write that requires an R2T
>> is sent, the open-iscsi initiator sometimes fails to respond to the
>> target's R2T.
>>
>> After inspecting the code, I am convinced it is caused by the lack of
>> fairness in the *libiscsi **iscsi_data_xmit* routine, which always
>> respond to an R2T request if the immediate/unsolicited data support is
>> large ~128KB. I've seen instances where a delay on an R2T is only a few
>> seconds and other instances where no response is received in 180 seconds.
>>
>> If the host is doing a prefill operation with large writes that can be
>> completed with immediate data alone and a large write that requires an R2T
>> is sent, the open-iscsi initiator sometimes fails to respond to the
>> target's R2T.
>>
>> After inspecting the code, I am convinced it is caused by the lack of
>> Do you have NOPs enabled? If so, do you see this issue with them disabled?
> I seriously dislike and advise against NOPs. I've never seen them actually
> help anything.
They could keep an connection alive that is passing through some firewall when there is no iSCSI I/O, maybe.
> I seriously dislike and advise against NOPs. I've never seen them actually
> help anything.
>
> Have you tried playing with this code, i.e. changing the order? Without
> looking deeply, are the R2Ts in the command queue and not in the requeue
> queue?
>
> What kind of load are you presenting to the server?
>
> What do you mean by "the immediate/unsolicited data support is large
> ~128KB"? What setting(s) did you change?
>
> You received this message because you are subscribed to the Google Groups
> "open-iscsi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to open-iscsi+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/open-iscsi/4c70b62c-467c-4860-a951-663fb881
> 58c7o%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages