iSCSI connection recovery timeout

[KUMAR NITISH]

Hi,

As I understand, below 3 timeouts contribute to the total time taken for marking an iSCSI connection unavailable for IOs.

noop_out_interval
noop_out_timeout
replacement_timeout 

/* timeouts in seconds */
#define DEF_LOGIN_TIMEO         15
#define DEF_LOGOUT_TIMEO        15
#define DEF_NOOP_OUT_INTERVAL   10
#define DEF_NOOP_OUT_TIMEO      15
#define DEF_REPLACEMENT_TIMEO   120

Can someone please explain why default values are higher? Do we really need to have these higher default values? Are these default values reduced from earlier open-iscsi versions to newer versions? If not, should these values get reduced significantly for the fact that newer network devices are faster?

Will higher values not cause the more time for IOs to failover to other available paths? What are the options for faster failover?


Thanks,
Nitish

[The Lee-Man]

Timeout settings, in general, are a bag of worms. There are no "right" settings because every installation can be different.

If you read (and re-read) the README file that comes with open-iscsi, and the iscsid.conf configuration file, they detail how some of these settings might change when using multipath, or using iSCSI as your root device.

The NOOP settings are evil, IMHO. I have NOOPs disabled for the distribution I support (SUSE/tumbleweed). It's a long story, but NOOPs are not implemented well IMHO, since they get mixed in with the rest of the I/O load. On a busy system, a NOOP may not go out right away ... it may be sitting in a queue. Bottom line, false positives (timeouts) can occur. When a NOOP timeout occurs, the connection gets reset. If I/O is occuring that just makes things worse! 

The login and logout timers are kind of self-explanatory. The replacement timeout is more complicated, and is mentioned in detail in the documents I mentioned.

One could spent a bit of time playing with these values, but I caution against making them too short, as this may cause false positives, especially during heavy I/O, or if you target server is busy.

Mike Christie may have more to say on these, and he's played with them far more than I have.