Re: How to recover iscsi connection if LAN switch goes down/failed for 1 hour or above

[Donald Williams]

Hello, 

What you were seeing was the stale mount info.   No "disk" is going to survive an hour disconnected.  Even a short disconnect will cause a SCSI disk error and Linux to remount the volume RO.   

 Best practices for iSCSI connectivity is redundant switches and configure MPIO to use both paths when up for better performance and re-route I/O when a path fails. 

 Regards, 

 Don 

On Thu, Apr 4, 2013 at 6:11 AM, parveen kumar <coolbu...@gmail.com> wrote:

I tried to find out for how many seconds/hours iscsi session try to re-establish the old connection if LAN disconnects between Host and Storage (or you can say for how many seconds/hours iscsi will try for login in target). 

In my case i had mounted the 1TB partition on my CentOS release 5.3 (Final) host/server from storage.

i am using that 1TB volume and able to write data on it.

Today i switched off the LAN switch for 1 hour, and after 1 hour i am able to see the iscsi partition mounted on my host/server and its also showing in "fdisk -l" but m not able to write data  no it :o

=========================================================

Below error when i switch off the LAN switch (the output of /var/log/messages)

=========================================================

Apr  4 13:51:15 master kernel: ping timeout of 5 secs expired, last rx 4559435595, last ping 4559440595, now 4559445595

Apr  4 13:51:15 master kernel:  connection1:0: iscsi: detected conn error (1011)

Apr  4 13:51:16 master iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)

Apr  4 13:51:18 master /usr/sbin/gmetad[4843]: data_thread() got not answer from any [cluster] datasource

Apr  4 13:51:58 master last message repeated 3 times

Apr  4 13:53:13 master last message repeated 4 times

Apr  4 13:53:16 master kernel:  session1: iscsi: session recovery timed out after 120 secs

Apr  4 13:53:16 master kernel: iscsi: cmd 0x2a is not queued (8)

Apr  4 13:53:16 master last message repeated 17 times

Apr  4 13:53:16 master kernel: iscsi: cmd 0x28 is not queued (8)

Apr  4 13:53:16 master kernel: iscsi: cmd 0x28 is not queued (8)

Apr  4 13:53:16 master kernel: iscsi: cmd 0x2a is not queued (8)

Apr  4 13:53:16 master last message repeated 3 times

Apr  4 13:53:16 master kernel: sd 14:0:0:6: SCSI error: return code = 0x00010000

Apr  4 13:53:16 master kernel: end_request: I/O error, dev sde, sector 531628095

Apr  4 13:53:16 master kernel: Buffer I/O error on device sde1, logical block 66453504

Apr  4 13:53:16 master kernel: lost page write due to I/O error on sde1

Apr  4 13:53:16 master kernel: iscsi: cmd 0x2a is not queued (8)

Apr  4 13:53:16 master last message repeated 2 times

Apr  4 13:53:16 master kernel: iscsi: cmd 0x28 is not queued (8)

Apr  4 13:53:16 master last message repeated 4 times

Apr  4 13:53:16 master kernel: iscsi: cmd 0x2a is not queued (8)

Apr  4 13:53:16 master kernel: sd 14:0:0:6: SCSI error: return code = 0x00010000

Apr  4 13:53:16 master kernel: end_request: I/O error, dev sde, sector 475005183

Apr  4 13:53:16 master kernel: Buffer I/O error on device sde1, logical block 59375640

Apr  4 13:53:16 master kernel: lost page write due to I/O error on sde1

Apr  4 13:53:16 master kernel: iscsi: cmd 0x2a is not queued (8)

Apr  4 13:53:16 master kernel: sd 14:0:0:6: SCSI error: return code = 0x00010000

Apr  4 13:53:16 master kernel: end_request: I/O error, dev sde, sector 475005175

Apr  4 13:53:16 master kernel: Buffer I/O error on device sde1, logical block 59375639

Apr  4 13:53:16 master kernel: lost page write due to I/O error on sde1

Apr  4 13:53:16 master kernel: iscsi: cmd 0x2a is not queued (8)

Apr  4 13:53:16 master kernel: sd 14:0:0:6: SCSI error: return code = 0x00010000

Apr  4 13:53:16 master kernel: end_request: I/O error, dev sde, sector 475005135

Apr  4 13:53:16 master kernel: Buffer I/O error on device sde1, logical block 59375634

Apr  4 13:53:16 master kernel: lost page write due to I/O error on sde1

Apr  4 13:53:16 master kernel: Buffer I/O error on device sde1, logical block 59375635

Apr  4 13:53:16 master kernel: lost page write due to I/O error on sde1

Apr  4 13:53:16 master kernel: Buffer I/O error on device sde1, logical block 59375636

Apr  4 13:53:16 master kernel: lost page write due to I/O error on sde1

===================================================

"dmesg" output when switch off the LAN switch

===================================================

iscsi: cmd 0x28 is not queued (6)

iscsi: cmd 0x28 is not queued (6)

sd 14:0:0:6: SCSI error: return code = 0x00010000

end_request: I/O error, dev sde, sector 633227487

sd 14:0:0:6: SCSI error: return code = 0x00010000

sd 14:0:0:6: rejecting I/O to device being removed

end_request: I/O error, dev sde, sector 633228031

Is there any parameter in iscsi file need to set that relogin/re-establish the session automatically whenever LAN switch comes up????

Thanks in advance.

====================================================================================================

--
You received this message because you are subscribed to the Google Groups "open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+...@googlegroups.com.
To post to this group, send email to open-...@googlegroups.com.
Visit this group at http://groups.google.com/group/open-iscsi?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mike Christie]

On 04/04/2013 05:11 AM, parveen kumar wrote:
> Apr 4 13:53:16 master kernel: session1: iscsi: session recovery timed
> out after 120 secs

You will want to update from Centos 5.3 to 5.7 to get some bug fixes.

For the 1 hour question, you want to set the
replacement_timeout/recovery_timeout higher. It is set to 120 secs by
default. There is more info the
/usr/share/doc/iscsi-initiator-utils*/REAMDE.


#
# See the iSCSI REAME's Advanced Configuration section for tips
# on setting timeouts when using multipath or doing root over iSCSI.
#
# To specify the length of time to wait for session re-establishment
# before failing SCSI commands back to the application when running
# the Linux SCSI Layer error handler, edit the line.
# The value is in seconds and the default is 120 seconds.
# Special values:
# - If the value is 0, IO will be failed immediately.
# - If the value is less than 0, IO will remain queued until the session
# is logged back in, or until the user runs the logout command.
node.session.timeo.replacement_timeout = 120

[Mike Christie]

Oh yeah, what you really want though is to use dm-multipath if you are
going to hit long down periods. With that you can also use the
queue_if_no_path feature with the multipath-tools.