iscsi disk become read only

[hemu]

Hi,

we are using virttual machine as NFS running cent os 5.0 & once in a
week week the iscsi lun becomes read only the log from cent os is
attached please suggest

Hemant

Jun 12 23:07:43 NFSMAIL kernel: session1: session recovery timed out
after 120 secs
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 ff c7 ed 50 00 00 10 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 4291292496
Jun 12 23:07:43 NFSMAIL kernel: Aborting journal on device sdb1-8.
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 00 00 08 00 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2048
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 0
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 00 00 0a 00 00 00 10 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2560
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 64
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 65
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 7d c0 09 58 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2109737304
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 263716907
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 7e c0 08 88 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2126514312
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 265814033
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 7e c0 09 18 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2126514456
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 265814051
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 7e c0 1c b8 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2126519480
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 265814679
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 7e c1 0a d8 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2126580440
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 265822299
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 81 00 09 e8 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2164263400
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 270532669
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10):
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:43 NFSMAIL kernel: 2a 00 81 c0 08 28 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2176845864
Jun 12 23:07:43 NFSMAIL kernel: Buffer I/O error on device sdb1,
logical block 272105477
Jun 12 23:07:43 NFSMAIL kernel: lost page write due to I/O error on
sdb1
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:43 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a
00 ff c4 08 00 00 00 08 00
Jun 12 23:07:43 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 4291037184
Jun 12 23:07:43 NFSMAIL kernel: JBD2: I/O error detected when updating
journal superblock for sdb1-8.
Jun 12 23:07:43 NFSMAIL kernel: journal commit I/O error
Jun 12 23:07:43 NFSMAIL kernel: journal commit I/O error
Jun 12 23:07:43 NFSMAIL kernel: journal commit I/O error
Jun 12 23:07:43 NFSMAIL kernel: EXT4-fs error (device sdb1):
ext4_journal_start_sb: Detected aborted journal
Jun 12 23:07:43 NFSMAIL kernel: EXT4-fs (sdb1): Remounting filesystem
read-only
Jun 12 23:07:46 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(No route to host)
Jun 12 23:07:52 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(No route to host)
Jun 12 23:07:52 NFSMAIL kernel: sd 3:0:0:0: [sdb] Unhandled error code
Jun 12 23:07:52 NFSMAIL kernel: sd 3:0:0:0: [sdb] Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jun 12 23:07:52 NFSMAIL kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00
81 01 12 10 00 00 08 00
Jun 12 23:07:52 NFSMAIL kernel: end_request: recoverable transport
error, dev sdb, sector 2164331024
Jun 12 23:07:52 NFSMAIL kernel: EXT4-fs error (device sdb1):
ext4_find_entry: reading directory #67633443 offset 0
Jun 12 23:07:58 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(No route to host)
Jun 12 23:08:04 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(No route to host)
Jun 12 23:08:10 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(No route to host)
Jun 12 23:08:13 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(Connection refused)
Jun 12 23:08:17 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(Connection refused)
Jun 12 23:08:21 NFSMAIL iscsid: connect to 192.168.2.11:3260 failed
(Connection refused)

[Mike Christie]

On 06/13/2012 02:19 AM, hemu wrote:
> Hi,
>
> we are using virttual machine as NFS running cent os 5.0 & once in a
> week week the iscsi lun becomes read only the log from cent os is
> attached please suggest
>
> Hemant
>
> Jun 12 23:07:43 NFSMAIL kernel: session1: session recovery timed out
> after 120 secs

Something happens to your network or target and we cannot connect to it
for over 2 minutes (the node.session.timeo.replacement_timeout). When
node.session.timeo.replacement_timeout seconds goes by the iscsi layer
will fail IO that is queued. And from there you will get IO errors up
the stack.

If this happens a lot you should use something like dm-multipath.

You also at the very least, if not using dm-multipath, increase the
node.session.timeo.replacement_timeout.

It looks like we are able to reach the target again at 23:08:17. After
you see those messages where we can at least reach the target but are
getting access issues, are we able to connect to the target again?