hostbyte=DID_TRANSPORT_DISRUPTED: network issues or?

[Mauricio]

      Now I was able to address my issue with the testbox, I can mount the LUN in that host without issues. So it is time to switch back to the problem box, which started having issues since the last reboot. I apply the solution used in the testbox and then restart the service:

[root@problembox ~]# systemctl restart iscsi

[root@problembox ~]#

And it acts like it is happy (so far; did not check dmesg or fdisk):

[root@problembox ~]# systemctl status iscsi

o iscsi.service - Login and scanning of iSCSI devices

   Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; vendor preset: disabled)

   Active: active (exited) since Thu 2021-11-25 23:21:40 EST; 9h ago

     Docs: man:iscsiadm(8)

           man:iscsid(8)

  Process: 3414 ExecStart=/usr/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=0/SUCCESS)

 Main PID: 3414 (code=exited, status=0/SUCCESS)

    Tasks: 0 (limit: 203741)

   Memory: 0B

   CGroup: /system.slice/iscsi.service

Nov 25 23:17:52 problembox systemd[1]: Starting Login and scanning of iSCSI devices...

Nov 25 23:21:40 problembox iscsiadm[3414]: Logging in to [iface: default, target: iqn.2000-01.com.synology-iSCSI:storage.01, portal: 192.168.10.18,3260]

Nov 25 23:21:40 problembox iscsiadm[3414]: Login to [iface: default, target: iqn.2000-01.com.synology-iSCSI:storage.01, portal: 192.168.10.18,3260] successful.

Nov 25 23:21:40 problembox systemd[1]: Started Login and scanning of iSCSI devices.

[root@problembox ~]#

[root@problembox ~]# ls -lh /dev/sd*

brw-rw----. 1 root disk 8,  0 Nov 25 21:42 /dev/sda

brw-rw----. 1 root disk 8,  1 Nov 25 21:42 /dev/sda1

brw-rw----. 1 root disk 8,  2 Nov 25 21:42 /dev/sda2

brw-rw----. 1 root disk 8,  3 Nov 25 21:42 /dev/sda3

brw-rw----. 1 root disk 8, 16 Nov 25 23:33 /dev/sdb

[root@problembox ~]# ls -l /dev/disk/by-path/|grep ip

lrwxrwxrwx. 1 root root  9 Nov 25 23:33 ip-192.168.10.18:3260-iscsi-iqn.2000-01.com.synology-iSCSI:storage.01-lun-0 -> ../../sdb

[root@problembox ~]#

Time to go probe the elephant in the room

[root@problembox ~]# fdisk -l /dev/sdb

fdisk: cannot open /dev/sdb: Input/output error

[root@problembox ~]#

What does dmesg has to tell me? The expected behaviour as seen in the testbox (mounting the very same LUN):

[root@testbox ~]# dmesg -T

[...]

[Thu Nov 25 19:58:00 2021] Loading iSCSI transport class v2.0-870.

[Thu Nov 25 19:58:00 2021] iscsi: registered transport (tcp)

[Thu Nov 25 19:58:00 2021] scsi host2: iSCSI Initiator over TCP/IP

[Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: Direct-Access     SYNOLOGY iSCSI Storage    3.1  PQ: 0 ANSI: 5

[Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: supports implicit TPGS

[Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: device naa.6001405e61f8c59d35fdd4481da3e1d3 port group 1 rel port 1

[Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: Attached scsi generic sg1 type 0

[Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: transition timeout set to 60 seconds

[Thu Nov 25 19:58:00 2021] scsi 2:0:0:0: alua: port group 01 state A non-preferred supports TOlUSNA

[Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] 754974720 512-byte logical blocks: (387 GB/360 GiB)

[Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Write Protect is off

[Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Mode Sense: 3b 00 00 00

[Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

[Thu Nov 25 19:58:00 2021]  sda: sda1

[Thu Nov 25 19:58:00 2021] sd 2:0:0:0: [sda] Attached SCSI disk

[root@testbox ~]#

Behaviour seen in the problembox

[root@problembox ~]# dmesg -T

[Thu Nov 25 23:17:51 2021] scsi host8: iSCSI Initiator over TCP/IP

[Thu Nov 25 23:17:51 2021] scsi 8:0:0:0: Direct-Access     SYNOLOGY iSCSI Storage    3.1  PQ: 0 ANSI: 5

[Thu Nov 25 23:17:51 2021] scsi 8:0:0:0: alua: supports implicit TPGS

[Thu Nov 25 23:17:51 2021] scsi 8:0:0:0: alua: device naa.6001405e61f8c59d35fdd4481da3e1d3 port group 1 rel port 1

[Thu Nov 25 23:17:51 2021] sd 8:0:0:0: Attached scsi generic sg1 type 0

[Thu Nov 25 23:18:02 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4300399244, last ping 4300404736, now 4300409856

[Thu Nov 25 23:18:02 2021]  connection4:0: detected conn error (1022)

[...]

[Thu Nov 25 23:31:56 2021]  connection4:0: detected conn error (1022)

[Thu Nov 25 23:31:56 2021] sd 8:0:0:0: [sdb] tag#76 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=72s

[Thu Nov 25 23:31:56 2021] sd 8:0:0:0: [sdb] tag#76 CDB: Read(10) 28 00 2c ff ff 80 00 00 08 00

[Thu Nov 25 23:31:56 2021] blk_update_request: I/O error, dev sdb, sector 754974592 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

[Thu Nov 25 23:32:10 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4301247437, last ping 4301252608, now 4301257728

[Thu Nov 25 23:32:10 2021]  connection4:0: detected conn error (1022)

[Thu Nov 25 23:32:22 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4301259759, last ping 4301264896, now 4301270016

[Thu Nov 25 23:32:22 2021]  connection4:0: detected conn error (1022)

[Thu Nov 25 23:32:35 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4301273035, last ping 4301278208, now 4301283328

[Thu Nov 25 23:32:35 2021]  connection4:0: detected conn error (1022)

[Thu Nov 25 23:32:48 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4301285359, last ping 4301290496, now 4301295616

[Thu Nov 25 23:32:48 2021]  connection4:0: detected conn error (1022)

[Thu Nov 25 23:33:00 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4301297653, last ping 4301302784, now 4301307904

[Thu Nov 25 23:33:00 2021]  connection4:0: detected conn error (1022)

[Thu Nov 25 23:33:13 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4301310922, last ping 4301316096, now 4301321216

[Thu Nov 25 23:33:13 2021]  connection4:0: detected conn error (1022)

[Thu Nov 25 23:33:13 2021] sd 8:0:0:0: [sdb] tag#81 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=73s

[Thu Nov 25 23:33:13 2021] sd 8:0:0:0: [sdb] tag#81 CDB: Read(10) 28 00 2c ff ff 80 00 00 08 00

[Thu Nov 25 23:33:13 2021] blk_update_request: I/O error, dev sdb, sector 754974592 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

[Thu Nov 25 23:33:13 2021] Buffer I/O error on dev sdb, logical block 94371824, async page read

[Fri Nov 26 00:46:06 2021]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4305683173, last ping 4305688576, now 4305693696

[Fri Nov 26 00:46:06 2021]  connection4:0: detected conn error (1022)

[root@problembox ~]#

Given that problembox is a baremetal server and testbox is a vm guest in a KVM server which is connected to the same switch as problembox, and that problembox has been working fine for 2 years until the very last reboot, are these network issues or something else.

[Ulrich Windl]

>>> Mauricio <raub...@gmail.com> schrieb am 26.11.2021 um 15:52 in Nachricht
<0c84ea13-e5f5-4755...@googlegroups.com>:

Did you notice: You are testing sdb, but the messages above are for sda!

> --
> You received this message because you are subscribed to the Google Groups
> "open-iscsi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to open-iscsi+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/open-iscsi/0c84ea13-e5f5-4755-8f34-3b81dd00
> 406cn%40googlegroups.com.

[The Lee-Man]

Yes, I believe your problems are network-related.

I would advise taking iscsi NOPs off the table -- if you have a slow connection, the error recovery involved in a ping timeout can screw up I/O big time.