iscsi target process not starting after upgrade to 14.4

[Kaya Saman]

Hi all,


I've got a strange problem with one of my systems after upgrading from
13.x to 14.4.


I have found this bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271460


which describes possibly the problem (maybe in a different context) and
a fix at least.


I don't have 2x ctld processes or the setup described but what I'm
finding is that the LUNs are inaccessible and showing as "connection
refused" on the Initiator node after a startup or reboot of the Target.


It's almost like the ctld process is hung or fails to start properly,
there is definitely a pid file and I can see output using the command:


ctladm portlist -v


Port Online Frontend Name     pp vp
0    NO     camsim   camsim   0  0  naa.5000000c046ef301
  Target: naa.5000000c046ef300
  All LUNs mapped
      port_type=8
1    YES    ioctl    ioctl    0  0
  All LUNs mapped
      port_type=4
2    YES    tpc      tpc      0  0
  All LUNs mapped
      port_type=8
3    YES    iscsi    iscsi    257 1 iqn.2008-10.com.xxx:target0,t,0x0101
  Target: iqn.2008-10.com.xxx:target0
  Initiator 0: iqn.1994-09.org.freebsd:xxx,i,0x8063a7ae15f0
  LUN 0: 0
      port_type=16
      cfiscsi_state=1
      cfiscsi_target=iqn.2008-10.com.xxx:target0
      ctld_portal_group_name=pg0
      cfiscsi_portal_group_tag=257
4    YES    iscsi    iscsi    257 2 iqn.2008-10.com.xxx:target1,t,0x0101
  Target: iqn.2008-10.com.xxx:target1
  Initiator 0: iqn.1994-09.org.freebsd:xxx,i,0x80e4e737c577
  LUN 1: 1
      port_type=16
      cfiscsi_state=1
      cfiscsi_target=iqn.2008-10.com.xxx:target1
      ctld_portal_group_name=pg0
      cfiscsi_portal_group_tag=257
5    YES    iscsi    iscsi    257 3 iqn.2008-10.com.xxx:target2,t,0x0101
  Target: iqn.2008-10.com.xxx:target2
  Initiator 0: iqn.1994-09.org.freebsd:.xxx.com,i,0x8047f6cf2784
  LUN 3: 2
      port_type=16
      cfiscsi_state=1
      cfiscsi_target=iqn.2008-10.com.xxx:target2
      ctld_portal_group_name=pg0
      cfiscsi_portal_group_tag=257
6    YES    iscsi    iscsi    257 4 iqn.2008-10.com.xxx:target3,t,0x0101
  Target: iqn.2008-10.com.xxx:target3
  Initiator 0: iqn.1994-09.org.freebsd:.xxx.com,i,0x80109ffe486f
  LUN 4: 3
      port_type=16
      cfiscsi_state=1
      cfiscsi_target=iqn.2008-10.com.xxx:target3
      ctld_portal_group_name=pg0
      cfiscsi_portal_group_tag=257


If I restart the ctld process, it fails to restart.


I end up having to use this command:


ctladm port -d iscsi -r -p DONTCARE -O cfiscsi_portal_group_tag=TAG -O
cfiscsi_target=TARGET


to remove each LUN then finally I can start the process and it works as
per usual.


One of the comments by Alan Somers was:


"I've confirmed that the cause of the problem is that ctld opens its
pidfile too late.  It reads the current list of targets from the kernel,
then reads the config file, then opens its pidfile, and then applies
changes based on the differences between the kernel's state and the
config file.  But the kernel's state could've changed before the pidfile
got opened.

I've hacked ctld to open the pidfile earlier and verified that this
fixes the problem.  However, doing it properly is hard, because the code
for opening the config file is intermingled with the code for
interacting with the kernel.  The biggest problem is the conf_pports
list, added in 057abcb00413010898f3046f7704444b8f537bab ."


....but I'm wondering if this is the case as ctld was working fine
before the upgrade and also works fine post upgrade on other systems
that are Targets and have been upgraded to 14.4, so it's an isolated
case on a single system.


I'm wondering if perhaps a certain file got corrupted upon merger during
the "upgrade" process or perhaps as the system had ZFS "snapshots"
installed something got rolled back somehow, even though I removed all
the snapshots post upgrade.


Can anyone suggest anything?


Many thanks.


Kaya