Logout target mounted as root

[Cheng Cheng]

Hi,

I am currently use iscsistart to mount the root filesystem in Debian. However, the system will fail to reboot due to failing to logout target. If I force to logout the target while umount rootfs, the system will not be able to read "reboot" as it resides on the iscsi volume.

Now I during rebooting (runlevel 6), I change the root filesystem to a ramfs with iscsiadm and related libraries loaded. However, after I chroot into the ramfs, I am unable to logout the target. I already have proc, sysfs, dev mounted to ramfs, and I can use "iscsiadm --mode session" to list connect session. But logout just fails. 

Can anyone share some hints into this issue?

Thanks!

Cheng

[Mike Christie]

On 4/24/14, 10:10 PM, Cheng Cheng wrote:
> Hi,
>
> I am currently use iscsistart to mount the root filesystem in Debian.

I think unbuntu supports iscsi boot/root now. You should check out how
they did it since it should be close to how you can do it in debian right?

> However, the system will fail to reboot due to failing to logout target. If
> I force to logout the target while umount rootfs, the system will not be
> able to read "reboot" as it resides on the iscsi volume.
>
> Now I during rebooting (runlevel 6), I change the root filesystem to a
> ramfs with iscsiadm and related libraries loaded. However, after I chroot
> into the ramfs, I am unable to logout the target. I already have proc,
> sysfs, dev mounted to ramfs, and I can use "iscsiadm --mode session" to
> list connect session. But logout just fails.
>

You should not have to logout the session. For Red Hat and I think SLES,
when the system is shutting down the root FS gets mounted as read-only.
iSCSI is left running. Eventually /sbin/shutdown is run and from there
the kernel is shutdown and if needed the scsi layer might have the iscsi
layer send some commands like a sync cache. So for this you just need to
leave the network up.

[Cheng Cheng]

Hi Mike,

Thanks for replying. 

You should not have to logout the session. For Red Hat and I think SLES, when the system is shutting down the root FS gets mounted as read-only. iSCSI is left running. Eventually /sbin/shutdown is run and from there the kernel is shutdown and if needed the scsi layer might have the iscsi layer send some commands like a sync cache. So for this you just need to leave the network up.

I did some research today around this. For Debian, yes, the root FS will be remounted as read-only before shutting down/rebooting. However, the system will then stuck at sync cache, even I have left network running. I have not looked into the Linux kernel to check whether iscsi layer will send logout command to the target, but I highly suspect it won’t. 

Here is what I am going to propose:

    1. I am now trying to extend iscsistart to be capable send logout command.

    2. In runlevel 0 and runlevel 6, after remounting the root FS as read-only, use pivot_root and chroot to change the root to a ramfs, similar to initramfs, which is loaded with iscsistart. Then we can logout gracefully by invoking iscsistart.

Please let me know what do you think of this.

Sincerely,

Cheng

[Mike Christie]

On 04/25/2014 11:54 PM, Cheng Cheng wrote:
> Hi Mike,
>
> Thanks for replying.
>
>> You should not have to logout the session. For Red Hat and I think
>> SLES, when the system is shutting down the root FS gets mounted as
>> read-only. iSCSI is left running. Eventually /sbin/shutdown is run and
>> from there the kernel is shutdown and if needed the scsi layer might
>> have the iscsi layer send some commands like a sync cache. So for this
>> you just need to leave the network up.
>
> I did some research today around this. For Debian, yes, the root FS will
> be remounted as read-only before shutting down/rebooting. However, the
> system will then stuck at sync cache, even I have left network running.
> I have not looked into the Linux kernel to check whether iscsi layer
> will send logout command to the target, but I highly suspect it won’t.

It will not logout of the target at this time. It cannot because
userspace is shutdown and iscsid is what sends the logout request.

A logout is *not* needed though. It is not what is causing your problem.

For RHEL/OEL and SUSE we do not send a logout and it works fine.

>
> Here is what I am going to propose:
>
> 1. I am now trying to extend iscsistart to be capable send logout
> command.
> 2. In runlevel 0 and runlevel 6, after remounting the root FS as
> read-only, use pivot_root and chroot to change the root to a ramfs,
> similar to initramfs, which is loaded with iscsistart. Then we can
> logout gracefully by invoking iscsistart.
>
> Please let me know what do you think of this.
>

It is nice to logout but it is not going to fix your problem.

If you are getting stuck at the sync cache stage then your network
probably went down. /sbin/shutdown probably shutdown the network on you.

[Mike Christie]

When you hit this problem, on the console around the time you see the
sync cache sent do you then see a conn error 1011 error message?

If you reconfig the cache settings on the target so a sync cache is not
needed do you hit this problem?

[Mike Christie]

As another test/workaround try deleting the scsi disks while the network
is still up. So you are running in your root ramfs, then do

echo 1 > /sys/block/sdXYZ/delete

for the iscsi disks. Then let the kernel shutdown.

[Cheng Cheng]

Hi Mike,

>>> You should not have to logout the session. For Red Hat and I think
>>> SLES, when the system is shutting down the root FS gets mounted as
>>> read-only. iSCSI is left running. Eventually /sbin/shutdown is run and
>>> from there the kernel is shutdown and if needed the scsi layer might
>>> have the iscsi layer send some commands like a sync cache. So for this
>>> you just need to leave the network up.
>>
>> I did some research today around this. For Debian, yes, the root FS will
>> be remounted as read-only before shutting down/rebooting. However, the
>> system will then stuck at sync cache, even I have left network running.
>> I have not looked into the Linux kernel to check whether iscsi layer
>> will send logout command to the target, but I highly suspect it won’t.
>
> It will not logout of the target at this time. It cannot because
> userspace is shutdown and iscsid is what sends the logout request.
>
> A logout is *not* needed though. It is not what is causing your problem.
>
> For RHEL/OEL and SUSE we do not send a logout and it works fine.

The reason I need the logout operation is not only limited to a gracefully shutdown. I also want to have the tgt side have a clean record for the target.

>> Here is what I am going to propose:
>>
>> 1. I am now trying to extend iscsistart to be capable send logout
>> command.
>> 2. In runlevel 0 and runlevel 6, after remounting the root FS as
>> read-only, use pivot_root and chroot to change the root to a ramfs,
>> similar to initramfs, which is loaded with iscsistart. Then we can
>> logout gracefully by invoking iscsistart.
>>
>> Please let me know what do you think of this.
>>
>
> It is nice to logout but it is not going to fix your problem.
>
> If you are getting stuck at the sync cache stage then your network
> probably went down. /sbin/shutdown probably shutdown the network on you.

I guess so. Before hitting /sbin/shutdown and right after remount root FS as read-only, the network is still up. So I guess /sbin/shutdown might be the problem.

Cheng

[Cheng Cheng]

It will not logout of the target at this time. It cannot because
userspace is shutdown and iscsid is what sends the logout request.

Do you mean even I extend iscsistart to handle logout and run from a ramfs, I still cannot sends the logout during shutting down? My understanding is that iscsistart is self-contained, and it will start the child process as iscsid for handling ipc calls.

Cheng

[Mike Christie]

On 04/26/2014 01:14 AM, Cheng Cheng wrote:
> Hi Mike,
>
>>>> You should not have to logout the session. For Red Hat and I think
>>>> SLES, when the system is shutting down the root FS gets mounted as
>>>> read-only. iSCSI is left running. Eventually /sbin/shutdown is run and
>>>> from there the kernel is shutdown and if needed the scsi layer might
>>>> have the iscsi layer send some commands like a sync cache. So for this
>>>> you just need to leave the network up.
>>>
>>> I did some research today around this. For Debian, yes, the root FS will
>>> be remounted as read-only before shutting down/rebooting. However, the
>>> system will then stuck at sync cache, even I have left network running.
>>> I have not looked into the Linux kernel to check whether iscsi layer
>>> will send logout command to the target, but I highly suspect it won't.
>>
>> It will not logout of the target at this time. It cannot because
>> userspace is shutdown and iscsid is what sends the logout request.
>>
>> A logout is *not* needed though. It is not what is causing your problem.
>>
>> For RHEL/OEL and SUSE we do not send a logout and it works fine.
>
> The reason I need the logout operation is not only limited to a gracefully shutdown. I also want to have the tgt side have a clean record for the target.
>

If you want to send a patch for iscsistart then that would be ok.

[Cheng Cheng]

Okay, then I will try to work on iscsistart this weekend for logout. This problem should be specific to the case of using iscsi volume as root FS.

Thanks,
Cheng

[Mike Christie]

On 04/26/2014 01:18 AM, Cheng Cheng wrote:
>
>> It will not logout of the target at this time. It cannot because
>> userspace is shutdown and iscsid is what sends the logout request.
>
> Do you mean even I extend iscsistart to handle logout and run from a
> ramfs, I still cannot sends the logout during shutting down? My

No. I mean as is. If you modify iscsistart so it does something similar
for logout as it does for login it should work ok. There might be some
tiny issues where some targets were disconnecting the connection before
sending the logout response. I am not 100% sure how that will work off
the top of my head.

[Ritesh Raj Sarraf]

The Root on iSCSI setup should work fine on Debian too. Just mark your
root device "_netdev" also. And as Mike said, we don't bother much about
logging out all session at shutdown. We need to ensure the last data
bits to get synced.



--
Given the large number of mailing lists I follow, I request you to CC me
in replies for quicker response