Re: Automatic deleting of LUNs possible?

[Michael Christie]

On Oct 30, 2012, at 7:31 PM, amit....@gmail.com wrote:

>
> I apologize if this has been asked in the list before, I did some
> research but did not find a clear answer/recommendation.
>
> Currently, I use udev to monitor new devices added by the open-iscsi
> initiator. This works great. However, when the target is disconnected I
> get timeout messages in the kernel log but the device nodes in /dev/
> still exists. Thus, no udev events.
>
> As noted in a few of the messages in the archives, doing a rescan does
> not automatically delete the LUNs. So the following script was
> suggested:
>
> scsi_sz=`sg_readcap /dev/sdko | grep address | sed s'/.*blocks=//'`
> kernel_sz=`cat /sys/block/sdko/device/block\:sdko/size`
>
> if (($scsi_sz==$kernel_sz)); then
> echo "Good."
> else
> echo "Do the deletion of the block device."
> echo 1 > /sys/block/sdko/device/delete
> fi
>
> This looks like it will do the trick. But I am wondering if this is still
> the recommended approach as the above suggestion was made in 2008.

I do not know about recommended, but it will work, and we have not implemented something like dev_loss_tmo FC behavior so it is one of the only options.

[Amit]

Michael Christie <michaelc@...> writes:
>
>
> On Oct 30, 2012, at 7:31 PM, amit.uttam@... wrote:
>

[snip]

> >
> > scsi_sz=`sg_readcap /dev/sdko | grep address | sed s'/.*blocks=//'`
> > kernel_sz=`cat /sys/block/sdko/device/block\:sdko/size`
> >
> > if (($scsi_sz==$kernel_sz)); then
> > echo "Good."
> > else
> > echo "Do the deletion of the block device."
> > echo 1 > /sys/block/sdko/device/delete
> > fi

> I do not know about recommended, but it will work, and we have not
> implemented something like dev_loss_tmo FC behavior so it is one of
> the only options.
>

Thanks for the response. I tried the above but it doesn't seem to work.
I still get the same size for the disk. Here is the steps I took:

1. Initiate a connection to target

$ cat /sys/block/sdc/device/block/sdc/size
4395405168

2. Poweroff the target

Messages in syslog:
11:37:43 kernel: [ 1292.202692] connection2:0: detected conn error(1020)
11:37:44 iscsid: Kernel reported iSCSI connection 2:0 error (1020) state (3)
11:38:19 iscsid: connect to 192.168.1.115:3260 failed (No route to host)
11:39:01 iscsid: last message repeated 6 times

No udev events as seen by udevadm monitor

3. Perform sg_readcap and readsize

$ sudo sg_readcap -v /dev/sdc | grep address | sed s'/.*blocks=//'
read capacity (10) cdb: 25 00 00 00 00 00 00 00 00 00
read capacity (10): transport: Host_status=0x0f is invalid
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]

$ cat /sys/block/sdc/device/block/sdc/size
4395405168

So the size is still the same. This is on kernel 3.6.0 on the target and
3.2.14 on the initiator.

Regarding dev_loss_tmo, this has to be implemented at the kernel level
correct? Since this feature is important to me, I can go ahead and try
to implement that.

Thanks,
Amit

[Boaz Harrosh]

Mike hi

Did we ever consider integrating iscsi with the hotplug infrastructure?

It could be nice if an admin could specify at configuration time if he
wants the device permanently attached but offline on error like today,
or if he wants an hot-unplug notification like a usb device.

Thanks
Boaz

[Mike Christie]

Yeah, I updated the iscsi dev loss tmo patch and posted it to the list.
I think I hit some bugs with lots of sessions and lots of removals at
the same time due to some workqueue/threading stuff or scsi locking
issues. I did not get a chance to look into more.

Before reposting I wanted to make the dev loss code generic and hook all
transport into it since srp and sas wanted it too.

[Mike Christie]

This just indicates there is n initial problem. During this time the
iscsi layer will not yet fail the device. When the recovery/replacement
timeout fires and you see this

session recovery timed out after %d secs

is when the iscsi layer will fail IO.

> No udev events as seen by udevadm monitor
>
> 3. Perform sg_readcap and readsize
>
> $ sudo sg_readcap -v /dev/sdc | grep address | sed s'/.*blocks=//'
> read capacity (10) cdb: 25 00 00 00 00 00 00 00 00 00
> read capacity (10): transport: Host_status=0x0f is invalid
> Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]
>

I am not sure if you are checking for the right thing. When you see the
session recovery timedout error above sg_readcap will fail like above or
will fail with

"No such device or address"

So at this time the sg_readcap output and the /sys...size value below
would not match. Even above it seems that if you printed the output of
sg_readcap then you would not see the correct size returned.


> $ cat /sys/block/sdc/device//block/sdc/size
> 4395405168


The /sys/ cap is going to be the same. It would only change if maybe you
rescanned the device when the transport is failed, because this would
case the kernel's read cap cmd to fail.


Instead of the above read cap checks , in 3.6 you can check if the
device state is "transport-offline" if it is then you can just delete
the device.

Something like this in pseudo code:

if /sys/block/sdc/device/state == "transport-offline" then
echo 1 > /sys/block/sdc/device/delete
endif

>
> So the size is still the same. This is on kernel 3.6.0 on the target and
> 3.2.14 on the initiator.
>
> Regarding dev_loss_tmo, this has to be implemented at the kernel level
> correct? Since this feature is important to me, I can go ahead and try
> to implement that.
>

Ok. Here is some info on the current status:

Implement iSCSI dev loss support.

Currently if a session is down for longer than replacement/recovery_timeout
seconds, the iscsi layer will unblock the devices and fail IO. Other
transport, like FC and SAS, will do something similar. FC has a
fast_io_fail tmo which will unblock devices and fail IO, then it has a
dev_loss_tmo which will delete the devices accessed through that port.

iSCSI needs to implement dev_loss_tmo behavior, because apps are beginning
to expect this behavior. An initial path was made here:
http://groups.google.com/group/open-iscsi/msg/031510ab4cecccfd?dmode=source


Since all drivers want this behavior we want to make it common. We need to
change the patch in that link to add a dev_loss_tmo handler callback to the
scsi_transport_template struct, and add some common sysfs and helpers
functions to manage the dev_loss_tmo variable.

[Boaz Harrosh]

On 11/02/2012 12:47 PM, Mike Christie wrote:
> Yeah, I updated the iscsi dev loss tmo patch and posted it to the list.
> I think I hit some bugs with lots of sessions and lots of removals at
> the same time due to some workqueue/threading stuff or scsi locking
> issues. I did not get a chance to look into more.
>
> Before reposting I wanted to make the dev loss code generic and hook all
> transport into it since srp and sas wanted it too.
>

Ha nice so you are already working on it? Nice that could come handy.
Do you have some documentation on it? like what timeout/configuration
controls when dev loss is triggered. Will it be a PM hot-unplug event
or do we need new udev rules for it?

Thanks
Boaz

[Amit]

Mike Christie <michaelc@...> writes:

> > Regarding dev_loss_tmo, this has to be implemented at the kernel level
> > correct? Since this feature is important to me, I can go ahead and try
> > to implement that.
> >
>
> Ok. Here is some info on the current status:
>
> Implement iSCSI dev loss support.
>
> Currently if a session is down for longer than replacement/recovery_timeout
> seconds, the iscsi layer will unblock the devices and fail IO. Other
> transport, like FC and SAS, will do something similar. FC has a
> fast_io_fail tmo which will unblock devices and fail IO, then it has a
> dev_loss_tmo which will delete the devices accessed through that port.
>
> iSCSI needs to implement dev_loss_tmo behavior, because apps are beginning
> to expect this behavior. An initial path was made here:
> http://groups.google.com/group/open-iscsi/msg/031510ab4cecccfd?dmode=source
>
> Since all drivers want this behavior we want to make it common. We need to
> change the patch in that link to add a dev_loss_tmo handler callback to the
> scsi_transport_template struct, and add some common sysfs and helpers
> functions to manage the dev_loss_tmo variable.
>

Looked at the patch and with a little modification I was able to apply
it to kernel 3.2.14. It works as advertised, thank you so much! The
device gets removed after a certain number of seconds and I get a udev
event that the device was removed.

Any reason why this patch is not in the mainline? The only thing I
noticed is that there are potential race conditions since the mutex was
removed.

Thanks again,
Amit

[Michael Christie]

I have to retest. With the original code there was a hang/lockup due to some threading/workqueue issue. That code has not been rewritten in the upstream kernel and I think those changes fix my issue.

[Michael Christie]

It will just work like the FC dev loss, but I wanted to move some of the code around. So there is just a timer that is set when the session/connection goes down. If the problem is not fixed within that timeout then the target and its devices are removed. That will send hot plug events like normal device deletion. Currently, tools like multipatthd handle this by removing the old paths.

I originally made this patch only for iscsi:

http://groups.google.com/group/open-iscsi/msg/031510ab4cecccfd?dmode=source

But at the last storage summit people were saying we should make it common code based off the core transport template, because SAS wanted it (and now SRP is adding it too).

Basically we just want to move the dev loss tmo and fast io fail callbacks to the scsi_transport and then move the timeout (setting of and handling) to common code. I also wanted to move some more handling of the command cleanup there too similar to how scsi_error.c handles cleanup/retry of commands.

I was working on this stuff for my old job (for the next RHEL), but since I have left I have not had a lot of time to work on iscsi initiator stuff.