Removing individual LUNs
Alex M.
the setup is working fine. However, when I change the size of one of
the underlying devices, the initiators (running open-iscsi 2.0-865)
don't pick up the change. I know that if I logout the entire session
and then log in again it works fine, but I have a number of active
LUNs that I can't afford to take offline just to resize one. So I'm
assuming I need a way to remove the lun on the initiator and then re-
add it. Unfortunately, I can't figure out how to do it. (I've already
deleted and readded the lun on the target side using ietadm which
worked fine.)
What I need is something along the lines of 'iscsiadm -m session --op
delete --lun 6'. Sadly, googling for "iscsiadm" constantly returns
results for the Solaris implementation which really isn't helping me
any. :) Any help is greatly appreciated.
P.S. Is there any way to have iscsiadm list all currently connected
luns? I can do iscsiadm -m node|session|etc but none of these are
granular enough to show all the luns being provided by that particular
target.
Mike Christie
> I have a Linux machine running as a target with LVM-backed devices and
> the setup is working fine. However, when I change the size of one of
> the underlying devices, the initiators (running open-iscsi 2.0-865)
> don't pick up the change. I know that if I logout the entire session
> and then log in again it works fine, but I have a number of active
> LUNs that I can't afford to take offline just to resize one. So I'm
> assuming I need a way to remove the lun on the initiator and then re-
> add it. Unfortunately, I can't figure out how to do it. (I've already
> deleted and readded the lun on the target side using ietadm which
> worked fine.)
>
> What I need is something along the lines of 'iscsiadm -m session --op
> delete --lun 6'. Sadly, googling for "iscsiadm" constantly returns
> results for the Solaris implementation which really isn't helping me
> any. :) Any help is greatly appreciated.
Do:
iscsiadm -m session -r $SID --rescan
you get the SID from iscsiadm -m session (it is the value in the []) or
if you do iscsiadm -m session -P 3 you can see which session lines with
with which lun.
or
iscsiadm -m node -T target --rescan
or you can just take the lazy way and do
iscsiadm -m session --rescan
which rescans everything.
The above commands will rescan existing devices and add new ones. It
will not remove old ones.
But if you have a disk mounted or have something like LVM or
dm-multipath using a disk then you probably need to run one of those
tools to pick up the change there.
Oh yeah I do not remember if this is in 865. It might only be in the
newer 865.X and new 869 releases on open-iscsi.org.
>
> P.S. Is there any way to have iscsiadm list all currently connected
iscsiadm -m session -P 3
will spit out lots of info including this.
> luns? I can do iscsiadm -m node|session|etc but none of these are
> granular enough to show all the luns being provided by that particular
> target.
http://www.open-iscsi.org/docs/README is for the newer 869 release, but
I think most of it applies to 865. If not then check out the README with
that release (older 865's may not have up to date READMEs though).
> >
Alex M.
> Do:
>
> iscsiadm -m session -r $SID --rescan
Having manually resized the filesystem on the target end, I now get
errors on the initiator since the filesystem extents are outside of
the apparent device extents.
> The above commands will rescan existing devices and add new ones. It
> will not remove old ones.
would be to delete the LUN on the target and then re-create it under a
_different_ LUN and then do a rescan. This is obviously not an ideal
solution as it's going to result in a large number of "zombie" LUNs on
the initiator as time goes on.
Konrad Rzeszutek
>
> On Jun 16, 4:29 pm, Mike Christie <micha...@cs.wisc.edu> wrote:
> > Do:
> >
> > iscsiadm -m session -r $SID --rescan
>
> Doing a rescan doesn't appear to pick up the new size of the devices.
> Having manually resized the filesystem on the target end, I now get
> errors on the initiator since the filesystem extents are outside of
> the apparent device extents.
That is not a iSCSI issue. It is the SCSI subsystem in the kernel. When you
give the rescan command iscsiadm passes in '- - -' to the /sys/class/scsi_host/hostX/rescan
which kicks off a kernel thread that re-scans for the LUNs. The logic in the kernel
is that if it finds that a Scsi_Host with a LUN exists already it returns that
one and doesn't do a full re-discover on it.
So you have to delete the Scsi_Host from userland and then follow it with a rescan.
But the reason this is done (and that the kernel doesn't do this automaticlly) is that this
operation is very dangerous. If you don't unmount any of the filesystems on those block
devices before you delete them, you will incur data corruption.
>
> > The above commands will rescan existing devices and add new ones. It
> > will not remove old ones.
>
> As you said, rescans don't pick up deleted LUN's, so my only option
> would be to delete the LUN on the target and then re-create it under a
> _different_ LUN and then do a rescan. This is obviously not an ideal
> solution as it's going to result in a large number of "zombie" LUNs on
> the initiator as time goes on.
You can delete the old block devices that have changed and do a SCSI rescan:
You have to delete the old devices (well, this script deletes _ALL_ so that might
not be that good):
for disk in `find /sys/class/iscsi_session/session*/device/target*/*:*:*:*/delete `
do
echo 1 > $disk
done
And then do the rescan:
#iscsiadm -m session --rescan
Unfortunatly this will delete _all_ of your block devices (sdc, sde, sdd, etc) that
were provided. But it will pick up your new one after the rescan. You can fiddle with the for loop
above and run 'sg_caps' on the block device, see if it has changed compared to
/sys/block/sdX/device/size and if so delete it:
scsi_sz=`sg_readcap /dev/sdko | grep address | sed s'/.*blocks=//'`
kernel_sz=`cat /sys/block/sdko/device/block\:sdko/size`
if (($scsi_sz==$kernel_sz)); then
echo "Good."
else
echo "Do the deletion of the block device."
echo 1 > /sys/block/sdko/device/delete
fi
Oh, please be carefull. These scripts will delete your block devices, so if you delete your
root one or any other one that is used - you are screwed. Make sure you first unmount
any of the filesystems on those block devices and sync it so that there are no
outstanding I/Os.
Alex M.
> You can delete the old block devices that have changed and do a SCSI rescan:
>
> You have to delete the old devices (well, this script deletes _ALL_ so that might
> not be that good):
>
> for disk in `find /sys/class/iscsi_session/session*/device/target*/*:*:*:*/delete `
> do
> echo 1 > $disk
> done
>
> And then do the rescan:
> #iscsiadm -m session --rescan
> root one or any other one that is used - you are screwed. Make sure you first unmount
> any of the filesystems on those block devices and sync it so that there are no
> outstanding I/Os.
As you said, removing devices blindly is quite dangerous, so I
certainly won't be automating this in any way. :)
It's a one-off operation that happens maybe once every 3 weeks or so,
and there are a number of other steps that happen before and after the
disk resize that require quite a bit of care to be taken so I'll be
doing it manually and with proper safety precautions. (doing an LVM
snapshot on the target side prior to doing anything just in case, etc)
Thanks very much for the help!
Mike Christie
> On Jun 16, 4:29 pm, Mike Christie <micha...@cs.wisc.edu> wrote:
>> Do:
>>
>> iscsiadm -m session -r $SID --rescan
>
> Doing a rescan doesn't appear to pick up the new size of the devices.
When you run that command what gets outputted to /var/log/messages. For
existing devices you should see:
Jun 23 10:08:44 max kernel: sd 0:0:0:0: [sda] Write Protect is off
Jun 23 10:08:44 max kernel: sd 0:0:0:0: [sda] Write cache: enabled, read
cache: enabled, doesn't support DPO or FUA
Jun 23 10:08:44 max kernel: sd 0:0:0:0: [sda] 195371568 512-byte
hardware sectors (100030 MB)
get spit out and the new size detected (this is only at the scsi layer),
because we do not just scan for devices we use the rescan attr for
existing devices.
However, there is issue outstanding I think. I think you then need a new
a newer kernel (like 2.6.26 or something) so that the block layer can
see the changes and pick up the scsi layer changes. Also see my comment
on FSs picking tihs up below.
> Having manually resized the filesystem on the target end, I now get
> errors on the initiator since the filesystem extents are outside of
> the apparent device extents.
I did not get what you said here? So do you have a FS mounted on the
initiator, then you resize and the FS on the target end (how do you do
that, do you mount it locally on the target)?
I think you have to unmount the FS when you resize the disk. I think we
can only handle the resizing at the disk level. The FS may not handle
this dynamically.
Alex M.
--snip--
> a newer kernel (like 2.6.26 or something) so that the block layer can
> see the changes and pick up the scsi layer changes. Also see my comment
> on FSs picking tihs up below.
upgrade since this is a Xen kernel.
> > Having manually resized the filesystem on the target end, I now get
> > errors on the initiator since the filesystem extents are outside of
> > the apparent device extents.
>
> I did not get what you said here? So do you have a FS mounted on the
> initiator, then you resize and the FS on the target end (how do you do
> that, do you mount it locally on the target)?
>
> I think you have to unmount the FS when you resize the disk. I think we
> can only handle the resizing at the disk level. The FS may not handle
> this dynamically.
then resize it, but this fails because the block device's size hasn't
been updated. So instead, I removed the LUN in question from the
initiator. I then resized the fs on the intiator and then recreated
the LUN. The target still hadn't picked up the new size of the block
device, so the filesystem failed a basic sanity check and failed to
mount. (fs size > block device size)
Some filesystems, namely XFS, can support on-line resizing. Actually,
with XFS, in order to use the xfs_growfs utility the fs *must* be
mounted.