Resizing a zpool as a VMware ESXi guest ...
Matthew Grooms
I am a long time user and advocate of FreeBSD and manage a several
deployments of FreeBSD in a few data centers. Now that these
environments are almost always virtual, it would make sense that FreeBSD
support for basic features such as dynamic disk resizing. It looks like
most of the parts are intended to work. Kudos to the FreeBSD foundation
for seeing the need and sponsoring dynamic increase of online UFS
filesystems via growfs. Unfortunately, it would appear that there are
still problems in this area, such as ...
a) cam/geom recognizing when a drive's size has increased
b) zpool recognizing when a gpt partition size has increased
For example, if I do an install of FreeBSD 10 on VMware using ZFS, I see
the following ...
root@zpool-test:~ # gpart show
=> 34 16777149 da0 GPT (8.0G)
34 1024 1 freebsd-boot (512K)
1058 4194304 2 freebsd-swap (2.0G)
4195362 12581821 3 freebsd-zfs (6.0G)
If I increase the VM disk size using VMware to 16G and rescan using
camcontrol, this is what I see ...
root@zpool-test:~ # camcontrol rescan all
Re-scan of bus 0 was successful
Re-scan of bus 1 was successful
Re-scan of bus 2 was successful
root@zpool-test:~ # gpart show
=> 34 16777149 da0 GPT (8.0G)
34 1024 1 freebsd-boot (512K)
1058 4194304 2 freebsd-swap (2.0G)
4195362 12581821 3 freebsd-zfs (6.0G)
The GPT label still appears to be 8G. If I reboot the VM, it picks up
the correct size ...
root@zpool-test:~ # gpart show
=> 34 16777149 da0 GPT (16G) [CORRUPT]
34 1024 1 freebsd-boot (512K)
1058 4194304 2 freebsd-swap (2.0G)
4195362 12581821 3 freebsd-zfs (6.0G)
Now I have 16G to play with. I'll expand the freebsd-zfs partition to
claim the additional space ...
root@zpool-test:~ # gpart recover da0
da0 recovered
root@zpool-test:~ # gpart show
=> 34 33554365 da0 GPT (16G)
34 1024 1 freebsd-boot (512K)
1058 4194304 2 freebsd-swap (2.0G)
4195362 12581821 3 freebsd-zfs (6.0G)
16777183 16777216 - free - (8.0G)
root@zpool-test:~ # gpart resize -i 3 da0
root@zpool-test:~ # gpart show
=> 34 33554365 da0 GPT (16G)
34 1024 1 freebsd-boot (512K)
1058 4194304 2 freebsd-swap (2.0G)
4195362 29359037 3 freebsd-zfs (14G)
Now I want the claim the additional 14 gigs of space for my zpool ...
root@zpool-test:~ # zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ
WRITE CKSUM
zroot ONLINE 0 0 0
gptid/352086bd-50b5-11e4-95b8-0050569b2a04 ONLINE 0 0 0
root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zroot 5.97G 876M 5.11G 14% 1.00x ONLINE -
The zpool appears to still only have 5.11G free. Lets reboot and try
again ...
root@zpool-test:~ # zpool set autoexpand=on zroot
root@zpool-test:~ # zpool online -e zroot
gptid/352086bd-50b5-11e4-95b8-0050569b2a04
root@zpool-test:~ # zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zroot 14.0G 876M 13.1G 6% 1.00x ONLINE -
Now I have 13.1G free. I can add this space to any of my zfs volumes and
it picks the change up immediately. So the question remains, why do I
need to reboot the OS twice to allocate new disk space to a volume?
FreeBSD is first and foremost a server operating system. Servers are
commonly deployed in data centers. Virtual environments are now
commonplace in data centers, not the exception to the rule. VMware still
has the vast majority of the private virutal environment market. I
assume that most would expect things like this to work out of the box.
Did I miss a required step or is this fixed in CURRENT?
Thanks,
-Matthew
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
Edward Tomasz Napierała
> All,
>
> I am a long time user and advocate of FreeBSD and manage a several
> deployments of FreeBSD in a few data centers. Now that these
> environments are almost always virtual, it would make sense that FreeBSD
> support for basic features such as dynamic disk resizing. It looks like
> most of the parts are intended to work. Kudos to the FreeBSD foundation
> for seeing the need and sponsoring dynamic increase of online UFS
> filesystems via growfs. Unfortunately, it would appear that there are
> still problems in this area, such as ...
>
> a) cam/geom recognizing when a drive's size has increased
> b) zpool recognizing when a gpt partition size has increased
>
> For example, if I do an install of FreeBSD 10 on VMware using ZFS, I see
> the following ...
>
> root@zpool-test:~ # gpart show
> => 34 16777149 da0 GPT (8.0G)
> 34 1024 1 freebsd-boot (512K)
> 1058 4194304 2 freebsd-swap (2.0G)
> 4195362 12581821 3 freebsd-zfs (6.0G)
>
> If I increase the VM disk size using VMware to 16G and rescan using
> camcontrol, this is what I see ...
AFAIK there is no way to do that. However, this should happen
automatically, if the "other side" properly sends proper Unit Attention
after resizing. No idea why this doesn't happen with VMWare.
Reboot obviously clears things up.
[..]
> Now I want the claim the additional 14 gigs of space for my zpool ...
>
> root@zpool-test:~ # zpool status
> pool: zroot
> state: ONLINE
> scan: none requested
> config:
>
> NAME STATE READ
> WRITE CKSUM
> zroot ONLINE 0 0 0
> gptid/352086bd-50b5-11e4-95b8-0050569b2a04 ONLINE 0 0 0
>
> root@zpool-test:~ # zpool set autoexpand=on zroot
> root@zpool-test:~ # zpool online -e zroot
> gptid/352086bd-50b5-11e4-95b8-0050569b2a04
> root@zpool-test:~ # zpool list
> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
> zroot 5.97G 876M 5.11G 14% 1.00x ONLINE -
>
> The zpool appears to still only have 5.11G free. Lets reboot and try
> again ...
online -e) should do the trick.
> root@zpool-test:~ # zpool set autoexpand=on zroot
> root@zpool-test:~ # zpool online -e zroot
> gptid/352086bd-50b5-11e4-95b8-0050569b2a04
> root@zpool-test:~ # zpool list
> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
> zroot 14.0G 876M 13.1G 6% 1.00x ONLINE -
>
> Now I have 13.1G free. I can add this space to any of my zfs volumes and
> it picks the change up immediately. So the question remains, why do I
> need to reboot the OS twice to allocate new disk space to a volume?
> FreeBSD is first and foremost a server operating system. Servers are
> commonly deployed in data centers. Virtual environments are now
> commonplace in data centers, not the exception to the rule. VMware still
> has the vast majority of the private virutal environment market. I
> assume that most would expect things like this to work out of the box.
> Did I miss a required step or is this fixed in CURRENT?
Filling PRs for those might be a good idea.
Garrett Cooper
> On Oct 16, 2014, at 1:10, Edward Tomasz Napierała <tr...@FreeBSD.org> wrote:
> "camcontrol rescan" does not force fetching the updated disk size.
> AFAIK there is no way to do that. However, this should happen
> automatically, if the "other side" properly sends proper Unit Attention
> after resizing. No idea why this doesn't happen with VMWare.
> Reboot obviously clears things up.
>
> [..]
Is open-vm-tools installed?
I ask because if I don't have it installed and the kernel modules loaded, VMware doesn't notify the guest OS of disks being added/removed.
Also, what disk controller are you using?
Cheers.
Michael Jung
I duplicated this behavior. According to gpart The virtual disk does
not grow
until the freebsd guest is rebooted.
FreeBSD freebsd10 10.0-RELEASE-p6 FreeBSD 10.0-RELEASE-p6 #0: Tue Jun 24
07:47:37 UTC 2014
ro...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
pkg info -- amd64 open-vm-tools-nox11-1280544_8,1 Open VMware tools for
FreeBSD VMware guests
ESXi reported -- Running, version:2147483647 (3rd-party/Independent)
ESXi-5.5-1331820(A00) Guest Hardware version 10
789 - S 0:00.54 /usr/local/bin/vmtoolsd -c
/usr/local/share/vmware-tools/
Id Refs Address Size Name
1 12 0xffffffff80200000 15f03b0 kernel
2 1 0xffffffff81a12000 5209 fdescfs.ko
3 1 0xffffffff81a18000 2198 vmmemctl.ko
4 1 0xffffffff81a1b000 23d8 vmxnet.ko
5 1 0xffffffff81a1e000 2bf0 vmblock.ko
6 1 0xffffffff81a21000 81b4 vmhgfs.ko
--mikej
Matthew Grooms
>
>> On Oct 16, 2014, at 1:10, Edward Tomasz Napierała
>> <tr...@FreeBSD.org> wrote:
>
>> "camcontrol rescan" does not force fetching the updated disk size.
>> AFAIK there is no way to do that. However, this should happen
>> automatically, if the "other side" properly sends proper Unit
>> Attention after resizing. No idea why this doesn't happen with
>> VMWare. Reboot obviously clears things up.
>>
>> [..]
>
> Is open-vm-tools installed?
>
> I ask because if I don't have it installed and the kernel modules
> loaded, VMware doesn't notify the guest OS of disks being
> added/removed.
>
VMware tools were not installed at the time. I'll try that, but I doubt
it will make a difference for resizing.
> Also, what disk controller are you using?
>
The ESXi 5.5 default controller for FreeBSD, LSI Logic Parallel ...
mpt0@pci0:0:16:0: class=0x010000 card=0x197615ad chip=0x00301000
rev=0x01 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = '53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI'
class = mass storage
subclass = SCSI
I'll try it with the LSI Logic SAS controller as well to see if that
makes a difference.
-Matthew
Matthew Grooms
> On 1010T1529, Matthew Grooms wrote:
>> All,
>>
>> I am a long time user and advocate of FreeBSD and manage a several
>> deployments of FreeBSD in a few data centers. Now that these
>> environments are almost always virtual, it would make sense that FreeBSD
>> support for basic features such as dynamic disk resizing. It looks like
>> most of the parts are intended to work. Kudos to the FreeBSD foundation
>> for seeing the need and sponsoring dynamic increase of online UFS
>> filesystems via growfs. Unfortunately, it would appear that there are
>> still problems in this area, such as ...
>>
>> a) cam/geom recognizing when a drive's size has increased
>> b) zpool recognizing when a gpt partition size has increased
>>
>> For example, if I do an install of FreeBSD 10 on VMware using ZFS, I see
>> the following ...
>>
>> root@zpool-test:~ # gpart show
>> => 34 16777149 da0 GPT (8.0G)
>> 34 1024 1 freebsd-boot (512K)
>> 1058 4194304 2 freebsd-swap (2.0G)
>> 4195362 12581821 3 freebsd-zfs (6.0G)
>>
>> If I increase the VM disk size using VMware to 16G and rescan using
>> camcontrol, this is what I see ...
> AFAIK there is no way to do that. However, this should happen
> automatically, if the "other side" properly sends proper Unit Attention
> after resizing. No idea why this doesn't happen with VMWare.
> Reboot obviously clears things up.
>
> [..]
>
All,
I know this is a very late follow up, but spent some more time looking
at this today and found some additional information that I found quite
interesting. I setup two VMs, one that acts as an iSCSI initiator (
CURRENT ) and another that acts as a target ( 10.2-RELEASE ). Both are
running under ESXi v5.5. There are two block devices on the initiator,
da1 and da2, that I used for resize testing ...
[root@iscsi-i /home/mgrooms]# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00> at scbus1 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 1.0> at scbus2 target 0 lun 0 (pass1,da0)
<VMware Virtual disk 1.0> at scbus2 target 1 lun 0 (pass2,da1)
<FREEBSD CTLDISK 0001> at scbus3 target 0 lun 0 (da2,pass3)
The da1 device is a virtual disk hanging off of a VMware virtual SAS
controller ...
[root@iscsi-i /home/mgrooms]# pciconf
...
mpt0@pci0:3:0:0: class=0x010700 card=0x197615ad chip=0x00541000
rev=0x01 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS1068 PCI-X Fusion-MPT SAS'
class = mass storage
subclass = SAS
[root@iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
Device Size: 10 G, Block Length: 512 bytes
[root@iscsi-i /home/mgrooms]# gpart show da1
=> 40 20971440 da1 GPT (10G)
40 20971440 1 freebsd-ufs (10G)
The da2 device is an iSCSI LUN mounted from my FreeBSD 10.2 VM running
ctld ...
[root@iscsi-i /home/mgrooms]# iscsictl
Target name Target portal State
iqn.2015-01.lab.shrew:target0 iscsi-t.shrew.lab Connected: da2
[root@iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
Device Size: 10 G, Block Length: 512 bytes
[root@iscsi-i /home/mgrooms]# gpart show da2
=> 40 20971440 da2 GPT (10G)
40 24 - free - (12K)
64 20971392 1 freebsd-ufs (10G)
20971456 24 - free - (12K)
When I increased the size of da1 ( the VMDK ) and then re-ran
'camcontrol readcap' without a reboot, it clearly showed that the disk
size had increased. However, geom failed to recognize the additional
capacity ...
[root@iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
Device Size: 16 G, Block Length: 512 bytes
[root@iscsi-i /home/mgrooms]# gpart show da1
=> 40 20971440 da1 GPT (10G)
40 20971440 1 freebsd-ufs (10G)
Here is the interesting bit. I increased the size of da2 by modifying
the lun size in ctld.conf on the target and then issued a /etc/rd.d/ctld
reload. When I re-ran 'camcontrol readcap' on the initiator without a
reboot, it also showed that the disk size had increased, but this time
geom recognized the additional capacity as well ...
[root@iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
Device Size: 16 G, Block Length: 512 bytes
[root@iscsi-i /home/mgrooms]# gpart show da2
=> 40 33554352 da2 GPT (16G)
40 24 - free - (12K)
64 20971392 1 freebsd-ufs (10G)
20971456 12582936 - free - (6.0G)
I was then able to resize the partition and then grow the UFS
filesystem, all without rebooting the VM ...
[root@iscsi-i /home/mgrooms]# gpart resize -i 1 da2
da2p1 resized
[root@iscsi-i /home/mgrooms]# gpart show da2
=> 40 33554352 da2 GPT (16G)
40 24 - free - (12K)
64 33554304 1 freebsd-ufs (16G)
33554368 24 - free - (12K)
[root@iscsi-i /home/mgrooms]# growfs da2p1
Device is mounted read-write; resizing will result in temporary write
suspension for /var/data2.
It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/da2p1, mounted on /var/data2, from 10GB to
16GB? [Yes/No] Yes
super-block backups (for fsck_ffs -b #) at:
21798272, 23080512, 24362752, 25644992, 26927232, 28209472, 29491712,
30773952, 32056192, 33338432
[root@iscsi-i /home/mgrooms]# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0p3 15G 1.2G 12G 9% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/da1p1 9.7G 32M 8.9G 0% /var/data1
/dev/da2p1 15G 32M 14G 0% /var/data2
It's also worth noting that the additional space was not recognized by
gpart/geom on the initiator until after the 'camcontrol readcap da2'
command was run. In other words, I'm skeptical that it was a Unit
Attention notification that made the right thing happen since it still
took manual prodding of cam to get the new disk geometry up into the
geom layer.
So what's the difference between the Virtual SAS block device vs the
iSCSI block device? I'm sure I have no idea. But in my mind this
invalidates two previous notions that were floated when I brought this
problem up late last year ...
1) There is in fact a command that can be manually run to force cam to
read new disk geometry. And when that new geometry is read, it is, at
least in some cases, passed on to geom.
2) While ESXi may or may not be issuing the correct SCSI notifications
to help the OS pick up the new disk geometry automatically, it surely
reports the new size when asked. Additionally, all the pluming is in
place to allow the entire disk, geom, fs resize process to work without
a reboot. I'm just not sure why it seems to work with iSCSI but doesn't
with the virtualized SAS controller.
Any thoughts?
Thanks,
-Matthew
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
Willem Jan Withagen
I remember doing this for a bhyve VM, and had the type same problem.
Getting gpart in the VM to actually pickup the new size required some
extra prodding (I like that word) or rebooting the VM.
I can remember reporting this:
tpoic: "resampeling of a ZVOL that has been resized"
and getting a fix from Andrey V. Elsukov...
Index: head/sys/geom/part/g_part_gpt.c
===================================================================
--- head/sys/geom/part/g_part_gpt.c (revision 282044)
+++ head/sys/geom/part/g_part_gpt.c (working copy)
@@ -760,7 +760,7 @@ g_part_gpt_resize(struct g_part_table *basetable,
struct g_part_gpt_entry *entry;
if (baseentry == NULL)
- return (EOPNOTSUPP);
+ return (g_part_gpt_recover(basetable));
entry = (struct g_part_gpt_entry *)baseentry;
baseentry->gpe_end = baseentry->gpe_start + gpp->gpp_size - 1;
Which went into the tree, but perhaps only in HEAD.
And that helped me getting the correct retasting of the GPART partitions.
Not sure if this snippet would help you to get around GEOM tasting the
new size.
--WjW
Matthew Grooms
Thanks for the response. I read the commit logs daily but I don't recall
that one. However, it looks like that change went in over five months
ago ( r284151 ) and I'm running these tests on a CURRENT ...
[mgrooms@iscsi-i ~]$ uname -a
FreeBSD iscsi-i.shrew.lab 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r291085:
Thu Nov 19 21:48:13 UTC 2015
ro...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
Something else is amiss here and I don't have the chops to fix it
myself. I tried compiling a custom kernel with CAMDEBUG and twiddling
with the camcontrol debug options to compare the output during the VMDK
vs iSCSI disk resize process. Nothing was obvious to my untrained eye.
I'd be more than happy to test patches or provide additional information
to anyone willing to help.
Thanks,
-Matthew
Matthew Grooms
similar to the camcontrol debug output ...
[root@iscsi-i /home/mgrooms]# sysctl kern.geom.debugflags=0x81
When I resize the iSCSI LUN and run the 'camcontrol readcap da2 -h', I
see this in the log output ...
[root@iscsi-i /home/mgrooms]# tail -f /var/log/messages
...
Nov 27 12:20:07 iscsi-i kernel: (pass3:iscsi1:0:0:0): Capacity data has
changed
Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff80973850,
0xfffff80003f4e000, 1, 0)
Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff8097a6b0,
0xfffff80003f42b60, 2, 0)
Nov 27 12:20:07 iscsi-i kernel: g_resize_provider_event(0xfffff800038c6700)
Nov 27 12:20:07 iscsi-i kernel: g_part_resize(da2)
Nov 27 12:20:07 iscsi-i kernel: GEOM_PART: da2 was automatically resized.
Nov 27 12:20:07 iscsi-i kernel: Use `gpart commit da2` to save changes
or `gpart undo da2` to revert them.
Nov 27 12:20:07 iscsi-i kernel: g_raid_taste(RAID, da2)
Nov 27 12:20:07 iscsi-i kernel: g_attach(0xfffff80003e64780,
0xfffff800038c6700)
Nov 27 12:20:07 iscsi-i kernel: g_detach(0xfffff80003e64780)
Nov 27 12:20:07 iscsi-i kernel: g_destroy_consumer(0xfffff80003e64780)
Nov 27 12:20:07 iscsi-i kernel:
g_destroy_geom(0xfffff800038c9c00(raid:taste))
Nov 27 12:20:07 iscsi-i kernel: g_label_taste(LABEL, da2)
However, when I resize the VMDK disk and run the 'camcontrol readcap da1
-h' command, I see nothing in the log output. So it would appear that
even though cam is reporting the new capacity in the command line
output, the this info is not being forwarded to geom in this case. Maybe
the VMware virtual SAS device ( mpt0 ) isn't reporting some special
capability bit to cam which causes it to squelch the info?
I'm not sure if this is useful but here is what the device info looks
like at boot time ...
mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem
0xfd4ec000-0xfd4effff,0xfd4f0000-0xfd4fffff irq 18 at device 0.0 on pci3
mpt0: MPI Version=1.5.0.0
...
da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 20480MB (41943040 512 byte sectors)
da0: quirks=0x40<RETRY_BUSY>
da1 at mpt0 bus 0 scbus2 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 20480MB (41943040 512 byte sectors)
da1: quirks=0x40<RETRY_BUSY>
...
da2 at iscsi1 bus 0 scbus3 target 0 lun 0
da2: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number MYSERIAL 0
da2: 150.000MB/s transfers
da2: Command Queueing enabled
da2: 16384MB (33554432 512 byte sectors)
Any thoughts?
Matthew Grooms
After sprinkling a bunch of printf's around to see what code paths were
being called, It's obvious that Edward was correct in assuming that ESXi
doesn't return any 'Unit Attention' sense information in response to a
'Read Capacity' request. This kinda makes sense as ESXi emulates SCSI-2
disk devices and, as far as I can tell, the 0x2A/0x09 ASC/ASCQ sense
code that denotes 'Capacity Data Has Changed' wasn't defined until the
SCSCI-3 spec. It's frustrating that the only way to get the scsci_da
code to call reprobe() is by receiving a command from the device. Would
something like this work? ...
1) Register a callback using xpt_register_async( daasync,
AC_REPROBE_DEVICE, path ) that calls reprobe()
2) Implement a new IOCTL in cam_xpt that camcontrol can call with the
bus:target:lun as the argument
3) have cam_xpt capture the IOCTL request and call xpt_async(
AC_REPROBE_DEVICE, path ) as a result
This way users would have the option of manually asking cam to
communicate the new size to geom. The only option now is one or more
reboots to gain access to the increased disk capacity. If this sounds
like a reasonable approach, I'll take a stab at implementing it.
Matthew Grooms
> I spent the day looking over the FreeBSD cam and scsi_da source code.
> After sprinkling a bunch of printf's around to see what code paths
> were being called, It's obvious that Edward was correct in assuming
> that ESXi doesn't return any 'Unit Attention' sense information in
> response to a 'Read Capacity' request. This kinda makes sense as ESXi
> emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09
> ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't
> defined until the SCSCI-3 spec. It's frustrating that the only way to
> get the scsci_da code to call reprobe() is by receiving a command from
> the device. Would something like this work? ...
>
> 1) Register a callback using xpt_register_async( daasync,
> AC_REPROBE_DEVICE, path ) that calls reprobe()
> 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the
> bus:target:lun as the argument
> 3) have cam_xpt capture the IOCTL request and call xpt_async(
> AC_REPROBE_DEVICE, path ) as a result
>
> This way users would have the option of manually asking cam to
> communicate the new size to geom. The only option now is one or more
> reboots to gain access to the increased disk capacity. If this sounds
> like a reasonable approach, I'll take a stab at implementing it.
>
cam, scsi or freebsd kernel development for that matter, so I'm sure it
could have been done a better way. In any case, I added a new command to
camcontrol that allows you to specify a bus, target and lun as an
argument. For example ...
# camcontrol readcap da1 -h
# gpart show da1
=> 40 58720176 da1 GPT (28G)
40 58720176 1 freebsd-ufs (28G)
Note, I resized the VMDK disk in ESXi. The camcontrol output shows the
size as 32G but geom thinks its 28G.
# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00> at scbus1 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 1.0> at scbus2 target 0 lun 0 (pass1,da0)
<VMware Virtual disk 1.0> at scbus2 target 1 lun 0 (pass2,da1)
# camcontrol reprobe 2:1:0
This generates an event that is captured by the scsci da device to
forces a reprobe. The kernel output looks almost identical to when the
'Unit Attention' sense data is received ...
Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes
or `gpart undo da1` to revert them.
Now that geom knows about the increased disk capacity, I can increase
the partition size and grow the fs ...
[root@iscsi-i /home/mgrooms]# gpart show da1
=> 40 67108784 da1 GPT (32G)
40 58720176 1 freebsd-ufs (28G)
58720216 8388608 - free - (4.0G)
# gpart resize -i 1 da1
da1p1 resized
# growfs da1p1
32GB? [Yes/No] Yes
# df -h
Filesystem Size Used Avail Capacity Mounted on
a drag to have to reboot production VMs to increase disk capacity when
it could be easily avoided. I'm not sure what the correct IOCTL should
look like. Maybe CAMIOCOMMAND is a better way to go? If someone with
some experience with the cam/scsi subsystems was willing to give me some
direction I'd be willing to try and rewrite the patch in a way that
would be commit worthy. I just need some direction.
Thanks,
-Matthew
Matthew Grooms
patch complete with man page updates. It communicates via CAMIOCOMMAND
instead of introducing a new ioctl value. I tried to model it after the
device reset option, hopefully with some degree of success. Functionally
it should be the same as the first patch.
Thanks,
-Matthew