Vmware High Disk Usage

[Jacque Waiden]

Brent looks like you are using VMware ESXi (or least when hovering over your performance charts they sure look similar), if so, get a copy of visualesxtop from vmware labs here (its free) and install, point it to your ESXi host and start, select the disk tab, narrow the windows for the device name columns so you can see the various queues and commands columns. Low and behold you should see something close to IOPs called commands (e.g. IO commands being issued) which is what an IOP is, e.g. disk I/O activity per time interval. If not using vmware, no worries, you can also get that from other tools including Perfmon on Windows, Spotlight on Windows/Unix among others.

As for calculating IOPs its the amount of data transferred (e.g. bandwidth/MByte/Data read/written, KB/MB read/written, etc) per time interval divided by the average IO size if known. If you know number of IO operations (e.g. IOPs) and data moved you can derive IO size. If you know IO size and IO rate you can figure out bandwidth etc.

Quick analysis looks like you are getting some cache hits (not clear if you are showing reads/writes) either at the drive level, the NAS/RAID level, or in the OS/hypervisor or combination. The drives performance will var y depending on the type, configuration, IO size, other factors. So a 15K SAS (newer generation) can do around 400 IOPs depending on some factors from the workload, to the type of drive, to the storage system being used, in some situations less, in others even higher.

@Hutch you are correct and if you need to use CLI, thats what you would do with ESXi e.g. use esxtop, however if you like or need different interface, thats why visualesxtop exists, basicly gives you same info, just with a different interface without having to know/remember the various esxtop commands etc. Likewise you see the various queues etc similar to what you can get out of esxtop, just organized and accessed differently.

@Canadianstorageguy thats a good simplified point about IOPs being more relevant with smaller IO sizes including both random or sequential along with latency/resposne time where there is less emphasis on bandwidth.

Perhaps something made a file within the filesystem that actually consists of lots of zeros? If this were the case then the inner filesystem's usage may be bigger than that of the outer VMDK containing it because VMware may detect runs of zeros and essentially store them in a more efficient manner within the file representing the VMDK. Not all data is equal...

When you delete files from your virtual machine, the disk space occupied by those files is not immediately returned to your host system. If a virtual disk has such empty space, you can use the Clean up disks command to return that space to the hard drive on a Microsoft Windows host.

The Clean up disks command is similar to the Compact command in the Workstation virtual machine settings and the shrink command provided by VMware Tools. The Clean up disks command has these advantages:

The Clean up disks command reclaims more disk space than the Compact command. The Clean up disks command reclaims disk space from the current state of the virtual machine, from any powered-off snapshots, and from any powered-on snapshots where the guest operating system is Windows XP or later and you have installed a version of VMware Tools that is compatible with Workstation 8 or later.

Unlike the Defragment command and the shrink command provided by VMware Tools, the Clean up disks command does not require any extra disk space on the host. The Clean up disks command operates directly on the virtual disk (.vmdk) files.

vSAN provides two different configurations: a hybrid configuration that leverages flash-based devices and magnetic disks and an all-flash configuration. The hybrid configuration uses server-based flash devices to provide a cache layer for optimal performance, while magnetic disks provide capacity and persistent data storage. This delivers enterprise performance and a resilient storage platform. The all-flash configuration uses flash for both the caching and capacity layers.

This document focuses on helping administrators to correctly design and size a vSAN cluster and answer some of the common questions around the number of hosts, number of disk groups, cache sizing, and number of capacity devices, along with detailed configuration questions to help correctly and successfully deploy vSAN.

A vSAN Ready Node is a validated server configuration in a tested, certified hardware form factor for vSAN deployment, jointly recommended by the server OEM and VMware. vSAN Ready Nodes are ideal as hyper-converged building blocks for larger data center environments looking for automation and needing to customize hardware and software configurations. Select vSAN Ready Node partners provide pre-installed vSAN on ready nodes.

There are a wide range of options for selecting a host model, storage controller as well as flash devices and magnetic disks, especially for the OSA. It is extremely important that you follow the VMware Compatibility Guide (VCG) to select these hardware components. This on-line tool is regularly updated to ensure customers always have the latest guidance from VMware available to them. A subset of the VCG, the vSAN VCG must also be consulted for for storage devices and controllers. For a list of vSAN ESA compatible storage devices see this vSAN VCG link.

The ESA is much easier to design for, as it is able to offer performance and efficiency capabilities relatively easily. Specifying server configurations through the ReadyNode program for ESA also provides for a simple, yet prescriptive approach to design. Recommendations for optimal performance in the ESA are also much easier than in the OSA.

Best Practice: Verify all software, driver and firmware versions are supported by checking the VCG and using the vSAN Health Service. The screen shot below shows an example of a controller driver that is not on the VCG.

As a recommended practice, VMware recommends deploying ESXi hosts with similar or identical configurations across all cluster members, including similar or identical storage configurations. This will ensure an even balance of virtual machine storage components across the disks and hosts cluster. While hosts that do not contribute storage can still leverage the vSAN datastore, having a cluster with fewer nodes contributing storage increases the capacity and performance impact when a node is lost. For this reason, VMware recommends balanced configurations within a cluster.

HCI Mesh supports the mounting of vSAN clusters by external clusters. This can allow for asymmetric compute scaling, as well as help increase storage utilization between clusters. For more information see the updated HCI Mesh Technote.

If the components are no longer available to purchase try to add equal quantities of larger and faster devices. For example, as 400GB SSDs become more difficult to find, adding a comparable in performance 800GB SSD should not negatively impact performance. To ensure levels of performance consistency, it is ideal, but not required to have hosts using storage devices with similar levels of performance capabilities. Mixing flash devices that using different bus protocols, or interfaces (SATA, SAS, NVMe) within or across hosts that comprise a vSAN cluster will generally only provide performance levels as fast as the slowest devices used (e.g. SATA). If one is transitioning existing hosts to newer, faster devices such as NVMe, these performance gains will likely only be realized when all of the slower devices in the hosts that comprise a cluster have been replaced with newer, faster devices.

New generations of servers can be mixed, but it is recommended to try to keep storage configurations balanced when possible. Be aware that EVC may need to be activated. Mixing of generations can also be used as a process for gradually migrating. Different servers and vendors can be mixed but it may add complexity to the management of the lifecycle on the hosts.

If host Rebuild Reservation is activated, it will base the storage reservation based on the assumption that the largest node within the cluster has failed. For clusters running the vSAN ESA in vSAN 8 U1, where the "Auto-Policy Management" capability is enabled, this may impact the effective storage policy that one can use for a given cluster. For more information, see the post: "Auto-Policy Management Capabilities with the ESA in vSAN 8 U1."

Best practice: Consider alternative solutions for asymmetric demand needs. Single socket servers can help with storage heavy workloads while deploying hosts with empty drive bays activates adding storage later on without the need to add additional nodes. Strategic approaches to purchasing can help.

vSAN provides customers with a storage solution that is easily scaled up by adding new or larger disks to the ESXi hosts, and easily scaled out by adding new hosts to the cluster. For clusters running the vSAN OSA, it is important to scale in such a way that there is an adequate amount of cache, as well as capacity, for workloads.

This allows customers to start with a very small environment and scale it over time, by adding new hosts and/or more disks. In many cases, scaling out by adding additional hosts to a VSAN cluster is preferred over adding or replacing drives in an existing host. Adding a host is non-disruptive as shown in this click-through demonstration: Scale Out by Adding a Host.

Adding additional drives into the existing hosts in a cluster can be a fast way to scale up capacity within the nodes in a cluster. This can be done by expanding existing disk groups by adding capacity devices, adding new disk groups entirely, or replacing existing devices if additional drive bays are not avalible.

This consideration is covered in depth throughout this guide. In particular, one should consider choosing hosts for a design that have additional disk slots for more capacity, as well as providing an easy way to install additional devices into these slots.

3a8082e126