Is this a possibility and is there any way that i can get the volumes back working - i could then delete out some actual vmware disks that reside on the vmfs datastores (both of these volumes are only used by our backup server) - but at the moment i'm just unable to do anything within the volumes to clear out some space.
Please note that 95% is the threshold set on our storevirtual to ensure data integrity on volumes.
Once cluster touches 95% mark, unavailability of space can lead to read/writes failing, thus raising a risk of data corruption.
There are few options that can be tried provided pre requisites are met.
The above action will give breathing space to cluster and bring the utilization down to an acceptable limit.
Volumes will become accessible,hence you need to take a back up immediately.
As soon as its done, you can start the cleaning process by removing unwanted data from the datastores.
While you are doing so, please check if space reclamation is enabled on the management group your store virtual nodes are running on.
If not then you enable them and then run unmap command on the esxi to reclaim space.
If you are running ESXI 6.5, it runs auto unmap provided you have datastores as VMFS 6, otherwise unmap will not work.
It is just frustrating because I do belive there is still almots 1TB of available space there but VMWare thinks that it has all ran out.when using thin volumes, does the consumed space just grow naturally i.e. at normal increments, or does it in anyway attemp to pre assign storage in chunks?
I generally suggest that for everyone who uses thin volumes (suggested) and overprovisioning (also typical), you should create at least one Thick provisioned LUN that would be large enough to save you in a situation like this where you weren't paying attention and the SAN filled up completely. Then it would take all of 30 seconds to delete that LUN and regain write access to the other luns. Just make sure that the full provitioned "emergency space" LUN is large enough that deleting that will give you enough time to either delete/migrate data off the SAN to free up space OR give you enough time to find add another node into the system.
If you are 100% out of space AND there is nothing youc an delete. YOu either have to switch a LUN to NR0 for long enough to free up space or spin up a VSA on any hardware you can find that can has at least as much raw storage capacity as your existing nodes and add that to the cluster. Switching to NR0 would be faster, but it does put your data at risk.
As mentioned in my earlier post this threshold is to protect data integrity. If threshold exceeds there are higher chances of data corruption which is why volumes become inaccessible the moment threshold exceeds.
Hewlett Packard Enterprise provides flexible deployment options to fit any requirement, including VSA software, bare metal storage arrays and hyper-converged systems. All-inclusive licensing lets you start small and support growth at all stages.
Based on the LeftHand operating system, HPE StoreVirtual Storage is a scale-out storage platform that provides data mobility across tiers and locations and between physical and virtual storage. It enables linear scaling of capacity and performance.
Common management and storage federation give you simplicity and flexibility in your virtual data centers. With the Centralized Management Console, you can manage hundreds of nodes in your environment with the same simplicity.
Snapshot and Remote Copy integration for virtualization platforms and Windows applications makes backup and recovery a snap. Plus, with proven 99.999 percent(1) availability and unique stretch cluster capabilities, you can sustain multiple concurrent failures and still keep data online and accessible to applications.
Try HPE StoreVirtual VSA free and transform server capacity into storage with the first software-defined virtual storage appliance to support all major industry-standard servers and multiple hypervisors.
The HP StoreVirtual 4330 Storage System, as a member of the HP StoreVirtual 4000 Storage Family, is a scale-out platform designed to meet the dynamic needs of virtualized environments. Based on the LeftHand operating system, the StoreVirtual 4330 SAN provides a virtualized pool of storage resources that enhances virtual environments, reduces costs, and simplifies management. The HP 4330 storage array delivers enterprise class functionality without enterprise level complexity, as it is easy to deploy, maintain, and grow without disruption to your business. The HP StoreVirtual 4330 storage array is offered in three factory specified configurations with up to 8TBs of storage, one E5-2620 six-core processor, and 32GB RAM. ILO 4 Advanced Pack remote management licensing is included at no cost, as well as the Smart Array Advanced Pack 2.0 for RAID 6 support. This system supports scalability up to 32 nodes in includes the LeftHand-based StoreVirtual Storage System software which features Storage Clustering, Network RAID 5/6/10, Thin Provisioning, Space Reclamation, Application Integrated Snapshots, SmartClone, and Remote Copy.
An instance store provides temporary block-level storage for your instance. This storage is located on disks that are physically attached to the host computer. Instance store is ideal for temporary storage of information that changes frequently, such as buffers, caches, scratch data, and other temporary content. It can also be used to store temporary data that you replicate across a fleet of instances, such as a load-balanced pool of web servers.
An instance store consists of one or more instance store volumes exposed as block devices. The size of an instance store as well as the number of devices available varies by instance type and instance size. For more information, see Instance store volumes.
The virtual devices for instance store volumes are ephemeral[0-23]. Instance types that support one instance store volume have ephemeral0. Instance types that support two or more instance store volumes have ephemeral0, ephemeral1, and so on.
The data on an instance store volume persists even if the instance is rebooted. However, the data does not persist if the instance is stopped, hibernated, or terminated. When the instance is stopped, hibernated, or terminated, every block of the instance store volume is cryptographically erased.
Therefore, do not rely on instance store volumes for valuable, long-term data. If you need to retain the data stored on an instance store volume beyond the lifetime of the instance, you need to manually copy that data to more persistent storage, such as an Amazon EBS volume, an Amazon S3 bucket, or an Amazon EFS file system.
There are some events that can result in your data not persisting throughout the lifetime of the instance. The following table indicates whether data on instance store volumes is persisted during specific events, for both virtualized and bare metal instances.
* If the new instance type supports instance store, the instance gets the number of instance store volumes supported by the new instance type, but the data does not transfer to the new instance. If the new instance type does not support instance store, the instance does not get the instance store volumes.
*** The data is included in the AMI bundle that is uploaded to Amazon S3. When you launch an instance from that AMI, the instance gets the instance store volumes bundled in the AMI with the data they contained at the time the AMI was created.
For instance types with NVMe instance store volumes, all of the supported instance store volumes are automatically attached to the instance at launch. For instance types with non-NVMe instance store volumes, such as C1, C3, M1, M2, M3, R3, D2, H1, I2, X1, and X1e, you must manually specify the block device mappings for the instance store volumes that you want to attach at launch. Then, after the instance has launched, you must format and mount the attached instance store volumes before you can use them. You can't attach an instance store volume after you launch the instance.
Some instance types use NVMe or SATA-based solid state drives (SSD), while others use SATA-based hard disk drives (HDD). SSDs deliver high random I/O performance with very low latency, but you don't need the data to persist when the instance terminates or you can take advantage of fault-tolerant architectures. For more information, see SSD instance store volumes.
The Amazon EC2 Instance Types Guide provides the quantity, size, type, and performance optimizations of instance store volumes available on each supported instance type. For more information, see the following:
You can use the describe-instance-types AWS CLI command to display information about an instance type, such as its instance store volumes. The following example displays the total size of instance storage for all R5 instances with instance store volumes.
HP StoreVirtual Storage, based on the LeftHand operating system, is a scale-out storage platform that is designed to meet the fluctuating needs of virtualized environments. Remote Copy asynchronous replicates thin provisioned, reservationless snapshots between primary and remote locations.
Silver Peak Replication Acceleration software fixes the problems that exist on the WAN, enabling more data to be replicated, in less time, over longer distances. Silver Peak software solves WAN problems by repairing dropped and out-of-order packets, overcoming the effects of latency, and providing real-time deduplication and compression of all data sent across the WAN. Silver Peak also provides a site-to-site AES-256 IPsec VPN that secures replication traffic without impacting performance. When the problems on the WAN have been repaired, it is easy to meet, maintain, and even reduce RPOs.
Once deployed, Silver Peak replication acceleration software enables any storage administrator to manage the Silver Peak implementation. Data is reported in storage-friendly terms, showing the data sent across the WAN in bytes instead of bits. Storage traffic is segmented by application, and device, to make management and reporting more intuitive. Reporting includes minute, hour, and daily intervals, and includes a 10 day chart to reflect performance data before and after any changes made during the weekend. With HP StoreVirtual and Silver Peak Replication Acceleration software, you can protect all of your business-critical data and replicate consistently, regardless of network problems or distance.
b1e95dc632