Fun with Hyper-V checkpoints

256 views
Skip to first unread message

Gantry Zettler

unread,
Jul 21, 2022, 8:50:11 AM7/21/22
to ntsys...@googlegroups.com
I have a Linux vm under a 2016 Hyper-V host that has a few orphaned checkpoints, meaning they don't show up in the Hyper-V manager GUI.  I've run into this before and they are always related to the Veeam backup, in this case all three avdhx files are small (100MB, 364MB and 2.5GB) compared to the main checkpoint file which does show in the GUI (273GB) and vhdx (785GB).  They also have a created date and modified date within an hour or so of each other, all of them from months ago.  

I've run into this before and normally I would use Merge-VHD to put them into their parent avhdx file, but in this case when using Inspect-VHD all three have the same parent checkpoint file - the main one that shows in the GUI.  When I've done this before it was a chain of checkpoint files, where each one had a different parent that went up the chain to the main vhdx file.  

When multiple checkpoints have the same parent, does the same logic apply to merge the most recent file first and keep going until you merge the oldest?  Googling also had someone (who replied to a lot of checkpoint related posts on Spiceworks) recommended simply moving these orphaned files to a temp location and see if anything is affected.  If not don't worry about merging them, not sure what is the best approach in this situation.  

Then once all that is done I'll (hopefully) have only one checkpoint which is available in the Hyper-V GUI that I can delete and merge back in.  But I have another issue - the current vhdx is 748GB and the max size is 750GB.  Googling told me absolutely do not expand a vhdx file when it has checkpoints.  But if I try to merge a 275GB checkpoint into a 748GB vhdx with a max size of 750GB, I'm worried we'll hit the space limit.  Keep in mind this actual VM only has 500GB of data as we moved a bunch of files to a different server.  

So what would be your approach for the orphaned checkpoints and merging the main checkpoint into a vhdx that is at its size limits?  Current thought is shut down the VM, copy all (a)vhdx files to a USB drive and try to merge the orphaned files in, newest to oldest.  After those are in export the VM to said USB drive and try to import into a new VM where I can make a new virtual disk with 1.5TB max or something like that.  Any suggestions would be welcome.  

PS - The VM runs fine and I do have working backups via Veeam which runs every few hours during the work day.  This is more of a "cleanup" job

Poppy Lochridge

unread,
Jul 21, 2022, 11:49:27 AM7/21/22
to ntsys...@googlegroups.com

My first attempt to see if I could get them to merge themselves would be to shut the VM down. Hyper-V should do a merge when the VM shuts down (and it’s much faster than doing a manual merge). I’d try that and see what I get from it, then if it didn’t work, I’d probably try the approach you outline.

 

--P

--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CAFZFFO3-rMub40AQueVyHshepj2rATOQ%2Bt%3DjHgB5UN7R35rY5w%40mail.gmail.com.

Gantry Zettler

unread,
Jul 21, 2022, 12:32:30 PM7/21/22
to ntsys...@googlegroups.com
Does it only try to merge with shutdowns and not reboots?  This server has been rebooted many times since all this started as I install kernel updates and restart, that said I may not have done an actual power down.  Will try that first, thanks...

Josh Doty

unread,
Jul 21, 2022, 2:01:23 PM7/21/22
to ntsys...@googlegroups.com

Hey Just ran into something similar to this last month.

We had a client with a 6TB VHDx and around 50 AVHDX’s that veam created. We powered down our VM, and begun merging snapshots from newest to oldest. However we came across a issue where the host ran out of storage mid-merge and this corrupted the disk.

We eventually had to do a full restore from Veeam and merge the current data on the corrupted disk to the healthy restored disk as restoring from backups took way longer than we expected.

 

We also noticed that during reboots and shutdowns the VM would fail to merge all of the avhdx’s that Veeam had created. It was only a manual merge that seemed to resolve this issue.

 

I’m going to adding to our health check scripts to check if the vm is using a avhdx cause this was a real week killer for us.

brett...@hotmail.com

unread,
Jul 22, 2022, 7:39:54 PM7/22/22
to ntsysadmin
A normal OS-level restart does not trigger the Hyper-V re-merge attempt. You must do a shutdown then restart it at Hyper-V level to trigger that.
I had a stubborn VM that wouldn't re-merge even this way. We ended up not being able to manually merge the disks (failed on the last one, no explainable reason shown) and we used the export we kept of the VM and re-imported it and the merge happened straight away and was successful.  

Philip Elder

unread,
Jul 23, 2022, 10:49:50 PM7/23/22
to ntsysadmin

There are scripts out there that check for the orphaned .AVHDX files that then trigger and then delete a CheckPoint which in most cases will then trigger the merge process for the orphans. If that doesn’t work, then there’s a couple more steps in PowerShell to take then trigger the CP and delete it to then start the merge process.

 

Note that there needs to be enough free storage to create a new parent .VHDX.

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Skype: MPECSInc.

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

Brian Illner

unread,
Jul 25, 2022, 10:39:04 AM7/25/22
to ntsys...@googlegroups.com

It would also be worth trying to use powershell to delete any lingering checkpoints:

 

Remove-VMSnapshot

 

(c’mon MS - standardize on whether it’s a snapshot or checkpoint and rename everything to match)

 

We’ve seen some after backups that don’t appear in the GUI, but do appear in PowerShell

 

BRIAN ILLNER | Senior Systems Administrator
864.250.9227 Office
864.679.2537 Fax


Canal Insurance Company
400 East Stone Avenue
Greenville, SC 29601

WARNINGAs the information in this transmittal (including attachments, if any) may contain confidential, proprietary, or business trade secret information, it should only be reviewed by those who are the intended recipients.  Unless you are an intended recipient, any review, use, disclosure, distribution or copying of this transmittal (or any attachments) is strictly prohibited.   If you have received this transmittal in error, please notify me immediately by reply email and destroy all copies of the transmittal.  While Canal believes this transmittal to be free of virus or other defect, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Canal (or its subsidiaries and affiliates) for any loss or damage arising therefrom.

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Philip Elder
Sent: Saturday, July 23, 2022 10:50 PM
To: 'ntsysadmin' <ntsys...@googlegroups.com>
Subject: RE: [ntsysadmin] Fun with Hyper-V checkpoints

 

CAUTION: This message was sent from outside of Canal Insurance. Please do not click links or open attachments unless you recognize the source of this email and know the content is safe. Please report all suspicious emails to "inf...@canal-ins.com" as an attachment.


Reply all
Reply to author
Forward
0 new messages