Veeam Failed To Download Disk Flat.vmdk
Divina Hujer
Failed to decrypt TLS data: (336130329) error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac Unable to retrieve next block transmission command. Number of already processed blocks: [2810]. Failed to download disk '.vhdx'. Reconnectable protocol device was closed. Failed to upload disk '.vhdx' Agent failed to process method DataTransfer.SyncDisk.
An error was received from the ESX host while powering on VM SERVER01.
Failed to start the virtual machine.
Module DiskEarly power on failed.
Cannot open the disk 'SERVER01-000001.vmdk' or one of the snapshot disks it depends on.
The system cannot find the file specified
The file that the error say is missing, is not there, but I could see that the disk SERVER01_1-flat.vmdk was updated latest at Dec 17 18:12 and that was after the snapshot removal was started, and that SERVER01_1-flat.vmdk was update latest at Dec 17 20:28, this means the SERVER01_1-flat.vmdk was fully consolidated, but the VMX file did not reflect this, so i just edited one line i the VMX file:
Every virtual disk is composed of two files, a "flat" or "delta" file containing the actual data and a descriptor file containing geometry and topology information. For example, vDiskname.vmdk and vDiskname-flat.vmdk or vDiskname-delta.vmdk. For more information, see Understanding virtual machine snapshots in VMware ESX (1015180).
Looking at the typical patterns within the datastore all -flat.vmdk files of the base disks are there and a dozen of -delta.vmdk files showing the different snapshots made. As a backup software was running there are a couple of snapshot files there but not all show up as the base disk is pretending to be locked so many broken or inconsistent snapshot are the result.
setup a PuTTY session to the host that has the frozen VM running.
use 'cd' to get into the LUN or share where the VM is held and where it's files are held. In this case, we will be using a machine by the name of FRX.
Once you get to the VM's folder and its contents we will view the state of the VM by typing the command: vmware-cmd VM.vmx getstate
you will see if the VM is on or off. If it's off, you're done. If it's on, we still have more work to do
try running the command:
vmware-cmd FRX.vmx stop hard
This command forces, well a hard stop on the process. More than likely, this will error out and you will not be able to stop the machine
The next task is to find the VMID of your particular VM
From your PuTTY session type in:
cat /proc/vmware/vm/*/names
a list will scroll of all the VMs that are running on that host. In the left column you will see a VMID. Remember the VMID of the frozen VM you are trying to fix.
Type in (note *VMID* is where you type in your VMID):
less -S /proc/vmware/vm/*VMID*/cpu/status
this will open up a screen where you will see information about the particular VM. Scroll to the right by pressing the right arrow. Close to the end of the row there will be a column that says:
vm.SOME-NUMBER
That SOME-NUMBER is usually 1 less than VMID. press q to quit.
Now it is time to kill the running process on that VM. To run the kill command on that ID type (note: *VMID-1* is the number found in the previous step that is represented by SOME-NUMBER):
/usr/lib/vmware/bin/vmkload_app -k 9 *VMID-1*
Give it a few minutes and continually check the state of the VM by typing: vmware-cmd FRX.vmx getstate
Hopefully in a few minutes you will see that your VM has powered off.
If you continue to have problems (such as I did) here are some more methods to try after doing the above. This is from my experiance so symptoms and results may differ.
Once the machine powered off, Virtual Center said the machine was 'invalid'. To remedy this I tried removing the VM from virtual center's inventory, going back into the LUN, and re-adding the VM to inventory.
At this point, Virtual center got stuck on the 'processing' stage and froze up. I had to reboot my virtual center server.
Upon reboot of my Virtual Center server, I tried re-adding the machine into inventory and it went much faster, I then tried to power on the machine, it would get to about 90% then fail. After some investigation I found out that the FRX-flat.vmdk file was locked by another host. A way to remedy this is by restarting the vpxa service on every host. I decided (well the VMWare rep decided) it was best to try cloning.
I opened up the VM settings under virtual center and removed the 'hard drive' that was locked from the machine. Clicked on OK and reconfigured the virtual machine.
I opened up a PuTTY session to the host that held the VM and changed directories until I got to the storage for the VM.
once I was in the folder of the VM I typed:
vmfsktools -i FRX-flat.vmdk FLR_new-flat.vmdk
this command clones the drive and makes sure it isn't locked by another host. At this point you will have to wait for the disk to clone and can take a while depending up the size of the disk. a 25gb drive took about 15 minutes. After the clone completed I went back to virtual center, and added a new 'hard disk' to the VM. This time selecting the newly created drive, clicked OK, and powered on the VM. We ran into one small problem because originally the SCSI Adapter was set to BusLogic because it was a P2V. Make sure the SCSI Adpater is set to LSI Logic, and you are good to go.
Hope this helps some people out there having problems.