VMDK Recovery

VMDK file repair is a custom data recovery service offered by Gillware. Our virtualization data recovery experts have extensive experience recovering all sorts of data problems with VMware Hypervisors, and in particular, VMDK files.

What do you do when a VMDK file is corrupted, inaccessible, or damaged? You need VMDK recovery. Corrupt VMware VMDK files can result in lost data. Before you download a VMDK recovery tool or VMDK recovery software, we highly recommend you read more about VMware VMDK files, how a corrupt VMDK file can cause problems, and how VMDK recovery works.

Using a VMDK recovery tool in an attempt to fix deleted, damaged, or corrupted VMDK files can result in a permanent inability to recover data from VMDK devices, and in general, it’s best to leave it to the VMDK recovery experts. Rather than trying to use a VMDK recovery tool on your own, get in touch with the experts in VMDK file systems at Gillware to help you with VMDK recovery.

Real Corruption vs Fake Corruption

Possible causes and what that means for file repair vs data recovery aka “Real Corruption” vs “Fake Corruption.” Sometimes a server environment and file system are perfectly healthy, and after a hard shutdown event or malicious malware event, the virtual machine files are legitimately corrupted. We call these hard or real corruptions.

However, sometimes a VMDK file can appear corrupted, with the wrong size, failing checksums, or broken delta chains, but the file isn’t corrupt… it’s being presented incorrectly by the hardware, and we call these fake corruptions. This is most often caused by issues with the underlying RAID array, where a stale drive has been forced into the mix, or a troubled drive is failing to read critical sectors. It can also happen when the cache drive has fallen offline or died, rendering the most recent data blocks in the physical volume unavailable.

If you attempt to repair a file that’s fake corrupted, it can and likely will cause permanent data loss. File system scans like FSCK are completely oblivious of the underlying hardware issues, they only see one layer of the error onion, and the methods they use the repair the file system is to truncate(kill) files that don’t look correct. After this, your file system is now valid, but the bad news is you permanently crushed your server files. The server file definition looked off to FSCK because there is a drive in the mix that has been offline for 6 months, so every 5th data block was blank or represented historical stale data. So FSCK kills the corrupted file definition, restoring the file system consistency. But when you ultimately tag in the experts like Gillware, who will notice the stale drive, it can be too late, because FSCK has corrupted the production data.

So if your data in these files is business-critical, the best advice is to pick up the phone and tag us in now at 877-624-7206, as all the steps in this guide have some risks associated. Data recovery efforts, without a forensic baseline of the underlying storage equipment, are not a zero-sum game. Well-meaning IT professionals, while simply trying to quickly resurrect the data, can cause permanent harm instead when their efforts fail.

Common VMDK repair Error Codes

There are a number of error codes and error messages that directly or indirectly relate to VMDK corruption issues. Some of the most common we see are:

• The file or directory is corrupted and unreadable
• There was a problem while trying to mount the virtual disk
• The disk has one or more internal errors that cannot be fixed.
• The virtual disk, ‘/path/to/your_corrupted_disk.vmdk’, is corrupted and cannot be repaired
• The virtual disk is corrupted but the repair process has failed.

What Is VMDK?

VMDK stands for Virtual Machine Disk. It is a proprietary file type format used by VMware as a disk container for every Virtual Machine disk. VMDK file data is used by nearly every software-based VMware Enterprise product on the market today, and you will find the format used extensively in the full version of VMware vSphere Suites, VMware Workstation, VMware Fusion, and VMware vSAN platforms, to name just a few.

VMware Virtual Machine file types use one or many VMDK files. You get better performance if you use multiple VMDK files, but a single VMDK is much easier to manage. In the latest release of VMware vSphere 7, a VMDK file can be grown to 62TB in size, but up until quite recently, the max size of a VMDK file was restricted to 2TB.

VMDK files are very flexible. The format can be set and written to disk at the time of creation – using all the allocated disk space no matter how small the Virtual Machine is. This approach is called either a flat file or thick provisioning. Alternatively, thin provisioning (or sparse file) allows the virtual machine to grow dynamically up to its allocated VMDK file data size.

VMware products are normally found in medium, large, or enterprise-grade organizations and each VMware configuration can vary from a small two or three VM ESXi host setup to thousands of ESXi hosts spanning around the globe. Despite the rise of cloud technology, VMware has proven very popular among enterprises, and VMware is now available as a service in the top three hyper-scale cloud providers.

VMware Storage Recovery

VMware works best when it is attached to enterprise-grade highly available storage solutions, such as a Storage Area Network (SAN). SAN technology, and in particular linked SAN solutions, provide a highly redundant hardware abstraction layer mapped to countless numbers of ESXi hosts.

A fairly recent VMware technology, VMware vSAN flips this approach on its head. vSAN can use SAN technology without a problem, but you can also virtually map any disk types to a host or add extra storage racks into one or multiple logical datastores.

VMware tech is usually very reliable, but as with everything, things can go wrong, especially if there are any Operating System software issues or unexpected, abrupt hardware faults. When things go wrong with VMware storage, it tends to be quite a spectacular outage and extensive file corruption.

In an ideal world, you will have a backup of your storage solution, either in the form of a storage snapshot, or your VMDK files being backed up at the file level. Enterprise customers running production workloads should also have SAN to SAN synchronous replication to protect against deleted data files.

You have probably found this page by searching for “VMDK file recovery,” “VMDK recovery,” or “recovered VMDK”, so it is quite likely that you are already in a pickle and suffering major issues with your VMware stack and your VMDK files.

If you are a small, medium, large, or enterprise customer, and your VMware infrastructure data contains business-critical files, and you have no backup, it is highly advisable to gracefully shut everything down. VMware VMDK files have heavy I/O usage and trying to fix the issue can actually make the situation a whole lot worse. So stop and call the experts at Gillware for leading VMware VMDK recovery services.

Gillware has provided professional and affordable services to recover VMDK file systems for well over a decade. As part of our business model, we don’t charge any fees for evaluations. In fact, we only charge for our VMDK file recovery efforts after we successfully recover data from VMDK files! And with our ISO-5 Class, 100 cleanroom workstations and SOC 2 Type II audited, GSA-contracted facilities, you can rest easy knowing there’s no safer place for your data files.

The experts here at Gillware include dedicated computer scientists who have studied all of the intricacies and ins and outs of VMDK files and a VMware virtual machine. We don’t just know how ESXi hypervisors and other VMware virtual machines work, we know how they stop working—and what to do when that happens. Our extensive knowledge of virtual machine architecture has helped us develop groundbreaking VMware file recovery techniques.

VMDK file types

There are many different types of VMDK files used by VMware software products, but they all have the extension .vmdk, and the kind of VMDK file type directly affects the ability to recover VMDK files.

Before we start this overview of VMDK files and how VMware file systems work, we highly recommend you do not proceed with VMDK recovery software or a VMDK recovery tool on your own. VMDK file recovery is complicated and, if done incorrectly, data from VMDK files may be permanently lost, with VMDK recovery rendered impossible.

If you’re taking the risk of an amateur fix with VMDK recovery software, then you can start here. For VMDK recovery software to work, you must first understand the different types of VMDK:

• VMDK Descriptor file – This file is only a few kilobytes in size and is essentially a VMDK text configuration file used by the Virtual Machine VMX to understand the disk size and geometry of the VMDK.
• VMDK Flat files – The flat file, which is displayed as your_server-flat.vmdk, is where the data of your virtual machine is actually kept. There is typically one VMDK flat file per Virtual disk on the server.
• VMDK Delta files – A VMDK delta file is created when a point-in-time snapshot is taken of the server. The delta file tracks all the changes made to the VMDK flat file. When the snapshot is deleted, the delta file merges into the flat VMDK file.
• RDM VMDK file – An RDM is a pass-thru disk for mapping to a physical device, usually kept on a storage attached network (SAN). The RDM.vmdk is used to track file I/O between the physical disk and the virtual machine.

Many other files make up a Virtual machine file system, but the ones mentioned above specifically relate to VMDK files. If the VMDK flat file is corrupt or you are experiencing issues with damaged RDM’s, then VMDK file recovery is very difficult – you need to contact the Gillware team to discuss our expert recovery. We advise that you should not attempt RDM or VMDK flat file recovery, as you will likely cause further corruption and affect a larger number of VMs.

If you have a corrupt VMDK descriptor file or VMDK snapshot delta – then it is possible to recover data from the VMDK files, and something you might be considering to attempt at home with a VMDK recovery tool, provided you can take the risk of losing all your data if things go wrong with your VMDK files.

VMDK descriptor files can be rebuilt, and VMDK delta files can be repaired – however, it must be mentioned that working on delta files may result in rolling back your virtual machine to the previous snapshot, therefore losing any new data. We will discuss these options later on.

VMware VMDK Recovery Software

A quick note about off-the-shelf VMDK recovery software and VMDK recovery tool options: before you or your team attempt any kind of VMware VMDK fix, you must decide whether to proceed with this extremely risky procedure. VMware server recovery is one of the toughest recovery solutions offered by Gillware; it is much, much harder than recovering data from filesystems such as fat, fat16, fat32. VMDK is a proprietary file type, and attempting to fix it at home with a VMDK recovery tool will almost certainly fail.

Even VMware themselves recommend that you ship the problem VMDK hosts to a VMDK recovery company like Gillware. You have been warned!

There are some specific scenarios where VMDK file recovery is possible. It also depends on which version of VMware you are running and what format your VMDKs are written in. If you are running vSphere 3.5 and early revisions of vSphere 4, VMware released a VMDK recovery tool that could repair a VMDK file in numerous scenarios. If you are running anything newer than vSphere 4.0.0, then this fix will not work, and considering vSphere 4.0 was deprecated in 2013, it is highly likely you are running vSphere 6 and above.

VMware stopped support of the VMDK recovery software tool with newer versions of vSphere and ESXi, but there is still a VMDK recovery tool built into the hypervisor image called vmfstools.

Another type of home fix is possible depending on which VMDK file has been deleted. If it is a flat-file then you will need a specialist data recovery company like Gillware, but if the Descriptor File of the VMDK or the Delta-VMDK is corrupt, there is a good chance that you can rebuild the VMDK files logically.

How to recreate a missing Virtual Machine Disk Descriptor File (VMDK)

VMware published a KB fix explaining how to recreate missing or corrupt descriptor files. This fix is intended to work with VMware Workstation and VMware vSphere. There are also video files that demo the process on the KB.

A datastore writes file data to the descriptor file that basically gives the VMDK an identity as well as defines the parameters of all VM media files. This problem will manifest itself with an error when the VM is started up.

Here is how to fix this problem:

Step 1 – Connect to the ESXi host command line. The host must have connectivity to the storage array. First, enable the SSH service from the Advanced Security in Center and then use your favorite SSH client (such as putty or iTerm2). You will need the ESXi host root password to log on.

Step 2 – Navigate to the VM Disk Array using the CLI – by default it will be located at:

cd /vmfs/volumes/myvmfsvolume/mydir

Step 3 – List the file system using the ls -l command. This will show you all the files that belong to the VM.

Step 4 – Next work out the VMs disk controller settings – you need this to work out the VMDK file size.

less *.vmx ~| grep -i virtualdev

Step 5 – List out the VMDK flat file and make a note of the VMDK size value in bytes. It will be in the format:

ls -l your_vm_flat.vmdk

Step 6 – Use the vmkfstools command to create a temporary descriptor file and an associated temporary flat file.

vmkfstools -c filesize -a scsi_type -d thin mytempvm.vmdk

Note: Input the filesize found in step 5

Step 7 – Rename the tempvmdk to the name of the original VMDK.

mv -i mytempvm.vmdk orginalvmdk-0001.vmdk

Step 8 – Use your favorite text editor (nano/vim etc) to replace the tempvm details with the original VM in the new descriptor file.

Step 9 – If the VM was not thin provisions, use your favorite text editor to remove the line: ddb.thinprovision = 2 from the descriptor file.

How to Recover from Corrupt VMDK Delta files

VMDK delta files are probably the easiest to fix, but as a word of caution, data loss is possible, especially if your datastore is low on disk space.

To fix a corrupt VMDK Delta, simply clone the VM to a new instance and delete the old one once the clone has been tested. This process will merge all the snapshots and will give you a clean VM.

The easiest way to do this is to log into Center, or find the VM in VMware Workstation, right-click on the VM, and click Clone. Once cloned power off the original server, power up the new one, and after successfully testing, delete the original.

Gillware VMware Data Recovery Services

As you can see from the process above, VMDK recovery can be hard, and frankly, fixing the descriptor file is one of the easier tasks to complete. Are you having problems with a corrupt VMDK file? Do you need help to recover data? This is where Gillware can come in and help you.

Due to the complexity of VMware VMDK recovery and corrupt VMDK files, we recommend you call Gillware to discuss your needs rather than attempting to recover data yourself or use a VMDK recovery tool. We don’t just know how ESXi hypervisors and a VMware virtual machine work. We know how they stop working—and what to do when that happens. Our extensive knowledge of virtual machine architecture has helped us develop groundbreaking VMWare data recovery techniques to recover VMDK files.

There are two types of VMware VMDK failure that we see: data loss and corrupt VMDK files caused by a physical hardware failure on the host. A bad sector on a physical hard drive’s platters will (if the sector happens to live inside the area taken up by the virtual disk) cause damage to the virtual machine as well. Even highly fault-tolerant RAID servers can crash due to hard disk failure, as we here at Gillware can easily attest. When that happens, all of the virtual machines you had on that server go up in smoke.

We also see file data loss due to logical failures, which might be a boot sector or partition table corruption.

Gillware VMware Data Recovery Process

When clients come to us with a VMware virtual machine they need data recovered from, those machines typically live on NAS, SAN, or RAID servers that have crashed. These servers typically have anywhere from four to four dozen hard drives inside of them, connected by some means into single large storage volumes (or into several large iSCSI targets).

Recovering VMDK file data from virtual machine disk files takes a lot of work. For starters, before our VMWare data recovery specialists can even touch the VMs, we must put all of those hard drives back together again.

After doing whatever it takes to rebuild the storage volumes (often involving repairs to two or more hard drives), our engineers can recover the virtual machines themselves, and after that, the file systems and files within the virtual machines. Our engineers must peel back layer after layer before reaching our client’s partition data files.

Once our engineers have pulled your virtual machines out of the wreckage, the next step in the VMware data recovery process is to pry open those virtual machines and examine their contents. To do this, our engineers mount the virtual machines on physical hard disks, just as if they were a hard disk pulled from any normal computer.

We use proprietary VMDK file recovery software known internally as “HOMBRE” to examine the GUID partition table and look for recovery settings and recovered files and folders using a bespoke recovery algorithm,

By cross-referencing status maps and carefully spot-testing files within the recovered virtual hard disks, our VMWare data recovery experts ensure that our clients get the best results possible from our data recovery efforts.