Synology RS3412xs Data Recovery Case Study: Deleted LUNs

In this data recovery case, the client had four deleted LUNs they needed recovered. These LUNs, or iSCSI targets, had lived on a Synology RackStation RS3412xs network-attached storage device.

A network-attached storage device, or NAS device, is an external device that connects to other computers by a network. Instead of a USB port, it has an ethernet port, and connects to a router instead of an individual computer. People with the right permissions can access the storage device remotely.

NAS devices almost always contain RAID arrays. This is especially true when the users are small and medium-sized businesses. Single-drive NAS devices are only designed for personal use. This NAS device actually served as a storage area network. The Synology RAID provided multiple people in the client’s organization with access to what looked like local disks, but were actually remote storage pools.

The ten four-terabyte hard drives in this data recovery client’s NAS device were linked in a RAID-6 configuration. Their array had 32 terabytes of total disk space. RAID-6 provides fault tolerance for up to two hard drive failures. But one thing no RAID array can protect its users from is file deletion.

Intro to LUNs

A LUN is a “logical unit number”. These numbers only see use when certain storage devices such as SANs use SCSI, iSCSI, or Fibre Channel protocols to transmit data. When using any of these protocols, a space on the storage volume gets assigned a unique iSCSI target number. “Logical unit number” literally refers to that unique number itself and nothing more. But “LUN” is also used as shorthand for the entire logical unit as well. A LUN can define a portion of a hard drive, an entire hard drive, or an entire RAID array. A single LUN is also known as an iSCSI target.

LUNs are a bit similar to virtual partitions. They are “soft” partitions. A soft partition will appear to a user and operating system as a discrete hard drive, but doesn’t actually correspond to a “hard” physical partition on a disk or a separate data storage device. It’s more like a single file that pretends to be its own hard drive. An administrator can create as many LUNs as needed out of the space on a network device. These LUNs can then be doled out to the users.

Challenges in Recovering Deleted LUNs

In most deleted file data recovery cases, data is not lost as soon as it is deleted. In Windows and Mac filesystems, for example, the data comprising a deleted file and the extents which point toward that file remain on the drive until they are overwritten by new data. If a Mac or PC does not see much use after data is deleted from its hard drive, it is relatively easy for our engineers to recover the deleted files.

This NAS device, however, was not using the Windows NTFS or Mac HFS+ filesystems. Rather, it used the Linux Ext4 filesystem. This is good for our engineers. Ext4 is an open-source, extremely well-documented filesystem. Many NAS devices will use their own proprietary filesystems. These filesystems have no documentation and are a pain and a half to unravel.

Ext4 does not automatically erase a deleted files. But Ext4 is particularly destructive with regards to its deleted files’ metadata. When the user deletes a file from an Ext4 partition, the filesystem scrubs the file’s directory entries from the its inodes. Ext4, unfortunately, destroys the information that tells us where the data for deleted files live by design.

Ext4 Deleted File Recovery

Sometimes, the old extents can be restored using the record Ext4 keeps of its journaling activities. The Ext4 filesystem has a feature called “filesystem journaling”. When the user wants to make a change to the disk—for example, creating, modifying, or deleting a file—the filesystem first puts a record of the change into a log file before making the change. This feature is to ensure data resiliency. Normally, if a hard drive loses power or a computer crashes in the middle of a write operation, the newly-written data can become corrupted. But with journaling enabled, the filesystem can “roll back” to just before it executed its last command. It can then do it again without any danger of writing corrupted data to the drive.

But the deletion of the user’s LUNs wasn’t in the log file. One feature of the Ext4 filesystem’s journaling capability that proved to be a drawback in this situation was that the log file is a set size. When the log file fills up, it loops around to the beginning. It starts writing the newest recorded changes at the beginning of the file, overwriting the oldest entries. The Synology NAS device had, unfortunately, seen continued use after the file deletion incident.

Most of the data from the deleted LUNs was still located somewhere on the hard drives. But without the directory entries, the data would prove difficult to find. These files were one terabyte in size each. In this filesystem, the LUN files were in no way contiguous. These deleted iSCSI targets were scattered in thousands of pieces all across the array.

Recovering the Data from Deleted LUNs

Our logical data recovery expert Greg Andrzejewski had one important lead he could follow to recover these deleted LUNs. For extremely large files such as these deleted LUNs, the extents pointing to these files must be stored in a tree structure. The inode for the file points to data blocks. These blocks in turn point to all of the extents that make up the file.

The directory entry for each file in an Ext4 filesystem provides the system with a file name and inode number. When the system has to find one of these files, it goes to the inode number. From there, it follows the tree all the way down. Upon deletion of these iSCSI targets, the exact inode numbers assigned to the deleted LUNs became impossible to find. When a user applies Ext4 to a volume, it creates tens of millions of inodes right at the start. Most of these inodes do not see use, because most users do not have tens of millions of files. Finding the inodes that pointed to the client’s deleted LUNs would be next to impossible.

However, our data recovery expert Greg could find the extent trees for the deleted LUNs. Ext4 didn’t wipe out these trees along with the directory entries. So it was merely a matter of finding all of the extent trees, then ruling out all of the ones connected to extant files. Greg was able to recover almost all of the 4 deleted LUNs. The filesystem had only overwritten a fraction of a percent of one deleted iSCSI target. Once Greg had combined the iSCSI targets into a single disk image, we could see all of the client’s files within the deleted LUNs.

Conclusion

The total control we have over our own recovery tools made this deleted LUN recovery case possible. In order to recover the four deleted LUNs, we had to search for all of their pieces. We then had to cross-reference them and correlate them together. This would not be possible without a powerful forensic analysis tool like HOMBRE and its relational database. With the help of our own custom software, we could recover all of the client’s files from their deleted LUNs. None of their critical files were missing or incomplete. Our data recovery technicians also found no evidence of file corruption. We rated this data recovery case a high 9 on our ten-point case rating scale.