Virtualizing TrueNas Scale in VMWare using RDM

TrueNAS SCALE is an open-source storage platform that offers file, block, and object storage services, which can be used for home or business data storage. It's built on Debian, inheriting its scalability and performance features, along with the ZFS filesystem that TrueNAS is known for.

As I recently navigated the challenge of migrating a traditional TrueNAS storage pool into my established VMware cluster, I found myself at a crossroads. The ideal path would have been to employ a spare Host Bus Adapter (HBA) to facilitate direct pass-through, but without one on hand, I was propelled to seek an alternative route.

My solution? Leaning on VMware's Raw Device Mapping (RDM). RDM is a neat feature that sidesteps the need for direct hardware access, allowing a virtual machine to deftly interact with physical drives. Though not as straightforward as using an HBA with PCI Passthrough, RDM offers the necessary workaround to keep TrueNAS's ZFS happy and performing well, all while meeting the goal of running in a VM.

Below I have played out the steps that can be taken to get this working within VMWare.

Prerequisites:

• A VMware vSphere environment.
• A VM with TrueNAS SCALE installer ISO attached.
• One or more additional hard disk drives (HDDs) or solid-state drives (SSDs) that are not part of any Datastore.
• If migrating from an existing environment like me, make sure to export the configuration in TrueNas.

Step 1: Prepare the Physical Disks

• Ensure that the disks you want to use for TrueNAS SCALE are not formatted or part of any VMFS datastore. These should be listed as "Available" drives in your ESXi configuration.

Step 2: Create RDM Pointers

1. Enable SSH on the ESXI Host.
2. SSH into your ESXi host.
3. Run ls /dev/disks -l to list all the available disks and identify the ones you want to map to TrueNAS SCALE.
4. Create an RDM pointer using the vmkfstools -z /vmfs/devices/disks/<disk_device> /vmfs/volumes/datastore_name/rdm_diskname.vmdk command, replacing <disk_device>, datastore_name, and rdm_diskname with appropriate values.
5. Repeat this step for each disk you want to map.

Step 3: Attach RDM Disks to the Virtual Machine

1. Right-click your TrueNAS SCALE VM and select 'Edit Settings'.
2. Click 'Add Hard Disk', choose 'Use an existing virtual disk', and browse to the location of the RDM pointer file you created. Select the file and finish the wizard.

Step 4: Install TrueNAS SCALE

• Boot the VM using the TrueNAS SCALE ISO as the boot media.
• Follow the installer prompts to install TrueNAS SCALE onto the desired Virtual Disk. For ease of backing up through VEEAM I have chosen to install it on a virtual disk, however booting of an RDM disk would work too.
• Once the installation is complete, remove the installation media and reboot the VM.

Step 5: Configure TrueNAS SCALE

After logging into TrueNAS SCALE's web interface, proceed with configuring your storage. Your RDM disks should be available for configuration within the TrueNAS SCALE UI, which you can utilize to set up pools, datasets, and shares.

In my case I just had to import the configuration exported from my bare metal instance, and voila, the pool was working as expected within the VM.

Benefits of Using RDM with TrueNAS SCALE:

• Direct Disk Access: TrueNAS SCALE will have direct access to the physical disks, which is crucial for the ZFS file system to ensure data integrity and performance.
• Flexibility: You can manage disk space and redundancy within TrueNAS SCALE rather than vSphere, offering more granular control over your storage configuration.
• Performance: Using RDMs can provide near-native disk performance, making it ideal for TrueNAS SCALE which is designed to manage and optimize disk I/O.

Important Considerations:

• Backup and DR: Due to the use of RDM backing up or using VMotion with the RDM disks will not be possible. As I am using S3 as backup storage directly within TrueNAS, this is a risk I am happy to accept.
• Lack of SMART Monitoring: By passing the disks trough RDM you will loose access to the SMART tests within TrueNAS. If this is a strict requirement, going with an HBA is recommended.