FreeNAS 9.3 now fully supports all 7 VAAI (vStorage APIs for Array Integration) block primitives:

Atomic Test & Set (ATS / Hardware Accelerated Locking)

XCOPY (Extended Copy / Clone)

Write Same (Zero)

Dead Space Reclamation (Delete / UNMAP)

Thin Provisioning LUN Reporting

Thin Provisioning Space Threshold Warning

Thin Provisioning Stun

Atomic Test & Set (ATS / Hardware Accelerated Locking)

ATS is a storage based locking mechanism to replace SCSI reservations on VMFS volumes when doing metadata updates. A SCSI reservation locks a datastore and prevents all other hosts from doing metadata updates on that datastore.

ATS is enabled by default on all FreeNAS iSCSI LUNS

Verify ATS support with esxcli storage core device vaai status get

[root@esx1:~] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f naa.6589cfc0000006948963c2aa5057083f VAAI Plugin Name: ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: supported

Verify ATS operations with esxtop. Press u f f g i o in esxtop. This will enable Device mode and all required fields.



ATS - Total number of ATS commands

- Total number of ATS commands ATSF - Total number of failed ATS commands.

XCOPY (Extended Copy / Clone)

Without VAAI clone and Storage vMotion operations are performed by the VMkernel. The XCOPY primitive allows to offload these requests to the storage array. This does not waste ESXi Host resources such as CPU cycles, DMA buffers and SCSI commands in the HBA queue.

XCOPY is enabled by default on all FreeNAS iSCSI LUNS

Verify XCOPY support with esxcli storage core device vaai status get

[root@esx1:~] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f naa.6589cfc0000006948963c2aa5057083f VAAI Plugin Name: ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: supported

Verify XCOPY operations with esxtop. Press u f f g i o in esxtop. This will enable Device mode and all required fields. A Virtual Machine was migrated with Storage DRS while the screenshot was taken.



Clone_RD - Total number of reads offloaded

- Total number of reads offloaded Clone_WR - Total number of writes offloaded

- Total number of writes offloaded Clone_F - Total number of failed CXOPY commands

- Total number of failed CXOPY commands MBC_RD/s - Current megabytes per second read

- Current megabytes per second read MBC_WR/s - Current magabytes per second written

Write Same (Zero)

When an eager-zeroed virtual disk is created, the entire space gets zeroed. The ESXi host has to write gigabytes of zeroes to the storage array which takes some time. By using the Write Same primitive, the ESXi host simply tells the storage array how much zeroes it should write. This would significantly shorten the time needed.

Write Same is enabled by default on all FreeNAS iSCSI LUNS

Verify Write Same support with esxcli storage core device vaai status get

[root@esx1:~] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f naa.6589cfc0000006948963c2aa5057083f VAAI Plugin Name: ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: supported

Verify Write Same operations with esxtop. Press u f f g i o in esxtop. This will enable Device mode and all required fields. An eager-zeroed virtual disk was created while the screenshot was taken.



Zero - Total number of ZERO commands offloaded

- Total number of ZERO commands offloaded Zero_F - Total number of failed ZERO commands

- Total number of failed ZERO commands MBZERO/s - Current megabytes per second zeroed

Dead Space Reclamation (UNMAP)

When a Virtual Machine on a thin-provisioned LUN is deleted, the storage array does not know that the blocks are unused, so it reports the space still as occupied. The UNMAP command enables an ESXi host to inform the storage array that space from a thin provisioned LUN can be reclaimed. This enables an array to reclaim space of a thin-provisioned datastore.

Dead Space Reclamation does only work if your iSCSI extent is a ZVOL created as sparse volume:



Verify Dead Space Reclamation support with esxcli storage core device vaai status get

[root@esx1:/vmfs/volumes/54b2d835-32f3bccf-e9d2-001b2193b3b0] esxcli storage core device vaai status get -d naa.6589cfc0000006948963c2aa5057083f naa.6589cfc0000006948963c2aa5057083f VAAI Plugin Name: ATS Status: supported Clone Status: supported Zero Status: supported Delete Status: supported

Dead Space is not automatically reclaimed. The unmap process must be triggered manually:

[root@esx1:~] vmkfstools -y /vmfs/volumes/esxi-sparse/ Try to unmap 152372 blocks in units of 200 blocks from volume /vmfs/volumes/esxi-sparse/.

Verify Dead Space Reclamation with esxtop. Press u f f g i o in esxtop.



Delete - Total number of UNMAP commands

- Total number of UNMAP commands Delete_F - Total number of failed UNMAP commands

- Total number of failed UNMAP commands MBDEL/s - Current megabytes per second reclaimed

VAAI Thin Provisioning Primitives

Thin Provisioning LUN Reporting

A simple VAAI primitive that allows an ESXi Host to determine if a LUN is thin-provisioned. Determine LUN status with esxcli storage core device list

[root@esx1:~] esxcli storage core device list -d naa.6589cfc0000006948963c2aa5057083f |grep Thin Thin Provisioning Status: yes

Thin Provisioning Space Threshold Warning

With the this VAAI primitive, a warning is displayed in the Event Log if a thin-provisioned datastore reaches a specific threshold. The threshold can be set in the extent configuration:



Thin Provisioning Stun

Without this VAAI primitive, all virtual machines would crashes if a datastore is full. With VAAI Stun, only those virtual machines requiring extra blocks of storage space are paused. After additional space is allocated to the thin-provisioned datastore, the paused virtual machines can be resumed.