I was wondering if it possible to speed up my Intel NUC based ESXi with Virtual SAN. The idea is that compared against vSphere Flash Read Cache, Virtual SAN can use the SSD not only as read cache but also as write buffer. This post explains how you can create a Virtual SAN Datastore on a single ESXi host from the command-line without a vCenter Server.

It goes without saying that this is neither the idea behind Virtual SAN nor officially supported by VMware. It also violates VMware's EULA if you are running Virtual SAN without a VSAN license. To assign a licence you need a vCenter Server and wrap the single ESXi into a Cluster.

My configuration for this test:

5th Gen Intel NUC NUC5i5MYHE

32GB Memory

SSD: 250GB Samsung 850 Evo M.2

HDD: 2.5" 1TB 5400rpm

vSphere ESXi 6.0

The guide starts with a fresh ESXi installation in evaluation mode and I will not cover licensing. According to this post it also works with the free ESXi Hypervisor license, but to be compliant with VMware's EULA you have to add the host to a vCenter Server and assign a Virtual SAN license.

First step is to enable VSAN traffic for the VMkernel interface

~ # esxcli vsan network ipv4 add -i vmk0

Next step is to identify the devices to be used for VSAN. Both, SSD and HDD, must not contain any data or partitions. The easiest way to identify devices is with the vdq -q command.

In this example, the HDD already contains a VMFS partition:

~ # vdq -q [ { "Name" : "t10.ATA_Samsung_SSD_850_EVO_M.2_250GB", "VSANUUID" : "", "State" : "Eligible for use by VSAN", "ChecksumSupport": "0", "Reason" : "None", "IsSSD" : "1", "IsCapacityFlash": "0", "IsPDL" : "0", }, { "Name" : "t10.ATA_ST9100AS", "VSANUUID" : "", "State" : "Ineligible for use by VSAN", "ChecksumSupport": "0", "Reason" : "Has partitions", "IsSSD" : "0", "IsCapacityFlash": "0", "IsPDL" : "0", }, ]

Use partedUtil to identify and delete all partions on ineligible drives. The partedUtil get command displays one partition (marked red) which can be deleted with the partedUtil delete command.

~ # partedUtil get /dev/disks/t10.ATA_ST9100AS 60801 255 63 976773168 1 2048 976768064 0 0 ~ # partedUtil delete /dev/disks/t10.ATA_ST9100AS 1

Next step is to add physical disks for VSAN usage. VSAN requires one SSD and one, or multiple HDDs. Use the esxcli vsan storage add command with the options -d HDD1 -d HDD2 -s SDD

~ # esxcli vsan storage add -d t10.ATA_ST9100AS -s t10.ATA_Samsung_SSD_850_EVO_M.2_250GB

To enable VSAN on the host simply create a new Cluster.

~ # esxcli vsan cluster new

Now you should see the vsanDatastore, but creating Virtual Machines fails with the following error message:

Create virtual machine

Cannot complete file creation operation.

This is caused by a missing redundancy defined in default policies. To fix this, change the default policies to hostFailuresToTolerate = 0 and forceProvisioning = 1.

~ # esxcli vsan policy setdefault -c cluster -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))" ~ # esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))" ~ # esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))" ~ # esxcli vsan policy setdefault -c vmswap -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))" ~ # esxcli vsan policy setdefault -c vmem -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"

You can verify the default policy with the following command:

~ # esxcli vsan policy getdefault

That's it. You can now use the Virtual SAN datastore on your single ESXi host.

~ # esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2015-11-09T10:26:45Z Local Node UUID: 563fc008-ab73-2e34-0075-005056bdaaf9 Local Node State: MASTER Local Node Health State: HEALTHY Sub-Cluster Master UUID: 563fc008-ab73-2e34-0075-005056bdaaf9 Sub-Cluster Backup UUID: Sub-Cluster UUID: 520afa76-164d-bab1-14ac-3cddab7a8570 Sub-Cluster Membership Entry Revision: 0 Sub-Cluster Member UUIDs: 563fc008-ab73-2e34-0075-005056bdaaf9 Sub-Cluster Membership UUID: db744056-5f49-77a1-7f1d-005056bdaaf9

~ # esxcli vsan storage list t10.ATA_ST9100AS Device: t10.ATA_ST9100AS Display Name: t10.ATA_ST9100AS Is SSD: false VSAN UUID: 5272476c-b445-4623-73f5-e2cd2a6265fd VSAN Disk Group UUID: 52b7be05-b875-a93e-24bc-62c2210948d7 VSAN Disk Group Name: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB Used by this host: true In CMMDS: false Checksum: 10071299334450500695 Checksum OK: true Emulated DIX/DIF Enabled: false t10.ATA_Samsung_SSD_850_EVO_M.2_250GB Device: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB Display Name: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB Is SSD: true VSAN UUID: 52b7be05-b875-a93e-24bc-62c2210948d7 VSAN Disk Group UUID: 52b7be05-b875-a93e-24bc-62c2210948d7 VSAN Disk Group Name: t10.ATA_Samsung_SSD_850_EVO_M.2_250GB Used by this host: true In CMMDS: false Checksum: 3233630865690359925 Checksum OK: true Emulated DIX/DIF Enabled: false

I've done some basic performance tests with a datastore directely on the HDD, and with the VSAN datastore. Here are the results:

The test was the Exchange 2007 Workload from VMware I/O Analyzer (8k 55%Read 80%Random).

VMFS Datastore on local HDD

IOPS: 226

Read IOPS: 125

Write IOPS: 101

MBPS: 1.77

Read MBPS: 0.98

Write MBPS: 0.79

Single-Node VSAN Datastore

IOPS: 5294

Read IOPS: 2911

Write IOPS: 2382

MBPS: 41.36

Read MBPS: 22.75

Write MBPS: 18.61