In Part 1 of this series, Usable Capacity Comparison between vSAN/VxRAIL and Nutanix, we’ve learned Nutanix provides more usable Capacity along with greater capacity efficiency, flexibility, resiliency, and performance when using Deduplication & Compression as well as Erasure Coding.

I provided another comparison of Usable Capacity between vSAN/VxRAIL and Nutanix using nodes equivalent to a DellEMC VxRAIL P-Series which are described by DellEMC as “high performance nodes optimised for heavy workloads.

The result was not surprisingly, 18.38TB (~16%) more usable capacity for Nutanix on a four node solution and a whopping 77.925TB (~31%) when the solution is scaled to 8 nodes.

Onto the topic of Memory usage, we have one of the easier (while still often misunderstood/misrepresented) topics to compare as the Nutanix CVM, being a Virtual Machine is configured with a static amount of memory.

By default, the Nutanix CVM is configured with between 24GB for the smallest nodes and 40GB for the largest nodes assuming aggressive recovery point objectives are required, otherwise they’re 32GB.

For vSAN/VxRAIL, VMware has published and updated a Knowledge base article titled “Understanding vSAN memory consumption in ESXi 6.0 U3, 6.5.0d, and later (2113954)”.

The article provides numerous examples, the first is a small One disk group hybrid configuration where the environment requires 16.478GB of RAM for vSAN.

Example 1: One disk group per host, hybrid configuration::

Formula:

HOST_FOOTPRINT + ( NumDiskGroups * ( DISKGROUP_FIXED_FOOTPRINT + DISKGROUP_SCALABLE_FOOTPRINT + ( CacheSize * CACHE_DISK_FOOTPRINT) + NumCapacityDisks * CAPACITY_DISK_FOOTPRINT)))

Example:

7100 + (1610 + 1228 + 600 * 10 + 3 * 180) = 16478 MB

The KB doesn’t provide a two disk group example, so I’ve taken a screenshot from my lab where it shows 33.29GB of memory used by the host for a two disk group, all flash configuration (6 x SATA-SSDs) without deduplication and compression enabled.

Returning to the KB we find example 4 which calculates a Three disk group configuration running deduplication and compression which uses 52.9GB of RAM.

Example 4: Three disk groups per host, dedup configuration:

Formula:

HOST_FOOTPRINT + NumDiskGroups * (DISKGROUP_FIXED_FOOTPRINT + DISKGROUP_SCALABLE_FOOTPRINT + CacheSize * CACHE_DISK_FOOTPRINT + NumCapacityDisks * CAPACITY_DISK_FOOTPRINT)

Example:

7100 + 3 * (1360 + 120 + 1310 + 600 * 20 + 3 * 160) = 52910 MB

Note: An additional 120MB is added in the diskgroup footprint for dedup.

Returning to my lab, I have a four disk group configuration running on Dell R740xd model with 4 x NVMe and 20 x SSD drives (5 SSDs per disk group) and we see vSAN and the host is using >70GBGB RAM with no VMs running and without deduplication and compression.

The exact figure for vSAN would be:

7100 + 4 * (1360 + 600 * 20 + 5 * 160) = 63740MB without Dedupe & Compression.

64GB just for vSAN.

It should be noted that vSAN supports up to 5 disk groups with 7 capacity drives per disk group, meaning vSAN/VxRAIL RAM usage will be higher than the examples provided if additional capacity drives and disk groups are used.

How much RAM will vSAN/VxRAIL use when scaled up to it’s maximum 5 disk groups with 7 capacity drives each?

Here is the formula.

7100 + 5 * (1360 + 600 * 20 + 7 * 160) = 80100MB

80GB of RAM required just for vSAN.

Note: This figure is without Dedupe & Compression.

As for Nutanix, what would drive a Controller VM to need (or benefit from) more than the 40GB maximum value out of the box?

If a node has a very large active working set (not to be confused with a large “dataset” which are typically mostly cold data), say 10s of TB of data being accessed throughout the day then having more RAM assigned to the CVM will allow the “medusa cache” to cache more metadata and ensure the fastest possible I/O.

So how do vSAN/VxRAIL and Nutanix compare?

The point many people are not aware (or in our competitors case, choose not to acknowledge) is that the Nutanix CVM and the 24-40GB or higher RAM which is assigned to it are not just providing a basic I/O layer such as vSAN.

The Nutanix CVM also proves a highly available and distributed management layer (PRISM) which includes data protection & replication, analytics and performance monitoring. The CVM resource usage should be compared with vSAN, plus vCenter, Update Manager, vSphere Replication & vRealize suite which use more memory resources as I’ve previous written.

But for the sake of this comparison, we can give VMware/vSAN the benefit of the doubt and just compare the memory usage of vSAN with the CVM.

Important to Note: VMware advises “vSAN consumes additional memory when the number of nodes in the cluster is greater than 32.”

The Nutanix CVM resource usage is designed to ensure linear scalability of all components in the Nutanix distributed platform and RAM is not required to be increased regardless of cluster size.

What do I recommend for Nutanix CVMs RAM?

My recommendations always trace back to the specific business requirements the environment is designed to achieve. In most cases, customers enjoy excellent consistent performance with CVMs with their default RAM (i.e.: <=40GB) even on large nodes with 24 drives.

For environments running business critical applications with large working sets and/or extremely latency sensitive workloads, I have previously recommended between 48-64GB of RAM for the CVM as the value of the additional memory can be significant in some cases, but in many cases the default memory of up to 40GB (typically 32GB) is fine.

If a customer uses a 24 drives system like in this example and the capacity requirements are high, let’s say 50TB+ but the active working set is only 1-2TB, then a CVM with 32GB RAM is more than sufficient to ensure excellent performance.

In this scenario, vSAN uses ~64GB regardless of the active working set.

On the flip side, a Nutanix customer using a 6 drive system with only 4TB per node but the same active working set of 1-2TB would also use a 32GB CVM.

The key here is it’s not the total capacity which drives CVM memory requirements, it’s the active working set and it’s metadata which the CVM ideally want’s to cache into RAM to ensure optimal performance.

If the CVM does not have enough RAM to cache the active working sets metadata, the result is that whatever metadata is not cached will not have optimal I/O performance. In many cases this is an insignificant difference in the real world, while for the top 5% of workloads, it can be a significant factor and a worthwhile investment to assign the CVM additional memory.

The CVM can easily be assigned more memory with a few clicks in PRISM and the system applies the changes to all CVMs in a rolling manner.

Summary:

Depending on the vSAN/VxRAIL configuration, memory usage will typically be equal or higher than the Nutanix CVM where the recommended multiple disk groups are used.

vSAN memory usage may be slightly lower than Nutanix CVM where customers have small single disk group configurations. However it’s important to note, single disk group vSAN configurations have significant resiliency issues and the single cache drive does restrict the performance which can be achieved.

On the other hand the Nutanix CVM provides significantly more native capabilities than vSAN even if we only talk at the storage layer with things like Storage Only nodes, in-line disk balancing and a granular distributed storage fabric.

Comparing only vSAN with the Nutanix CVM is far from apples for apples. Even if we accept this unfair comparison, as we’ve learned, Nutanix is typically using equal or less memory while providing a much more resilient & feature-rich solution.

Next up we compare Network Usage between Nutanix vs VMware vSAN/DellEMC VxRAIL!

Related Posts: