Heath

What is Multi-NIC vMotion?

Multi-NIC vMotion allows you to send multiple vMotion streams (even when migrating a single virtual machine), and if configured properly provides a higher overall throughput to the vMotion process. The configuration is straightforward, the vMotion service is configured on multiple VMkernel adapters, and each VMkernel adapter is associated with a single physical interface. With multiple gigabit interfaces the throughput of the vMotion migration scales linearly with amount of adapters used, with 10G you will very likely hit some other performance barriers once you have more than two 10G interfaces involved.





This feature was added in ESXi 5.0 and remains mostly unchanged. In my experience it is not a heavily used feature, although I depend on it in our production environment and have been running it since early 2012 in 5.0 and now 5.5.

Why do I need it?

There are two scenarios where the available throughput for vMotion can come into play. The first is simple, hosts with a large amount of RAM and a very large number of VMs. The second can be a little more complicated, virtual machines under heavy load that are dirtying memory pages very rapidly.





The benefit of mulit-NIC vMotion when dealing with large hosts with many virtual machines is obvious. We have many hosts with 1TB of ram, and a few hosts that have 2TB of ram. Placing these hosts into maintenance mode would take FOREVER over a single gig interface, and quite some time over a single 10G interface.





The second use case is for migrating a large single VM under heavy load. During vMotion the guests memory is copied, then a delta is copied containing the pages that were changed or "dirtied" during the first copy. If the guest is dirtying memory pages faster than they can be copied this will cause Stun During Page Send to kick in and slow the rate that the guest OS can dirty the memory pages so the copy can finish. This stun has caused us some problems with big databases servers, and the only way to avoid it is to throw more bandwidth at vMotion.





Dive into configuration after the break.

How do I turn it on?

For the "how to" I'll discuss the VMware distributed switch. If you want to run mulit-NIC vMotion on the VMware standard switch it should be simple to figure out from the distributed switch info.





The first step to enable multi-NIC vMotion is to create additional VMkernel interfaces, and enable the vMotion service on them. These additional interfaces should be placed in the same IP subnet as the existing VMkernel vMotion interfaces. In my case I already have an existing VMkernel interface for vMotion in the vMotion-A port group with an IP address of 192.168.2.186. We will be adding a second port group and VMkernel interface for Multi-NIC vMotion.

1. Create a second vMotion port-group













Next create a new port group, named vMotion-B, using the same VLAN as the vMotion-A port group. From the Networking tab click on the "Create a New Distributed Port Group" icon.









I'll create the port group with default settings, simply setting the name to vMotion-B and the VLAN. I will come back later and modify the failover settings.





2. Create an additional vMotion VMkernel adapter.











Return to the Hosts and Clusters tab and navigate to the VMkernel adapter settings for the host. Click on the Add Host Networking to add the second VMkernel Interface.