I should have ordered balloons and streamers, because Monday was VM creation day on my VMware cluster.

In addition to a 3-node production-licensed vSphere cluster, I run a 10-node cluster specifically for academic purposes. One of those purposes is building and maintaining classroom environments. A lot of professors maintain a server or two for their courses, but our Information Assurance program here goes above and beyond in terms of VM utilization. Every semester, I’ve got to deal with the added load, so I figured if I’m going to document it, I might as well get a blog entry while I’m at it.

Conceptually, the purpose of this process is to allow an instructor to create a set of virtual machines (typically between 1 and 4 of them), collectively referred to as a ‘pod’, which will serve as a lab for students. Once this set of VMs is configured exactly as the professor wants, and they have signed off on them, those VMs become the ‘Gold Images’, and then each student gets their own instance of these VMs. A class can have between 10 and 70 students, so this quickly becomes a real headache to deal with, hence the automation.

Additionally, because these classes are Information Assurance courses, it’s not uncommon for the VMs to be configured in an insecure manner (on purpose) and to be attacked by other VMs, and to generally behave in a manner unbecoming a good network denizen, so each class is cordoned off onto its own VLAN, with its own PFsense box guarding the entryway and doing NAT for the several hundred VMs behind the wall. The script needs to automate the creation of the relevant PFsense configs, too, so that comes at the end.

I’ve written a relatively involved PowerShell script to do my dirty work for me, but it’s still a long series of things to go from zero to working classroom environment. I figured I would spend a little time to talk about what I do to make this happen. I’m not saying it’s the best solution, but it’s the one I use, and it works for me. I’m interested in hearing if you’ve got a similar solution going on. Make sure to comment and let everyone know what you’re using for these kinds of things.

The process is mostly automated hard parts separated by manual staging, because I want to verify sanity at each step. This kind of thing happens infrequently enough that I’m not completely trusting of the process yet, mostly due to my own ignorance of all of the edge cases that can cause failures. To the right, you’ll see a diagram of the process.

In the script, the first thing I do is include functions that I stole from an awesome post on Subnet Math with PowerShell from Indented!, a software blog by Chris Dent. Because I’m going to be dealing with the DHCP config, it’ll be very helpful to be able to have functions that understand what subnet boundaries are, and how to properly increment IP addresses.

I need to make sure that, if this powershell script is running, that we are actually loading the VMware PowerCLI commandlets. We can do that like this:

if ( ( Get-PSSnapin -name VMware.VimAutomation.Core -ErrorAction SilentlyContinue ) -eq $null ) {

Add-PSSnapin VMware.VimAutomation.Core

}

For the class itself, this whole process consists of functions to do what needs to be done (or “do the needful” if you use that particular phrase), and it’s fairly linear, and each step requires the prior to be completed. What I’ve done is to create an object that represents the course as a whole, and then add the appropriate properties and methods. I don’t actually need a lot of the power of OOP, but it provides a convenient way to keep everything together. Here’s an example of the initial class setup:

$IA = New-Object psobject

# Lets add some initial values

Add-Member -InputObject $IA -MemberType NoteProperty -Name ClassCode -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name Semester -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name Datastore -Value “FASTDATASTORENAME”

Add-Member -InputObject $IA -MemberType NoteProperty -Name Cluster -Value “IA Program”

Add-Member -InputObject $IA -MemberType NoteProperty -Name VIServer -Value “VSPHERE-SERVER”

Add-Member -InputObject $IA -MemberType NoteProperty -Name IPBlock -Value “10.0.1.0”

Add-Member -InputObject $IA -MemberType NoteProperty -Name SubnetMask -Value “255.255.0.0”

Add-Member -InputObject $IA -MemberType NoteProperty -Name Connected -Value $false

Add-Member -InputObject $IA -MemberType NoteProperty -Name ResourcePool -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name PodCount -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name GoldMasters -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name Folder -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name MACPrefix -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name ConfigDir -Value “”

Add-Member -InputObject $IA -MemberType NoteProperty -Name VMarray -Value @()

These are just the values that almost never change. Since we’re using NAT, and we’re not routing to that network, and every class has its own dedicated VLAN, we can use the same IP block every time without running into a problem. The blank values are there just as placeholder, and those values will be filled in as the class methods are invoked.

At the bottom of the script, which is where I spend most of my time, I set per-class settings:

$IA.ClassCode = “ia1234”

$IA.Semester = “Fall-2014”

$IA.PodCount = 35

$IA.GoldMasters = @(

@{

vmname = “ia1234-win7-gold-20141014”

osname = “win7”

tcp = 3389

udp = “”

},

@{

vmname = “ia1234-centos-gold-20141014”

osname = “centos”

tcp = “”

udp = “”

},

@{

vmname = “ia1234-kali-gold-20141014”

osname = “kali”

tcp = “22”

udp = “”

}

)

We set the class code, semester, and pod count simply. These will be used to create the VM names, the folders, and resource groups that the VMs live in. The GoldMaster array is a data structure that has an entry for each of the gold images that the professor has created. It contains the name of the gold image, plus a short code that will be used to name the VM instances coming from it, and has a placeholder for the tcp and udp ports which need forwarded from the outside to allow internal access. I don’t currently have the code in place that allows me to specify multiple port forwards, but that’s going to be added, because I had a professor request 7(!) forwarded ports per VM in one of their classes this semester.

As you can see in the diagram, I’m using Linked Clones to spin up the students’ pods. This has the advantage of saving diskspace and of completing quickly. Linked clones operate on a snapshot of the original disk image. Rather than actually have the VMs operate on the gold images, I do a full clone of the VM over to a faster datastore than the Ol’ Reliable NetApp.

We add a method to the $IA object like this:

Add-Member -InputObject $IA -MemberType ScriptMethod -Name createLCMASTERs -Value {

# This is the code that converts the gold images into LCMASTERs

# Because you need to put a template somewhere, it makes sense to put it

# into the folder that the VMs will eventually live in themselves (thus saving

# yourself the effort of locating the right folder twice).

Param()

Process {

… stuff goes here

}

}

The core of this method is the following block, which actually performs the clone:

if ( ! (Get-VM -Name $LCMASTERName) ) {

try {

$presnap = New-snapshot -Name (“Autosnap: “ + $(Get-Date).toString(“yyyMMdd”)) -VM $GoldVM -confirm:$false

$cloneSpec = new-object VMware.Vim.VirtualMachineCloneSpec

$cloneSpec.Location = New-Object VMware.Vim.VirtualMachineRelocateSpec

$cloneSpec.Location.Pool = ($IA.ResourcePool | Get-View).MoRef

$cloneSpec.Location.host = ($vm | Get-VMHost).MoRef

$cloneSpec.Location.Datastore = ($IA.Datastore | Get-View).MoRef

$cloneSpec.Location.DiskMoveType = [VMware.Vim.VirtualMachineRelocateDiskMoveOptions]::createNewChildDiskBacking

$cloneSpec.Snapshot = ($GoldVM | Get-View).Snapshot.CurrentSnapshot

$cloneSpec.PowerOn = $false

($GoldVM | Get-View).cloneVM( $LCMasterFolder.MoRef, $LCMASTERName, $cloneSpec)

Remove-snapshot -Snapshot $presnap -confirm:$false

}

catch [Exception] {

Write-Host “Error: “ $_.Exception.Message

exit

}

} else {

Write-Host “Template found with name $LCMasterName — not recreating”

}



(apologies for the lack of indentation)

If you’re interested in doing this kind of thing, make sure you check out the docs for the createNewChildDiskBacking setting.

After the Linked Clone Masters have been created, then it’s a simple matter of creating the VMs from each of them (using the $IA.PodCount value to figure out how many we need). They end up getting named something like $IA.ClassCode-$IA.Semester-$IA.GoldMasters[#].osname-pod$podcount which makes it easy to figure out what goes where when I have several classes running at once.

After the VMs have been created, we can start dealing with the network portion. I used to spin up all of the VMs, then loop through them and pull the MAC addresses to use with the DHCP config, but there were problems with that method. I found that a lot of the time, I’ll need to rerun this script a few times per class, either because I’ve screwed something up or the instructor needs to make changes to the pod. When that happens, EACH TIME I had to re-generate the DHCP config (which is easy) and then manually insert it into PFsense (which is super-annoying).

Rather than do that every time, I eventually realized that it’s much easier just to dictate what the MAC address for each machine is, and then it doesn’t matter how often I rerun the script, the DHCP config doesn’t change. (And yes, I’m using DHCP, but with static leases, which is necessary because of the port forwarding).

Here’s what I do:

Add-Member -InputObject $IA -MemberType ScriptMethod -Name assignMACs -Value {

Param()

Process {

$StaticPrefix = “00:50:56”

if ( $IA.MACPrefix -eq “” ) {

# Since there isn’t already a prefix set, it’s cool to make one randomly

$IA.MACPrefix = $StaticPrefix + “:” + (“{0:X2}” -f (Get-Random -Minimum 0 -Maximum 63) )

}

$machineCount = 0

$IA.VMarray | ForEach-Object {

$machineAddr = $IA.MACPrefix + “:” + (“{0:X4}” -f $machineCount).Insert(2,”:”)

$vm = Get-VM -name $_.name

$networkAdapter = Get-NetworkAdapter -VM $vm

Write-Host “Setting $vm to $machineAddr”

Set-NetworkAdapter -NetworkAdapter $networkAdapter -MacAddress $machineAddr -Confirm:$false

$IA.VMarray[$machineCount].MAC = $machineAddr

$IA.VMarray[$machineCount].index = $machineCount

$machineCount++

}

}

}

As you can see, this randomly assigns a MAC address in the vSphere range. Sort of. The fourth octet is randomly selected between 00 and 3F, and then the last two octets are incremented starting from 00. Optionally, the fourth octet can be specified, which is useful in a re-run of the script so that the DHCP config doesn’t need to be re-generated.

After the MAC addresses are assigned, the IPs can be determined using the network math:

Add-Member -InputObject $IA -MemberType ScriptMethod -Name assignIPs -Value {

# This method really only assigns the IP to the object.

Param()

Process {

# It was tempting to assign a sane IP block to this network, but given the

# tendancy to shove God-only-knows how many people into a class at a time,

# lets not be bounded by reasonable or sane. /16 it is.

# First 50 IPs are reserved for gateway plus potential gold images.

$currentIP = Get-NextIP $IA.IPBlock 2

$IA.VMarray | ForEach-Object {

$_.IPAddr = $currentIP

$currentIP = Get-NextIP $currentIP 2

}

}

}

This is done by naively giving every other IP to a machine, leaving the odd IP addresses between them open. I’ve had to massage this before, where a large pod of 5–6 VMs all need to be incremental then skip IPs between them, but I’ve done those mostly as a one-off. I don’t think I need to build in a lot of flexibility because those are relatively rare cases, but it wouldn’t be that hard to develop a scheme for it if you needed.

After the IPs are assigned, you can create the DHCP config. Right now, I’m using an ugly hack, where I basically just print out the top of the DHCP config, then loop through the VMs outputting XML the whole way. It’s ugly, and I’m not going to paste it here, but if you download a DHCPD XML file from PFsense, then you can basically see what I’m doing. I then do the same thing with the NAT config.

Because I’m still running these functions manually, I have these XML-creation methods printing output, but it’s easy to see how you could have them redirect output to a text file (and if you were super-cool, you could use something like this example from MSDN where you spin up an instance of IE:

$ie = new-object -com “InternetExplorer.Application”

$ie.navigate(“http://localhost/MiniCalc/Default.aspx")

… and so on

Anyway, I’ve spun up probably thousands of VMs using this script (or previous instances of it). It’s saved me a lot of time, and if you have to manage bulk-VMs using vSphere, and you’re not automating it (using PowerCLI, or vCloud Director, or something else), you really should be. And if you DO, what do you do? Comment below and let me know!

Thanks for reading all the way through!