We’re very pleased to announce that starting with the Mitaka release we have introduced a new Openstack Nova Compute Driver, the Hyper-V Failover Cluster Driver, offering host based fault tolerance to tenants. The main advantage is that non high-available applications running in virtual machines can rely on the cluster for fault tolerance, a very common scenario in the enterprise IT world.

This feature is based on Windows Server Failover Clustering (WSFC) and is very useful when you need services that are always available, even if the Hyper-V compute hosts go offline due to planned maintenance or unplanned hardware or software issues. Thanks to this feature, the virtual machines are fault-tolerant and get automatically migrated to other nodes in the same cluster, where they continue to work unaffected.

All of this is now integrated with OpenStack, with the Hyper-V Compute Driver seamlessly notifying Nova about any failovers in the cluster, ensuring that instance status is properly up-to-date.

Windows Clustering is fully supported and automatically configured in our OpenStack Hyper-V hyper-converged product offer.

In the remaining part of this post, we’re showcasing the Failover Clustering feature in a quick demo that will work on any Mitaka OpenStack deployment. All you need is a minimum of two Hyper-V nodes (joined to an Active Directory domain) to add to your deployment.

Firstly, let’s boot some instances on the Hyper-V Cluster.

nova boot --image cirros_vhd --flavor m1.small test1 nova boot --image cirros_vhd --flavor m1.small test2 nova boot --image cirros_vhd --flavor m1.small test3 1 2 3 nova boot -- image cirros_vhd -- flavor m1 .small test1 nova boot -- image cirros_vhd -- flavor m1 .small test2 nova boot -- image cirros_vhd -- flavor m1 .small test3

We can see the 3 instances with nova list:

And the same 3 instances in the Failover Cluster Manager:

Now, we’re going to forcefully restart the HV12R202 Hyper-V host server where two of the three instances are hosted. Being a planned restart, the instances hosted on this node will live migrate to other Hyper-V nodes in the cluster without any downtime. This is the so-called “node drain” feature. If instead the Hyper-V server undergoes an unplanned shutdown, the instances will failover to other nodes.

PS > shutdown -r -t 0 1 PS > shutdown -r -t 0

The instances have been live migrated to the other nodes in the cluster as we can see in the Cluster Manager shown in the following snapshot:



A snippet from the Hyper-V Nova Compute logs, showcasing the occurring failover events.

2016-04-22 07:50:08.865 4108 INFO hyperv.nova.cluster.clusterops [-] Checking instance failover instance-00000001 to HV12R201 from host HV12R202. 2016-04-22 07:50:09.009 4108 INFO nova.compute.manager [-] [instance: 2991095f-9fc0-4a64-9b35-ef5637e1757a] VM Started (Lifecycle Event) 2016-04-22 07:50:09.586 4108 INFO nova.compute.manager [req-f9e2d6cd-8ad1-42d8-82e1-5e98bd052596 - - - - -] [instance: 2991095f-9fc0-4a64-9b35-ef5637e1757a] During the sync_power process the instance has moved from host HV12R202 to host HV12R201 2016-04-22 07:50:10.141 4108 INFO hyperv.nova.cluster.clusterops [-] Instance instance-00000001 failover to HV12R201. 2016-04-22 07:50:11.142 4108 WARNING oslo.service.loopingcall [-] Function 'hyperv.nova.cluster.clusterops._looper' run outlasted interval by 0.28 sec 2016-04-22 07:50:11.150 4108 INFO hyperv.nova.cluster.clusterops [-] Checking instance failover instance-00000003 to HV12R201 from host HV12R202. 2016-04-22 07:50:12.130 4108 INFO hyperv.nova.cluster.clusterops [-] Instance instance-00000003 failover to HV12R201. 1 2 3 4 5 6 7 2016 - 04 - 22 07 : 50 : 08.865 4108 INFO hyperv .nova .cluster .clusterops [ - ] Checking instance failover instance - 00000001 to HV12R201 from host HV12R202 . 2016 - 04 - 22 07 : 50 : 09.009 4108 INFO nova .compute .manager [ - ] [ instance : 2991095f - 9fc0 - 4a64 - 9b35 - ef5637e1757a ] VM Started ( Lifecycle Event ) 2016 - 04 - 22 07 : 50 : 09.586 4108 INFO nova .compute .manager [ req - f9e2d6cd - 8ad1 - 42d8 - 82e1 - 5e98bd052596 - - - - - ] [ instance : 2991095f - 9fc0 - 4a64 - 9b35 - ef5637e1757a ] During the sync_power process the instance has moved from host HV12R202 to host HV12R201 2016 - 04 - 22 07 : 50 : 10.141 4108 INFO hyperv .nova .cluster .clusterops [ - ] Instance instance - 00000001 failover to HV12R201 . 2016 - 04 - 22 07 : 50 : 11.142 4108 WARNING oslo .service .loopingcall [ - ] Function 'hyperv.nova.cluster.clusterops._looper' run outlasted interval by 0.28 sec 2016 - 04 - 22 07 : 50 : 11.150 4108 INFO hyperv .nova .cluster .clusterops [ - ] Checking instance failover instance - 00000003 to HV12R201 from host HV12R202 . 2016 - 04 - 22 07 : 50 : 12.130 4108 INFO hyperv .nova .cluster .clusterops [ - ] Instance instance - 00000003 failover to HV12R201 .

The changes are consistent with Nova and the instances location has been updated.

Of course, you can try out this feature yourself. We’re going to explain the setup in the following part.

Requirements

OpenStack Mitaka deployment

Active Directory

At least 2 Hyper-V compute nodes in the same Active Directory, running Windows Hyper-V / Server 2012 R2 or newer, including Nano Server

Install the OpenStack Hyper-V Compute Driver on the Hyper-V nodes

Optionally, you can add Scale-Out File Server storage nodes, also joined to the Active Directory domain and part of the same cluster.

Setup

Run the following PowerShell script on a server in the Active Directory domain. This script will configure the Failover Clustering service on each Hyper-V compute node along with a SMB share, where the instances’ disks and configurations will be hosted on, since they will have to be accessible from each node. The shared storage used by Hyper-V can be either based on SMB or CSV, for this basic demo we’ll just use a simple SMB share, take a look at our OpenStack Hyper-V hyper-converged offer for a full Scale-Out File Server storage configuration.

######################## # Windows Server 2012 R2 # setting up server for share # please setup your own configurations here. $hyperv_nodes = @("HYPERV_NODE_01.DOMAIN", "HYPERV_NODE_02.DOMAIN") $cluster_name = "CLUSTER_NAME" $cluster_user = "DOMAIN\USER" $smb_path = "S:\SMB_SHARE" $smb_name = "SMB_SHARE" # install locally Install-WindowsFeature File-Services, FS-FileServer # this will install the features necessary, including the Failover Clustering feature. Invoke-Command -ComputerName $hyperv_nodes {Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering} New-Cluster –Name $cluster_name -Node $hyperv_nodes # Create share folder MD "$smb_path" # Create file share New-SmbShare -Name $smb_name -Path $smb_path -FullAccess $cluster_user # Set NTFS permissions from the file share permissions Set-SmbPathAcl –ShareName $smb_name 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ######################## # Windows Server 2012 R2 # setting up server for share # please setup your own configurations here. $hyperv_nodes = @ ( "HYPERV_NODE_01.DOMAIN" , "HYPERV_NODE_02.DOMAIN" ) $cluster_name = "CLUSTER_NAME" $cluster_user = "DOMAIN\USER" $smb_path = "S:\SMB_SHARE" $smb_name = "SMB_SHARE" # install locally Install-WindowsFeature File-Services , FS-FileServer # this will install the features necessary, including the Failover Clustering feature. Invoke-Command -ComputerName $hyperv_nodes { Install-WindowsFeature File-Services , FS-FileServer , Failover-Clustering } New-Cluster – Name $cluster_name -Node $hyperv_nodes # Create share folder MD "$smb_path" # Create file share New-SmbShare -Name $smb_name -Path $smb_path -FullAccess $cluster_user # Set NTFS permissions from the file share permissions Set -SmbPathAcl – ShareName $smb_name On each Hyper-V compute node, we must configure the nova-compute service to use the Hyper-V Cluster Driver along with a shared instances configuration path. Update these configuration options with your own values and add them to their appropriate sections:

# add extra configs in nova.conf [DEFAULT] compute_driver=hyperv.nova.cluster.driver.HyperVClusterDriver instances_path=\\SHARE_HOST_IP\SHARE_PATH\Instances 1 2 3 4 # add extra configs in nova.conf [ DEFAULT ] compute_driver = hyperv . nova . cluster . driver . HyperVClusterDriver instances_path = \ \ SHARE_HOST_IP \ SHARE_PATH \ Instances (Optional) Install the OpenStack Mitaka Cinder Storage driver, enabling SMB3 block storage volumes on clustered storage with Windows Server Scale-Out File Server enabled. Done! Now you can create and use your OpenStack Hyper-V instances as usual, with the additional peace of mind provided by the Hyper-V Clustering feature working behind the scenes.

We hope you enjoy this feature and if you have any questions or suggestions, please let us know!