Configuring Hyper-V Cluster Networks and NIC Teaming

26 Jan

I recently was working on a 2012 cluster that had some issues. When a lot of live migrations were taking place, a node would just drop out of the cluster with the error:

HOSTNAME: The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk.

Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Cluster node 'HOSTNAME' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

It seemed that the heartbeat was getting lost with the high volume of traffic and so at this point I looked to add extra virtual networks.

I decided upon 3 virtual networks:

  • Management & VMs – VLAN: 20, Weight: 40
  • Live Migration: VLAN: 21, Weight: 20
  • Cluster: VLAN: 22, Weight: 40

I performed the changed on one node at a time, each time draining all of the roles from that node before changing the network settings.

Steps:

Get rid of the previous virtual switch:

Remove-VMSwitch -Name "External"

Create the new virtual switch:

New-VMSwitch "TeamSwitch" –NetAdapterName "PhysicalTeam" –MinimumBandwidthMode Weight –AllowManagementOS $true
Set-VMSwitch -Name "TeamSwitch" -DefaultFlowMinimumBandwidthWeight 40

I set the teaming mode to be ‘Switch Independent’ and the load balancing mode to ‘Hyper-V Port’:

nicteaming

At this stage I also had to set the fixed IP address here.

The VM’s will all use the TeamSwitch but now we need the other networks for Live Migration and Cluster

# Live Migration
Add-VMNetworkAdapter –ManagementOS –Name "Live Migration" –SwitchName "TeamSwitch"
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "Live Migration" -Access -VlanId 21
Set-VMNetworkAdapter -ManagementOS -Name "Live Migration" -VmqWeight 90 -MinimumBandwidthWeight 20

# Cluster
Add-VMNetworkAdapter –ManagementOS –Name "Cluster" –SwitchName "TeamSwitch"
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "Cluster" -Access -VlanId 22
Set-VMNetworkAdapter -ManagementOS -Name "Cluster" -VmqWeight 80 -MinimumBandwidthWeight 30

I didn’t script this but I then had to set the IP addresses for each new adaptor.

Next was to rename the cluster networks:

(Get-ClusterNetwork | Where-Object {$_.Address -eq "000.000.90.0"}).Name = "Management"
(Get-ClusterNetwork | Where-Object {$_.Address -eq "000.000.91.0"}).Name = "Migration Network"
(Get-ClusterNetwork | Where-Object {$_.Address -eq "000.000.92.0"}).Name = "Cluster Network"

I also decided to use QoS to give priority to the heartbeat:

New-NetQosPolicy "Cluster" -IPDstPort 3343 -PriorityValue 6

# You can view the QoS settings with
get-NetQosPolicy

networks

Next was to set the roles of each network adaptor, ie. which would be used for live migration etc:


migrationnetwork managementnetwork clusternetwork

livemigrationsettings

Done, please let me know if you have any suggestions here.