Capacity Planner for Hyper-V Replica

Hyper-V in Windows Server 2012 includes a new capability called Hyper-V Replica. Hyper-V Replica allows administrators to replicate their virtual machines from a primary server/cluster to a replica server/cluster. The Capacity Planner for Hyper-V Replica guides the IT administrator to design the server, storage and network infrastructure which is required to successfully deploy Hyper-V Replica.

clip_image001

After reviewing the license terms, Click on ‘I accept the license term’ and click on ‘Next’.

clip_image002

Before proceeding from this page, ensure that a Hyper-V Replica server/cluster has been enabled to receive replication traffic from this primary server/cluster. As part of collecting various metrics, the capacity planner attempts to send a temporary VHD from the primary server/cluster to the replica server/cluster. This allows the tool to study the network characteristics of the link between the primary and replica server.

If your primary or replica server is part of a cluster, ensure that the Hyper-V Replica Broker role is added to the cluster.

clip_image003

Specify the following parameters in this screen and click ‘Next’:

Primary Server/Cluster details:

a. For a standalone primary server, enter the server name or FQDN.

b. If your primary server is part of a cluster, enter the FQDN of the (primary cluster) Hyper-V Replica Broker Client Access Point (CAP).

Replica Server/Cluster details:

a. For a standalone replica server, enter the server name or FQDN.

b. If your replica server is part of a cluster, enter the FQDN of the (replica cluster) Hyper-V Replica Broker Client Access Point.

Estimated WAN Bandwidth:

a. Enter the estimated WAN bandwidth link speed between the primary and replica server/cluster.

Duration of collecting metrics:

a. Enter an appropriate interval for which the metrics need to be collected. It is highly recommended that the tool is run during ‘production hours’ which ensures that the most representative data is collected. Running the tool for a short duration (eg: 10mins) may not give quality data.

clip_image004

The tool connects to the primary server and enumerates the virtual machines which are running on the primary. Ensure the following:

1) You are an administrator on the primary server/cluster. Remote-WMI is used to enumerate the virtual machines on the primary server – ensure that the right set of firewalls and permissions are set to allow this call to execute.

2) Ensure that replication has not been enabled on any of the VMs which are on the primary server/cluster.

3) Ensure that the VMs on the primary server/cluster are running.

The following details needs to be provided in the page:

1) Temporary VM location: As part of collecting various metrics, the tool creates a temporary VM on your primary server/cluster and enables replication on the VM. This allows the tool to study the network characteristics between the primary and replica server. Provide a location on the primary server/cluster in which this VHD/VM can be created. In a clustered deployment, ensure that the location is accessible from all the nodes in the cluster.

2) (Optional) Certificate: If your primary and replica servers are in a workgroup [or] if certificate based authentication is being used in your Hyper-V Replica environment, you should provide the required certificate in this page.

3) Select VMs and VHDs: You can select the VMs and VHDs on which the metrics need to be collected. If you are not planning to enable replication on any specific VM/VHD, you can uncheck the VM in this screen.

Click ‘Next’ after providing all the inputs.

clip_image006

The tool now captures the metrics in the background. The tool will run for a few minutes beyond the duration of the run. You can continue to operate on your VM during the duration of the run. Once completed, the screen will look as follows:

clip_image008

Click on ‘View Report’ to go over the recommendations.

Cheers,


Marcos Nogueira
http://blog.marcosnogueira.org
Twitter: @mdnoga

Management Tasks for Hyper-V Replica

The Hyper-V Manager interface is used to manage standalone Hyper-V Primary, Replica servers, and the virtualized workloads running on those servers.   The Failover Cluster Manager interface is used if the Primary or Replica servers are part of a Hyper-V Failover Cluster. Hyper-V Replica management tasks can be categorized as follows:

  • Hyper-V Server Primary Site Management Tasks
  • Hyper-V Server Replica Site Management Tasks
  • Virtual Machine Primary Site Management Tasks
  • Virtual Machine Replica Site Management Tasks
  • Modifying Virtual Machine Replication Settings

Note: In the above list, Hyper-V Failover Cluster can be substituted for ‘Hyper-V Server’.

Hyper-V Server Primary Site

Management tasks involving the Hyper-V Server at a Primary Site include:

Ensure the Hyper-V server (Hyper-V Failover Cluster) at the Primary site is configured as a Replica server to support Reverse Replication for a Planned Failover event

To configure the Hyper-V server at the Primary site as a Replica server:

  1. In the Hyper-V Manager interface, Click on Hyper-V Settings in the Actions pane
  2. In the Hyper-V Settings dialog box, Click on Replication Configuration
  3. In the Details pane, Select Enable this computer as a Replica server
  4. Choose an Authentication method to include the port that will be used (if not using the default port)
  5. Configure Authorization and storage.  This includes designating a specific location to store replica virtual machine files if the default location is not to be used.  Should you not desire to allow all Hyper-V Primary servers to be serviced, you have the option to allow only specific Hyper-V servers (Primary servers) to send replication requests. Click Apply or OK when finished

Note:  In a Replica cluster, use the Hyper-V Replica Broker role to configure the cluster nodes for replication.

Monitor the Replication Health of virtual machines configured for replication

To monitor the Replication Health of a virtual machine configured for replication:

  1. Open Hyper-V Manager
  2. In the details pane, Right-click on one of the Column Headings and select Add\Remove Columns
  3. Choose Replication Health in the Available Columns list, click Add to move it to the Displayed Columns list
  4. Move the new column to the desired location in the listing and click OK

Monitor Hyper-V Replica specific Performance counters using Performance Monitor

To monitor Hyper-V Replica performance:

  1. Click the Start button,  then click Run and type perfmon.msc and press ENTER
  2. In the navigation tree, expand Monitoring Tools, and then click Performance Monitor
  3. In the menu bar above the Performance Monitor graph display, either click the Add button (+) or right-click anywhere in the graph and click Add counters from the menu. The Add Counters dialog box opens
  4. In the Available Counters section, select counters to view in the Performance Monitor display.  The counters for Hyper-V Replica are virtual machine specific and are listed under Hyper-V F Counter VM
  5. Choose the desired counters and instances (virtual machines) then click the Add button to add the counters
  6. When finished, click OK

For more information Performance Monitor, visit the Performance Monitor Getting Started Guide.

Evaluate Hyper-V Replica log data using the Microsoft-Windows-Hyper-V-VMMS\Admin log

To review Hyper-V Replica log data:

  1.  In the Server Manager Menu Bar, Click on Tools and choose Event Viewer from the list
  2. In the navigation tree, expand Application and Services Logs,  expand Microsoft, expand Windows, expand Hyper-V-VMMS
  3. Click on Admin 

    Hyper-V Replica event messages are registered in the Hyper-V-VMMS channel.

Hyper-V Server Replica Site

Management tasks involving the Hyper-V Server at a Replica Site include:

Ensure the Hyper-V server (Hyper-V Failover Cluster) at the Replica site is configured as a Replica server

To configure the Hyper-V server at the Primary site as a Replica server:

  1. In the Hyper-V Manager interface, Click on Hyper-V Settings in the Actions pane
  2. In the Hyper-V Settings dialog box, Click on Replication Configuration
  3. In the Details pane, Select Enable this computer as a Replica server
  4. Choose an Authentication method to include the port that will be used (if not using the default port)
  5. Configure Authorization and storage.  This includes designating a specific location to store replica virtual machine files if the default location is not to be used.  Should you not desire to allow all Hyper-V Primary servers to be serviced, you have the option to allow only specific Hyper-V servers (Primary servers) to send replication requests. Click Apply or OK when finished

Note:  In a Replica cluster, use the Hyper-V Replica Broker role to configure the cluster nodes for replication.

Monitor the Replication Health of virtual machines configured for replication

To monitor the Replication Health of a virtual machine configured for replication:

  1. Open Hyper-V Manager
  2. In the details pane, Right-click on one of the Column Headings and select Add\Remove Columns
  3. Choose Replication Health in the Available Columns list, click Add to move it to the Displayed Columns list
  4. Move the new column to the desired location in the listing and click OK

Monitor Hyper-V Replica specific Performance counters using Performance Monitor

To monitor Hyper-V Replica performance:

  1. Click the Start button,  then click Run and type perfmon.msc and press ENTER
  2. In the navigation tree, expand Monitoring Tools, and then click Performance Monitor
  3. In the menu bar above the Performance Monitor graph display, either click the Add button (+) or right-click anywhere in the graph and click Add counters from the menu. The Add Counters dialog box opens
  4. In the Available Counters section, select counters to view in the Performance Monitor display.  The counters for Hyper-V Replica are virtual machine specific and are listed under Hyper-V Replica  Counter VM
  5. Choose the desired counters and instances (virtual machines) then click the Add button to add the counters
  6. When finished, click OK

For more information Performance Monitor, visit the Performance Monitor Getting Started Guide.

Evaluate Hyper-V Replica log data using the Microsoft-Windows-Hyper-V-VMMS\Admin log

To review Hyper-V Replica log data:

  1. In the Server Manager Menu Bar, Click on Tools and choose Event Viewer from the list
  2. In the navigation tree, expand Application and Services Logs,  expand Microsoft, expand Windows, expand Hyper-V-VMMS
  3. Click on Admin

Hyper-V Replica event messages are registered in the Hyper-V-VMMS channel.

Virtual Machine – Primary Site

Management tasks involving virtual machines at the Primary Site include:

Planned Failover – This action initiates a failover of a virtual machine from a Primary to a Replica server.  This is a ‘planned’ event as opposed to a Failover action, which is unplanned.  Since it is a ‘planned’ event, there should be no data loss.  This action executes a series of checks prior to executing the failover.  One check determines if the Primary server has also been configured as a Replica server.  This is done because the assumptions are first, the virtual machine being failed over to a Replica server will eventually be moved back to the Primary server and second, the Primary server will become the Replica server for the virtual machine that is being failed over.  This action provides an Administrator the flexibility to execute the failover of a virtual machine to a replica server in a controlled manner before a disaster occurs

To execute a Planned Failover for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected, Right-click and select Replication and then choose Planned Failover
  4. A list of Pre-Requisites and Actions is presented.  If the Virtual Machine has not been shut down and the Primary Server not configured as a Replica Server, complete those tasks before proceeding. By default, Start the replica virtual machine after Failover is checked (uncheck if this is not the desired action for the virtual machine after a Planned Failover completes)
  5. Click on the Failover button.
  6. If the Failover is successful, a pop-up dialog box appears reporting the Failover completed successfully (Note: If the option to start the virtual machine after the Planned Failover was left checked, then the virtual machine will be started on the Replica server).  Close the dialog box.
  7. If the Planned Failover does not complete successfully, review the information contained in the General Methodology for troubleshooting the virtual machine Failover process in the troubleshooting section.

Pause Replication – This action pauses replication for the selected virtual machine.  The Replication Health column in the Hyper-V Manager interface (if selected for display) reflects a Warning Status

To Pause Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine that is not paused
  3. With the virtual machine selected, Right-click and select Replication and then choose Pause Replication
  4. The Replication Health, if visible, in Hyper-V Manager, will be updated and the Replication Health for the virtual machine will indicate a Warning.  The State column still shows the Virtual Machine as Running
  5. Right-click on the Virtual Machine, select Replication and then click on View Replication Health.  The health report reflects an accurate Replication State which should be Replication Paused

Resume Replication (Available only if replication has been paused for a virtual machine) – This action resumes replication for the selected virtual machine (the action must be executed in the same site where replication was Paused).  The Hyper-V Replica Network Services component re-establishes a connection to the Replica server (if needed) and replication resumes.  If the virtual machine was in a Resynch Required state, Resume Replication performs a resynchronization.  A resynchronization essentially compares blocks between the Primary and Replica VHDs and then sends the delta blocks to the Replica. Scenarios where this can happen include, but may not be limited to, a failure occurred on the Primary server when changes were being made to the replication log or, if the Primary is a Failover Cluster, an unplanned cluster failover occurred.  The Replication Health column in Hyper-V Manager interface (if selected for display) reflects a status of Normal

To Resume Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a paused virtual machine
  3. With the virtual machine selected, Right-click and select Replication and then choose Resume Replication
  4. The Replication Health, if visible, in Hyper-V Manager, is updated and the Replication Health for the virtual machine is Normal

View Replication Health – This action provides data about the replication events for a virtual machine.

A Replication Health Report can be saved as a CSV file.  A Replication Health Report indicates if it is being viewed as either a Primary or a Replica virtual machine (see a sample of a Replication Health Report on a Replica virtual machine later in this guide)

To view Replication Health for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected, Right-click and select Replication and then choose View Replication Health
  4. The Replication Health Report for the virtual machine is displayed. The report can be saved as a CSV file by clicking on Save as … Button

Remove Replication – This action stops replication for the virtual machine.  All connections for the virtual machine to the Replica server are terminated.  The Replication Health in Hyper-V Manager on the Primary server, if selected for viewing, is Not Applicable.  A corresponding action must be accomplished on the Replica server.  Failure to execute this same action on the Replica server will result in errors should a Hyper-V Administrator attempt to re-enable replication for the virtual machine (more information is provided in the Troubleshooting section)

To Remove Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected,  Right-click and select Replication and choose Remove Replication
  4. Acknowledge the pop-up Warning by clicking on Remove Replication
  5. The Replication Health column, if displayed, indicates Not Applicable for the virtual machine
  6. Connect to the Replica server and execute Steps 1-5.  This will remove replication for the virtual machine on the Replica server and will initiate a merge for all the replica information for the virtual machine
  7. The Replication Health column, if displayed, indicates Not Applicable for the virtual machine
  8. Additional cleanup action is required on the Replica server.  In Hyper-V Manager, Right-click on the virtual machine and choose Delete.  Acknowledge the pop-up Warning by clicking on Delete.  This removes the virtual machine reference in Hyper-V Manager.  Some data files remain on the Replica server in the storage location specified for the replication data.  To recover storage space, manually remove the data.

Enable Replication (Available only if replication is not enabled for a virtual machine) – This action enables replication for a virtual machine

To Enable Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected,  Right-click and select Enable Replication
  4. Review the information in the Before You Begin screen and click Next
  5. In the Specify Replica Server screen, provide the name for the Replica Server using the Fully Qualified Domain Name (FQDN) or NetBIOS format.   There is also the option to Browse Active Directory for the server.  If the Replica Server is configured correctly, the Specify Connection Parameters screen is populated.  If not, an error is registered and an option to Configure Server is available to configure the server to be a Replica Server. If data compression is not desired, Uncheck the box Compress the data that is transmitted over the network. Click Next
  6. In the Choose Replication VHDs screen, ensure all disks to be replicated are Checked and then click Next (i.e. uncheck those disk you do not want replicated.  An example might be a disk functioning as a repository for the virtual machine page file)
  7. In the Configure Recovery History screen, select as desired. For an explanation of the options, review the section on Enabling a virtual machine for replication.  Click Next
  8. In the Choose Initial Replication Method screen, select as desired. For an explanation of the options, review the section on Enabling a virtual machine for replication.  Click Next
  9. Review the information in the Summary screen, and click Finish

Once replication has been enabled for a virtual machine, the Replication Health column, if visible, in Hyper-V Manager will be updated.  Once the Initial Replication (IR) has been completed, the Replication Health for a virtual machine will be Normal.

Virtual Machine – Replica Site

Management tasks involving virtual machines at the Replica Site include:

Failover – This action executes a process that starts a virtual machine on the Replica server using a replica (Recovery Point) selected by the Hyper-V Administrator.  This is an unplanned event unlike the Planned Failover action, which is a planned event.  Executing a Failover for a virtual machine could result in data loss depending on which recovery point is selected

To Failover a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected,  Right-click and select Replication and then choose Failover
  4. In the next screen, choose a recovery point from the drop-down listing of all the recovery points associated with the virtual machine and then click Failover
  5. The virtual machine starts and the Replication Health indicates Warning.  If the Primary server remains the same, the Replication Health for the virtual machine that was recovered is also Warning. To complete the process and remove the Warnings, either Cancel Failover or configure Reverse Replication and allow Initial Replication to completeIf a new Replica Server is needed, configure Reverse Replication to the new Replica server.

Test Failover – This action allows a Hyper-V Administrator to test a virtual machine on the Replica server without interrupting the production workload running on the Primary server.  The network configuration for the test virtual machine is disconnected by default so as not to interfere with the production workload. If network connectivity is to be tested, the recommendation is to create a separate test network and connect the test virtual machine to that network. The virtual machine created and started has the same name as the original virtual machine with a modifier of Test added on to the end
mgmt_rep

To start a Test Failover for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected,  Right-click and select Replication and then choose Test Failover
  4. In the next screen, choose a recovery point from the drop-down listing of all the recovery points associated with the virtual machine and then click Test Failover.  A new test virtual machine is created but is not started.  At this point, the virtual machine can be started and then a connection can be made to the virtual machine and a verification process can be completed

Stop Test Failover (Available only if a test is already running for the selected virtual machine) – This action stops a test that is in progress for the selected virtual machine.  The virtual machine is stopped and deleted from Hyper-V Manager (Note:  If the Test Failover is being executed on a Replica cluster, the Test-Failover role that is created in Failover cluster Manager will have to be manually deleted)

To stop a Test Failover for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select the virtual machine being tested, not the test virtual machine that is running. Right-click on the virtual machine, select Replication and then choose Stop Test Failover
  3. The test virtual machine is stopped if it is running and is removed from Hyper-V Manager as the test is completed

Pause Replication – This action pauses replication for the selected virtual machine.  The Replication Health column in Hyper-V Manager, if selected for viewing, indicates a Warning

To Pause Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine that is not paused
  3. With the virtual machine selected, Right-click and select Replication and then choose Pause Replication
  4. The Replication Health, if visible, in Hyper-V Manager, is updated and indicates a Warning

Resume Replication (Available only if replication has been paused for a virtual machine on the Replica server) – This action resumes replication for the selected virtual machine.  If a ‘resynch’ is required for the virtual machine, that action will be initiated on the Primary server.  The Replication State column, if selected for viewing in Hyper-V Manager, indicates Replication Enabled

To Resume Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a paused virtual machine
  3. With the virtual machine selected, Right-click and select Replication and then choose Resume Replication
  4. The Replication Health, if visible, in Hyper-V Manager, is updated and indicates Normal

View Replication Health – This action provides data about the replication events for a virtual machine.

A Replication Health Report can be saved as a CSV file.  A Replication Health Report indicates if it is being viewed on either a Primary or a Replica server

To View Replication Health for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected, Right-click and select Replication and then choose View Replication Health
  4. The Replication Health Report for the virtual machine is displayed. The report can be saved as a CSV file by clicking on Save as…

Remove Replication – If a Remove Replication action is executed on the Replica server, a corresponding action must be executed on the Primary Server. This action stops replication for the virtual machine.  Prior to re-enabling replication, the virtual machine must be deleted in Hyper-V Manager on the Replica server.  This destroys the virtual machine on the Replica Server.  If the virtual machine is not deleted, a Replication error is reflected in Hyper-V Manager and associated error logs are registered (more information is provided in the Troubleshooting section)

To Remove Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine
  3. With the virtual machine selected,  Right-click and select Replication and then choose Remove Replication
  4. Acknowledge the pop-up Warning by clicking on Remove Replication
  5. The Replication Health column, if displayed, indicates Not Applicable
  6. A process will start to merge all recovery point data into the VHD that was initially replicated
  7. Connect to the Primary server.  In Hyper-V Manager, the virtual machine Replication Health indicates Normal
  8. Execute Steps 1-5.  This removes replication for the virtual machine on the Primary server
  9. The Replication Health column, if displayed, now indicates Not Applicable on the Primary server.  If Step 8 is not accomplished before the next 5 minute replication interval, Replication Health will indicate Critical
  10. Additional cleanup action is required on the Replica server.  In Hyper-V Manager, Right-click on the virtual machine and choose Delete.  Acknowledge the pop-up Warning by clicking on Delete.  This removes the virtual machine reference in Hyper-V Manager.  Some data files remain on the Replica server in the storage location specified for the virtual machine.  To recover storage space, manually remove this data

Cancel Failover – This action is available if a Failover action was executed for a virtual machine.  This allows a Hyper-V Administrator to cancel the Failover action if, for example, he decides the recovery point chosen was not the desired one.  After cancelling the Failover, another recovery point can be selected and another Failover process initiated. A Failover can only be cancelled if the virtual machine state is Failed over – Waiting Completion.  If a Reverse Replication has been completed, the Failover can no longer be cancelled

To Cancel Failover for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine that has a Replication Health of Warning but viewing the Health Report shows Failover Complete
  3. With the virtual machine selected,  Right-click, select Replication and then choose Cancel Failover
  4. Acknowledge the pop-up Warning by clicking on Yes.  On the Replica server, the virtual machine is turned off and the Replication Health indicates NormalOn the Primary server, the Replication Health shows Critical.
  5. To clear the Critical health state,  right-click on the virtual machine and select Replication and then choose Resume Replication

Reverse Replication – This action is available if a Failover action was executed for a virtual machine.  This allows the Hyper-V Administrator to designate a Hyper-V server as a Replica server for the virtual machine that was recovered

To enable Reverse Replication for a virtual machine:

  1. Open Hyper-V Manager
  2. In the details pane, select a virtual machine that has a Health of Warning
  3. With the virtual machine selected, Right-click, select Replication, and then choose Remove Recovery Points This merges all the recovery points into the original VHD.  This completes Failover but Replication Health still indicates Warning
  4. With the virtual machine selected, Right-click, select Reverse Replication
  5. Complete the Reverse Replication wizard by either selecting the Primary server (the default) as the Replica y server or choosing another Replica server.  Keep in mind that if the selected Hyper-V server has not been enabled as a Replica server and the appropriate firewall rule enabled, the Reverse Replication process will fail

Remove Recovery Points – This action is available only during a Failover scenario.  When this action is executed, all recovery points (snapshots) for a Replica virtual machine are deleted.  When the action is executed, a pop-up dialog box is presented to the user indicating all recovery points will be removed and Cancel Failover will no longer be available.  The user must acknowledge the pop-up by clicking either Yes or No.  If Yes is selected, the Failover is committed and the recovery points are merged down into the base VHD for the virtual machine.  At this point Reverse Replication can be configured to clear the Warning for Replication Health and an Initial Replication can begin to the new Replica Server

Troubleshooting Hyper-V Replica

Introduction to Troubleshooting Hyper-V Replica

This section explains how to troubleshoot Hyper-V Replica.  Use this guide when:

  • You have problems with connectivity between Primary and Replica servers
  • You have problems enabling a virtual machine for replication
  • You have problems with virtual machine replication whether it is Initial Replication (IR) or Delta Replication (DR)
  • You have problems executing management actions associated with virtual machines on a Primary or Replica server
  • You have problems with the Replication Broker configured in a Hyper-V Failover Cluster.
  • You need to collect Performance monitoring data for replicated virtual machines.

Tools for Troubleshooting Hyper-V Replica

Utilities and Commands for Troubleshooting Hyper-V Replica

Performance Monitor

Performance Monitor contains Hyper-V counters specific to Hyper-V Replica.  These counters monitor replication statistics for configured virtual machines.  The specific counter is Hyper-V Failover Replication Counter VM.  The data that can be collected for each selected virtual machine includes:

  • Average Replication Latency
  • Average Replication Size
  • Last Replication Size
  • Network Bytes Received
  • Network Bytes Sent
  • Replication Count
  • Replication Latency
  • Resynchronized Bytes
Hyper-V Replica Integration into the Hyper-V Best Practice Analyzer (BPA)

Rules pertaining to Hyper-V Replica are included in the Hyper-V Best Practice Analyzer. The following BPA Rule details are provided to assist with troubleshooting:

Summary

Detail

Rule Title A Replica server must be configured to accept replication requests
Severity Red
Category Configuration
Issue This computer is designated as a Hyper-V Replica server but is not configured to accept incoming replication data from primary servers.
Impact This server cannot accept replication traffic from primary servers.
Resolution Use Hyper-V Manager to specify which primary servers this Replica server should accept replication data from.

Summary

Detail

Rule Title Replica servers should be configured to identify specific primary servers authorized to send replication traffic
Severity Yellow
Category Configuration
Issue As configured, this Replica server accepts replication traffic from all primary servers and stores them in a single location.
Impact All replication from all primary servers is stored in one location, which might introduce privacy or security problems.
Resolution Use Hyper-V Manager to create new authorization entries for the specific primary servers and specify separate storage locations for each of them. You can use wildcard characters to group primary servers into sets for each authorization entry.

Summary

Detail

Rule Title Compression is recommended for replication traffic
Severity Yellow
Category Configuration
Issue The replication traffic sent across the network from the primary server to the Replica server is uncompressed.
Impact Replication traffic will use more bandwidth than necessary. This impacts the following virtual machines:<List of VMs>
Resolution Configure Hyper-V Replica to compress the data transmitted over the network in the settings for the virtual machine in Hyper-V Manager. You can also use tools outside of Hyper-V to perform compression.

Summary

Detail

Rule Title Configure guest operating systems for VSS-based backups to enable application-consistent snapshots for Hyper-V Replica
Severity Red
Category Configuration
Issue Application-consistent snapshots require that Volume Shadow Copy Services (VSS) is enabled and configured in the guest operating systems of virtual machines participating in replication.
Impact Even if application-consistent snapshots are specified in the replication configuration, Hyper-V will not use them unless VSS is configured. This impacts the following virtual machines:<List of VMs>
Resolution Use Hyper-V Manager to install integration services in the virtual machine.

Summary

Detail

Rule Title Integration services must be installed before primary or Replica virtual machines can use an alternate IP address after a failover
Severity Red
Category Configuration
Issue Virtual machines participating in replication can be configured to use a specific IP address in the event of failover, but only if integration services are installed in the guest operating system of the virtual machine.
Impact In the event of a failover (planned, unplanned, or test), the Replica virtual machine will come online using the same IP address as the primary virtual machine. This configuration might cause connectivity issues. This impacts the following virtual machines:<List of VMs>
Resolution Use Hyper-V Manager to install integration services in the virtual machine.

Summary

Detail

Rule Title To participate in replication, servers in failover clusters must have a Hyper-V Replica Broker configured
Severity Red
Category Configuration
Issue For failover clusters, Hyper-V Replica requires the use of a Hyper-V Replica Broker name instead of an individual server name.
Impact If the virtual machine is moved to a different failover cluster node, replication cannot continue.
Resolution Use Failover Cluster Manager to configure the Hyper-V Replica Broker. In Hyper-V Manager, ensure that the replication configuration uses the Hyper-V Replica Broker name as the server name.

Summary

Detail

Rule Title Virtual hard disks with paging files should be excluded from replication
Severity Yellow
Category Configuration
Issue Paging files should be excluded from participating in replication, but no disks have been excluded.
 Impact Virtual hard disks that experience a high volume of input/output activity will unnecessarily require much greater resources to participate in replication. This impacts the following virtual machines:\n{0}
Resolution If you have not already done so, create a separate virtual hard disk for the Windows paging file. If initial replication has already been completed, use Hyper-V Manager to remove replication. Then, configure replication again and exclude the virtual hard disk with the paging file from replication.

Summary

Detail

Rule Title Configure the Failover TCP/IP settings that you want the Replica virtual machine to use in the event of a failover
Severity Yellow
Category Configuration
Issue Replica virtual machines configured with a static IP address should be configured to use a different IP address from their primary virtual machine counterpart in the event of failover.
Impact Clients using the workload supported by the primary virtual machine might not be able to connect to the Replica virtual machine after a failover. Also, the primary virtual machine’s original IP address will not be valid in the Replica virtual machine network topology.
Resolution Use Hyper-V Manager to configure the IP address that the Replica virtual machine should use in the event of failover. This impacts the following virtual machine(s): <List of VMs>

Summary

Detail

Rule Title Authorization entries should have distinct tags for primary servers with virtual machines that are not part of the same security group.
Severity Yellow
Category Configuration
Issue The server will accept replication requests for the replica virtual machine from any of the servers in the authorization list associated with the same replication tag as of the VM.
Impact There might be privacy and security concerns with a virtual machine accepting replication from primary servers belonging to different authorization entries. This impacts the following authorization entries:<List of VMs>
Resolution Use different tags in the authorization entries for primary servers with virtual machines that are not part of the same security group. Modify the Hyper-V settings to configure the replication tags.

Summary

Detail

Rule Title Certificate-based authentication is configured, but the specified certificate is not installed on the Replica server or failover cluster nodes
Severity Red
Category Configuration
Issue The security certificate that Hyper-V Replica has been configured to use to provide certificate-based replication is not installed on the Replica server (or any failover cluster nodes).
Impact In the event of a cluster failover or move to another node, Hyper-V replication will pause if the new node does not also have the appropriate certificate installed. This impacts the following nodes: <List of nodes>
Resolution Install the configured certificate on the Replica server (and all associated nodes in the failover cluster, if any).

Summary

Detail

Rule Title Replication is paused for one or more virtual machines on this server
Severity Yellow
Category Operation
Issue Replication is paused for one or more of the virtual machines. While the primary virtual machine is paused, any changes that occur will be accumulated and will be sent to the Replica virtual machine once replication is resumed.
Impact As long as replication is paused, accumulated changes occurring in the primary virtual machine will consume available disk space on the primary server. After replication is resumed, there might be a large burst of network traffic to the Replica server. This impacts the following virtual machines: <List of VMs>
Resolution Confirm that pausing replication was intended. If replication was paused to address low disk space or network connectivity, resume replication as soon as those issues are resolved.

Summary

Detail

Rule Title Initial replication is complete, but no test failover has been attempted
Severity Red
Category Operation
Issue No test failovers have been attempted since completing initial replication.
Impact A test failover confirms that failover will succeed and that all workload operations on the primary virtual machine continue properly after failover to the Replica virtual machine. This impacts the following virtual machines: <List of VMs>
Resolution Use Hyper-V Manager to conduct a test failover.

Summary

Detail

Rule Title There has been no test failover in at least one month
Severity Yellow
Category Operation
Issue Test failovers should be carried out at least monthly to verify that failover will succeed and that virtual machine workloads will operate as expected after failover.
Impact A test failover confirms that failover will succeed and that all workload operations on the primary virtual machine continue properly after failover to the Replica virtual machine. This impacts the following virtual machines: <List of VMs>
Resolution Use Hyper-V Manager to conduct a test failover.

Summary

Detail

Rule Title Certificate-based authentication is recommended for replication.
Severity Yellow
Category Configuration
Issue One or more virtual machines selected for replication are configured for Kerberos authentication.
Impact The replication network traffic from the primary server to the replication server is unencrypted. This impacts the following virtual machines:<List of VMs>
Resolution If another method is being used to perform encryption, you can ignore this. Otherwise, modify the virtual machine settings to choose certificate-based authentication.

Summary

Detail

Rule Title Configure a policy to throttle the replication traffic on the network
Severity Yellow
Category Configuration
Issue There might not be a limit on the amount of network bandwidth that replication is allowed to consume.
Impact Network bandwidth could become completely dominated by replication traffic, affecting other critical network activity. This impacts the following ports: <List of Ports>
Resolution If you use another method to throttle network traffic, you can ignore this. Otherwise, use Group Policy Editor to configure a policy that will throttle the network traffic to the relevant port of the Replica server.

Summary

Detail

Rule Title Resynchronization of replication should be scheduled for off-peak hours.
Severity Yellow
Category Configuration
Issue Resynchronization of replication for the primary VMs is not scheduled for off-peak hours.
Impact Replication logs and Replication Point Objective will increase when the VM is in a resynchronize-required state for a longer time. At the same time, resynchronization will affect the IOPS bandwidth on the primary and the replica server, hence might affect production workloads.
Resolution Use Hyper-V Manager VM Replication settings to configure the auto-resynchronize replication window of the primary VM within the off-peak hours.

Summary

Detail

Rule Title VHDX-based virtual hard disks are recommended for virtual machines that have recovery history enabled in replication settings.
Severity Yellow
Category Configuration
Issue VHD-based virtual hard disks are being used for the virtual machines that are enabled for replication with recovery history turned on.
Impact Under some circumstances, the VHDs on the replica server could experience consistency issues. This impacts the following virtual machine(s): <List of VMs>
Resolution Use the new virtual hard disk format (VHDX) for the virtual machines that are enabled for replication with recovery history turned on. You can convert a virtual hard disk from VHD format to VHDX format. The VHDX format has reliability mechanisms that help protect the disk from corruptions due to system power failures. However, do not convert the virtual hard disk if it is likely to be attached to an earlier release of Windows at some point. Windows releases earlier than {1} do not support the VHDX format.

Summary

Detail

Rule Title Recovery snapshots should be removed after failover.
Severity Yellow
Category Operation
Issue A failed over virtual machine has one or more recovery snapshots.
Impact Available space may run out on the physical disk that stores the snapshot files. If this occurs, no additional disk operations can be performed on the physical storage. Any virtual machine that relies on the physical storage could be affected. This impacts the following virtual machines: <List of VMs>
Resolution For each failed over virtual machine, use the Complete-VMFailover cmdlet in Windows PowerShell to remove the recovery snapshots and indicate Failover completion.

Summary

Detail

Rule Title A large number of recovery points has been configured
Severity Yellow
Category Configuration
Issue Hyper-V Replica has been configured to store more than nine previous recovery points.
Impact Maintaining too many recovery points could cause the Replica server to run out of available disk space. This impacts the following virtual machines: <List of VMs>
Resolution Review the number of recovery points configured, taking into account factors such as the number of virtual machines on the server and the oldest recovery point that is really required.

General Methodology for Troubleshooting Hyper-V Replica

Hyper-V Replica connectivity issues between Primary and Replica servers

Symptom:  Hyper-V Replica functionality is disrupted and the Hyper-V VMMS\Admin log reports general network connectivity errors between the Primary and Replica server
  1. Verify the Replica server is booted and running.
  2. Check network connectivity and name resolution functionality between the Primary and Replica server by executing ping and nslookup tests.  If ping test fails, resolve network connectivity issues.  If name resolution fails, check DNS
  3. Ensure the Replica server is listening on the Replica Server Port.  This can be accomplished by running a netstat -ano command on the Replica server after verifying the  appropriate firewall rule has been Enabled or the custom firewall rule has been configured to allow Inbound communications on the configured port
    troblehooting_Rep
  4. Inspect the System Event Log on the Primary and Replica servers to determine if there is any failure condition associated with network functionality
  5. Run the Hyper-V Best Practice Analyzer (BPA) and inspect the report for any configuration or operational issues

Configuring a virtual machine for replication

Symptom:  Configuring a virtual machine for replication fails.
  1. Verify the Replica server is booted and running.
  2. Check network connectivity between and name resolution functionality the Primary and Replica server by executing a ping and nslookup tests.  If the ping test fails, resolve network connectivity issues. If name resolution fails, check DNS
  3. Ensure the Replica server is listening on the Replica Server Port and the Authentication Type is configured correctly.
  4.  If the Replica server configuration matches the parameters entered in the Enable Replication wizard,  verify the Firewall on the Replica server has been configured to allow Inbound communications on the Replica Server Port
  5. Inspect the System Event Log on the Primary and Replica servers to determine if there is any failure condition associated with network functionality
  6. Inspect the Hyper-V VMMS\Admin Log for any events related to network connectivity on both the Primary and Replica servers

Virtual machine Planned Failover process

A virtual machine Planned Failover process is a planned event where a running virtual machine on the Primary server is moved to a designated Replica server.

Symptom:  The Check that virtual machine is turned off Pre-Requisite test fails.
  1. Ensure the virtual machine has been shut down prior to executing a Planned Failover to a Replica server
Symptom:  The Check configuration for allowing revers replication test fails.
  1. Ensure the Primary server has also been configured as a Replica server.  The assumption is that if a Planned Failover is executed to a Replica server, the virtual machine will use the Primary server as the new Replica server.   This configuration in the virtual machine is included as part of the Planned Failover process
Symptom:  Send un-replicated data to Replica server fails.
  1. Verify network connectivity to the Replica server using the procedures outlined in the Hyper-V Replica connectivity issues between Primary and Replica servers section

Configuring a virtual machine for Reverse Replication

Symptom:  Reverse Replication configuration for a virtual machine results in a failure.
  1. Verify network connectivity to the Hyper-V server being used as a Replica server using the procedures outlined in the Hyper-V Replica connectivity issues between Primary and Replica servers section

Initial Replication (IR) for a virtual machine

Symptom:  Initial Replication (IR) for a virtual machine fails.
  1. Verify network connectivity to the Replica server using the procedures outlined in the Hyper-V Replica connectivity issues between Primary and Replica servers section
  2. Ensure the protocol configuration between the Primary and Replica server match
  3. Verify the Primary server is authorized to replicate with the Replica server this includes verifying the Security Tags match
  4. Ensure the Authentication method matches between the Primary and Replica server
  5. If there is an error on the Replica server indicating there is insufficient storage space,   verify there is sufficient storage space available on the drive hosting the virtual machine replica file(s).  If there is insufficient storage space, add additional storage space

Delta Replication (DR) for a virtual machine

Symptom:  Delta Replication (DR) for a virtual machine fails
  1. Verify network connectivity to the Replica server using the procedures outlined in the Hyper-V Replica connectivity issues between Primary and Replica servers section
  2. Ensure the protocol configuration between the Primary and Replica server match
  3. Verify the Primary server is authorized to replicate with the Replica server
  4. Ensure the Authentication method matches between the Primary and Replica server
  5. Check for any error(s) on the Replica server indicating there is insufficient storage space available to host the virtual machine replica files
  6. Check for any error(s) on the Replica server indicating the virtual machine files could not be located
Symptom:  Application-consistent replicas are not generated by the Primary server and replicated to the Replica server
  1. Verify the virtual machine has been configured to replicate application-consistent replicas to the Replica server
  2. Verify the Integration Services version of the Guest matches what is installed in the Host (if there is a mismatch, a Warning message will be registered in the Hyper-V-Integration Admin log)
  3. Check the virtual machine Integration Services and verify the Backup (Volume snapshot) integration component is enabled in the Guest
  4. Review the system event log in the Guest and determine if there any errors pertaining to the Volume Shadow Copy Service (VSS)
  5. Test VSS in the Guest by executing a backup of the operating system
  6. Execute a backup on the Hyper-V host and verify the Guest can be backed up

Replication Broker issues

Symptom:  When enabling a virtual machine for replication, a connection to the Client Access Point (CAP) being used by the Hyper-V Replica Cluster Replication Broker cannot be made.
  1. Ensure all the resources supporting the Hyper-V Replica Clustering Replication Broker are Online in the cluster.  If there are any failures for the resources in the group, troubleshoot the failures using standard Failover Cluster troubleshooting procedures
  2. Move the resource group containing the Hyper-V Replica Clustering Replication Broker to another node in the cluster and attempt to enable replication for a virtual machine using the Client Access Point for the Hyper-V Replica Clustering Replication Broker

Guest IP functionality

Symptom:  After initiating a Failover for a virtual machine, the configured Failover TCP/IP settings for the virtual machine in the Replica server are not implemented and a connection to the virtual machine cannot be made.
  1. Ensure the Integration Components in the virtual machine have been updated.  This problem could occur in down-level operating systems running in a virtual machine on a Windows Server 2012 Hyper-V server
  2. Check the Hyper-V-Integration\Admin event log for an Event ID: 4010 Warning message reporting a problem with the Hyper-V Data Exchange functionality with the virtual machine experiencing this problem.  Additionally, an Event ID: 4132 Error message will be recorded indicating a problem applying IP settings to a network adapter in the virtual machine experiencing this problem
  3. Update the Integration Components in the virtual machine

Hyper-V Failover Cluster as a Primary or Replica Server

Failover Clustering has proven its value in making virtualized workloads highly available.  We saw this in Windows Server 2008 using Quick Migration and then in Windows Server 2008 R2 with the addition of Live Migration. Failover Clustering can also play an important role as a Replica Cluster.  To accommodate this, a new role has been added in Failover Clustering called the Hyper-V Replica Broker.  A new resource type, Virtual Machine Replication Broker, was added to support this new Role.

Failover Replication Broker Architecture

The Hyper-V Replica Broker runs in a Replica cluster and provides a Replica server name (connection point (a.k.a.  Client Access Point (CAP))) for initial virtual machine placement when contacted by a Primary server. After a virtual machine is initially replicated to the Replica Cluster, the Hyper-V Replica  Broker provides the virtual machine to Replica Server (cluster node) mapping to ensure the Primary server can replicate data for the virtual machine to the correct node in the cluster in support of mobility scenarios on the Replica side (e.g. Live\Quick Migration, or Storage Migration).

The Hyper-V Replica Broker is used to configure the replication settings for all nodes in the cluster.  In standalone Hyper-V servers, the Hyper-V Manager is used to configure replication settings.  The Failover Cluster Manager is used to configure replication settings in the Replica cluster.  Using the Hyper-V Replica Role, Replication Settings across the entire cluster are set.

clu1

 The replication settings are the same as those for standalone Hyper-V servers.

conf2

 Network Considerations for Hyper-V Replica Scenarios

There are scenarios where the Replica server, or Replica cluster, will reside at a Disaster Recovery (DR) site located across a WAN link and the DR site uses a completely different network-addressing scheme than the Primary site.  In this configuration, when virtual machines are failed over to a DR site, a new IP configuration will be needed for each network configured in the virtual machine.  To accommodate this scenario, there is built-in functionality in Hyper-V Replica where virtual machines network settings can be modified to include configuration information for a different network at a DR site.  To take advantage of this, the Hyper-V Administrator must modify the network configuration for each replicated virtual machine on the Replica server.  If connectivity to networks at the replica site is required, the settings for all networks a virtual machine is connected to must be modified. The Hyper-V Administrator can provide both IPv4 and IPv6 configuration information for a virtual machine.  The Failover TCP/IP setting, which is available after replication is enabled for the virtual machine, is used to provide the network configuration information in the virtual machine.

clu3

The addressing information provided is used when a Failover action (Planned Failover or Failover) is executed.  The configuration of the Guest virtual machine IP settings in this manner only applies to Synthetic Network Adapters and not Legacy Network Adapters.  The operating system running in the Guest virtual machine must be one of the following – Windows Server “8” Beta, Windows Server 2008 R2, Windows Server 2008, Windows Server 2003 SP2 (or higher), Windows 7, Vista SP2 (or higher), and Windows XP SP2 (or higher).  The latest Windows Server “8” Beta Integration Services must be installed in the virtual machine.

The information is reflected in the virtual machine configuration file located on the Replica server.

clu4