How to extend Azure Service Fabric to on-premise?

You can deploy a Service Fabric cluster on any physical or virtual machine running the Windows Server operating system, including ones residing in your on-premises datacenters. You use the standalone Windows Server package available for download from the Azure online documentation website (see here).

To deploy an on-premises Service Fabric cluster, perform the following steps:

  1. Plan for your cluster infrastructure. You must consider the resiliency of the hardware and network components and the physical security of their location because in this scenario, you can no longer rely on the resiliency and security features built in to the Azure platform.
  2. Prepare the Windows Server computer to satisfy the prerequisites. Each computer must be running Windows Server 2012 or Windows Server 2012 R2, have at least 2 GB of RAM, and have the Microsoft .NET Framework 4.5.1 or later. Additionally, ensure that the Remote Registry service is running.
  3. Determine the initial cluster size. At a minimum, the cluster must have three nodes, with one node per physical or virtual machine. Adjust the number of nodes according to your projected workload.
  4. Determine the number of fault domains and upgrade domains. The assignment of fault domains should reflect the underlying physical infrastructure and the level of its resiliency to a single component failure. Typically, you consider a single physical rack to constitute a single fault domain. The assignment of upgrade domains should reflect the number of nodes that you plan to take down during application or cluster upgrades.
  5. Download the Service Fabric standalone package for Windows Server.
  6. Define the cluster configuration. Specify the cluster settings before provisioning. To do this, modify the ClusterConfig.json file included in the Service Fabric standalone package for Windows Server. The file has several sections, including:
    o    NodeTypes. NodeTypes allow you to divide cluster nodes into groups represented by a node types and assign common properties to the nodes within each group. These properties specify endpoint ports, placement constraints, or capacities.
    o    Nodes. Nodes determine the settings of individual cluster nodes, such as the node name, IP address, fault domain, or upgrade domain.
  7. Run the create cluster script. After you modify the ClusterConfig.json file to match your requirements and preferences, run the CreateServiceFabricCluster.ps1 Windows PowerShell script and reference the .json file as its parameter. You can run the script on any computer with connectivity to all the cluster nodes.
  8. Connect to the cluster. To manage the cluster, connect to it via http://IP_Address_of_a_node:19080/Explorer/index.htm, where IP_Address_of_a_node is the IP address of any of the cluster nodes.

 

Cheers,

Marcos Nogueira
azurecentric.com
Twitter: @mdnoga

Cluster Shared Volumes (CSV) errors on Hyper-V Cluster

In a failover cluster, virtual machines can use Cluster Shared Volumes that are on the same LUN (disk), while still being able to fail over (or move from node to node) independently of one another. Virtual machines can use a Cluster Shared Volume only when communication between the cluster nodes and the volume is functioning correctly, including network connectivity, access, drivers, and other factors.

You probably didn’t notice any issues with your VMs, but If you are getting the following events on your Hyper-V Cluster nodes, regarding the CSV volume:

Warning – Disk 153

“The description for Event ID 153 from source disk cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

The following information was included with the event:

DeviceHarddisk5DR5

the message resource is present but the message is not found in the string/message table

Information – Microsoft-Windows-FailoverClustering 5121

“The description for Event ID 5121 from source Microsoft-Windows-FailoverClustering cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Volume2

Cluster Disk 2 – Volume2

Error – Microsoft-Windows-FailoverClustering 5120

“The description for Event ID 5120 from source Microsoft-Windows-FailoverClustering cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Volume1

Cluster Disk 1 – Volume1

STATUS_DEVICE_BUSY(80000011)

That means there has been an interruption to communication between a cluster node and a volume in Cluster Shared Volumes. This interruption may be short enough that it is not noticeable, or long enough that it interferes with services and applications using the volume

How to resolve it

CSV – Review events related to communication with the volume

To perform the following procedure, you must be a member of the local Administrators group on each clustered server, and the account you use must be a domain account, or you must have been delegated the equivalent authority.

To open Event Viewer and view events related to failover clustering:

1. If Server Manager is not already open, click Start, click Administrative Tools, and then click Server Manager. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

2. In the console tree, expand Diagnostics, expand Event Viewer, expand Windows Logs, and then click System.

3. To filter the events so that only events with a Source of FailoverClustering are shown, in the Actions pane, click Filter Current Log. On the Filter tab, in the Event sources box, select FailoverClustering. Select other options as appropriate, and then click OK.

4. To sort the displayed events by date and time, in the center pane, click the Date and Time column heading.

CSV – Check storage and network configuration

To perform the following procedures, you must be a member of the local Administrators group on each clustered server, and the account you use must be a domain account, or you must have been delegated the equivalent authority.

Gathering information about the condition and configuration of a disk in Cluster Shared Volumes

To gather information about the condition and configuration of a disk in Cluster Shared Volumes:

1. Scan appropriate event logs for errors that are related to the disk.

2. Review information available in the interface for the storage and if needed, contact the vendor for information about the storage.

3. To open the failover cluster snap-in, click Start, click Administrative Tools, and then click Failover Cluster Manager. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Yes.

4. In the Failover Cluster Manager snap-in, expand the console tree and click Cluster Shared Volumes. In the center pane, expand the listing for the volume that you are gathering information about. View the status of the volume.

5. Still in the center pane, to prepare for testing a disk in Cluster Shared Volumes, right-click the disk, click Take this resource offline, and then if prompted, confirm your choice. Repeat this action for any other disks that you want to test.

6. Right-click the cluster containing the Cluster Shared Volumes, and then click Validate This Cluster.

7. On the Testing Options page, select Run only tests I select.

8. On the Test Selection page, clear the check boxes for System Configuration and Network. This leaves the tests for Cluster Configuration, Inventory, and Storage. You can run all these tests, or you can select only the specific tests that appear relevant to your situation.

NOTE: If you run the Storage tests you will have downtime in your Cluster. Not recommend if you are troubleshooting on a production environment.

9. Follow the instructions in the wizard to run the tests.

10. On the Summary page, click View Report.

11. Under Results by Category, click Storage, click any test that is not labelled as Success, and then view the results.

12. Scroll back to the top of the report, and under Results by Category, click Cluster Configuration, and then click List Cluster Network Information. Confirm that any network that you intend for communication between nodes and Cluster Shared Volumes is labelled either Internal use or Internal and client use. Confirm that other networks (for example, networks used only for iSCSI and not for cluster network communication) do not have these labels.

13. If the information in the report shows that one or more networks are not configured correctly, return to the Failover Cluster Manager snap-in and expand Networks. Right-click the network that you want to modify, click Properties, and then make sure that the settings for Allow the cluster to use this network and Allow clients to connect through this network are configured as intended.

14. To bring disks back online, click Cluster Shared Volumes and, in the center pane, right-click a disk, and then click Bring this resource online. Repeat this action for any other disks that you want to bring online again.

Verifying settings for a network designated for network communication with Cluster Shared Volumes

To verify settings for a network designated for network communication with Cluster Shared Volumes:

1. Click Start, click Control Panel, click Network and Internet, and then click Network and Sharing Center.

2. In the Tasks pane, click Change adapter settings.

3. Right-click the connection you want, and then click Properties.

4. Make sure that the following check boxes are selected:

  • Client for Microsoft Networks
  • File and Printer Sharing for Microsoft Networks

Verifying that the required NTLM authentication is allowed

1. On a node in the cluster, to see the security policies that are in effect locally, click Start, click Administrative Tools, and then click Local Security Policy.

2. Navigate to Security SettingsLocal PoliciesSecurity Options.

3. In the center pane, click the Policy heading to sort the policies alphabetically.

4. Review Network security: Restrict NTLM: Add remote server exceptions for NTLM authentication and the items that follow it. If items related to “server exceptions” are marked Disabled, or other items have specific settings, a policy may be in place that is interfering with NTLM authentication on this server. If this is the case, contact an appropriate administrator (for example, your administrator for Active Directory or security) to ensure that NTLM authentication is allowed for cluster nodes that are using Cluster Shared Volumes.

Opening Event Viewer and viewing events related to failover clustering

To open Event Viewer and view events related to failover clustering:

1. If Server Manager is not already open, click Start, click Administrative Tools, and then click Server Manager. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Yes.

2. In the console tree, expand Diagnostics, expand Event Viewer, expand Windows Logs, and then click System.

3. To filter the events so that only events with a Source of FailoverClustering are shown, in the Actions pane, click Filter Current Log. On the Filter tab, in the Event sources box, select FailoverClustering. Select other options as appropriate, and then click OK.

4. To sort the displayed events by date and time, in the center pane, click the Date and Time column heading.

Finding more information about the error codes that some event messages contain

To find more information about the error codes that some event messages contain:

1. View the event, and note the error code.

2. Look up more information about the error code in one of two ways:

NET HELPMSG errorcode

How to verify it

Confirm that the Cluster Shared Volume can come online. If there have been recent problems with writing to the volume, it can be appropriate to monitor event logs and monitor the function of the corresponding clustered virtual machine, to confirm that the problems have been resolved.

To perform the following procedures, you must be a member of the local Administrators group on each clustered server, and the account you use must be a domain account, or you must have been delegated the equivalent authority.

Confirming that a Cluster Shared Volume can come online

To confirm that a Cluster Shared Volume can come online:

1. To open the failover cluster snap-in, click Start, click Administrative Tools, and then click Failover Cluster Manager. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Yes.

2. In the Failover Cluster Manager snap-in, if the cluster you want to manage is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.

3. If the console tree is collapsed, expand the tree under the cluster you want to manage, and then click Cluster Shared Volumes.

4. In the center pane, expand the listing for the volume that you are verifying. View the status of the volume.

5. If a volume is offline, to bring it online, right-click the volume and then click Bring this resource online.

Using a Windows PowerShell command to check the status of a resource in a failover cluster

To use a Windows PowerShell command to check the status of a resource in a failover cluster:

1. On a node in the cluster, click Start, point to Administrative Tools, and then click Windows PowerShell Modules. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Yes.

2. Type: Get-ClusterSharedVolume

If you run the preceding command without specifying a resource name, status is displayed for all Cluster Shared Volumes in the cluster.

MPIO on Hyper-V Server

On the previous version of Windows Server (prior Windows Server 2012) you have to download and install MultiPath I/O (MPIO). Since Windows Server 2012 MPIO is a feature that you can enable. Because it’s a feature that comes with the server, means that you will have the PowerShell cmdlets available.

Use of the MPIO module in Windows PowerShell requires an “elevated” PowerShell window, opened with Administrator privileges.

How to do it

 

Installing MPIO using the GUI

If you have Hyper-V Servers, you don’t have GUI on the server, but you can do it remotely from other server or from you RSAT installed on Windows 8.1, using the Server Manager Console. Just follow the steps.

1. Open Server Manager Console

2. Browse the Hyper-V Server that you want to enable the MPIO. To do that click on All Servers and then click on the Hyper-V Server.

image

3. Right-Click on the Hyper-V Server and click on Add Roles and Features

4. Click 4 times Next (to go to features windows)

image

5. On the Select features window, select Multipath I/O and click next.

image

6. Click Install to enable the feature.

Installing and Managing MPIO using PowerShell

Enable or Disable the MPIO Feature

If the MPIO feature is not currently installed, use the following command to enable the MPIO feature:

Enable-WindowsOptionalFeature –Online –FeatureName MultiPathIO

clip_image007

To disable the MPIO feature, use the following command

Disable-WindowsOptionalFeature –Online –FeatureName MultiPathIO

Listing commands available in the MPIO module

The commands available in the MPIO module can be listed using get-command as shown below

clip_image009

Full help and example content for the MPIO module is available via the following method:

  • In PowerShell, after importing the MPIO module or using any MPIO cmdlet, updated help can be downloaded from the internet by running the following command:
    • Update-Help

Tips and Tricks

Configuring MPIO using PowerShell

If these steps are performed prior to connecting devices of the desired BusType, you can typically avoid the need for a restart.

  • Install the MPIO feature on a new Windows Server 2012 installation.
  • Configure MPIO to automatically claim all iSCSI devices.
  • Configure the default Load Balance policy for Round Robin.
  • Set the Windows Disk timeout to 60 seconds.

Here is what this script would look like:

# Enable the MPIO Feature

Enable-WindowsOptionalFeature –Online –FeatureName MultiPathIO

# Enable automatic claiming of ISCSI devices for MPIO

Enable-MSDSMAutomaticClaim -BusType iSCSI

# Set the default load balance policy of all newly claimed devices to Round Robin

Set-MSDSMGlobalLoadBalancePolicy -Policy RR

# Set the Windows Disk timeout to 60 seconds

Set-MPIOSetting -NewDiskTimeout 60

Restoring Hyper-V and Virtual Machines using Windows Server Backup

Cluster, Hyper-V Replica, Storage Migration, and other features makes for a higher availability for Hyper-V virtual machines, but it is impossible to provide 100 percent assurance in any system.

When we lose a host computer, for example, not only is a single system offline, but also all the virtual machines that are running in there. It does not matter if it is only an application, a virtual machine, or the entire Hyper-V host that needs to be restored. In simple words, you have to be prepared to bring your systems online as soon as possible in case of a failure.

One of the benefits of Windows Server Backup is that it is easy to use and it is present in the OS already. With a couple of clicks, your hosts and its VMs are up and running.

 

How to do it

In the following steps, you will see steps how to restore a backup containing virtual machines and the host components using Windows Server Backup:

1. If the computer on which you restore the data does not have Windows Server Backup installed, proceed with the following steps. In case it is already installed, go to step 7.

2. To install the Windows Server Backup feature, open Server Manager from the taskbar.

3. In the Server Manager Dashboard, click on Manage and select Add Roles and Features.

4. In the Before you begin page, click on Next four times.

5. In Select features, select Windows Server Backup, as shown in the following screenshot:

clip_image002

6. Click on Next and then click on Install. Wait for the installation to be completed.

7. To open WSB, in the start menu, type wbadmin.msc.

8. In the Windows Server Backup console, in the pane on the right-hand side, click on Restore.

9. In the Recovery Wizard window, under Getting Started, select This server if the backup is stored locally, or select A backup stored on another location if the backup is in a storage or network folder, as shown in the following screenshot. Then click on Next.

clip_image004

10. In the Specify Location Type window, you can select from the options Local drives or Remote shared folder. Select the option where your backup files are situated and click on Next.

11. In the Select Backup Location window, click on the drop-down list to select the backup location where the files are placed, and then click on Next.

12. In the Select Server window, select the server which you want to restore and click on Next.

13. In Select Backup Date, select the month, day, and time for restoring. If you want to check the content of the backup, click on the hyperlink in front of Recoverable items. Click on Next when done.

14. In the Select Recovery Type page, select what you want to recover. You can choose from the following options

  • Files and folders
  • Hyper-V
  • Volumes, Applications
  • System State

To recover virtual machine and the host components, select Hyper-V, as shown in the following screenshot, and click on Next:

clip_image006

15. On the Select Items to Recover page, select the virtual machine that you want to restore. If you need to recover the Hyper-V settings, check the Host Component option and click on Next.

16. In the Specify Recovery Options, you can select from the following options

  • Recover to original location
  • Recover to alternate location
  • Copy to folder

Select the options that suits your environment and click on Next.

17. In the Confirmation window, verify the recovery items that will be restored and click on Recover. Wait until the restore is complete and click on Close to close the Recovery Wizard.

18. To check the recovered items, in the Windows Server Backup console, under Messages, double click on the recovery event.

19. To check the recovered files, click on View list of all recovered files in the Application recovery window.

20. To check whether the restoration was successful, open the Hyper-V Manager and check whether the restored virtual machines are listed.

Backing up Hyper-V and virtual machines using Windows Server Backup

Previous versions of Hyper-V (like 2008, 2008 R2 and 2008 R2 SP1) had complications and incompatibilities with the built-in backup tool, forcing the administrators to acquire other solutions for backing up and restoring.

Since Windows Server 2012, Windows Server comes with a feature known as Windows Server Backup (WSB), which has full Hyper-V integration, allowing you to back up and restore your server, applications, Hyper-V, and virtual machines. WSB is easy and provides for a low cost scenario for small and medium companies.

How to do it

The following steps will show you how to install the Windows Server Backup feature and how to schedule a task to back up your Hyper-V settings and virtual machines:

1. To install the Windows Server Backup feature, open Server Manager from the taskbar.

2. In the Server Manager Dashboard, click on Manage and select Add Roles and Features.

3. On the Before you begin page, click on Next four times.

4. Under the Add Roles and Features Wizard, select Windows Server Backup from the Features section, as shown in the following screenshot:

clip_image002

5. Click on Next and then click on Install. Wait for the installation to be completed.

6. After the installation, open the Start menu and type wbadmin.msc to open the Windows Server Backup feature.

7. To change the backup performance options, click on Configure Performance from the pane on the right-hand side in the Windows Server Backup console.

8. In the Optimize Backup Performance window, we have three options to select from, as shown in the following screenshot:

  • Normal backup performance
  • Faster backup performance
  • Custom

clip_image004

9. In the Windows Server Backup console, in the pane on the right-hand side, select the backup that you want to perform. The two available options are:

  • Backup Schedule to schedule an automatic backup
  • Backup Once for a single backup.

The next steps will show how to schedule an automatic backup.

10. In the Backup Schedule Wizard, in the Getting Started page, click on Next.

11. In the Select Backup Configuration page, select Full Server to back up all the server data or click on Custom to select specific items to back up. If you want to backup only Hyper-V and virtual machines, click on Custom and then Next.

12. In Select Items for Backup, click on Add Items.

13. In the Select Items window, select Hyper-V to back up all the virtual machines and the host component. You can also expand Hyper-V and select the virtual machines that you want to back up. When finished, click on OK.

14. Back to the Select Items for Backup, click on Advanced Settings to change Exclusions and VSS Settings.

15. In the Advanced Settings window, in the Exclusions tab, click on Add Exclusion to add any necessary exclusions.

16. Click on the VSS Settings tab to select either VSS full Backup or VSS copy Backup as shown in the following screenshot. Click on OK.

clip_image006

17. In the Select Items for Backup window, confirm the items that will be backed up and click on Next.

18. In the Specify Backup Time page, select Once a day and the time for a daily backup or select More than once a day and the time and click on Next.

19. In the Specify Destination Type page, select the one of the options Back up to a hard disk that is dedicated for backups (recommended), Back up to a volume, or Back up to a shared network folder, as shown in the following screenshot and click on Next.

clip_image008

NOTE: If you select the first option, the disk you choose will be formatted and dedicated to storing the backup data only.

20. In Select Destination Disk, click on Show All Available Disks to list the disks, select the one you want to use to store your backup, and click on OK. Click on Next twice.

21. If you have selected the Back up to a hard disk that is dedicated for backups (recommended) option, you will see a warning message saying that the disk will be formatted. Click on Yes to confirm.

22. In the Confirmation window, double-check the options you selected and click on Finish.

23. After that, the schedule will be created. Wait until the scheduled time to begin and check whether the backup has been finished successfully.

Tips and Tricks

The Windows Server Backup feature, introduced in Windows Server 2008, has many limitations such as no tape support, no advanced schedule options, fewer backup options, and so on. When we talk about Hyper-V in this regards, the problem is even worse. Windows Server 2008 has minimal support and features for it.

Since Windows Server 2012, the same feature is available with some limitations; however, it provides at least the core components to back up, schedule, and restore Hyper-V and your virtual machines.

By default, WSB is not installed. The feature installation is made by Server Manager. After its installation, the feature can be accessed via console or command lines. Before you start the backup of your servers, it is good to configure the backup performance options you want to use.

Faster Backup

By default, all the backups are created as normal. It creates a full backup of all the selected data. This is an interesting option when low amounts of data are backed up. You can also select the Faster backup performance option. This backs up the changes between the last and the current backup, increasing the backup time and decreasing the stored data. This is a good option to save storage space and backup time for large amounts of data, especially large virtual machines.

Backup Schedule

A backup schedule can be created to automate your backup operations. In the Backup Schedule Wizard, you can back up your entire server or a custom selection of volumes, applications, or files. For backing up Hyper-V and its virtual machines, the best option is the customized backup, so that you don’t have to back up the whole physical server.

When Hyper-V is present on the host, the system shows Hyper-V, and you will be able to select all the virtual machines and the host component configuration to be backed up. During the wizard, you can also change the advanced options such as exclusions and Volume Shadow Copy Services (VSS) settings.

WSB has two VSS backup options, VSS full backup and VSS copy backup. When you opt for VSS full backup, everything is backed up and after that, the application may truncate log files. If you are using other backup solutions that integrate with WSB, these logs are essential to be used in future backups such as incremental ones. To preserve the log files you can use VSS copy backup so that other applications will not have problems with the incremental backups.

After selecting the items for backup, you have to select the backup time. This is another limitation from the previous version, only two schedule options, namely Once a day or More than once a day. If you prefer to create different backup schedule such as weekly backups, you can use the WSB command lets in PowerShell.

Backup Destination

Moving forward, in the backup destination type, you can select between a dedicated hard disk, a volume, or a network folder to save your backups in. When confirming all the items, the backup schedule will be ready to back up your system. You can also use the option Backup once to create a single backup of your system.