Investigate a cluster resource that keeps moving between nodes

A cluster resource moving between nodes is rarely a configuration error in the resource itself. It is usually a symptom of underlying instability. The cluster service is reacting to a perceived failure, and your job is to find the root cause before the resource moves again. The first thing to check is the system log on the nodes, specifically the events generated by the Failover Clustering service.

Check the system log for cluster events

Open Event Viewer on the affected nodes and navigate to Windows Logs > System. Filter the log by the source FailoverClustering. You are looking for a sequence of events that explains why the resource failed and why the cluster decided to move it.

Event 1069 is the primary indicator of a resource failure. The description states: “Cluster resource ‘%1’ in clustered role ‘%2’ failed.” The cluster service uses the failure policies for the resource and role to decide whether to restart the resource on the same node or move the group to another node. If you see this event followed by a resource start error on a different node, the cluster has already made the decision to move.

However, Event 1069 does not tell you why the resource failed. You need to look for preceding events that indicate a node-level or storage-level issue. Event 1135 is critical here: “Cluster node ‘%1’ was removed from the active failover cluster membership.” This event suggests the node lost communication with other active nodes. The guidance in the documentation recommends checking for hardware or software errors related to network adapters and verifying the network components to which the node is connected Failover Clustering system log events.

If the issue is storage-related, look for Event 1038: “Ownership of cluster disk ‘%1’ has been unexpectedly lost by this node.” This event indicates that the node holding the disk resource lost ownership, which can trigger a failover. The documentation advises checking your storage configuration Failover Clustering system log events.

Correlate with Health Service faults

If the cluster is running on Storage Spaces Direct, the Health Service provides a more granular view of hardware and configuration issues. The Health Service constantly monitors the cluster and generates faults that are designed to be precise and actionable Health Service faults.

Run the following PowerShell command to see any current faults:

Get-StorageSubSystem Cluster* | Debug-StorageSubSystem

This cmdlet returns any faults which affect the overall Storage Spaces Direct cluster. Most often, these faults relate to hardware or configuration. If there are no faults, this cmdlet will return nothing Health Service faults.

Each fault contains five important fields: Severity, Description of the problem, Recommended next step(s), Identifying information for the faulting entity, and its physical location. For example, a fault might look like this:

Severity: MINOR
Reason: Connectivity has been lost to the physical disk.
Recommendation: Check that the physical disk is working and properly connected.
Part: Manufacturer Contoso, Model XYZ9000, Serial 12345789
Location: Seattle DC, Rack B07, Node 4, Slot 11

The Health Service can also assess the potential causality among faulting entities to identify and combine faults which are consequences of the same underlying problem. For example, if a server is down, it is expected that any drives within the server will also be without connectivity. Therefore, only one fault will be raised for the root cause - in this case, the server Health Service faults.

Investigate network and storage connectivity

Once you have identified the relevant events and faults, focus on the specific components mentioned. If Event 1135 or Event 1553 (“This cluster node has no network connectivity”) is present, verify the network configuration. Check for hardware or software errors related to the network adapters. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges Failover Clustering system log events.

If Event 1038 or a Health Service fault related to physical disk connectivity is present, verify the storage configuration. The documentation advises running the Validate a Configuration wizard to check your storage configuration for events like 1035 and 1037, which indicate mount or filesystem failures Failover Clustering system log events. For Event 1038, the primary step is to ensure the disk is accessible and properly connected.

Verify the resource state

After addressing the underlying issues, verify that the resource is stable. Use the Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet to check the resource and group state. Ensure that the resource is online and that no new failure events are being generated Failover Clustering system log events.

If the resource continues to move, review the failure policies for the resource and role. The cluster service may be configured to restart the resource on the same node or move the group to another node. Adjusting these policies may help, but only after the root cause has been resolved.