HA Heartbeats not seen on all interfaces
Why it happens and how to fix it
High availability (HA) is essential for any critical NetScaler deployment, ensuring that your services remain online even if a node fails. In a typical HA setup, two NetScaler appliances work together: one operates as the primary node, handling connections and server management, while the other acts as the secondary node, monitoring the primary’s health. If the primary node becomes unavailable, the secondary takes over to maintain uninterrupted service.
The secondary node keeps tabs on the primary by sending periodic heartbeat messages (health checks). If these heartbeats are not received, the secondary retries for a specified period before deciding to take over (a process known as failover).
The Risk: Heartbeats on only one interface
While HA provides resilience, relying on a single interface for heartbeats introduces risk. If that interface fails, both nodes may lose the ability to determine which should be primary, potentially resulting in both acting as primary and causing downtime, a classic split-brain scenario.
Best practice: Always enable HA on all interfaces, or at least two, to avoid a single point of failure.
Why are heartbeats missing on some interfaces?
By default, HA packets are sent and received over what NetScaler considers the native (untagged) VLAN. In environments with tagged interfaces and channels, it’s common to find that HA packets are not seen on all interfaces. This typically happens because the switch ports are configured to only allow tagged VLANs, causing untagged heartbeat packets to be dropped.
How to verify missing heartbeats
You can check for missing HA heartbeats in several ways:
1. Command Line Check:
Run show ha node to see which interfaces are sending and receiving heartbeats. If heartbeats are missing, you’ll see output like:
2. Failover Testing:
During a manual failover, you may receive a warning such as:
“[WARNING]: Force Failover may cause configuration loss, peer health not optimum. Reason(s):
– HA heartbeats not seen on some interfaces.”
3. NetScaler Alerts:
Look for alerts such as:
4. Packet Capture:
Capture HA traffic on the relevant interfaces to confirm whether heartbeat packets are being transmitted and received. For example, you may see that HA heartbeats are only sent via native VLAN 1:
VLAN configuration example
Let’s clarify with a sample VLAN configuration:
In this setup, LA/4 is connected to trunked/tagged switch ports with allowed VLANs. If LA/4 sends untagged HA heartbeats, these are dropped by the switch, as it expects only tagged traffic.
Resolution: Ensure heartbeats on all required interfaces
To resolve this, you have two main options:
Switch-side: Configure switch ports to allow both a native VLAN and tagged VLANs. However, this is not always preferred by network teams.
NetScaler-side: Set the channel to tag all traffic and untag one VLAN, making it the native VLAN for that interface or channel. This ensures that HA heartbeats are sent using the correct VLAN.
Commands to tag all on the channel and untag the VLAN:
Remark: This configuration applies to both channels and interfaces.
Now, or sample configuration will look like:
After these changes, you can verify that heartbeats are now sent via LA/4 on VLAN 847:
Best practices and final thoughts
Always enable HA heartbeats on at least two interfaces.
Avoid relying on VLAN 1 for critical HA traffic; use dedicated VLANs where possible.
Regularly verify HA status using both CLI commands and monitoring alerts.
If you see warnings or alerts about missing heartbeats, investigate VLAN tagging and switch port configuration immediately.
Ensuring that HA heartbeats are visible on all required interfaces is vital for a resilient NetScaler deployment. Following these best practices helps prevent split-brain scenarios and keeps your services running smoothly.
Blubyte: Your partner for expert NetScaler management, configuration, and troubleshooting. Deliver smarter. Scale faster. Stay secure.
Click, and set up a direct call with one of our top experts.