HA Heartbeats not seen on all interfaces
Why it happens and how to fix it
High availability (HA) is essential for any critical NetScaler deployment, ensuring that your services remain online even if a node fails. In a typical HA setup, two NetScaler appliances work together: one operates as the primary node, handling connections and server management, while the other acts as the secondary node, monitoring the primary’s health. If the primary node becomes unavailable, the secondary takes over to maintain uninterrupted service.
The secondary node keeps tabs on the primary by sending periodic heartbeat messages (health checks). If these heartbeats are not received, the secondary retries for a specified period before deciding to take over (a process known as failover).
The Risk: Heartbeats on only one interface
While HA provides resilience, relying on a single interface for heartbeats introduces risk. If that interface fails, both nodes may lose the ability to determine which should be primary, potentially resulting in both acting as primary and causing downtime, a classic split-brain scenario.
Best practice: Always enable HA on all interfaces, or at least two, to avoid a single point of failure.
Why are heartbeats missing on some interfaces?
By default, HA packets are sent and received over what NetScaler considers the native (untagged) VLAN. In environments with tagged interfaces and channels, it’s common to find that HA packets are not seen on all interfaces. This typically happens because the switch ports are configured to only allow tagged VLANs, causing untagged heartbeat packets to be dropped.
How to verify missing heartbeats
You can check for missing HA heartbeats in several ways:
1. Command Line Check:
Run show ha node to see which interfaces are sending and receiving heartbeats. If heartbeats are missing, you’ll see output like:
> sh ha node
1) Node ID: 0
IP: <NSIP>
Node State: UP
Master State: Primary
Fail-Safe Mode: OFF
INC State: DISABLED
Sync State: ENABLED
Propagation: ENABLED
Enabled Interfaces : 0/1 LA/4
Disabled Interfaces : None
HA MON ON Interfaces : 0/1 LA/4
HA HEARTBEAT OFF Interfaces : None
Interfaces on which heartbeats are not seen : LA/4
Interfaces causing Partial Failure: None
SSL Card Status: UP
Sync Status Strict Mode: DISABLED
Hello Interval: 200 msecs
Dead Interval: 3 secs
Node in this Master State for: 172:22:5:10 (days:hrs:min:sec)
2) Node ID: 1
IP: <NSIP>
Node State: UP
Master State: Secondary
Fail-Safe Mode: OFF
INC State: DISABLED
Sync State: SUCCESS
Propagation: ENABLED
Enabled Interfaces : 0/1 LA/4
Disabled Interfaces : None
HA MON ON Interfaces : 0/1 LA/4
HA HEARTBEAT OFF Interfaces : None
Interfaces on which heartbeats are not seen : LA/4
Interfaces causing Partial Failure: None
SSL Card Status: UP
Local node information:
Critical Interfaces: 0/1 LA/4
Done
2. Failover Testing:
During a manual failover, you may receive a warning such as:
“[WARNING]: Force Failover may cause configuration loss, peer health not optimum. Reason(s):
– HA heartbeats not seen on some interfaces.”
3. NetScaler Alerts:
Look for alerts such as:
haNoHeartbeats (haNicsMonitorFailed = "LA/4", haLastNicMonitorFailed = "0/1", nsPartitionName = default)
4. Packet Capture:
Capture HA traffic on the relevant interfaces to confirm whether heartbeat packets are being transmitted and received. For example, you may see that HA heartbeats are only sent via native VLAN 1:
No. Time Source Destination Protocol Length Info
172 9.888482254 <source_ip> <destination_ip> UDP 381 3003 -> 3003 Len=304
Frame 172: 381 bytes on wire (3048 bits), 381 bytes captured (3048 bits)
NetScaler Packet Trace
Operation: TXB (0xad)
Nic No: 2
Activity Flags: 0x00000000
Capture Flags: 0x00000000
Errorcode: No Error (0x00)
App: TEST (0x17)
Core Id: 0
Vlan: 1
PcbDevNo: 0x00000000
Linked PcbDevNo: 0x351fcdd0
The Vlan field reads 1, NetScaler’s native, untagged VLAN, confirming the heartbeat left on the default VLAN instead of a tagged one.
VLAN configuration example
Let’s clarify with a sample VLAN configuration:
VLAN ID: 1
Interfaces : 0/1 LO/1
Channels : LA/4
VLAN ID: 831 VLAN Alias Name: DATA831
Channels : LA/4(T)
IPs : <IP_address> Mask: 255.255.255.240
VLAN ID: 847 VLAN Alias Name: DATA847
Channels : LA/4(T)
IPs : <IP_address> Mask: 255.255.255.240
In this setup, LA/4 is connected to trunked/tagged switch ports with allowed VLANs. If LA/4 sends untagged HA heartbeats, these are dropped by the switch, as it expects only tagged traffic.
Resolution: Ensure heartbeats on all required interfaces
To resolve this, you have two main options:
- Switch-side: Configure switch ports to allow both a native VLAN and tagged VLANs. However, this is not always preferred by network teams.
- NetScaler-side: Set the channel to tag all traffic and untag one VLAN, making it the native VLAN for that interface or channel. This ensures that HA heartbeats are sent using the correct VLAN.
Commands to tag all on the channel and untag the VLAN:
set channel LA/4 -tagall ON
bind vlan 847 -ifnum LA/4
Remark: This configuration applies to both channels and interfaces.
Now, or sample configuration will look like:
VLAN ID: 1
Interfaces : 0/1 LO/1
VLAN ID: 831 VLAN Alias Name: DATA831
Channels : LA/4(T)
IPs : <IP_address> Mask: 255.255.255.240
VLAN ID: 847 VLAN Alias Name: DATA847
Channels : LA/4
IPs : <IP_address> Mask: 255.255.255.240
After these changes, you can verify that heartbeats are now sent via LA/4 on VLAN 847:
No. Time Source Destination Protocol Length Info
221 1.366413384 <source_ip> <destination_ip> UDP 381 3003 -> 3003 Len=304
Frame 221: 381 bytes on wire (3048 bits), 381 bytes captured (3048 bits)
NetScaler Packet Trace
Operation: TXB (0xad)
Nic No: 1
Activity Flags: 0x00000000
Capture Flags: 0x00000000
Errorcode: No Error (0x00)
App: TEST (0x17)
Core Id: 0
Vlan: 847
PcbDevNo: 0x00000000
Linked PcbDevNo: 0xd889c623
The Vlan field now reads 847, confirming the heartbeat left tagged on the dedicated VLAN instead of the native one.
Best practices and final thoughts
- Always enable HA heartbeats on at least two interfaces.
- Avoid relying on VLAN 1 for critical HA traffic; use dedicated VLANs where possible.
- Regularly verify HA status using both CLI commands and monitoring alerts.
- If you see warnings or alerts about missing heartbeats, investigate VLAN tagging and switch port configuration immediately.
Ensuring that HA heartbeats are visible on all required interfaces is vital for a resilient NetScaler deployment. Following these best practices helps prevent split-brain scenarios and keeps your services running smoothly.
Blubyte: Your partner for expert NetScaler management, configuration, and troubleshooting. Deliver smarter. Scale faster. Stay secure.
Click, and set up a direct call with one of our top experts.
