NetScaler· Stefan Jordanov

HA Heartbeats not seen on all interfaces

Why it happens and how to fix it

High availability (HA) is essential for any critical NetScaler deployment, ensuring that your services remain online even if a node fails. In a typical HA setup, two NetScaler appliances work together: one operates as the primary node, handling connections and server management, while the other acts as the secondary node, monitoring the primary’s health. If the primary node becomes unavailable, the secondary takes over to maintain uninterrupted service.

The secondary node keeps tabs on the primary by sending periodic heartbeat messages (health checks). If these heartbeats are not received, the secondary retries for a specified period before deciding to take over (a process known as failover).

The Risk: Heartbeats on only one interface

While HA provides resilience, relying on a single interface for heartbeats introduces risk. If that interface fails, both nodes may lose the ability to determine which should be primary, potentially resulting in both acting as primary and causing downtime, a classic split-brain scenario.

Best practice: Always enable HA on all interfaces, or at least two, to avoid a single point of failure.

Why are heartbeats missing on some interfaces?

By default, HA packets are sent and received over what NetScaler considers the native (untagged) VLAN. In environments with tagged interfaces and channels, it’s common to find that HA packets are not seen on all interfaces. This typically happens because the switch ports are configured to only allow tagged VLANs, causing untagged heartbeat packets to be dropped.

How to verify missing heartbeats

You can check for missing HA heartbeats in several ways:

1. Command Line Check:

Run show ha node to see which interfaces are sending and receiving heartbeats. If heartbeats are missing, you’ll see output like:

> sh ha node
1)      Node ID:        0
        IP:             <NSIP>
        Node State: UP
        Master State: Primary
        Fail-Safe Mode: OFF
        INC State: DISABLED
        Sync State: ENABLED
        Propagation: ENABLED
        Enabled Interfaces : 0/1 LA/4
        Disabled Interfaces : None
        HA MON ON Interfaces : 0/1 LA/4
        HA HEARTBEAT OFF Interfaces : None
        Interfaces on which heartbeats are not seen : LA/4
        Interfaces causing Partial Failure: None
        SSL Card Status: UP
        Sync Status Strict Mode: DISABLED
        Hello Interval: 200 msecs
        Dead Interval: 3 secs
        Node in this Master State for: 172:22:5:10 (days:hrs:min:sec)
2)      Node ID:        1
        IP:             <NSIP>
        Node State: UP
        Master State: Secondary
        Fail-Safe Mode: OFF
        INC State: DISABLED
        Sync State: SUCCESS
        Propagation: ENABLED
        Enabled Interfaces : 0/1 LA/4
        Disabled Interfaces : None
        HA MON ON Interfaces : 0/1 LA/4
        HA HEARTBEAT OFF Interfaces : None
        Interfaces on which heartbeats are not seen : LA/4
        Interfaces causing Partial Failure: None
        SSL Card Status: UP

Local node information:
        Critical Interfaces: 0/1 LA/4
 Done

2. Failover Testing:
During a manual failover, you may receive a warning such as:
“[WARNING]: Force Failover may cause configuration loss, peer health not optimum. Reason(s):
– HA heartbeats not seen on some interfaces.”

3. NetScaler Alerts:
Look for alerts such as:

haNoHeartbeats (haNicsMonitorFailed = "LA/4", haLastNicMonitorFailed = "0/1", nsPartitionName = default)

4. Packet Capture:
Capture HA traffic on the relevant interfaces to confirm whether heartbeat packets are being transmitted and received. For example, you may see that HA heartbeats are only sent via native VLAN 1:

No.    Time           Source           Destination      Protocol  Length  Info
172    9.888482254    <source_ip>      <destination_ip>  UDP       381     3003 -> 3003 Len=304

Frame 172: 381 bytes on wire (3048 bits), 381 bytes captured (3048 bits)
NetScaler Packet Trace
    Operation: TXB (0xad)
    Nic No: 2
    Activity Flags: 0x00000000
    Capture Flags: 0x00000000
    Errorcode: No Error (0x00)
    App: TEST (0x17)
    Core Id: 0
    Vlan: 1
    PcbDevNo: 0x00000000
    Linked PcbDevNo: 0x351fcdd0

The Vlan field reads 1, NetScaler’s native, untagged VLAN, confirming the heartbeat left on the default VLAN instead of a tagged one.

VLAN configuration example

Let’s clarify with a sample VLAN configuration:

VLAN ID: 1
 Interfaces : 0/1 LO/1
 Channels : LA/4

VLAN ID: 831    VLAN Alias Name: DATA831
 Channels : LA/4(T)
 IPs : <IP_address>    Mask: 255.255.255.240

VLAN ID: 847    VLAN Alias Name: DATA847
 Channels : LA/4(T)
 IPs : <IP_address>    Mask: 255.255.255.240

In this setup, LA/4 is connected to trunked/tagged switch ports with allowed VLANs. If LA/4 sends untagged HA heartbeats, these are dropped by the switch, as it expects only tagged traffic.

Resolution: Ensure heartbeats on all required interfaces

To resolve this, you have two main options:

  • Switch-side: Configure switch ports to allow both a native VLAN and tagged VLANs. However, this is not always preferred by network teams.
  • NetScaler-side: Set the channel to tag all traffic and untag one VLAN, making it the native VLAN for that interface or channel. This ensures that HA heartbeats are sent using the correct VLAN.

Commands to tag all on the channel and untag the VLAN:

set channel LA/4 -tagall ON
bind vlan 847 -ifnum LA/4

Remark: This configuration applies to both channels and interfaces.

Now, or sample configuration will look like:

VLAN ID: 1
 Interfaces : 0/1 LO/1

VLAN ID: 831    VLAN Alias Name: DATA831
 Channels : LA/4(T)
 IPs : <IP_address>    Mask: 255.255.255.240

VLAN ID: 847    VLAN Alias Name: DATA847
 Channels : LA/4
 IPs : <IP_address>    Mask: 255.255.255.240

After these changes, you can verify that heartbeats are now sent via LA/4 on VLAN 847:

No.    Time           Source           Destination      Protocol  Length  Info
221    1.366413384    <source_ip>      <destination_ip>  UDP       381     3003 -> 3003 Len=304

Frame 221: 381 bytes on wire (3048 bits), 381 bytes captured (3048 bits)
NetScaler Packet Trace
    Operation: TXB (0xad)
    Nic No: 1
    Activity Flags: 0x00000000
    Capture Flags: 0x00000000
    Errorcode: No Error (0x00)
    App: TEST (0x17)
    Core Id: 0
    Vlan: 847
    PcbDevNo: 0x00000000
    Linked PcbDevNo: 0xd889c623

The Vlan field now reads 847, confirming the heartbeat left tagged on the dedicated VLAN instead of the native one.

Best practices and final thoughts

  • Always enable HA heartbeats on at least two interfaces.
  • Avoid relying on VLAN 1 for critical HA traffic; use dedicated VLANs where possible.
  • Regularly verify HA status using both CLI commands and monitoring alerts.
  • If you see warnings or alerts about missing heartbeats, investigate VLAN tagging and switch port configuration immediately.

Ensuring that HA heartbeats are visible on all required interfaces is vital for a resilient NetScaler deployment. Following these best practices helps prevent split-brain scenarios and keeps your services running smoothly.

Blubyte: Your partner for expert NetScaler management, configuration, and troubleshooting. Deliver smarter. Scale faster. Stay secure.

Click, and set up a direct call with one of our top experts.

Tagged: Reactive IT, Managed IT services, Fixed-fee IT, IT cost savings, Downtime prevention, IT budgeting, Proactive IT management, NetScaler support, IT productivity, IT strategy