NetScaler VIPs and Services Flapping after 13.1 b59.19 Upgrade

Issue and Background

Recent firmware updates released for NetScaler in June 2025 address CVE 2025-6543, along with CVE-2025-5349 and CVE-2025-5777. For customers opting to remain on the major release of 13.1, upgrading to 13.1 b59.19 is the target build. For release notes of this build, please refer to this link.

Working recently with a customer running multiple pairs of NetScaler (briefly Citrix ADC) VPX pairs, a condition was encountered on one pair that presented itself as periodic failures of all back-end services being monitored and their associated VIPs. This occurred roughly every 10-20 minutes.

Below are additional conditions:

• Very basic and appliance configurations (NetScaler Gateway, load balancing VIPs), and very low transaction load
• No network or hypervisor changes occurred around the time of the issue presenting itself, no correlated VM live migration events
• The issue was present on both nodes of the HA pair, consistent experience when failing over between them
• Availability of NSIPs, CLI, the web admin UI were unaffected
• Active ICA connections through NetScaler Gateway remained available and failed over during forced HA events
• Reboots of the appliances did not resolve the issue
• No double entries of default gateways were present in the routing tables
• Ping was possible from SNIP to back-end servers via CLI
• Upstream firewalls (active and passive) hosting the default gateway used by the appliances had the correct ARP entries for the floating IPs (SNIPs, VIPs) for each node when primary, ruling out stale ARP entries
• HA status of both nodes in the pair were healthy, not flapping, and synchronizing successfully
• No core or crash dumps on the appliance
• The NetScalers were deployed in an isolated internal network, ruling out DDoS attacks from the Internet
• The issue did not occur on other near-identically configured appliances in the environment, previously upgraded to 13.1 b59.19, this was isolated to one pair
• CPU load remained low, RAM utilization 40%

Root Cause and Resolution

After investigating the range of usual suspects based on the symptoms and yielding no results, a less likely but once identified, logical cause was identified.

When performing configuration and hardware comparisons between working pairs and the non-working pairs, it was discovered the VPXs had remained with their default RAM allocation of 2GB from their initial deployment in contrast to their other counterparts in the environment deployed with 8GB. While 2GB has been the default in most hypervisor images upon provisioning and remains the bare minimum NetScaler documentation outlines, it is possible this is no longer sufficient. We have typically recommended a minimum of 4GB on VPXs even in non-production and POC use cases. More is typically recommended in our configurations when accommodating for load requiring multiple packet processing engines (PPEs) and features (IP Reputation, etc.).

It was somewhat perplexing that the RAM utilization did not go much past 40% and no evidence of core or crash dumps were seen. Prior to the firmware upgrade, the appliances had been in use for several years and had transitioned between at least three major firmware releases, A deeper analysis of memory and process behaviour in the logs is pending, to further zero in on the root cause and determine if a memory leak was to blame.

Upon aligning the RAM allocation to 8GB on each VPX to align with the broader environment’s standard (we suspect 4GB would have been sufficient for their use case and aligned to NetScaler’s “recommended” starting point), the issue disappeared on both nodes of the pair and no recurrence of the issue materialized in 24 hours.

While this was evidently an oversight from whomever initially deployed the pair, we would emphasize that 2GB on a VPX running any current 13.1 or 14.1 firmware and beyond, regardless of intended use case, is ill advised, and would suggest starting with 4GB minimum, not 2GB, to avoid surprises that mimic other more common culprits of such behaviours.