NetScaler Bug in Azure Instances Fixed

Bug Details

Recently, a customer noticed that one of their Azure instances of NetScaler was showing as “Down” in the NetScaler Console Service. Attempts to remote into the appliance via SSH and via the web were unsuccessful, as the connections timed out. This appliance was running as the secondary instance in an HA pair, so we opened an SSH connection to the Primary appliance. When we ran “show ha node,” the secondary node showed as “Up” and synchronized.

A ping run from the local machine to the IP address of the secondary node was successful.

SSH from the Primary to the Secondary failed.

Upon opening a console to the secondary, all communications outbound from the secondary seemed to work correctly. SSH from the secondary to the primary was successful, but SSH from the secondary to the primary failed. The shell on the secondary was launched and nstcpdump.sh was run to indicate what traffic was being seen. ICMP replies to pings could be seen but attempts to connect via SSH or the web GUI (on HTTPS) were not seen by the nstcpdump.sh command.

Firmware Fix

On September 8^th, 2025, NetScaler released firmware 14.1 build 51.72 that included some new features and quite a few bug fixes. The release notes indicated that one of these (NSPLAT-33590) fixed an issue where “NetScaler VPX instances running on Microsoft Azure might experience SSH and TCP session failures. This issue can occur randomly after a VPX reboot or upgrade and is caused due to changes in Azure hypervisor layer.” This is the bug that gave the behavior experienced by the customer. To upgrade the secondary appliance, we had to perform the following steps:

1. Upload the firmware to the primary appliance.
   1. Since the secondary appliance could not be accessed, the firmware had to be placed somewhere the affected appliance could access
   2. The firmware was placed in the /var/nsinstall directory for easy transfer
2. From the Serial Console on the affected appliance
   1. Launch the Shell
   2. Run “scp <username>@<NSIP>:/var/nsinstall/build-14.1-51.72_nc_64.tgz /var/nsinstall”
      1. Example scp nsroot@10.100.100.21:/var/nsinstall/build-14.1-51.72_nc_64.tgz /var/nsinstall
   3. Wait for the file to copy to the affected NetScaler
   4. Installations should always be run from a separate folder, so we recommend
      1. cd /var/nsinstall
      2. mkdir ./build-14.1-51.72
      3. mv build-14.1-51.72_nc_64.tgz ./build-14.1-51.72
      4. cd ./build-14.1-51.72
      5. tar -zxvf build-14.1-51.72_nc_64.tgz
      6. 
   5. Now the installation can be run by executing “./installns”
      1. 
   6. Once the firmware is installed, reboot the appliance.
      1. 
3. Upon completion of the reboot, the appliance should now be accessible.

Note: The primary appliance will also need to be upgraded. Since the secondary appliance is unreachable, it is recommended that you execute the set ha node – HAStatus STAYPRIMARY on the primary node and set ha node – HAStatus STAYSECONDARY on the affected appliance so that during the reboots, the services do not attempt to failover to the secondary appliance.

Note: This will require an outage, so ensure the work is done during a period where the affected resources can be offline for a short period.

Note: Do not remove the secondary (affected) appliance from the HA pair. If the appliances are in an HA pair and one of them is removed, the primary appliance will stop responding to the Azure Load Balancer (ALB) monitor probes at port 9000, and the ALB will mark the instances both as offline and not forward any traffic.

Recommendation

It is highly recommended that your Azure instances be upgraded in a timely manner to prevent any appliances from being affected by the bug. Once affected, the recovery steps are entailed, and the upgrade is more difficult. Additionally, an outage must be taken to complete the recovery.

For customers who are still running firmware version 13.1, there was a similar firmware release for this version as well (13.1 build 60.26), with the same bug fix included.