April 16, 2022

Introduction and Background

This is the second post in our series on upgrading to Citrix ADX SDX bundle 13.1 from 12.1 with the first located here. This issue followed the resolution of the first)

The issue covered in this post deals with upgrade tasks not completed successfully during the XenServer 6.x to 7.x upgrade (aka platform upgrade) despite the appliance telling us otherwise. As some post-install tasks did not complete, we’re unable to power on the VPX instances. They are left in a “halted” state, and attempts to power them on via the SVM management console are greeted with an internal error with the text:

"INTERNAL_ERROR xenopsd internal error: Unix.Unix_error(Unix.ENODEV. "write". "")

We also see this halted from within the XenServer host.

Attempts to log into the XenServer hypervisor and start the VM using the xe start-vm name-label=XXXX command also fail with the same error code.

Environment:

Citrix ADC SDX 14030 FIPS appliance
Base SDX bundle of 12.1 b58.14
Target SDX bundle of 13.1 b21.50 (issue has been seen on earlier 13.1 builds as well)

Root Cause and Resolution

The root cause here likely comes down to lax coding and release testing so hopefully, this is resolved in a later SDX upgrade bundle. Doing a little digging, this has come up recently on some recent, earlier 13.1 upgrade bundles. There is a post-upgrade script that does not get successfully run (postinst.sh) on the XenServer host after its transition from XenServer 6.x. Once we run this script and reboot, our VMs start (after overcoming a minor cosmetic glitch we’ll get to at the end).

Step 1 – Confirm Issue in Logs

Confirm the issue we want to resolve is indeed present by checking the SVM logs at this location: /var/mps/log/mps_control.log

Run this command to grep on the log file:

cd /var/mps/log
more mps_control.log | grep postinst

If we see an output as below, you’re facing the same issue.

Step 2 – Copy Postinst.sh From SVM to XenServer

SSH’d into the SVM appliance (using nsroot), drop to shell. Copy the upgrade script from the SVM to XenServer. There is an APIPA address from within the SVM that can be used to access the underlying host without needing to SSH directly to the IP of the hypervisor.

scp /var/mps/xs6_upgrade_files/postinst.sh root@169.254.0.1:/

Step 3 – Invoke Script in XenServer and Reboot

While logged into the SVM’s shell, we can connect to the XenServer hypervisor and run the script that got missed during the formal platform upgrade. Once upgraded, reboot the hypervisor (will also reboot the SVM).

## Run these commands individually ##
ssh root@169.254.0.1
sh /postinst.sh 71000 450085 >/var/log/postinst.log 2>&1

##Reboot once script completes and drops back on prompt, may only take a second or two##
reboot

Step 4 – Incorrect States in SVM

Post reboot the instance states should no longer say “HALTED” but may in some cases display cosmetic anomalies such as showing down, or VM state running and instance state down. An example below post-reboot of the SDX.

The instance is actually running. If we attempt to boot the instance from XenServer CLI it will advise us its state is already running and if we revisit the page, its status will change to reflect its true state. Or, simply editing the VM instance and then clicking the “back” button also seems to clear the state misrepresentation.

Conclusion

Citrix ADC (NetScaler) SDX bundle upgrades can go sideways from time to time. Thankfully we were able to clear these hurdles within our change window and hopefully this issue will be sorted in a later release. Huge thanks to Alejandro Portal at Citrix for his assistance and in going above and beyond in helping sort this one.

Michael Shuster

Michael is Ferroque's founder and a noted Citrix authority, overseeing operations and service delivery while keeping a hand in the technical cookie jar. He is a passionate advocate for end-user infrastructure technology, with a rich history designing and engineering solutions on Citrix, NetScaler, VMware, and Microsoft tech stacks.