Introduction and Background
This is the second post in our series on upgrading to Citrix ADX SDX bundle 13.1 from 12.1 with the first located here. This issue followed the resolution of the first)
The issue covered in this post deals with upgrade tasks not completed successfully during the XenServer 6.x to 7.x upgrade (aka platform upgrade) despite the appliance telling us otherwise. As some post-install tasks did not complete, we’re unable to power on the VPX instances. They are left in a “halted” state, and attempts to power them on via the SVM management console are greeted with an internal error with the text:
"INTERNAL_ERROR xenopsd internal error: Unix.Unix_error(Unix.ENODEV. "write". "")
We also see this halted from within the XenServer host.
Attempts to log into the XenServer hypervisor and start the VM using the xe start-vm name-label=XXXX command also fail with the same error code.
- Citrix ADC SDX 14030 FIPS appliance
- Base SDX bundle of 12.1 b58.14
- Target SDX bundle of 13.1 b21.50 (issue has been seen on earlier 13.1 builds as well)
Root Cause and Resolution
The root cause here likely comes down to lax coding and release testing so hopefully, this is resolved in a later SDX upgrade bundle. Doing a little digging, this has come up recently on some recent, earlier 13.1 upgrade bundles. There is a post-upgrade script that does not get successfully run (postinst.sh) on the XenServer host after its transition from XenServer 6.x. Once we run this script and reboot, our VMs start (after overcoming a minor cosmetic glitch we’ll get to at the end).
Step 1 – Confirm Issue in Logs
Confirm the issue we want to resolve is indeed present by checking the SVM logs at this location: /var/mps/log/mps_control.log
Run this command to grep on the log file:
cd /var/mps/log more mps_control.log | grep postinst
If we see an output as below, you’re facing the same issue.
Step 2 – Copy Postinst.sh From SVM to XenServer
SSH’d into the SVM appliance (using nsroot), drop to shell. Copy the upgrade script from the SVM to XenServer. There is an APIPA address from within the SVM that can be used to access the underlying host without needing to SSH directly to the IP of the hypervisor.
scp /var/mps/xs6_upgrade_files/postinst.sh firstname.lastname@example.org:/
Step 3 – Invoke Script in XenServer and Reboot
While logged into the SVM’s shell, we can connect to the XenServer hypervisor and run the script that got missed during the formal platform upgrade. Once upgraded, reboot the hypervisor (will also reboot the SVM).
## Run these commands individually ## ssh email@example.com sh /postinst.sh 71000 450085 >/var/log/postinst.log 2>&1 ##Reboot once script completes and drops back on prompt, may only take a second or two## reboot
Step 4 – Incorrect States in SVM
Post reboot the instance states should no longer say “HALTED” but may in some cases display cosmetic anomalies such as showing down, or VM state running and instance state down. An example below post-reboot of the SDX.
The instance is actually running. If we attempt to boot the instance from XenServer CLI it will advise us its state is already running and if we revisit the page, its status will change to reflect its true state. Or, simply editing the VM instance and then clicking the “back” button also seems to clear the state misrepresentation.
Citrix ADC (NetScaler) SDX bundle upgrades can go sideways from time to time. Thankfully we were able to clear these hurdles within our change window and hopefully this issue will be sorted in a later release. Huge thanks to Alejandro Portal at Citrix for his assistance and in going above and beyond in helping sort this one.
Michael Shuster is Ferroque Systems’ Chief Architect and noted Citrix authority. A passionate virtualization and digital workspaces advocate, he has designed, engineered, or otherwise advised clients on Citrix, VMware, and Microsoft technology platforms across the globe.