August 22, 2020

Issue and Background

A customer encountered problems with accessing the ADM console after a power failure at their data center. The nsroot credentials failed to work and accessing the gui resulted in page not found error.
The following troubleshooting steps were performed to recover access:

• Login to the Citrix ADM with nsrecover credentials ( to reset ADM password follow this, in our case /flash/mpsconfig/.recover was missing therefore we could not reset nsroot password )
• Run /mps/mas_recovery/mas_recovery.py script
• When the Citrix ADM cleanup utility screen is shown, type y and enter key to proceed
• The screen lists the list of files in the database. This step determines connectivity with the postgresql database. In this case, we got a “connection to the database could not be established” error

Citrix Support recommended verifying File system integrity of the ADM appliance as a next step, the following steps were followed to verify file system integrity of the ADM appliance:
• Restart the appliance, Press Ctrl+C simultaneously to interrupt the boot, till the screen to press spacebar is displayed

• Run boot -s command to start kernel in single user mode

• Press Enter when the prompt for entering full pathname of shell or return for /bin/sh is displayed.

• The prompt of the appliance changes to \u@ ( this is different from what was mentioned in the support article CTX122845)

• Verify if flash drive is mounted using df -k command, if not mount the flash drive using: mount dev/ad0s1a /flash

• Run fsck -t ufs /dev/ad0s1a (-t ufs is used with fsck after version 10.5)
• Verify if flash drive is mounted using df -k command, if not mount the flash drive using: mount dev/ad0s1e /var

• Run fsck -t ufs /dev/ad0s1e (-t ufs is used with fsck after version 10.5)
• File system integrity of the ADM appliance was validated with zero errors

Resolution

It was later determined that a library(libavrocpp.so.1.8.2.0) was missing from the ADM, which was replaced from a working appliance, however the ADM appliance was still inaccessible.

Access to the ADM appliance was restored by upgrading the appliance to Citrix ADM 13.0 Build 61.48, the root cause was found to be database corruption. The upgrade was performed using the following steps:
• Download the Citrix ADM upgrade image file from Citrix Website
• Upload the image to the ADM using nsrecover credentials
• Logon to the ADM appliance using nsrecover credentials
• Navigate to the folder where the image file resides and unzip the file

• Run the following script to upgrade the appliance: ./installmas

Chetan Kini

Chetan is a consultant focused on virtualization and cloud technologies. Chetan has years of experience in professional and managed services and has specialized in end-user computing with organizations across North America.