March 25, 2021

Issue

A customer was having issues when upgraded to 13.0 71.40 firmware and higher, that the disk size was getting filled quickly up to 100%. On closer inspection, log files filled up too large and were not being rotated correctly causing the hard drive to balloon up uncontrollably.

Root Cause

There are missing entries in the /etc/newsyslog.conf file that enables cron to rotate specific log files hourly. Some entries were removed in version 13.0 71.40 and up (confirmed 76.29 does not have this fixed) possibly as an oversight. There are two entries missing; httperror-vpn.log and httpaccess-vpn.log. For some, but not all customers, it appears httpaccess-vpn.log fills up the drive, and an immediate symptom is negatively impacting the loading of the management GUI or even Citrix Gateway or AAA-TM pages by presenting blank screens, while SSH to appliance still works.

This bug has been raised to engineering and hopefully will be addressed in a future build.

Special thanks to Marco Schirrmeister at Citrix for his input.

Resolution

To immediately address the matter if encountered, this command will empty the file, or jump down to step 2 for a similar command:

echo "" > /var/log/httpaccess-vpn.log

The longer-term fix involves adding the cron job configs for log rotation back into the newsyslog.conf file, and creating an rc.netscaler entry (modifying it on primary will replicate it to secondary) to add the lines back to that file on reboots, as that conf file will revert back on each reboot. Note that once this is fixed in later firmware releases, I’d recommend removing the rc.netscaler entries and let the code work as intended.

Step 1

Start off by seeing how much disk space is available and trying to determine which log files could be responsible for this.

We can see that the /var/ partition is fully at capacity at 109%. The second find command shows any file larger than 1GB which in this case is the httpaccess-vpn.log file which is of interest for this issue.

df -h && find /var/log/ -size +1G -exec ls -lh {} \;

Step 2

To give the system much-needed breathing room issue this command to clear out the 12GB httpaccess-vpn.log file effectively erasing all the logs inside.

file=/var/log/httpaccess-vpn.log; echo "" > $file && ls -l $file

Step 3

This one-liner can be used to check if the httpaccess-vpn.log and httperror-vpn.log rotate rules exists in /etc/newsyslog.conf already. If it does not, then it will be added otherwise it will inform you that it already exists.

After, it will create the directory /var/custom and copy newsyslog.conf from memory (/etc/) to the hard drive in the newly created /var/custom directory.

The last action would be appending the startup rule in /flash/nsconfig/rc.netscaler to overwrite newsyslog.conf making this change persist reboots.

The picture is colour-coded for easy readability to see the changes that took place.

Note that a reboot is not needed to implement these configs. A reboot would be wise to confirm rc.netscaler inserts the log rotation line entries into newsyslog.conf ideally, but the existence of the entries in the conf file is sufficient to get the logs rotating again on the next interval.

Note: Modify the Citrix ADC config files at your own risk. We strongly recommend backing up the rc.netscaler file and the nsconfig directory prior to making changes (or via Citrix ADM). Ferroque Systems nor its affiliates will be held liable for your use of the commands in this article.

grep -i httpaccess-vpn.log /etc/newsyslog.conf; [[ $? = 1 ]] && echo "/var/log/httpaccess-vpn.log             600  25    100  *     ZB   /var/run/httpd.pid 30" >> /etc/newsyslog.conf || echo "httpaccess-vpn.log already being rotated"; grep -i httperror-vpn.log /etc/newsyslog.conf; [[ $? = 1 ]] && echo "/var/log/httperror-vpn.log              600  25    100  *     ZB   /var/run/httpd.pid 30" >> /etc/newsyslog.conf || echo "httperror-vpn.log already being rotated"; mkdir /var/custom;  cp /etc/newsyslog.conf /var/custom/newsyslog.conf; printf 'cp /var/custom/newsyslog.conf /etc' >> /flash/nsconfig/rc.netscaler;

Step 4

There are two ways to go about testing; either wait for the cron job that runs every hour exactly or for anyone as impatient as me, run the newsyslog command manually to archive every file mentioned in newsyslog.conf.

Michael Wieloch

Michael is a multi-disciplinary consultant with expertise in digital workspaces, networking, security, virtualization, DevOps, and Linux. When not engineering systems or building ZFS clusters, he enjoys cruising the streets on his motorcycle.