Reference: Citrix Product Log Files Locations

Introduction

Often as Citrix Engineers and Administrators, we troubleshoot issues, and that involves inspecting log files. Recently, I had a request from one of our customers to provide them with log file locations of all the Citrix products they use, and surprisingly I found that there was no centralized repository for log file locations. With that being said, I have written this article, which attempts to cover log file locations of each Citrix product. It serves as a single point of reference for log file locations for various Citrix products and stands to be a living reference and will require updates over time. We will do our best to maintain this article on an ongoing basis and community feedback is always welcome.

Citrix Virtualization

Citrix Licensing

The following table contains important log file locations for Citrix licensing Server:

Citrix Delivery Controller (CVAD aka On-Prem Only)

The following table contains important log file locations for Citrix Desktop Delivery Controller:

For more information on Windows event log messages for Citrix Virtual Apps & Desktops, please refer to this article.

Citrix Cloud Connector (CVADS aka Citrix Cloud Only)

The following table contains log file location for Citrix Cloud Connector:

Citrix Workspace Environment Manager

The following table contains important log file locations for Citrix Workspace Environment Manager (WEM) Agent & Infrastructure components:

Citrix StoreFront

The following table contains important log file locations for Citrix StoreFront:

Note: In addition to verbose tracing on the StoreFront, Fiddler trace can be particularly useful when troubleshooting network issues between StoreFront services and Receiver for Web. For this, Fiddler trace should be enabled in StoreFront (Advanced settings), loopback must also be disabled. The confidentiality and integrity of outbound SSL connections which are proxied through fiddler maybe compromised during traffic capture. More information on how to perform the trace can be found here.

Citrix Workspace App

The following table contains important log file locations for Citrix Workspace App (Previously known as Citrix Receiver):

Citrix Profile Management

The following table contains important log file locations for Citrix Profile Management (CPM):

Note: %SystemRoot%\System32\Logfiles\UserProfileManager is the default location for the path to save the log file, this can be changed to a UNC path. Citrix Virtual Desktops Machine Creation Services uses a local persistent folder which is mapped to the C drive (C:\Program Files\Citrix\PVS\Service\PersistedData).For non-persistent instances (PVS, MCS), Citrix recommends using a centralized file share to store log file(NTFS and SMB share permissions should be set for domain computers for read/write) or redirecting log files to a persistent drive attached to the VM so logs can be retained between reboots (this is commonly recommended for VDA event logs as well). Citrix UPM Log Parser can be leveraged to analyze the log files generated by Citrix Profile Management. Its installation and usage can be found here.

Citrix Virtual Delivery Agent

The following table contains important log file locations for Citrix Virtual Delivery Agent (VDA):

Citrix Provisioning Services

The following table contains log file locations for Citrix Provisioning Services (PVS):

Citrix Director

The following table contains log locations for Citrix Director:

Citrix App Layering

The following table contains log locations for Citrix App Layering:

Citrix Federated Authentication Services

The following table contains log locations for Citrix Federated Authentication Services (FAS):

Citrix Hypervisor (XenServer)

The following table contains log locations for Citrix Hypervisor:

Citrix Networking

Citrix ADC (NetScaler)

The following table contains the important log file and dump file locations for Citrix ADC:

Citrix ADC (NetScaler) – SDX

The following table contains log locations for Citrix ADC SDX – SVM and Citrix Hypervisor (previously XenServer):

Citrix ADM (MAS)

The following table contains the important log file locations for Citrix Application Delivery Management (ADM):

Citrix SD-WAN

The following table contains log locations for Citrix SD-WAN:

Citrix Mobility

Citrix Content Collaboration (ShareFile)

The following table contains log file locations for Citrix Content Collaboration (previously ShareFile) Applications:

Citrix Endpoint Management (XenMobile)

The following table contains log file information for Citrix Endpoint Management (previously XenMobile):

===Supplemental Log Guidance===

ADC Troubleshooting

Citrix ADC provides several diagnostic tools to view console messages, event messages, and download traces.  Most of these utilities are available directly within the Citrix ADC GUI. This section focuses on command line diagnostic tools which can be leveraged in troubleshooting the Citrix ADC.

Command Line Diagnostics

nstrace.sh:  nstrace.sh is a Citrix ADC utility (script file) that allows network administrators to take ADC traces from the appliance.  The Citrix ADC trace captures all traffic going through the ADC appliance at any given time. The syntax for nstrace.sh is /ADC/nstrace.sh –sz0 –tcpdump 1.

This syntax will automatically create a trace file in the /var/nstrace directory. The administrator can press Ctrl-C on the keyboard to stop the trace. This trace can then be downloaded to a local host (via PSCP, WinSCP) and viewed on any packet capture program (i.e. Wireshark).

nstcpdump.sh: The /ADC/nstcpdump.sh script is a utility that emulates tcpdump syntax on ADC interfaces.  The main benefit of using nstcpdump.sh includes its filtering ability. Below is an example:

/ADC/nstcpdump.sh –w /var/nstrace/ftp.pcap host 192.168.1.1 and host 192.168.1.2 and tcp.port==21

The above filter allows for all FTP traffic between 192.168.1.1 and 192.168.1.2 to be captured on the Citrix ADC and downloaded to /var/nstrace/ftp.pcap.  This file can be downloaded to a local machine and viewed on Wireshark for easier analysis.

Local SYSLOG:  ADC stores all log files locally on the appliance under /var/log/ns.log. Details regarding authentication errors or ADC can be viewed by running the following command:

>shell

# cat /var/log/ns.log

tail –f /var/log/ns.log

Authentication Debugging Tools:  ADC provides a debugging utility to check for authentication successes and failures. Group extraction can also be validated using this utility. User details including group extraction can be viewed on screen by typing the following command, while a user is logging on to the VPN:

>shell

# cat /tmp/aaad.debug

The output of the command will provide helpful details on the authentication scheme used, success, failures, and cause of failures (i.e. incorrect password, bad bindings etc.).

Console Message Diagnostics:  The Citrix ADC provides useful console messages that can shed light on ADC performance. For example, using the following command below can easily identify IP address conflicts and duplex mismatches:

> shell

# nsconmsg -K newnslog -d consmsg (live)

or

# nsconmsg -K newnslog –K /var/nslog/newnslog -d consmsg (from latest file)

Event Message Diagnostics:  The Citrix ADC provides useful event messages that can provide insight on the status of configured ADC services and high availability. For example, the status of a specific configured service or notification of a failover or a reboot can be identified by running the command below:

> shell

# nsconmsg -K newnslog -d event (live)

or

# nsconmsg -K newnslog –K /var/nslog/newnslog -d event (from latest file)

 EPA Logging

By default, EPA plugin as well as VPN plugin does not log anything related to client machine for security reasons. From Citrix ADC version 11.0.64.34 onward, Citrix has introduced “EPA verbose logging” to enhance EPA troubleshooting. This setting is enabled on the Citrix ADC globally, and new EPA scan attempts will result in logs on the client. It is recommended once troubleshooting or pilot have concluded, to remove this logging setting from the Citrix ADC, so bad actors cannot gain insight via client logs, as to what the EPA scan is looking for.

The EPA scan failure logs are written in human readable language, so troubleshooting can be done on the user end without involving the administrator.

The following can be achieved using this EPA feature:

• Provide verbose logging of which EPA scans passed/failed on the ADC
• Display human readable logs for EPA scan failure on the client machine

Through CLI the following command can be run on the ADC for PreAuth and PostAuth EPA logging:

>set vpn param -clientSecurityLog ON

Note: vpn param must be set for PreAuth and PostAuth logging. If the clientSecurityLog is modified in a SessionAction whose Session Policy has ClientSecurity as the rule, the clientSecurityLog value in SessionAction will not be honored. All these settings must be configured at a global level under Citrix Gateway.

For Windows Vista, 7, 8, 8.1, and 10 the log file location is:

C:\Users\<username>\AppData\Local\Citrix\AGEE

For Windows XP:

C:\Documents and Settings\All Users\Application Data\Citrix\AGEE

For Mac OS:

~/Library/Application Support/Citrix/EPAPlugin/

All failed EPA scans are logged as error messages and successful scans are logged as debug messages. By default, error messages are logged in ns.log and to log debug messages in ns.log the loglevel needs to be increased to DEBUG. This can be done via the command line using the following command:

>set audit syslogParams -loglevel ALL

Note: It is recommended to remove debug logging once troubleshooting has concluded.

Debugging can be disabled using the following command:

>set audit syslogParams -loglevel EMERGENCY ALERT CRITICAL ERROR WARNING NOTICE INFORMATIONAL

The same process can be done via the Graphical user interface by:

• Navigating to Configuration > System > Auditing > Settings > Change Auditing Syslog Settings
• Check DEBUG under Log levels during troubleshooting
• Uncheck DEBUG from Log Levels after troubleshooting is complete

StoreFront Troubleshooting

Fiddler Trace

In addition to verbose tracing on the StoreFront, Fiddler trace can be particularly useful when troubleshooting network issues between StoreFront services and Receiver for Web. For this, Fiddler trace should be enabled in StoreFront (Advanced settings), loopback must also be disabled. The confidentiality and integrity of outbound SSL connections which are proxied through fiddler maybe compromised during traffic capture The procedure of obtaining a fiddler trace must be performed only in a non-production environment and It should be noted that in case explicit authentication is enabled in Receiver for Web, user passwords will appear in the trace as clear text.

To obtain a fiddler trace of the network traffic between Receiver for Web Proxy and the StoreFront services, carry out the following steps:

• Log in to the StoreFront server as local user with admin privileges
• Edit the web.config file for Receiver for Web and enable fiddler tracing: <proxy enabled = “true” processName=”Fiddler” port=”8888” />
• The web.config file is usually located in C:\inetpub\wwwroot\Citrix\StoreWeb\web.config
• Run IIS Manager and click on Application pools under the server node. Then select the application pool named Citrix Receiver for Web and click Advanced Settings. Change the Application Pool identity to custom account and specify the same account used to log into the StoreFront server
• Install and run Fiddler on StoreFront Server (logged in with elevated privileges)
• From Fiddler’s Tools menu, select Fiddler Options. On the HTTPS tab, select Decrypt HTTPS traffic check box.
• Run Fiddler on the StoreFront server and export all Fiddler sessions, after reproducing the problem
• After the trace has been captured, log in to the StoreFront server with the same local admin account used while installing Fiddler and follow steps to restore system to its previous state
• Run IIS Manager and reset identity for Citrix Receiver for Web application pool to the built-in account ApplicationPoolIdentity
• Uninstall Fiddler
• Run certmgr.msc and remove the Fiddler root certificate “DO_NOT_TRUST_FiddlerRoot” from the Trusted Root Certification Authorities store
• Edit the web.config file for the Receiver for Web site and disable Fiddler tracing: <proxy enabled = “false” processName=”Fiddler” port=”8888” />

Citrix Profile Management Log Parser

Citrix Profile Management log parser can be used to analyze log files generated by Citrix Profile Management.  This section describes how to use the tool to analyze logs.

The zip file used to install the tool can be found here.

To use the tool, perform the following steps:

• Launch the tool and enter the name of the remote machine on which the log parser will try to locate the log file
• On the Open File dialog box, specify the location of the log file
• If the machine can be reached, the tool tries to locate the UserProfileManager.log file in <systemroot>\system32\LogFiles\UserProfileManager.
• If the file is not foun, the tool looks into HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\UserProfileManager\PathToLogFile to retrieve the log file location
• If no machine name is specified, the local machine is used instead to prompt for a log file
• The log parser displays the first error found in the log file, the errors are highlighted in red, the logon events appear in green and the logoff events in blue. Service starting and stopping events appear in grey.
• Find button can be used to search text, filter information, warning and error messages.
• Filtering based on Date, Time, Type, Username, Domain, Session ID or Thread ID is also possible. “Reset filter” option will clear all previously selected filters.
• Filtering based on time span between two lines is also possible, this allows for setting custom time span (above 60 seconds) or use the preconfigured time spans of 5, 10 o 30 seconds. The display will be updated to show the time span selected.

Enabling Audit Log within Citrix Provisioning Services

Citrix Provisioning provides an auditing tool that records configuration on components within provisioning farms. The auditing tools saves the information to the provisioning database. It provides Citrix Engineers and Administrators with a way to troubleshoot and monitor any recent changes impacting system behavior or performance. Enabling audit logging is a Citrix leading practice on PVS deployments.

To enable auditing within Citrix Provisioning Services, perform the following steps:

• In the Citrix Provisioning console, right click on the PVS farm and select Farm Properties option.
• On the Options tab, under auditing, check the Enable auditing checkbox

The following managed objects within a Citrix Provisioning deployment are audited:

• Farm
• Site
• Provisioning servers
• Collection
• Device
• Store
• vDisks

Recorded tasks include the following:

• Citrix Provisioning console
• MCLI
• SOAP Server
• PowerShell

CDFControl

CDFControl is an event tracing tool geared towards capturing Citrix Diagnostic Facility (CDF) trace messages from various Citrix tracing providers.  This tool is particularly useful when troubleshooting issues on specific components of the Citrix environment, such as on the delivery controllers, storefronts, or VDAs. The trace needs to be captured on the components and uploaded to Citrix Insight (https://cis.citrix.com) to be analyzed. Citrix Insight has smart capabilities to identify any discrepancies in configuration and suggests fixes as well.

This utility is available for download on the Citrix Downloads page. It is also available with the download of Citrix Scout utility, which is also equally useful for collecting trace files and troubleshooting various Citrix component issues.

The following steps need to be performed to collect a trace using CDFControl at system startup:

• Start CDFControl and select Options from the Tools menu.
• Specify the trace file path in the Startup trace file path for capturing startup trace section. Then click Save.
• Select the Trace Categories as recommended by Citrix Technical Support.
• With administrator privileges, select Startup Tracing and click Enable from the Tools menu.
• After clicking Enable, the animated bar starts scrolling. This does not affect the procedure.
• Close the CDFControl utility and restart the system after the Startup Tracing is enabled.
• Start the CDFControl utility. After the system restarts and the error appears, disable the Startup Tracing option by selecting Disable.
• Disable the Startup Tracing option by selecting Startup Tracing from the Tools menu and clicking Disable as described in the previous steps.
• Stop the Citrix Diagnostics Facility COM server service.
• Collect the trace log file (.etl) for analysis in the specified file path as set by the initial steps.
• Start the Citrix Diagnostics Facility COM server service.
• Upload trace log file(s) to Citrix Insight (https://cis.citrix.com) to be analyzed

How to Collect Diagnostic Data on Citrix SD-WAN Appliance

While troubleshooting issues relating to Citrix SD-WAN, engineers or administrators must often collect diagnostic data on the Citrix SD-WAN appliance. This section covers the steps required to collect Diagnostic data on the Citrix SD-WAN appliance and how to download it locally for viewing.

To collect diagnostic data, follow the steps listed below:

• Logon to Citrix SD-WAN UI and navigate to Configuration> System Maintenance> Diagnostics> Diagnostic Data
• Scroll-down and click Create New
• Only 5 diagnostic packages ca exist on the system at any given time, delete any unwanted packages
• Select the file from drop down and click Download Selected
• To analyze logs, uncompress the vw_sts_dir.zip file and refer the table to look for relevant log files

How to Configure Logging on Citrix Endpoint Management

This section describes the steps required to configure logging on Citrix Endpoint Management.

Perform the following steps to configure logging on Citrix Endpoint Management:

• In the XenMobile console, click the wrench icon to open the Support page
• Under Log Operations, click the Log Settings option.
• Within Log Settings, the following options can be specified:
• Log Size: Controls the size of the log file and the maximum log file backups retained in the database.
• Log level: This option is used to change log levels (like Fatal,Error,Warning,Info) or persist settings (log levels persist after reboot)
• Custom Logger: This option allows administrators to setup custom logging; custom logs require a class name and a log level.