Issue and Background
Recently while working with one of our managed services customers, an unexpected error crept into the environment affecting users of a specific forest trusted by the infrastructure (hosting) domain. Immediately upon successfully authenticating to Citrix Gateway and being passed to StoreFront, or authenticating directly TO StoreFront users immediately receive the following message “Your logon has expired. Please log on again to continue.”
No known changes occurred to the Citrix environment including patching of OS or Citrix around the time the issue commenced manifesting itself. No errors in StoreFront logs were noted either, and StoreFront security logs indicated the login was successful for the user.
Environment particulars are as follows:
- Citrix XenApp Site 7.15 LTSR CU5
- Two Citrix Sites (one per data centre), users with issue access resources only from one Site (Site A)
- StoreFront 3.12 LTSR CU5
- StoreFront and VDAs in Domain A
- Users with logon issue in Domain B (two-way trust between domains)
We troubleshot the issue through various means including the following, without improvement:
- CTX204766 (No improvement)
- Adding DNS suffix search list for Domain B on the StoreFront servers
- Rebooting Controllers and StoreFront servers
- Rebooting Domain B’s Domain Controllers
- Confirmed the computer-level security setting “Access this computer from the network” had not been altered (checked via RSOP and gpedit.msc) and locked down to groups that would prevent the login from occurring as outlined in this Reddit post
- Checked GPO modification dates, no changes for months on any related GPOs
- Validated between DCs that trusts were still valid and operational
- Performed tests from the StoreFront servers in Site A where the users connect via test-netconnection PoSH command to confirm all TCP AD ports (other than RPC port checks) were open
After a battery of various tests we worked on a hunch that there may be issues enumerating against the Delivery Controllers being aggregated into StoreFront. Two sets of Controllers were present; one for each Site. This was not immediately suspected as a probable cause as the platform worked fine for most of the year, since the aggregation was implemented up until recently.
As the users of Domain B only access resources in Site A (whereas other users of the platform in Domain A do use resources from both Site A and Site B), we elected to throw in User Farm Mapping as a means to better control enumeration for users of Domain B while leaving users of Domain A unaffected.
Sure enough, once this was propagated, users of Domain B could successfully log in once more.
The root cause appeared to be AD communication issues between the Site B XML brokers and Domain B’s Domain Controllers, suspecting something at the firewall or routing level changed more recently.
In this case, this fix is not deemed a “workaround” as the users do not “need” to enumerate against Site B at present time, and leveraging User Farm Mapping actually helps reduce communication flows to only those that are critical to the user’s needs.
For more details on User Farm Mapping and Multi-Site aggregation, I encourage reading of Sarah Steinhoff’s TechZone article on the subject in addition to Citrix Docs. Amongst other things, in AD environments where this is feasible, using user groups to isolate XML enumeration without using separate Stores can simplify deployments, improve StoreFront login times modestly, and avoid unecessary cross-data centre traffic.