Introduction and Background

Recently, we were engaged by a customer to architect and assist with POC deployment for a multi-tenant CVAD and ADC platform for a large multi-national software organization. For those that know our areas of specialization, we love multi-tenant Citrix platforms. It throws such a wrench into conventional architectural planning and extends well beyond Citrix into AD architecture and into areas of security and controls outside the norm. We’ve architected, deployed, and migrated various multi-tenant Citrix platforms over the years, and assessed such platforms for SMB-focused CSP customers through to billion-dollar healthcare software companies.

As part of a platform modernization initiative, this particular customer wanted to minimize the administrative burden of managing customer user accounts in their hosting environment. To achieve this, Citrix FAS was included in the architecture, to allow each customer to bring their own IDP to the Citrix platform. Yes, this requires shadow accounts of course, but once provisioned these accounts don’t require the level of maintenance and support as compared to managed user accounts in the hosting environment. Customer users get to use a native authentication platform, have one less credential to remember, and the hosting provider drastically reduces management overhead. Now, such a platform does require more moving parts (FAS, Microsoft CA infra), additional onboarding tasks (SAML SP setup, shadow accounts, UPN suffixes, etc.) but most would agree the benefits ultimately outweigh these added efforts. And FAS’s use of virtual smart cards (configured for in-session use) provides the additional potential to provide Windows authentication to applications within the user sessions, to avoid users potentially needing to know the shadow account credentials (which would ultimately bring one back to square one).

FAS has several configurations and dependencies for a typical single domain deployment, however FAS becomes further complicated by the customer’s desire to split out functional components of their revised platform to employ a user forest, and an infrastructure forest. This is largely due to the dependency on Microsoft CA infrastructure thus the complexity that comes along with multi-forest certificate issuance. Two-way trusts are a must in this setup not just for FAS, but for RDS License CAL issuance as well I should note; one-way trusts are a non-starter. Citrix does have a blog that covers the ins and outs of this setup thankfully (which also covers integrating into selective auth trusts, which we didn’t need), and hats off to Roger for writing it up in such detail. There are lots of opportunities for error in multi-forest FAS setup on account of the PKI infra, but following the guidance and a fair bit of patience, and you’ll get there. And once it’s online, set it and forget it (well, aside from tracking your certificate expiry for domain controller certificates and the FAS authorization certificates). That’s the one thing I can say for FAS in my experience at least, once it’s configured and functional, it’s quite solid and rarely needs much attention.

Issue

As mentioned, especially in multi-forest setups, there are additional configurations required for the Microsoft CA infrastructure, and more opportunity for kinks to present themselves to be worked out. Before getting into the issue further, I’ll list out the particulars of the platform:

Multi-forest (user forest, infra forest) with two-way transitive trust in-place
Domain Controllers of both forests running Windows 2012 R2
All other infrastructure (including Citrix VDAs) running on Windows Server 2016
Two-tier Microsoft Enterprise PKI infrastructure with multiple issuing CAs
Citrix components operating on 1912 LTSR CU1
Multiple firewalls between components (for our CAs this meant locking down DCOM port to something static)
FAS and Microsoft CA infrastructure residing in same domain (required)

Citrix has done quite a good job of documenting configuration and troubleshooting for FAS, and although this particular issue was not documented, the various error codes and symptoms from other articles and blogs were useful in helping pinpoint (or eliminate) focus areas.

In our case, despite having FAS configured correctly per Citrix documentation (both Citrix Docs and the multi-forest blog references earlier), we continued to run into “failed to launch xxxx” errors related to FAS in StoreFront’s event viewer (error code 28) when launching applications. Note that this event covers various error conditions, and in our case it was an “unknown error”. Fun!

This was also accompanied by event ID 1, corresponding to an “access denied” response received from FAS.

This generally means we need to check out FAS’s event logs to get the real scoop on what’s going on. And we found correlated errors to the above event, in form of event\error code 102; “Exception: The encryption type requested is not supported by the KDC”.

Resolution

We spent some time combing through the environment to determine if Kerberos had been hardened in some fashion as to create such an error. No group policies within the environment were found to be hardening Kerberos and looked default. We did not take the time to investigate the encryption configurations on the servers as the customer felt these too had not been hardened, but it would have been a logical step as we were clearly dealing with an anomalous situation.

Ultimately we turned our attention after input from a colleague within Citrix and customer, to hard setting the permitted Kerberos encryption types in the infrastructure domain. We did so via GPO in “Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Configure encryption types allowed for Kerberos”.

We explicitly set all encryption types other than DES, ensured this applied to all components, ran GPUPDATE /FORCE on the infrastructure domain’s Domain Controllers, VDAs, StoreFront, CAs, and FAS servers, restarted KDC service on both user and infra Domain Controllers for good measure. Our intent was to circle back and harden further to just AES and confirm we were still operational. This modified KDC configuration was not required in the user domain.

After running through those steps, we could both log directly into StoreFront and launch resources (as any Store enabled for FAS will make use of the FAS configurations), as well as via Citrix Gateway using SAML successfully, and without further errors.

This is likely an obscure issue not common to most environments (even in multi-forest setups), but do hope it saves the odd person a few hours of troubleshooting in the future.

Subscribe