Issue: Citrix Workspace App Authentication Service Could Not Be Contacted

Issue and Background

Recently while working with a customer undergoing a transition from F5 APM to Citrix Gateway for access to Citrix resources with App Protection, we were tasked with replicating their authentication flow off APM. It looked something like this:

EPA Scan > RSA > LDAPS > Group Membership Check

This was working no problem for browser-initiated authentication (which they’d been using for years), but the requirement of App Protection meant authenticating at Citrix Workspace App (CWA) which allows authentication sequence to be protected by the feature. There’s some work being done to find a workable solution for browser access, but the logic as to why CWA is needed is sound.

Some Environment Details:

• Citrix ADC 13.0 b70.44 and b76.29 and corresponding EPA client versions (both had been attempted, the issue may exist with other builds and firmware versions)
• Citrix Virtual Apps and Desktops (CVAD) 1912 LTSR CU2 with App Protection
• Workspace App for Windows 1912 CU2 and CU3, and Workspace App for Windows 2102
• Citrix Gateway with AAA-TM and nFactor, and a RfWebUI-based theme on AAA and Gateway vServers
• Citrix ADC in DMZ sandwiched between two Palo Alto devices (front-end being transparent)

Although we had no issues with the authentication flow on browser-initiated authentication to the Citrix Gateway, we continued to encounter very sporadic issues with Workspace App failing to connect. We’d pass all the nFactor factors successfully, get to select a Store, and post that step CWA would bomb usually with the error “The Authentication Service could not be contacted.”

After quadruple checking the StoreFront configs, we peeled off our security hardening (SSL protocol and cipher hardening, IP reputation, geoblocking, rate-limiting) and could still produce the same issue. App Protection itself wasn’t a culprit here, as all it does in this scenario from the client end is add an additional header (x-citrix-appprotection-capable: true) into the communication in order for apps to launch from protected Delivery Groups. Disabling App Protection on StoreFront and the Delivery Group had no effect on the issue.

Sporadic issues are the bane of any IT professional’s existence. And unfortunately, no logging in the path was giving any real smoking gun clue as to what was going on. StoreFront logs were clear, CDF traces confirmed the client connection was detected as external, ruling out a beaconing issue.

We were seeing this error however in the traces:

Error Throwable created: CProtocolException: Response did not contain the expected cookie: pwcount

Firewall logs did not indicate any NGFW security was interfering with the flow either but did seem to show the client (i.e. CWA) was resetting the connection.

Root Cause and Resolution

In this particular instance, the resolution was very simple, although not immediately obvious.

After tearing down the configs to the barebones basics and layering combinations of configs back on, we isolated the issue to the EPA factor in the flow. EPA scans always passed. But something about EPA was tripping up the CWA authentication and Store integration sequence randomly. In any combination we tried, if EPA was not involved, we’d get 6 for 6 or more successful configurations (with CWA resets in between, on various workstations). Soon as EPA was involved (EPA to RSA then LDAPS, or EPA then LDAPS) we’d start getting our friendly “Authentication Service could not be contacted” error again.

So the root cause here was EPA being the first factor in the authentication flow. Didn’t bother browser-based authentication any, but CWA, another story. Citrix Receiver and Workspace App have in some configurations required special consideration with Citrix Gateway with or without nFactor in the past so additional modification was not an out of the ordinary need.

We introduced a new passthrough factor into the nFactor flow: instead of EPA being the first factor, a NOAUTHN factor was first, followed by EPA. This configuration has the effect of establishing the authentication process without forcing a challenge in the very first factor of the flow which alleviated the intermittent failure of the CWA to Citrix Gateway integration. Any session would still require successfully passing the authentication factors so this is not considered a weakening of security. A connection still needs to pass the various authentication factors to gain access to any authorized resources once properly authenticated.

Using a non-authenticating factor at the beginning or in the middle of an nFactor flow is common, either to overcome specific issues such as in this case (which we would refer to as a passthrough factor) or to alter authentication flows (dubbed a selector factor) based on some piece of data such as host header, URL, client IP, etc.

Making the first factor a passthrough factor in this manner remedied the issue after continual testing, and after layering back on the security configurations including Bot Management (a super cool new 13.0 feature that also works on Citrix Gateway and AAA-TM), we were able to deem it stable and in compliance with requirements.

I suspect this resolution will likely be applicable to other scenarios involving Citrix Workspace App authentication at Citrix Gateway when using nFactor (which everyone should be using at this point). I want to once again send a shout-out to our ADC Hero at Citrix, Rene Gamache for the NOAUTHN suggestion.