Issue and Background
Yesterday, Citrix released a security bulletin outlining several “Low” and “Medium” risk vulnerabilities as covered in CTX281474 and outlines the firmware versions available for download which mitigate those vulnerabilities.
Included in the Citrix ADC 13.0 build 64.35 release are additional security enhancements to SSO to block by default weaker authentication methods for SSO. The supporting documentation located here, outlines basic authentication, digest authentication, and NTLM without the Negotiate NTLM2 key or negotiate sign as the affected “weaker” methods impacted, whilst other methods such as SAML, Kerberos, OAuth and form-based authentication are supposedly unaffected. As of September 18th, 2020 at least, those documented statements may not be entirely accurate.
Testing reveals that SSO functionality is broken when attempting to access Citrix StoreFront through Citrix Gateway and an error of “Cannot complete your request” appears when using SAML. The Citrix Gateway configuration tested against is purely SAML so this came as a surprise, albeit it is possibly an interaction issue with IIS. Citrix has noted this behaviour in CTX281592 and points back to the Citrix Docs on the subject of SSO enhancements. This doc does cover how to override the default security measures in the new firmware build using traffic policies bound to the Citrix Gateway vServer.
It is possible that this security enhancement may make its way into other builds such as 12.x so this guidance may become valid for more than 13.0 in the future.
In this article, we’ll walk through how to quickly fix the issue and restore a Citrix Gateway to service.
SSH into the ADC appliance and enter into shell mode.
The syslog message generated we are looking for is “SSO FAIL forwarding to client because of weak SSO user email@example.com”. To quicken this process grep through the cleartext ns.log with:
grep -n "SSO FAIL" /var/log/ns.log*
If this does not produce any hits, the logs might have been rotated in which you can use the zgrep command
zgrep -n "SSO FAIL" /var/log/ns.log*
The output shows the filename : line_number : content of the line.
We have 3 matches from my failed attempts so to investigate further, we’ll follow line number 72, 2054 and 2074 of the file ns.log.11. The cat command will be used or alternatively the zcat for gzip compressed logs. Vi can also be used as an alternative.
cat /var/log/ns.log.11 | less -N +72
(type q to quit after investigating)
Now that we have confirmation SSO is the cause of this issue, type exit to enter the ADC command line interface. A new vpn policy will need to be attached so let’s find out the name of the currently added vpn vservers.
show ns runningConfig | grep "add vpn vserver"
A new traffic action and policy will need to be added. Copy and paste the following 3 commands changing <vpn_server_name> with the vserver name found in the command above.
add vpn trafficAction sso_vpn_traffic_action http -SSO ON add vpn trafficPolicy sso_vpn_traffic_policy TRUE sso_vpn_traffic_action bind vpn vserver <vpn_vserver_name> -policy sso_vpn_traffic_policy -priority 100
At this point, re-test authentication to Citrix Gateway and confirm SSO passthrough works. Be sure to save your config!
Michael Wieloch is part of Ferroque’s technology services team with a seasoned background in datacentre operations. Michael is an avid Linux enthusiast and specializes in networking, security, virtualization, and operational monitoring & alerting integrations. When not thwarting cyberattacks, engineering hypervisors, and building ZFS clusters, Michael can be found cruising the streets on his motorcycle.
Great article Michael,
Does anyone know the security implications of this work-around? I see we are potentially re-enabling some of the SSO methods that Citrix deemed insecure.
You have a type in your code snip for step 3. The second line does not reference the correct name of the traffic action configured in line 1. The second line should instead read:
Hi Jeff, good catch. We had modified a couple names when publishing and must have missed one. Fixed.
Great article Michael,
This would work perfectly fine in most cases, but will break SSL VPN in case you have one enabled. Specifically, TCP services such as RDP, SMB, etc. (HTTP/HTTPs should be fine).
I’d suggest you use a different expression instead of just “TRUE” so it doesn’t affect the TCP services:
add vpn trafficPolicy sso_vpn_traffic_policy “HTTP.REQ.METHOD.EQ(post) || HTTP.REQ.METHOD.EQ(get) && false” sso_vpn_traffic_action
Thanks a lot!! I started having the “week SSO user” when connecting to XenMobile after upgrading Netscaler from 12.0 to 12.1. It didn’t have any traffic policiy before but adding them solved it