HowTo: Front Citrix Gateway with Citrix ADC AppFW (WAF)

Notice: This article can be considered obsolete with the introduction of WAF on Citrix/NetScaler Gateway on the latest builds of 13.1 and 14.1 firmware. It is being left in place for posterity. We encourage customers to use the native WAF protection for AAA / Gateway vServers as per the documentation found here.

Introduction

Web Application Firewalls (WAF or AppFW) are big business in today’s increasingly security-conscious landscape. Going well above the call of traditional firewalls or next-generation firewalls (NGFWs), WAFs are designed to well… as the name would imply… protect web apps from threats. They’re designed to operate at L7 of the OSI model and protect applications from attack vectors that standard firewalls cannot as they often lack insight into the architecture of web apps themselves. The focus on web application security for organizations large and small is only going to increase as time goes on, and there are many capable vendors providing on-prem and cloud-hosted WAF solutions.

One of the advantages of Premium Citrix ADC licensing is the inclusion of Citrix Web Application Firewall which provides a convenient means of implementing WAF directly on the appliance fronting web apps, which can yield improved performance vs. external WAF solutions. Security Insight (part of Citrix ADM) provides customers visibility into security analytics telemetry coming from Citrix AppFW-protected web apps, further increasing the value proposition within the Citrix Application Delivery and Security ecosystem.

The intent of this post isn’t to delve into the workings of Citrix AppFW too much, but specifically on how we can use the in-built WAF functionality to front Citrix Gateway. In the field, the general consensus is WAF on Citrix Gateway is a bit pointless, as the majority of Gateway’s use is ICA proxy traffic. The web portions of Citrix Gateway are routinely hardened between builds as necessary and in general, is pretty tightly coded. With that said, we’re all painfully aware of vulnerabilities affecting Citrix Gateway in recent history, and some organizations have sought to take things a step further and throw WAFs in front of Citrix Gateway to help mitigate potential attacks.

This has proven somewhat challenging for Citrix Gateway, as Citrix Gateway really doesn’t like anything else terminating the client’s TLS session other than itself. And a good WAF typically needs to be able to see what’s going on within the encrypted flow to be of any real use. It is possible to get Citrix Gateway successfully fronted by external solutions such as Imperva, but it’s also quite easy to get going with Citrix AppFW right on the appliance itself. Doing so overcomes the common challenges of an external device decrypting and re-encrypting the Citrix ICA traffic to apply WAF to protect the web traffic portion of Citrix Gateway as it’s all happening on the same device (the Citrix ADC). It’s not something most consultants are likely to implement by default when deploying a greenfield build, but we’ve certainly been asked to do so (usually to appease a customer’s security team with broad stoke “WAFs on all web apps” mandates), and certainly have done so with success.

As with any WAF implementation sequence, test, test, and test again. It’s important to test all possible functions of a web app when using a WAF to ensure nothing is impacted. For Citrix, this may include all possible login sequences, browser login vs. Citrix Workspace App, working through the start-to-finish setup of Workspace App (adding Store, launching apps) on different devices and interfacing with all possible functions of StoreFront. All of this is easily done in 10-15 minutes but it’s important to highlight it as you kick the tires of your WAF configuration (once it’s bound and you’re monitoring the ADC logs for violations).

Citrix AppFW Overview

This won’t be all that detailed of an overview but I do want to cover some basics. Citrix’s WAF solution is divided up into two key focus areas:

• Known Attacks. Derived from signatures for known threats across 13 or more categories that are updated periodically from a SNORT threat database. This database can be manually or automatically updated on Citrix ADC (but does require the ADC to have Internet access to Amazon S3 (https://s3.amazonaws.com) to pull down said signature updates). Administrators create a signature and bind them to an AppFW profile. You can create numerous signature profiles each with different category configurations (and granularly enable or disable signature IDs if they cause issues with an app) to optimize performance and security controls for different web apps.
• Unknown Attacks. These are a set of security checks (the number varies by type of web app configuration chosen when creating the AppFW profile, such as Web 2.0, etc.) that allow administrators to configure deep learning of the web app’s behaviours in order for the WAF to be “trained” on what normal operating patterns are for the application being protected via a “learning” mode, which administrators can then use to create relaxation rules against and then subsequently enable blocking of unrecognized patterns. It is these security checks that allow customers to help thwart zero-day attacks on their web apps.

Citrix AppFW is comprised of three key components we configure:

• Signature Profile. Contains the “known attacks” database in which administrators can enable or disable entire categories of protections, or individual protection rule IDs (we’ll touch on that later as it relates to Citrix Gateway, as there are a few tunings we need to do).
• AppFW Profile. Here we configure our security checks for unknown attacks including enabling and disabling of learning mode, blocking of unrecognized access patterns to the web apps, working with relaxation rules (patterns we’ve learned and deployed as accepted behaviours), and tuning baseline profile configurations, such as selecting an associated signature profile to be used with the AppFW profile.
• AppFW Policy. Here we select our AppFW profile and define an expression to be used for the WAF to act on. This is important especially for our use case here, as we can’t bind WAF policies to Citrix Gateway at this time (not entirely surprising given the general consensus on the utility of WAF on Citrix Gateway), so we will add an expression that affects Citrix Gateway only and bind the WAF policy globally, so as not to interfere with other web apps or needlessly inspect Citrix ADC admin GUI traffic.

As with any WAF, it is very important to know the application you’re trying to protect, or have access to people who do. Throwing all known protections and deep protections for unknown threats for the sake of doing so will not only cause you a world of pain but sucks up the horsepower on the appliance, which could otherwise be serving more important traffic vs. scanning traffic with protections that have absolutely nothing to do with the type of web app being used (i.e. if you’re fronting an IIS web app, you’re probably not going to need Drupal or WordPress protections).

Tip 1: By accessing websites when WAF is in “learn mode” and engaging testers to test every possible aspect and function of the web application, WAF is trained to understand the expected behaviours of the apps. When “block mode” is enabled once learning is complete, unrecognized behaviours are blocked. As updates are made to applications (code, etc.) administrators can relax or learn new behaviours and whitelist them as well. Administrators can also set designated client IPs to “trust” for learning.

Tip 2: As with any AppFWWAF solution, they are not a “set it and forget it” solution. Changes to web apps, updates to known protections signatures, changes to authentications, changes to authentication flows fronting the web app (ex. nFactor and AAA-TM), or the type of clients interfacing with the web app (e.g. browsers vs. say Citrix Workspace App) could all potentially negatively impact the availability or functionality of the web app being protected. Thus, as such changes come about throughout the platform and web app lifecycle, it is important to test the changes in advance, so any new impacts can be mitigated in the production WAF profile and signatures to avoid impacting production users.

Did I really just write all of this?!? 1,300 words in and I’m just on the intro. No wonder my colleagues get anxious when I submit design documents for peer review 😉

Now, on to the good stuff…

Step 1 – Create Enable Citrix Web Application Firewall

Probably the easiest step; on your Premium-licensed ADC, navigate to System > Settings > Configure Advanced Features, and check (enable) the Citrix Web App Firewall box.

Step 2 – Create AppFW Signature Profile

Navigate to Security > Citrix Web App Firewall > Signatures

Next, we’ll create our custom signature profile. I like to duplicate the in-built default profile out of habit and go from there. Before doing so, I like to click (enable) the checkbox beside “*Default Signatures” and hit the “Update Version” button, to ensure the profile starts off being created based off the latest signature database version available. Once done, click “Add” with the default signatures profile still selected.

Provide a name for the profile. By default, none of the known protections will be enabled as noted by the X in the “ENABLED” column. Here we will toggle the protections we want to use for Citrix Gateway, as many here do not apply, and as mentioned above, enabling them will just waste processing cycles. On the left-hand side of the page, toggle hiding the categories until only these remain:

• web-misc
• web-cgi
• web-php
• web-client
• web-activex
• web-struts
• HTTP.sys
• web-shell-shock

With the irrelevant protection rules now hidden, from the “Select Action” menu, select “Enable All” and “Yes” when prompted to confirm.

Now, we need to disable some protection rules (IDs) from our signature profile, as they do cause issues when interfacing with Citrix Gateway. In the “Click here to search…” field at the top of the columns, select “ID” and then individually search and then disable (from the “Select Action” menu once the rule is checked off in the left-most column) the following rules:

• 1048
• 1147
• 2067
• 20528
• 20616
• 20620
• 999945
• 999960
• 17279 (Noted to block HTML5 client circa signature version 117 and firmware 13.1 v51.15)

If you then modify your search filter to the parameter “Enabled” with value “OFF”, your list should appear like so:

Click “OK” at the bottom of the page, and voila, you have your WAF profile.

Step 3 – Create AppFW Profile

Navigate to Security > Citrix Web App Firewall > Profiles

Now, we’ll configure a baseline profile that will include our security checks and bind our signature profile to the AppFW profile. Click the “Add” button (don’t have any existing in-built profile selected) and give the profile a name, select Web Application and XML Application options, and Advanced defaults.

On the right-hand menu, select “Profile Settings” and scroll to the bottom. Under “Bound Signatures”, select the new signature profile we created in the prior step.

Now, on the right-hand menu select “Security Checks” and check (enable) the various checkboxes as shown below.

Now, you may ask yourself why this looks so light. And that is because we’re looking to start with a baseline, and some are not considered relevant to the Citrix Gateway as a web app. You can absolutely enable additional security checks in learn mode and tune them accordingly, but some may not be worth the trouble to maintain. Start URL for example can be very tedious to tune and get working let alone keep working due to the nature of how Citrix Gateway generates URIs for example. Others, such as form field consistency or field formats, may not be relevant if ADC is not intaking such data (say for example if using only SAML for authentication).

With that said, the following have been observed as breaking the Citrix Gateway either for both browser and Workspace App access, or Workspace App Access alone:

• Cookie Hijacking (even with just Log and Stats enabled seemed to break the app strangely, so be mindful of that kind of glitch when working through security checks).
• Content-type.
• Web Services Interoperability.
• XML Message Validation (some intermittent authentication issues with Citrix clients noted).
• XML Format (believe it or not, observed violations when enabled).

Note that in this configuration, I intentionally have skipped over throwing these checks into learn mode prior to enabling blocking. That is because most, if not all, won’t have anything to learn. If you run into problems, definitely enable learn on any checks you can, re-run your tests, and after selecting “Learned Rules” on the right side of the page, select the security check and then “Edit” to see the learned rules to deploy.

Also, once the WAF policy is created and bound to the ADC in the next step, you can test and then return to the AppFW profile, navigate to “Security Checks” section and you can select a check, and click the “Log” button to see any blocks (can have a bit of a lag, be patient). Usually, we’d do this if encountering an error with an app, and we go through the security checks one-by-one and check the logs to see if there are any violations.

At this point, I’m going to jump ahead for a minute (in order for us to tune an item in the profile before we save and create it). To illustrate the above testing process, the configuration described above was saved and bound to a profile that was put into service, resulting in some errors with Workspace App connecting successfully to a Gateway with a customer. As you can see, this was with the buffer overflow check. It’s complaining that the threshold for header length is beyond the configured default maximum of 4096.

This is a check that does not have a learnable function, but we can edit the check using the “Action Settings” function and modify that parameter. In this case for the example, I selected a bit liberal value of 6144 chars.

With this out of the way, you can select the “Save & Close” button for the “Security Check” section, and then “Done” to complete the creation of the profile.

If required to do so, after the WAF configuration is in place, feel free to come back and tune the profile if you wish to tinker more; just be sure to thoroughly test after each configuration adjustment, learn the behaviours (if applicable), and with blocking enabled test again and again. But please do so only if necessary, and only after this initial (and relatively secure) config has been duly tested and verified to be successful.

Step 4 – Create AppFW Policy

Navigate to Security > Citrix Web App Firewall > Policies > Firewall

Now to tie things all together, create a policy and select the profile we just created, and define an expression. We’re using HTTP.REQ.IS_VALID so ICA traffic is ignored, and in this demo, we’re going to filter based upon the domain name (probably could be more granular for the FQDN, but this is just an example). If you’re worried about using merely host headers, you could modify this in other ways, such as IP etc., and just confirm once bound that the policy is getting hits. Again, we’re creating this expression as we’ll be binding this WAF policy globally and only want it to affect certain traffic.

HTTP.REQ.IS_VALID && HTTP.REQ.HOSTNAME.CONTAINS("<YOUR_FQDN_HERE>")

Click “OK” to complete the creation of the policy.

Step 5 – Bind AppFW Policy Globally

Back on the policy page, click the “Policy Manager” button and select “Default Global” and click “Continue”.

Click “Add Binding” and select the Citrix Gateway WAF policy.

Click “Done” and your WAF configuration is now in force.

Step 6 – Testing and Troubleshooting

At this point, go ahead and validate all possible functionality of the Citrix Gateway, including from browser, from Workspace App (if authenticating via Workspace App is applicable to your deployment), navigating around StoreFront, launching resources, logging out, etc.

If you do run into issues, you can flip on the learning mode in the profile and re-test, and examine the log results. If you disable blocking on all the checks and issues are still encountered, it may be an issue with a known protection rule. This is fairly easy for which to monitor, and akin to SSH into the ADC, drop to the shell, and run this command to monitor as a user is testing to generate the error condition if there is a block:

tail -f /var/log/ns.log | grep -i "blocked"

In the example below, we see the rule of ID 1147 was triggered and the request blocked. This provided the details necessary to go back to our signatures profile, search the ID, and disable the rule as we did earlier.

Security Insight can be useful for visualizing these violations as well; however, as we cannot enable Security Insight on a Citrix Gateway, it isn’t much help to us. With that said, if we fronted Citrix Gateway with a content switch (as we would with Unified Gateway), we could enable Security Insight on that object, bind the WAF policy directly to that vServer (shortening the expression to HTTP.REQ.IS_VALID) and overcome this. Should I find time to lab and document that out, I’ll amend this post.

Conclusion

That pretty much wraps up this guide. AppFW can be a little daunting at first, so do take your time with it to get comfortable if this is your first rodeo with the product. Hopefully, this has been a useful walk-through for some, in implementing a basic Citrix AppFW configuration on Citrix Gateway. In conjunction with other recommended hardening techniques on Citrix ADC (Bot Management, geo-blocking, IP Reputation, rate limiting, SSLTLS hardening, HTTP tuning, etc.), AppFW can further augment the security posture of your Citrix Gateway vServers.