September 15, 2024

Introduction

As attackers become increasingly sophisticated in both manual and automated attacks, financial services and e-commerce platforms strive to minimize fraud, while companies contend with remote workers attempting to bypass geolocation restrictions. This evolving threat landscape has driven a growing demand for robust IP intelligence capabilities to enhance protection and control access to externally facing web applications.

Although many security solutions offer a wide array of features, including real-time IP reputation services, they often lack the depth of data or the update frequency required by some customers for effective IP classification.

Many customers facing the aforementioned challenges subscribe to one or more threat intelligence services to layer on IP intelligence in front of their web apps to mitigate cyber risks, strengthen compliance, and enhance their overall security posture.

NetScaler, with its Premium edition licensing, comes with numerous security capabilities out of the box, including web app firewall, bot protection, API gateways, geoblocking, DDoS protection, rate limiting, real-time IP reputation, and several more. NetScaler’s IP Reputation feature leverages Webroot’s BrightCloud threat intelligence service and updates every five minutes provided the feature is enabled and NetScaler is able to access the BrightCloud API URL through your firewall. IP Reputation is easy to set up and easy to apply and is an excellent first line of defence on NetScaler.

Note: While this article is not about NetScaler’s IP Reputation feature, we want to take a moment to highlight that for IP Reputation to work properly, it requires 4 GB of RAM per packet engine. This allows NetScaler to cache in memory the reputation data so packets can be filtered through with a negligible impact on performance. You can check how many you have by running “stat CPU” from the CLI, or by dropping to the shell and running the top command. So if your appliance is configured with three packet engines, your appliance should have at least 12 GB of RAM for this purpose, in addition to RAM required for other baseline functionality and features (another 4 GB is a good starting point). This is more of a concern with VPX instances than it is with MPXs, which often come with sizeable amounts of RAM as standard, whereas with VPXs, administrators right-size the RAM to suit the use case.

While IP Reputation is great, and Ferroque encourages its use for most customers using NetScalers to front web apps on the Internet, unless another device upstream is handling this task, when it comes to detecting anonymous proxies and VPN networks, it is not particularly robust. BrightCloud has a great reputation, and other security appliances also rely on BrightCloud integration to provide real-time IP reputation protection to customers, but it just so happens this is an area the service is a little weaker on. This also illustrates why some firms subscribe to and layer on multiple threat intelligence services to gain access to a broader range of coverage across various areas of interest, as some providers may be stronger than others in maintaining and identifying certain categories of IPs or in the case of geolocation databases, have more granular data as compared to NetScaler’s built-in database.

So how is this accomplished on NetScaler? How can other databases for, say, anonymous proxies and popular VPN services be integrated into the NetScaler security framework for a website? This is what we will cover in this article, along with detailed deployment steps.

Disclaimer: Despite best efforts, some explanations for terms used in this guide may not be the best. The author is not a bona fide authority in IP quality detection. We welcome feedback on any definitions requiring massaging to improve accuracy for all readers.

Use NetScaler HTTP Callouts for Proxy and VPN Detection APIs

In the AppExpert pane, you will find the HTTP Callouts feature. This feature allows administrators to embed logic into a request flow to send data to a server for evaluation and take action on the response based on the data received back. In our case, whether or not a client IP attempting to access our website is an IP deemed invalid based on our criteria or is legitimate.

“Would this not slow down the website access?” you ask? That’s a great question and a valid concern. IP Reputation caches its data in RAM, surely this solution would be exponentially slower. While these solutions may add a fraction of a second more to an initial access request, NetScaler can be configured to cache the response received thanks to the Integrated Caching feature, and avoid the callout for subsequent requests from that IP for the caching period. Many security-conscious professionals would argue the barely noticeable page load impact would far outweigh the benefits of thwarting cyber attacks and blocking end-users from masking their true IP and location (based on their IP).

“Surely these services are not free.” you may also ask. Indeed, many of them are not free or are free for very limited use. In our demo here, we will use the IP Quality Score service (IPQS), one of several threat intelligence services that Netflix is alleged to use for proxy and VPN detection. This service has been selected for our use case for its apparent strength in detecting our IP category of interest and its free tier, which provides 5,000 API calls (IP lookups) monthly. Great for very small deployments that do not require the most advanced intelligence sets and proofs-of-concept. Their pricing plan may appeal to many, with different tiers priced per IP lookup request based on the level of IP intelligence sought.

“What about the quality of the data? Might these services incorrectly categorize some public IPs our legitimate users might egress through as proxies?” Absolutely. I can guarantee that will happen. Heck, in our testing, we were able to generate what we felt were false positives. While BrightCloud seems to highlight its anonymous proxy detection capabilities, do a little more reading, and it seems it’s mostly focused on web-based proxy services and Tor (dark web) exit nodes. Other threat intelligence providers seem to have a broader view of a proxy. An IP may be detected as a VPN or a Tor exit node, which will almost always be flagged as a proxy, or neither and still flagged as a proxy. And being a real-time platform, they may be re-classified later if deemed erroneous.

These larger subscription-based threat intelligence services also tend to classify data centres (also referred to as ASNs) as proxies, which is quite an expansive list, and ISPs may use them as well. But realistically, how many of your legitimate users will access your apps from within a public cloud or a commercial data centre? Probably some, but for the average use case, perhaps not a significant number.

Just as with email threat intelligence and SPAM filters, the answer to these concerns is initial tuning and ongoing adjustments when required. Do you want to focus only on VPN and Tor IP risks? You can scrutinize based on those and not on the broader “proxy” category. Do you wish to treat iCloud Private Relay IP (an IP anonymizing proxy service for Apple devices) addresses as legitimate? No problem, you can whitelist the ISP or Organization tags on these types of services. Are some ASNs used by residential ISPs come up as proxy? You can exclude them as well.

Using the same logic, if your users employ a ZTNA solution such as Cato, ZScaler, or Prisma, which are similarly categorized as proxies and VPNs, you can exclude them and treat them as legitimate based on their ISP or Organization tag in the HTTP callout response. And if your organization has IPs or blocks of IPs from which legitimate users originate but appear on the subscription lists as a VPN or proxy (or even if they do not), you can add them to a bypass list on the NetScaler using a data set, reducing API calls (and cost). Being proxy is a broad category, it may be best to start with a category such as “VPN” to avoid unexpected denials from what would otherwise be considered legitimate users.

Getting Ready

In this article, we will cover the following core configuration tasks. We will assume your NetScaler is already online, has Premium entitlements, and has an externally facing web app available to bind our configuration to. In our case, we will use a Citrix Gateway vServer, but it could be any HTTP/s web app. We will also assume you have signed up for a free account with ipqualityscore.com and have collected your API key used to make the IP lookup calls. While this example uses ipqualityscore.com, your organization may already have a subscription to a similar service or prefer another. Other services will likely use similar constructs to make API calls for IP lookups, and hopefully, the steps outlined can be adapted easily to other providers.

Fair Warning: As you may have gathered from the introduction, this article is going to be rather verbose. Some of the configurations we are making can be a little complicated or simple but not commonly used by the average administrator, so some context, rationale, and considerations on the various steps we felt were warranted. Each step starts with the objective and context, followed by the configuration steps. So with that in mind, if you are confident and seasoned in the nuances of these steps, feel free to skip the preambles.

Enabling NetScaler features
Configuring a non-addressable load-balanced VIP for the API service
Configuring the HTTP callout
Configuring a data set for whitelisting networks (to bypass API calls)
Configuring a pattern set for web crawlers (to avoid wasting API calls on them)
Configuring an HTTP page for access denied messages (optional alternative to dropping invalid requests)
Configuring a responder policy to invoke the HTTP callout and filter exceptions (ex. allow iCloud proxies)
Configuring a custom log action to alert when a request was dropped/denied (optional)
Binding the responder policy to the vServer (plus guidance on binding order in relation to other IP filtering policies)
Testing different networks and monitoring API usage

Step 1 – Enable NetScaler Features

If not already enabled on your appliance, enable Responder and Integrated Caching in addition to Load Balancing, which is likely already enabled. Responder allows us to implement logic and actions on requests from IPs to our web app. Integrated Caching will allow us to cache responses we receive from the IP intelligence service for a period of time so subsequent requests from that IP can be sourced from RAM and save money on API calls. When callout responses received from the API service are cached, they are stored in the Integrated Caching built-in Content Group labelled calloutContentGroup. This Content Group is configured with a default 200 MB memory usage limit. Assuming each cached response is 4096 bytes at most, a total of 51,200 cache objects (more specifically, IPs in our case) can fit into 200 MB. A fair bit of mileage.

Step 2 – Build a VIP for the IP Intelligence / Threat Detection Service

Preamble

The HTTP callout feature allows us to specify either an IP/port combination for the external web service or a VIP on the NetScaler for this purpose. The VIP must be an HTTP load-balancing VIP. While specifying an IP and port might seem simple enough, it prevents us from applying additional security measures to validate the external server and encrypt both our payload and API key and, equally important, would eliminate any service load balancing of the host address as the field can only be an IP. In our case, there are three possible IPs.

Non-authoritative answer:
Name:	www.ipqualityscore.com
Address: 172.67.72.12
Name:	www.ipqualityscore.com
Address: 104.26.2.60
Name:	www.ipqualityscore.com
Address: 104.26.3.60

As covered in CTX310762, we will create a DNS-based server on the NetScaler (this requires that the NetScaler DNS client is functioning properly), allowing the service provider to continue handling its load balancing (and avoid mishaps of their IPs changing entirely later on). We will then create a service of type SSL. Because of the SSL/TLS security configuration of IPQS (as seen below in its SSLLabs scan), we need to tune some settings at the service level. Otherwise, SSL handshakes will fail from the NetScaler. We will then bind this service to a non-addressable VIP of type HTTP so that the callout can use it.

This configuration will allow us to send our callout requests to IPQS with encryption and also allow (as per the example in the CTX article, note that was not tested with IPQS) to perform server authentication against the server to further validate it is legit (based on certificate) and not spoofed through DNS manipulation.

Steps

Navigate to Traffic Management > Load Balancing > Servers and create a new server with the following parameters:

It will appear as so once completed:

CLI Command:

add server srv_ipqualityscore www.ipqualityscore.com

Navigate to Traffic Management > Load Balancing > Services and create a new service with the following parameters:

IPQS has its own mechanisms for load balancing its service among servers, and we are only specifying one server to begin with, so the default TCP monitor can be left in place. A secure HTTP monitor can be used, but it may not offer much benefit and may even introduce a risk of failure if the response code (currently 200 against the header) is changed in the future. Note an HTTPS monitor will only work once the SSL parameters of the service are correctly set. Again, leaving the default TCP monitor that gets bound upon service creation is probably the best approach here.

IPQS uses SNI, as touched on earlier, meaning multiple SSL/TLS certificates are bound to its service. On the NetScaler, either via an SSL profile (backend type) or SSL parameters, select SNI Enable and specify the common name of www.ipqualityscore.com on our newly created service. While the common name of the certificates does not include that name, it is valid thanks to the wildcard certificate presented from www.ipqualityscore.com, and we will need to specify that as the common name for the SSL handshake to work, being the NetScaler will attempt to communicate to www.ipqualityscore.com via IP when sending HTTP callout requests.

If, for example, we try to access one of the IPQS IPs directly in our browser, the connection will fail as the common name is not being sent alongside the handshake request with the Server Name Indication (SNI) extension. IPQS supports TLS1.2 and TLS1.3 only, so we can specify that as well while we are here.

Once you click OK and then Done and return to the Services page, it will appear as such below. If the IP address is not resolving (it may be different from what is displayed below), check your DNS configuration both on the NetScaler and upstream. It is possible a security appliance is configured to block external DNS requests from some network segments.

CLI Commands:

add service svc_www.ipqualityscore_443 srv_ipqualityscore SSL 443
set ssl service svc_www.ipqualityscore_443 -ssl3 DISABLED -tls1 DISABLED -tls11 DISABLED -dtls1 DISABLED -SNIEnable ENABLED -commonName www.ipqualityscore.com

Navigate to Traffic Management > Load Balancing > Virtual Servers and create a new LBVS with the following parameters and click OK:

Click No Load Balancing Virtual Server Service Binding and bind the service we created in the last step.

Click Done to return to the Virtual Server page. We should see the VIP online as so:

CLI Commands:

add lb vserver lbvs_ipqualityscore_80 HTTP 0.0.0.0 0
bind lb vserver lbvs_ipqualityscore_80 svc_www.ipqualityscore_443

Our VIP is now configured, and we can move on to the next step.

Step 3 – Creating the HTTP Callout

Preamble

This is the most complicated step in the process. It is easy to configure, but understanding what to configure and for what syntax can be a little challenging. And it will likely vary across threat intelligence providers. The HTTP Callout allows us to send a payload to an external web service (in this case IPQS) via GET or POST to allow the service to validate our query against its data set and respond. We can also define what we want to extract from the response in event there is a large amount of irrelevant information we do not need to retain.

IPQS’s proxy detection documentation tells us their request URL is as follows:

https://ipqualityscore.com/api/json/ip/YOUR_API_KEY_HERE/USER_IP_HERE, and we can suffix the URL (making it a URI) with parameters (starting with ‘?’) to pass through with additional parameters. In IPQS’s case, these can include the level of strictness, speed to respond, etc. Full details are in their documentation. Worth noting in their example the www is omitted (whereas we are specifying it in our common name, but we will not be using this request format anyway).

To improve the security of our API calls to IPQS we will want to obfuscate the API key a little better. Although not a substitute for outright encrypting the calls over HTTPS, it is ideal to configure. In our case, we will use the header method. And in their example, the path does start with www for whatever reason. IPQS supports JSON and XML, our configuration will use JSON. So we will receive and parse results back from our callout requests in JSON format.

When we run a query to IPQS, it will come back in JSON format like so:

{"success":true,"message":"Success","fraud_score":84,"country_code":"CA","region":"Ontario","city":"Milton","ISP":"Cato Networks","ASN":13150,"organization":"Cato Networks","is_crawler":false,"timezone":"America\/Toronto","mobile":false,"host":"45.62.189.125","proxy":true,"vpn":true,"tor":false,"active_vpn":true,"active_tor":false,"recent_abuse":true,"bot_status":true,"connection_type":"Premium required.","abuse_velocity":"Premium required.","zip_code":"N\/A","latitude":43.51,"longitude":-79.88,"request_id":"QkPj41sya2"}

We can also configure the caching of responses on the callout. This means NetScaler will cache the response in memory (specified in seconds) which both speeds up subsequent requests from the same source IP and eliminates another call (and cost) calling out to IPQS.

Our challenge will be to embed variables into the HTTP Callout and properly craft the request URI to IPQS so that it functions properly and nets us a valid response.

Steps

Navigate to AppExpert > HTTP Callouts and click Add.

There is a lot to configure here, and just as much we do not need to. Following IPQS documentation under “Additional Request Options” we are working to format our request to use the following path:

https://www.ipqualityscore.com/api/json/ip

Omitting https://, the first part will map to our “Host Expression”. The rest of the path is the “URL Stem Expression”. The API key will be sent stored in a header called IPQS-KEY (specified in IPQS docs so it must look for it), and additional parameters such as fast, strict, etc. we will specify as Parameter variables.

The variable we need to pass over will be the client IP making the request to our web app, which we will specify in the callout with the placeholder “CLIENT.IP.SRC”.

Below we summarize what we will fill out:

Name:
- ipqs_callout
Server to receive callout:
- Virtual Server
  - lbvs_ipqualityscore_80
Request to send to the server:
- Request Type: Attribute-Based
- Method: GET
- Host Expression: “www.ipqualityscore.com”
- URL Stem Expression: “/api/json/ip”
- Headers
  - IPQS-KEY
    - Value: <YOUR_API_KEY_FROM_IPQS>
- Parameters
  - ip
    - CLIENT.IP.SRC
  - fast [Note: Optional]
    - 1
  - strictness [Note: Optional]
    - 1
- Scheme: https
Server Response
- Return Type: TEXT
- Expression*: HTTP.RES.BODY(1000)
- Cache Expiration Time (in secs)**: 3600

*We could modify this expression to extract only data from the JSON response we want to retain and in theory reduce the caching load.

For example, using this expression instead of HTTP.RES.BODY

"proxy:" + HTTP.RES.BODY(1000).CONTAINS("proxy") + ", is_crawler:" + HTTP.RES.BODY(1000).CONTAINS("is_crawler") + ", vpn:" + HTTP.RES.BODY(1000).CONTAINS("vpn") + ", tor:" + HTTP.RES.BODY(1000).CONTAINS("tor") + ", country_code:" + HTTP.RES.BODY(1000).CONTAINS("country_code") + ", region:" + HTTP.RES.BODY(1000).CONTAINS("region") + ", city:" + HTTP.RES.BODY(1000).CONTAINS("city") + ", ISP:" + HTTP.RES.BODY(1000).CONTAINS("ISP")

Would return this:

host:45.62.189.125, is_crawler:false, vpn:true, tor:false, proxy:true, country_code:CA, region:Ontario,city: Milton,ISP:Cato Networks

Instead of this:

{"success":true,"message":"Success","fraud_score":84,"country_code":"CA","region":"Ontario","city":"Milton","ISP":"Cato Networks","ASN":13150,"organization":"Cato Networks","is_crawler":false,"timezone":"America\/Toronto","mobile":false,"host":"45.62.189.125","proxy":true,"vpn":true,"tor":false,"active_vpn":true,"active_tor":false,"recent_abuse":true,"bot_status":true,"connection_type":"Premium required.","abuse_velocity":"Premium required.","zip_code":"N\/A","latitude":43.51,"longitude":-79.88,"request_id":"QkPj41sya2"}

Roughly a difference of ~300 bytes.

However, in preliminary testing, we found the ISP was coming out as “true” and not the actual value. And we cared about that value (will see why later). This can probably be sorted but for the purposes of the example we reverted to HTTP.RES.BODY(1000). The request responses are around 600 bytes so 1000 is a fair value.

** 3600 translates to 24 hours. The maximum in seconds we can specify equates to 8,760 hours. Choose a value that works best for you that balances cost and performance. Note that while we can configure a caching timeout here, it will do absolutely nothing if Integrated Caching is not enabled on NetScaler.

Here are the settings configured within the callout, and remember the parameters other than ‘ip’ are optional, explore the documentation to find what suits you best:

CLI Commands:

add policy httpCallout ipqs_callout -vServer lbvs_ipqualityscore_80 -returnType TEXT -hostExpr "\"www.ipqualityscore.com\"" -urlStemExpr "\"/api/json/ip\"" -headers IPQS-KEY("xxxxxxxxxxxxxxxxxxxxxxxx") -parameters ip(CLIENT.IP.SRC) fast(1) strictness(1) -scheme https -resultExpr "HTTP.RES.BODY(1000)" -cacheForSecs 3600

Step 4 – Configuring a Bypass Data Set (Optional)

Preamble

This step is not a hard requirement, but we recommend implementing it. If you have client requests coming from trusted IP blocks, there may be little reason to send the request to IPQS and consume an API call. Similarly, there may be instances where IP blocks or specific IPs are flagged as VPNs or proxies or whatever it is you wish to block by default, either due to erroneously being flagged or perhaps originating from trusted data centres or ZTNA platform networks, and you also want them to be whitelisted and bypass an API call via HTTP callout to IPQS.

This is where implementing a data set comes in handy, so we can bind a Responder policy to your web app with a higher priority than the HTTP Callout invoking policy and by using END when matching the policy, it will skip the HTTP Callout policy bound with a lower priority.

Steps

You can define one in AppExpert > Data Sets. As this is not a key focus, we will skip the GUI visual and provide the commands to make a simple data set and a corresponding Responder policy (that takes no action but, once bound, allows us to escape the HTTP Callout policy). You can then add individual IPs or IP ranges to the data set later.

CLI Commands:

add policy dataset vpnblocker_whitelist_dataset ipv4
add responder policy trusted_network_traffic_policy "CLIENT.IP.SRC.TYPECAST_TEXT_T.CONTAINS_ANY(\"vpnblocker_whitelist_dataset\")" NOOP

Step 5 – Configuring a Web Crawler Pattern Set (Optional)

Preamble

This is another optional step, but is worthwhile considering. Most websites online, unless they have been set to explicitly tell bots not to index them, will be hit periodically with web crawlers from numerous sources. Compiled from ChatGPT, here is a list of some common ones. Crawler bots will consume API calls, so if you wish to bypass them or outright block them (if your site shouldn’t be indexed to begin with), we can use a pattern set to do this. The steps below will make a pattern set for the following known crawler bot user-agents:

Googlebot bingbot Yahoo! Slurp Baiduspider YandexBot DuckDuckBot Sogou Spider Exabot SeznamBot MJ12bot AhrefsBot SemrushBot DotBot LinkedInBot facebookexternalhit Applebot Pinterestbot

Important: User agents are easily spoofed. So while it may seem convenient to bypass, it may provide an attack vector to bypass your IP threat intelligence security controls. We would recommend either dropping requests identified with them via a Responder policy action or omitting this step and allowing the API calls.

As bots in general can generate significant traffic on websites depending on their content, it is highly recommended to employ a bot management platform that handles bots before reaching the HTTP Callout Responder to eliminate significant load on the API calls. NetScaler Web App Firewall combined with Bot Management can help achieve this.

Steps

You can define one in AppExpert > Pattern Sets. As this is not a key focus, we will skip the GUI visual and provide the commands to make a simple pattern set that can then be used in a Responder policy.

add policy patset crawler_bots
bind policy patset crawler_bots Googlebot -index 1
bind policy patset crawler_bots bingbot -index 2
bind policy patset crawler_bots "Yahoo! Slurp" -index 3
bind policy patset crawler_bots Baiduspider -index 4
bind policy patset crawler_bots YandexBot -index 5
bind policy patset crawler_bots DuckDuckBot -index 6
bind policy patset crawler_bots "Sogou Spider" -index 7
bind policy patset crawler_bots Exabot -index 8
bind policy patset crawler_bots SeznamBot -index 9
bind policy patset crawler_bots MJ12bot -index 10
bind policy patset crawler_bots AhrefsBot -index 11
bind policy patset crawler_bots SemrushBot -index 12
bind policy patset crawler_bots DotBot -index 13
bind policy patset crawler_bots LinkedInBot -index 14
bind policy patset crawler_bots facebookexternalhit -index 15
bind policy patset crawler_bots Applebot -index 16
bind policy patset crawler_bots Pinterestbot -index 17
add responder policy drop_crawler_bots "HTTP.REQ.HEADER(\"user-agent\").SET_TEXT_MODE(IGNORECASE).CONTAINS_ANY(\"crawler_bots\")" DROP

Step 6 – Configuring an HTTP Page for Invalid Requests

Preamble

While technically optional as well, rolling out this solution by hard-dropping invalid client requests when focusing on VPN and proxies is bound to result in some false positives or matches to networks that the organization would rather treat as trusted (as outlined earlier). By pairing our HTTP Callout Responder Policy (which we will create later, but this step comes first), we can display to legitimate and illegitimate users a message indicating they appear to be accessing the website from an unauthorized network and have been denied access.

This will greatly assist in troubleshooting and issue resolution for instances of legitimate users being denied based on their IP being flagged as invalid by the system. Again, if we are dealing with a very sensitive platform and a very savvy set of end-users and support desk personnel, a dropping of traffic might suffice, paired with a SYSLOG message indicating the request was blocked for matching in IQPS (covered in a later step).

We can easily create and store an “Access Denied” page on NetScaler, including images encoded in Base64 (to avoid image hosting, thanks Juliano!) and optionally provide guidance to users for self-service (such as disable VPN anonymizers on their endpoint, iCloud Private Relay, etc.) and how to log a support ticket and what information to include to expedite investigation and resolution (which might result in adding an IP or IP range to the data set created earlier).

Steps

Navigate to AppExpert > Responder > HTML Page Imports and click Add.

Specify a name (our example uses vpn_denied_page) and select “Import From: Text” and click Continue.

Under File Contents, paste in an HTML page customized to your organization’s styling and with whatever information, links, and messages you require. Below is a very ugly example but one that renders well on web and desktop browsers. It features a header, subheader, and an image encoded in Base64 (omitted from example). Logos and other images can be embedded in the HTML as Base64 encoded images. Tools to convert PNG and JPEG images are widely available online, free, and easy to use.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Access Denied</title>
    <style>
        body {
            display: flex;
            flex-direction: column;
            justify-content: center;
            align-items: center;
            text-align: center;
            height: 100vh;
            margin: 0;
            padding: 20px;
            background-color: #f8f9fa;
            font-family: Arial, sans-serif;
        }
       h1 {
            color: #dc3545;
            font-size: 2rem;
            margin-bottom: 1rem;
        }

        h2 {
            color: #000000;
            font-size: 1.2rem;
            max-width: 100%;
            word-wrap: break-word;
            margin-bottom: 1rem;
        }

        img {
            max-width: 100%;
            height: auto;
        }

        @media only screen and (max-width: 768px) {
            h1 {
                font-size: 1.5rem;
            }

            h2 {
                font-size: 1rem;
            }
        }

        @media only screen and (max-width: 480px) {
            h1 {
                font-size: 1.2rem;
            }

            h2 {
                font-size: 0.9rem;
            }
        }
    </style>
</head>
<body>
<div>
    <h1>Access Denied</h1>
<br>
    <h2>Our system has detected your request as originating from an unauthorized network such as a VPN or anonymizer proxy service, both of which are strictly prohibited.</h2>
<br>
<img src="data:image/jpeg;base64,<YOUR_BASE64_CODE>" type="image/jpeg"/>
</div>
</body>
</html>

Click Done, wait a few seconds, and you should be returned to the Responder HTML Pages page.

With this out of the way, we can create the actual Responder policy and action that will call on this HTML page when the HTTP Callout receives a match from IPQS, identifying it as a VPN or proxy.

Step 7 – Configuring the HTTP Callout Responder Policy & Action

Preamble

We use a Responder to link the HTTP Callout to our web app vServer on the NetScaler. We will first start by creating an action to send matching responses from IPQS to the Access Denied page we created previously. If you just want to drop the traffic, you can omit the action and configure that action directly on the Responder policy.

When we create our Responder policy expression, we will define the evaluation logic we wish to use when the HTTP Callout is invoked and the response is received.

In our example, we are going to tell the Responder to ignore (thus bypass) client IPs originating from the iCloud Private Relay service and our SASE/ZTNA platform from Cato Networks so that legitimate users on our managed endpoints are not blocked from connecting. We could alternatively add the known IP blocks to our bypass data set to reduce API calls to IQSP, but as you can see from the Cato example here, this would be quite extensive, and new ones are likely added periodically.

To mitigate this to some extent, many ZTNA platforms provide a set of public IPs dedicated to a customer which can be configured to egress traffic from toward a particular website, which would reduce the configuration in the data set but may also result in a slightly less optimal routing path to your web app. As is often the case, there are minor trade-offs when working to balance performance, management overhead, and security. You may not wish to treat the entirety of your ZTNA platform’s (if you use one at all) IP blocks as trusted and only those assigned directly to you, for example.

Steps

Navigate to AppExpert > Responder > Actions and click Add.

Configure the Action per the screenshot below. Click Create, and you will return to the prior page.

Navigate to AppExpert > Responder > Policies and click Add.

Create a Responder policy with the configuration found below, modified to suit your needs. For example:

As the proxy category is very broad, you may wish to consider checking for VPN only, but we suggest doing your research on how the threat intelligence provider categorizes IPs to better understand what the result may be. Data centres used by known VPN services may not get flagged, for example.
If you do not need to bypass any ISPs or private relays (ZTNA platforms will override the ISP of a user’s home Internet) you can omit the examples we have for both and stick initially to the HTTP Callout filter on VPN or proxy.

The expression used in the example (again, edit to suit your needs and skip the other qualifiers after the first one):

SYS.HTTP_CALLOUT(ipqs_callout).CONTAINS("\"proxy\":true,") && SYS.HTTP_CALLOUT(ipqs_callout).CONTAINS("\"ISP\":\"iCloud Private Relay\",").NOT && SYS.HTTP_CALLOUT(ipqs_callout).CONTAINS("\"ISP\":\"Cato Networks\",").NOT

Note: The filter parameter in the expression assumes the content of the HTTP Callout Response has been unaltered. If it has been altered, similar to one of the examples outlined in Step 4, the query parameter would be different, as we would no longer be filtering through JSON-formatted text, which is what the above expression is expecting.

CLI Command:

add responder policy drop_vpn_traffic_policy q/SYS.HTTP_CALLOUT(ipqs_callout).CONTAINS("\"proxy\":true,") && SYS.HTTP_CALLOUT(ipqs_callout).CONTAINS("\"ISP\":\"iCloud Private Relay\",").NOT && SYS.HTTP_CALLOUT(ipqs_callout).CONTAINS("\"ISP\":\"Cato Networks\",").NOT/ drop_vpn_traffic_action

Step 8 – Create a Custom Log Action (Optional)

Preamble

In this optional step, we can inject a custom SYSLOG message when the Responder policy matches, which may assist in auditing violations or troubleshooting end-user issues for legitimate requests to the site. We will use the variables of the client IP alongside the destination host (our web app) in the log message to assist in tracing.

Steps

Back on the Responder Policy we created, click Add next to Log Action.

Choose the log level suitable for you. Informational may not be appropriate if info logs are not sent to your SIEM platform from NetScaler (albeit info logs are what contain key auditing information for SSLVPN and HDX proxy connections).

For the expression, use the following as a reference to create one suitable for your needs.

"[VPNDENY] " + CLIENT.IP.SRC + " was denied access to " + HTTP.REQ.HOSTNAME + " as this IP was flagged as originating from a proxy by www.ipqualityscore.com"

Click OK and select the Log Action on the previous screen. Click OK to commit the modifications.

CLI Commands (bind to the policy after):

add audit messageaction drop_vpn_traffic_log_act INFORMATIONAL "\"[VPNDENY] \" + CLIENT.IP.SRC + \" was denied access to \" + HTTP.REQ.HOSTNAME + \" as this IP was flagged as originating from a proxy by www.ipqualityscore.com\"" -logtoNewnslog YES

Step 9 – Binding Responder(s) to Your Web App vServer(s)

Preamble

All the components are now in place; hopefully, the callout is functional. Before binding the HTTP Callout, it is best to first discuss optimizing overall security filtering on the web app. As mentioned earlier, using a third-party threat intelligence service in a production environment will likely incur some costs. While the actual cost will depend on the volume of API calls and the class of intelligence you are looking to receive, you will nonetheless want to optimize and reduce what gets sent to it. For example:

Trusted Networks. If we have a list of trusted networks that should always be permitted to access the web app, use a responder that calls a data set to bypass and stop evaluating further policies such as the HTTP Callout.
IP Reputation. As a service included with the NetScaler at no additional cost, use it to weed out a preliminary set of malicious IPs from BrightCloud’s real-time database. Either send the offenders to an HTML page or drop their connections. If an upstream security device performs this function for the edge of your network as standard, use that service, but be sure it is configured to apply to your web app hosted on the NetScaler downstream from it. In either case, they should reduce the amount of invalid requests from reaching the NetScaler’s HTTP Callout policy.
Geoblocking. Similar considerations to IP Reputation. If you are only allowing traffic from known regions, block the others. Crafting the expression is important (our example uses a series of .NOTs for permitted countries with an action of dropping traffic). Again, if an upstream security device handles this and is confirmed to affect the web app on the NetScaler, leave it configured there and do not bother doing it on the NetScaler. Trust but verify.
Crawlers. If we do not want the web app to be indexed on the Internet, use the pattern set we created and drop the traffic.

In practice, bind the abovementioned responder policies first, with the VPN blocking HTTP Callout policy coming afterward. These will help reduce the number of requests to IPSP for IPs we already trust or deem invalid through other means.

Steps

Navigate to the vServer on your NetScaler and bind the policies in the order appropriate for your use cases.

Note: Each of these were bound with a GOTO EXPRESSION parameter of “END” which is what you will also want.

Step 10 – Testing

Now the fun begins. You will want to have a test plan in place to validate different scenarios and expected outcomes. Use the following as a starting point:

Access to the web app is not blocked from remote locations such as residences without private relays, proxies, or VPN clients in use
Access to the web app is not blocked and denied page received when iCloud Private Relay is enabled on an Apple device (if set to bypass in the expression)
Access to the web app is not blocked and denied page received when ZTNA network is detected as ISP (if set to bypass in the expression)
Access to the web app is blocked and denied page received when accessed from different VPN clients. We strongly recommend trying numerous different ones from different PoPs, provided those PoPs are within your geoblocking boundaries (otherwise, the geoblocking policy may drop the traffic before it has a chance to hit the HTTP Callout)
Add an IP from a VPN client you had a denied session on to the VPN bypass list and reattempt access to the page, confirm the web app is not blocked
Confirm Integrated Caching is caching responses from IQPS
Confirm logs are generated when the HTTP Callout result matches the denial criteria

In our example, the passing requests start to load a Citrix Gateway page. Those who were detected as originating from an invalid network (in this case various Windscribe PoPs in the US and Canada) we receive a cute little denial page.

We confirmed VPN client connections were denied and logged in SYSLOG with the following message:

31) 09/14/2024:20:49:37  [XXXXXXX] Informational 0-PPE-0 : default RESPONDER Message 47199 0 :  “66.115.182.136 was denied access to [XXXXXXX] as this IP was flagged as originating from a proxy by www.ipqualityscore.com”

Let us check out the Integrated Caching function. Navigate to Optimization > Integrated Caching > Content Groups. Observe the memory usage and 304 cache hits to calloutContentGroup after your preliminary testing.

Go up a level to Integrated Caching, click View Cache Objects under the Cache Object header, and select All.

Click on one of the Locator items in the calloutContentGroup and click Details.

From here you can peruse the contents of the HTTP Callout. Please note that the HttpCalloutResult only displays the first few bytes of the payload in this view.

Back in our IQSP account, we can see over the course of the day the calls made to the service using our API key.

Conclusion

We hope this article will be informative to IT professionals looking to further enhance the security and compliance of their web apps. This example is but one of many IP threat intelligence platforms that could be loaded into NetScaler, and it is possible to layer on multiple if needed. Please reach out if you have questions or need assistance in integrating your preferred IP intelligence service with NetScaler for IP blocklists, geolocation services, and more.

References

The following resources were of value when implementing the above solution.

Michael Shuster

Michael is Ferroque's founder and a noted Citrix authority, overseeing operations and service delivery while keeping a hand in the technical cookie jar. He is a passionate advocate for end-user infrastructure technology, with a rich history designing and engineering solutions on Citrix, NetScaler, VMware, and Microsoft tech stacks.