**** UPDATE MAY 2021 – As of Citrix ADC Firmware 13.0 b76.x, Citrix now automates the creation of KEK files to enhance the security of deployments as a default configuration. Note this will likely impact moving hashed passwords and passphrases between ADC deployments as a result. As of b79.x, the system kek creation commands are no longer present in the code. ****
This guide is intended to shed a little light on an interesting security mechanism in Citrix ADCs which seldom gets much attention. Mentioned in the Citrix Secure Deployment Guide for ADCs, is a line item about creating a system master key for “data protection”. In the context of ADCs this means certificate key pair passphrases, LDAP passwords, RPC passwords, RADIUS secret keys, etc.
You may be inclined to say “Yeah, so what?”. And I hear you, anyone who has looked at an ns.conf file will note the passwords and passphrases are already hashed. But here’s where it gets a little errr… grey area-ish. See, anyone who has worked with ADCs for a while will likely know that one can build a brand new ADC, throw in ns.config code from another appliance that includes the hashed passwords, and voilà, encrypted credentials are now on the other appliance. How? Because all ADCs share a default hashing mechanism for credentials that allows this to occur. This makes migrations or cloning configs between like systems easy. But it’s a double-edged sword as I eluded to in an earlier post on sanitizing ns.conf files which can be facilitated with our web-based passphrase scrubber and IP obfuscator tool.
Being that ns.conf files are merely text files easily stored or emailed around, it’s easy to not give second thoughts to those files ending up in unauthorized hands. Beyond internal system details being exposed in these files (server names, IPs, account names, etc.) hashed passphrase information could be exploited as well. For example, a hashed LDAP BIND account password (normally non-expiring at that), could be transposed onto a new appliance with the LDAP server set as plaintext, and all it would take would be a Wireshark trace from the ADC to capture in plaintext the account password. Feeling uneasy yet?
Anyhow, this is where the KEK key concept comes in. Essentially they’re an AES 256 master key an administrator creates on ADC appliances with a passphrase of their choosing. Upon rebooting, the existing passphrases will be re-encrypted with this new master key as noted by the introduction of “-kek” beside the hashed passwords in the ns.conf files.
Recently working with a large financial organization with a very hyperactive security team, they mandated these custom AES master keys be created across the full complement of ADC appliances as part of the greenfield build’s security hardening. We wanted to take this opportunity to better document the process and what actually happens when they’re implemented.
So cool, we have means to move off the built-in default keys for hashing sensitive information in ns.conf files. But how does it all work? Well, my colleague Reubin spent some time diving deeper into its mechanics to convey what’s going on under the hood, what are the pieces, how do we break it, and how portable are these little guys. So shout out to Reubin for his tenacity on this one and for helping write a good chunk of this post.
Creating the AES Master Key (KEK) Files
NOTICE: BEFORE CREATING KEK FILES BE SURE TO BACK UP YOUR NSCONFIG DIRECTORY!
Before KEK is implemented hashed passwords will show up in nsconfig like the following:
To create system master key run the following command on the primary node of the HA pair and enter a passphrase with a recommended minimum of 8 characters.
I’d suggest rebooting for good measure, and be sure you check both HA nodes in the pairs to confirm they’re still syncing correctly. This may necessitate resetting RPC credentials on each appliance worst case, but an easy few second task.
OK, What Just Happened?
Two little files got created after running this command under /flash/nsconfig named F1.key and F2.key. These files are used by the appliance to encrypt sensitive data such as passwords and passphrases within the ns.conf file. The files should propagate to your secondary node in the pair as well.
If you re-examine the ns.conf file you’ll see a new addition toward the end of any string with a hashed password and the hash itself has changed.
What If I Delete The Files?
When F1.key and F2.key are deleted and the ADC reboots, hashed passwords revert to the previous encryption method and are still functional on the ADC. Authentication to Gateway is successful, as is login, etc. The nsconfig looks exactly the same prior to setting the master key.
Let’s Break Stuff!
So, one of the main goals of these KEK files is to hamper portability and decryption of sensitive data within the ns.conf files. Does it work? Do things break if you try and port over KEK’d (gamers rejoice) ns.conf files to another appliance? Yup.
However, if you port over your F1.key and F2.key files to the other appliance, you can in fact decrypt them and leverage these configs.
In testing, it does not matter if the passwords in ADC were configured before the KEK implementation or after the KEK implementation. If the keys are deleted from the ADC, the hashed passwords are still functional and can be copied over to another appliance as if nothing happened. So really the only thing securing the hashed password from being copied over are those 2 keys. If a user has ssh access and is aware of these keys nothing is going to prevent them from copying the information. That being said, we’re more prone to downloading copies of ns.conf files than we are other key files (no pun intended) of the appliances. With this in mind, it is still essential to apply appropriate security controls to backups of the appliances which could contain these KEK files and classify them as sensitive as one might a PFX certificate file, private certificate key, or spreadsheet of internal IPs and hostnames.
Is there a case for custom AES master keys after seeing how they behave? Absolutely and beyond a doubt. Every implementation of ADCs should have this implemented in order to avoid using the default hashing functionality of the appliances. Your custom keys could be made portable, sure, but it’s a heck of a lot harder to decrypt hashed configs if you don’t have the necessary keys, than if you’re able to leverage a private key common to base firmware.
So there you have it. I hope this is of use to someone, and for the shy and uninitiated, it provides some additional awareness in repercussions when implementing.
For more security guidance on Citrix ADCs, be sure to visit:
I really appreciate the level of detail in this article.
[…] Citrix ADC Security: create system KEK files to encrypt ns.conf passwords – Michael Shuster […]
Great article Mike the new FIPS 8900 boxes require you configure KEK for compliance
Glad you liked it, brother! Gotta love that new FIPS firmware… force KEK, remove RADIUS. What a trade off! 😉
Have you had any success in copying kek keys and configs from older versions (like 12.1) to versions of 13.0 with the setting on by default? I am working on this for a client and am not having any luck. Even moving the F1.key and F2.key files to a 13.0 82.45 VPX results in an “Error while decrypting the argument”. It seems that the 13.0 builds with it on by default now put the keys in the /nsconfig/keys folder, and append a timestamp to them , which is passed to the command with the “-suffix” switch in the config (this… Read more »
No I have not had the opportunity to test this. My best suggestion would be to try a build earlier than b76.x and then upgrade from there to 82.45 but with the more recent CVEs out it may not be possible to get the older versions publicly any longer.