Issue and Background

Mr. Reubin Huckle had a fantastic title for this article of “Dude Seriously, What The FAS: Shopping Adventures in the NTAuth Store” however he insisted this would not rank highly for SEO and I’ve adopted a lamer, more relevant title. You win this time, Google.

This troubleshooting guide is focusing on a pretty uncommon failure cause for a fairly common “Incorrect username or password” error a user may get on the Windows logon screen when launching a Citrix session. Note that depending on the Windows OS this error may present itself as “The user name or password is incorrect”. It generally indicates smart card SSO is broken, which is a core underpinning of Citrix FAS’s SSO architecture. If you have enabled Kerberos logging on the VDA (quick registry change that requires no reboot) you will note this Event ID 4768 error with the condition “KDC_ERR_PADATA_TYPE_NOSUPP“. Sarah Steinhoff has this error condition mentioned in her ever-popular “Troubleshooting the Federated Authentication Service” blog on Citrix Blogs.

You may also get Error ID 8 with the following message: The domain controller rejected the client certificate of user username@domain.com, used for smart card logon. The following error was returned from the certificate validation process: A certification chain processed correctly, but one of the CA certificates is not trusted by the policy provider.

-or-

You may also get Error ID 9 with the following message: The client has failed to validate the domain controller certificate for domaincontroller@domain.com. The following error was returned from the certificate validation process: A certification chain processed correctly, but one of the CA certificates is not trusted by the policy provider.

The environment in our case was built on CVAD & FAS 1912 CU3, Windows Server 2019 VDAs. The Domain and Forest Functional Levels were Windows Server 2008 R2 although Domain Controllers were newer.

As mentioned, there are quite a number of Citrix KBs related to the “incorrect username or password” logon message indicating FAS SSO is broken, but nearly none of them that came up, nor their root causes were relevant to us.

The actual cause of our particular issue eluded us for some time, and we felt perpetually beat up by Microsoft PKI infrastructure, which is the underpinning of Citrix FAS (and thus why most error conditions with FAS end up being Microsoft-related). There was Citrix KB found shortly after we resolved our issue via a Microsoft TechNet discussion, that had not come up before, perhaps due to modest differences in error messages or event IDs or just bad SEO. The resolution was pretty much the same as what will be described here as a resolution.

Incorrect Username and password error when using FAS to single sign on with VDA with Kerberos Event ID 8

Root Cause and Resolution

The cause of the issue was the NTAuth store as found in the VDA’s registry. It was devoid of any of the necessary intermediate certificates of our CAs who are issuing our virtual smart card certificates. Despite those intermediate CA certificates being present on the local computer’s certificates store (as validated by snap-in), the Domain Controllers in the environment having been issued the sub CA for Kerberos\Smart Card\Domain Controller use, and the issuing\subCA certificates being present in the domain’s Enterprise PKI NTAuth store (which had been validated several times), and the VDA receiving the sub CA certificates from AD per policy, they were missing from the registry. Even a gpupdate /force and a certutil -pulse did not help sort out the missing certificates.

We confirmed this through two methods.

Method 1: Registry

If you navigate onto the VDA’s registry and locate the path here, in a normal setup you should see entries for one or more certificates representing trusted CAs.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseCertificates\NTAuth\Certificates

Method 2: CertUtil

certutil -viewstore -user -enterprise NTAuth

When running that command, if you receive a screen pop-up that indicates no certificates are available, you are affected by this issue of NTAuth certificates missing.

The steps to resolve the issue are found within the two references from Citrix and Microsoft mentioned earlier, but essentially to resolve the issue, we need to get the trusted intermediate certificates (i.e. those of our issuing\subCAs used by FAS and the DCs) into the NTAuth registry location. Once these were in place, our FAS solution worked immediately. This did mean we also had to ensure the master images were updated with this configuration as well.

Step 1 – Export the Intermediate CA Certificates

Launch MMC.EXE and add in the Certificates snap-in under ‘Local Computer’. Navigate to Intermediate Certificate Authorities and export the internal CA certificates trusted by the computers on the network. In our case, there was a total of 4, but more than likely 2 is more common for most. Exporting them in DER format is fine, and place them on the desktop or someplace easy to access as we will be calling them via CMD in the next step.

Step 2 – Import the CA Certificates via CertUtil into NTAuth

From an Administrator command prompt, run the following command, replacing ‘filename’ with the CA certificate files you just exported. Repeat the process for the other files.

certutil -enterprise -addstore NTAuth filename

Step 3 – Validate with CertUtil or Registry the NTAuth Store is Populated

Once the files are imported, you can validate them again by the two methods mentioned earlier.

Method 1: Registry

From RegEdit navigate to the following path:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseCertificates\NTAuth\Certificates

You should now see entries you just imported.

Method 2: CertUtil

Run the following command again, which should bring up a window with your imported certificate (s) for your CA(s).

certutil -viewstore -user -enterprise NTAuth

Step 4 – Test FAS SSO by Launching Citrix Resource

With the CA certificates now imported into the right location on your VDA (and now presumably on all VDAs that could host a session when testing), go ahead and launch a published app or desktop. It should proceed normally for SSO. If it fails, double-check your work. If your work was good, you have other lingering Kerberos\CA issues in the environment, but you have certainly cleared one vital hurdle to a successful launch. As we had already overcome CRL and Domain Controller firewall issues, this was the silver bullet.

Do not forget to update your master images with this change as well, as needed.

Subscribe