Continuous Evolution: Citrix Profile Management Q4 2023

Introduction

It’s been just under three years since we brought you my last blog on profile management. To generalize, it did not feel like we had a lot of modernization immediately after we published that blog. Sure, FSLogix introduced the Disk Compaction feature in release 2010 and three subsequent hotfixes to stabilize performance, but the modernization the product initially brought to us in 2019 seems to have plateaued. Citrix Profile Management (CPM) seemed almost to be on maintenance, releasing disk-based support for .OST files in 2021, and then support for OneDrive for Business (OD4B) in Q3 2023, but this seemed like an attempt to keep pace with another product rather than drive that historic innovation in the EUC/EUI space to which we have been accustomed. Moreover, I did not think there was sufficient guidance from Citrix in the new container-based features that were being released. In the field, I have seen some great and tenured Citrix Engineers (who have been on the tools since MetaFrame) experience issues bridging the gap between file-based and container-based profile management solutions. If I can help just one team avoid some of the challenges bridging file and container-based profile management storage, then I will have achieved what I set out to accomplish.

In November 2023, two notable events happened:

• Citrix published the following blog highlighting new storage management features: Yarshin He’s blog, Use Citrix Profile Management to effectively manage your user profile storage, and this piqued our interest in the internal Slack channels within Ferroque. The blog highlights three features specific to storage optimization within CPM:
  • Disk Compaction
  • File De-duplication
  • Profile container sizing and auto-expansion
• Ferroque attended PTEC at Citrix HQ in Fort Lauderdale: In addition to some great PTEC sessions, Michael Shuster was able to speak with Wayne Liu, Director of Product Management at CSG, about participation in a Ferroque webinar, dedicated to outlining the continuous evolution on the CPM product roadmap from v2203 to v2411

If you haven’t already, please see the following link to register/view our CPM Continuous Evolution webinar delivered on January 18, 2024.

• MASTERCLASS: EUI Innovations in Container-Based Profiles – CPM 2203 and Beyond

Additionally from Ferroque, please see the following blog: Profile Management Innovation – A Wishlist. In this Ferroque blog, Michael Shuster summarizes 20 years of profile management strategies, brainstorming discussions with Citrix Product Managers at PTEC ’23, and specific functionality enhancements he would like to see in CPM moving forward. Feel free to review and add your own items to this wish list!

Present Day Trends

Fast forward to the present day, and we are firmly entrenched in the era of container-based profile management. Since 2019, the choice for which container-based profile management solution to use has been a foregone conclusion: FSLogix. This has been a solid solution for scenarios wherein there is a business requirement to include M365 apps and data into user profiles, where we are modernizing VDI workloads and optimizing them to run the latest collaboration and productivity tools, and you or your customer already has the entitlement to the FSLogix product (which describes the majority of deployments we have seen). All customers need to be able to provide is storage to accommodate the user and application container(s). Once customers become accustomed to the new reality that profile sizes will likely be measured in GBs and not MBs (as they were in CPM’s file-based profile management solution), then they are off to the races.

In October 2022, we started to see some long-awaited storage optimization offered in FSLogix with VHD Disk Compaction, which reclaimed white space that naturally collects on VHDXs, which themselves can grow in size to accommodate growing profiles but cannot shrink if/when a user’s profile shrinks. In the field, I have heard some of my favorite Microsoft Black Belts offer real-world examples of savings offered through disk compaction, but in the field, customers were not necessarily blown away by this feature as their profiles were relatively new, and disk compaction did not always run on profiles as intended.

Writing profiles to primary and secondary storage locations with FSLogix Cloud Cache was a particularly refreshing feature when we first read about it. Profile consistency across primary and secondary DR sites has never been particularly a straightforward exercise in architecture and design, and hopes were high that FSLogix Cloud Cache was about modernizing this particular piece of profile management. Practically, the feature succeeds in writing the profiles to multiple locations, but the only evolution we have seen with this particular feature is in the product documentation, which is now updated in Microsoft documentation to exclaim:

• BCDR (Business Continuity and Disaster Recovery) events are rarely graceful. Depending upon the circumstances, user profile data may not be guaranteed to be intact.
• Users signing into session hosts in the failover region could experience data loss or even worse, container corruption. The situation amplifies the need to use storage platforms like OneDrive or SharePoint for critical user data.

Additionally, Microsoft documentation also mentioned that Cloud Cache is:

• An IO-intensive system can easily cause a network and/or storage bottleneck to the restored location.

On the immediate point above, once a primary region has recovered from a failure, FSLogix addresses the update of the primary storage target by completely copying the contents of the secondary storage location. In the event that you are copying to a primary storage region that is geographically separated by WAN connections, Cloud Cache will find your network and storage bottlenecks quickly and ruthlessly, consuming as much bandwidth and IOPS as possible. And we sadly have heard of more than one occasion of a primary storage region becoming overwhelmed and failing when it came back online due to unthrottled delta replication.

In large enterprises, it is only a matter of time until the most appealing recovery model for Cloud Cache is to have no profile recovery configured, and instead recreate the user profiles in a secondary storage location. Microsoft is already quite clear that we should not have any critical data stored within FSLogix profiles. For large enterprise customers in Azure, the case against Cloud Cache is further argued effectively when CPS Architects insist on Azure NetApp Storage with Premium Tiered storage for profile storage, the cost of keeping profiles hosted in a second region, as well as the cost of network ingress/egress charges to keep profiles updated (mind you it is worth customers doing the math on the last one, as it ultimately may only work out to a few Starbucks lattes per month and hopefully not a sticking point, but the general sentiment is customers do not like being nickled and dimed).

With the increased considerations for storage, IOPS, bandwidth to storage, and bandwidth between primary and secondary sites, it should be obvious that container-based profile management is often only as resilient as the bare metal that is supporting the solution. Customers who have the best success at architecting container-based profile management understand the following:

• Detailed XML Redirect files. The default size of a user profile in FSLogix is 30 GB, which is an obscene amount of storage when compared to a typical CPM created profile. Customers with a detailed XML redirect file and intimate knowledge of what is written into a user’s profile have success creating user profiles >1GB.
• IOPS Bottlenecks. In many environments, IOPS is the bottleneck that administrators will likely run into, and there are a number of ways a customer can achieve this:
  • Not understanding the IOPS requirements. Customers can be forgiven on this one, as in my opinion there is not great guidance on this. Chances are if you have researched this you have come across another Ferroque blog from EUI Hall of Famer, Zeljko Macanovic, Scaling FSLgoix Containers for Citrix VDAs with Azure Files. In this blog, Zeljko shares some estimates for IOPS, and in my engagements, these are good conservative estimates for the purpose of sizing storage for your profiles based on IOPS demands (provided by Microsoft):
    • Logon/logoff: 50 IOPS per user
    • Average Use: 30 IOPS per user
    • Steady State: 10 IOPS per user
    • Important to note: I strongly recommend that you measure the performance of FSLogix/CPM profile disks in your environment. As I mentioned previously, the above numbers are good conservative estimates, but I have observed measurements in customer environments wherein average IOPS clock in well under these estimates.
    • FSLogix User Profile – Average State: 7 IOPS
    • FSLogix Application Profile – Average State: 2 IOPS
  • Available IOPS. Your storage (or your customer’s storage) will only have a finite amount of IOPS that is available. If traditional storage is in scope, then IOPS sizing should be a relatively straightforward exercise. But if you are using cloud-based storage, we will need to be much more considerate of available IOPS. For example, if target storage is an instance of Azure Files, the advertised 100,000 available IOPS with Premium storage will become available only when the full 100 TB of that instance has been provisioned.
• Cloud Storage Considerations. Sticking with Azure as an example here, if you are scoping proper enterprise workloads crossing the 5,000 to 10,000 user threshold, you should be scoping Azure NetApp Files and Premium Tiered Storage for hosting container-based profiles.
• Resiliency – Container-Based Profile Management. The profile management solution that is implemented is ultimately only as resilient as the storage that is used to host the profiles. Administrators should be encouraged to continue to configure appropriate folder redirection to ensure critical files and folders are available and backed up on appropriate file services.
• Lack of new features in FSLogix. With respect to FSLogix, the evolution of the product and the introduction of new features has plateaued. The last significant feature update was Disk Compaction which became available in release v2010. Customers seemed to struggle to get this enabled and tuned to the point where they experienced the intended results, and Microsoft has released two hotfixes to stabilize the release. In fact, the argument can be made potentially that we are losing more features with FSLogix than we are gaining, see Microsoft documentation FSLogix feature deprecation. RIP FRXTray Utility.
• Cloud Cache Recovery. Writing to two different storage locations has obvious benefits, but if you experience a disruptive storage location event, Microsoft’s own product documentation has been updated with some important disclaimers.
  • Active-Active Storage Locations: Word for word from Microsoft, “BCDR failover events are never graceful, and depending on the circumstances of the event, user profile data isn’t guaranteed to be intact. Users who sign-in to Session Hosts in the failover region could experience data loss or at worse container corruption. This situation amplifies the need to use storage platforms like OneDrive or SharePoint for critical data.” Profile corruption could come in many forms, including corrupt .OST files, and I have recommended to customers in this particular scenario the use of Cloud Chache to be removed and profiles to be redirected to appropriate containers at each data center. Certainly, the time to create a new profile or update an existing profile is a preferred scenario to experiencing storage and network bottlenecks while FSLogix profiles are replicated to the recovered storage location.
  • Customers with Active-Active Cloud Cache configurations to storage locations hosted in geographically separate storage locations will notice that when a failed storage location is recovered, significant IOPS and bandwidth are consumed on storage and links as the FSLogix profiles from the secondary location are copied in their entirety (not just the deltas) to the recovered storage location. In this case, the Cloud Cache feature configured for Active-Active locations between data centers/storage locations seems to be an inappropriate configuration as the recovery scenario is particularly bumpy. In short, Cloud Cache appears better suited for configuration within a single data center across multiple storage locations where high bandwidth is available.
• New CPM features from v2203 to v2311 do not appear widely recognized currently in the EUI community. This is really the primary motivator for this blog, and I encourage you to join me, and key members from the CPM product management team, Wayne Liu and Liang Yang, on our upcoming webinar, January 18, 2024, EUI Innovations in Container-Based Profiles – CPM 2203 and Beyond. In summary, life in container-based profile management has been just okay. We are integrating M365 apps into non-persistent VDI with FSLogix, but the EUI community in general has not been blown away by any innovation with the product in the past year. The introduction of disk compaction was an interesting feature introduced by FSLogix in October 2022, but otherwise, new features with that product have been hard to come by.

Meanwhile, Citrix has quietly been adding features to CPM since 2019, quietly adding functionality and features to the point where we need to seriously rethink our leading practices in profile management.

From File to Container-Based Profile Management

Citrix first started to offer container-based features in CVAD 7.18, with the introduction of .OST redirects and search index files. From v2203 to v2311, CPM has seen significant new features that in the opinion of many in the EUI community, will make CPM the preferred profile management solution from 2024 onward. Let us take a moment to summarize the key features and releases that have modernized the product:

CPM General Updates

The CPM 2311 release marks the bookend of an 18-month window wherein Citrix has aggressively added features to the product, which requires us to completely rethink profile management planning and strategizing to which we have grown accustomed over the past five years.

With these updates, Ferroque considers CPM to have been sufficiently modernized wherein it is now our preferred profile management solution in both file and container-based profile management solutions.

CPM Hybrid Profile Management

Advancements from the Citrix product team have given the EUI community its first truly Hybrid Profile Management Solution. The combined technology advantage offered in CPM now offers flexibility to apply a best-fit profile solution, regardless of the use case.

• File-based profile roaming
• Container-based Profiles

Profile bloat is now addressed not only with careful dissemination of exclusions applied to the AppData folder in the user’s profile but also via many storage features, mainly disk compaction and file de-duplication.

CPM Container Access Modes

Three access models are now supported in the CPM Container, one method is exclusive to CPM, called Multi-Session Writeback Access

1. Multi-Session Access.
   • CPM allows only one session (in read/write mode) to write changes to the container. Changes in other sessions (in read-only mode) are discarded upon user logoff.
   • 
2. Multi-Session Writeback Access is exclusive to Citrix Profile Management
   • Enabled via CPM policy setting “Enable multi-session write-back for profile containers”.
   • All user sessions are available to write to the user store.
   • 
3. Exclusive Access
   • Enabled via CPM policy “Enable exclusive access to VHD containers.
   • Only one user session is allowed to access a user’s .VHD container and writes data change into the .VHD container directly.
   • 

CPM Smart Selection of Best User Store

This feature is available in CPM in both file and container-based solutions, defining multiple storage locations for a user’s profile to be stored. The closest comparable feature in FSLogix would be Cloud Cache, and while both products offer the ability to write into multiple storage targets, how CPM operates is distinct.

• At user logon, CPM is connected to the primary site first and fails over to the replicated store(s) if the primary site is not available.
• Profile data is synchronized to both storage locations upon logoff.
• Policy settings are available to change the selection method of the primary store based upon:
  • The user store with the latest profile version.
    • User store with the earliest configured store in the list.
    • The store with the best access performance.

CPM Profile Migration Tool

Initially introduced in v2308, CPM offers a supported migration tool to smoothly facilitate the migration to a CPM user profile container from FSLogix, Citrix Roaming Profile, and Windows Local Profiles. Pre-requisites to use this tool include the setup and access to the Citrix user store, including the creation of appropriate Windows ACLs.

At the time of this article, the migration tool is PowerShell-based, but it is my understanding that a GUI-based solution is on the product roadmap.

CPM User-Level Policy Settings

Selfishly, I think this is one of the most interesting features in CPM, and the immediate use case I foresee is with a manufacturing customer who will be deploying Plant Floor PCs across a number of sites. Users primarily access desktops with generic accounts and launch sessions that are heavily governed by GPOs that govern profile management as well as internet browser functionality. These users also access published desktops with named accounts that allow them to launch sessions which enable access to the M365 OWA, and a wider ability to run web apps.

User Level Policy Settings would allow this customer the ability to accommodate both use cases into a single provisioned machine catalog while accommodating use case scenarios that call for different profile management strategies. Typically, profile management policies have been deployed using computer-based policies, which means our machine catalogs are dedicated to specific use cases (and profile management solutions).

While the use cases for this feature are admittedly narrow, the extra color provided here is important since my customer will carefully consider every opportunity to reduce their machine footprints wherever possible.

CPM Storage Management

With container-based profile management solutions, in-scope storage must be the subject of additional scrutiny, not just for the amount of storage required to host the solution, but also an expectation that other bottlenecks in IOPS and bandwidth must be considered as notable factors impacting the performance and resiliency of any container-based profile management solution. Given that increased dependence upon storage performance, any container-based profile management solution we are implementing should provide storage optimization features themselves. These features have been a long time coming, and we are relieved to see Citrix picking up the baton and driving this innovation.

File De-duplication

The functionality of this storage management feature has changed slightly. Initially introduced, identical files stored across multiple profiles were moved to a shared storage location. As of the v2311 release, file de-duplication has been supported in file-based profile management, and also within the CPM container as well.

This feature is enabled by policy and specifying file types to include and exclude from the scope of the de-duplication policy. The results of the de-duplication policy will be environment-dependent, with administrators strongly recommending customizing the de-duplication file settings based on the use case. Generally, Citrix recommends de-duplicating files that change infrequently, such as:

• Program Files that users install.
• Installer or image files that users download.

Files that are not recommended to be included in this policy include:

• Documents that users may update frequently and redirected already in CPM settings.
  • Documents*.elsx
  • Documents*.docx
• Document types with the potential to change frequently.
  • AppDataLocal*.dat

CPM Disk Compaction

Disk compaction is nothing new in container-based profile management. Unsupported scripts were offered by several people in the EUI community, including scripts offered by prominent members of the FSLogix and AVD product team, before FSLogix offered disk compaction as a fully supported feature in the v2010 release of the product.

Citrix’s take on this feature differs from FSLogix with two methods to kick off disk compaction on a user’s .VHDX file.

• Once the free space ratio in a profile falls below a certain percentage (default is 20%).
• The number of logoffs since the last compaction was performed (default is 5).

CPM Disk Auto-expansion

Granular settings are now available in CPM to assist in automatically allocating additional .VHDX space. Triggers can be customized for the following auto-expansion properties:

• Auto-expansion trigger threshold (%).
• Auto-expansion increment (GB).
• Auto-Expansion Limit (GB).

CPM Application Management

I find a couple of items particularly intriguing here. On App Access Control, I like the flexibility that we are going to have in how we apply the rules. I can think of one Citrix Enterprise Architect in particular who winces when any customer mentions they need/want to rely upon FSLogix App Masking to limit the visibility of applications.

Additionally, I always find the OD4B integration conversations interesting. How we hydrate and de-hydrate OD4B at an enterprise level has real consequences in large enterprise environments for underlying storage and needs to be considered.

From a reporting perspective, the potential value here for understanding the exact details of how container-based profiles are performing is vital in support of scaling that a particular workload or environment may experience. Details on .VHDX performance has been too difficult to come by, and I can think of several customers who would have benefitted from detailed reporting.

CPM App Access Control

App Access Control has been introduced to provide more granular control in how applications are hidden from users, machines, and processes, with single image management in mind.

Applications can be assigned by:

• AD and Azure AD users/user groups.
• AD, Azure AD, non-domain joined machines.
• Azure AD, and non-domain joined machine catalogs.

The App Access Control feature performs rule-based management, rules can be configured via:

• PowerShell – Rule Generator.
• WEM Tool Hub – Rule generator.

Once the rules for App Access Control have been generated, admins can deploy these rules via:

• Citrix Studio Policies.
• WEM Administration Console.

Native Outlook Experience

The big “need to know” items here are the support for concurrent sessions (mentioned previously in this article under CPM container access methods) and Citrix being the only profile management solution that offers multi-session writeback access where multiple sessions are allowed to write changes into a user profile.

Also of note is the Automatic Mode Switch feature. With Outlook, cached exchange mode is set automatically, when the application is launched. However, if the application becomes detached from the container hosting the .OST file, Outlook will automatically switch to Online mode. When the container becomes available again, Outlook automatically switches to Cached Exchange mode.

OneDrive for Business (OD4B) Cache Roaming

CPM redirects the OD4B cache folder into the CPM profile container. This IO-level redirection is transparent to OneDrive and ensures that OneDrive folders roam with users by default. The result is that:

• Cached files will be ready to use when users logon to new machines.
• Saves on machine local storage.

Concurrent user sessions are also notably supported now in the OneDrive container.

CPM Report Insight

As we move into 2024 and beyond, accurate reporting on container-based solutions is something that could potentially be very valuable. The resiliency and performance of container-based profile management solutions is only as good as the underlying storage upon which the solution depends. I would love to see CPM reporting detailed IOPS metrics for containers as well as individual profiles. This visibility in reporting would go a long way in assisting administrators and engineers in understanding the IOPS requirements that are being placed on storage solutions, when additional cloud storage (IOPS) needs to be provisioned, and when aging on-prem network storage can be retired in a dignified fashion.

Currently, insight into CPM container performance is available via WEM, with high-level reporting on overall metrics for:

• Disk usage.
• Mounting failure rates.
• Itemization of a large file list in the container.

Coming soon, administrators can expect additional functionality around:

• Scheduled data export.
• Granular-level insights for:
  • Specific users.
  • Specific time periods.

Conclusion

Citrix Profile Management is now the EUI leader in file-based and container-based profile management solutions. The release of CPM 2311 marks the bookend of an 18-month window wherein Citrix has modernized the product to the point where its functionality either meets or exceeds the performance offered by its closest competitor in the EUI space, Microsoft FSLogix.

Those familiar with my previous blog on profile management published in 2021 will recognize this as a complete shift in our takeaways at the time, wherein we expected that Citrix Consulting would begin to lead with FSLogix concerning profile management solutions in any given virtualization solution.

Ferroque’s preference for leading with CPM in architecting profile management solutions is based on the following;

• Product evolution. Throughout CPM releases of v2203 to v2311, CPM has added features that now exceed the comparable functionality offered in FSLogix. Comparatively, FSLogix appears to be deprecating more features rather than offering new innovative features.
• Storage optimization: In container-based profile management, the resiliency and durability of the profile management solution that is implemented is contingent upon the performance of underlying storage. With that, features that can optimize storage are a welcome addition to the product.
• Enhanced logging and reporting. Functional updates also include updates log collection and reporting to provide greater depth. If profile management requires troubleshooting, the additional depth in log collection and reporting will facilitate quick resolution of issues. Based upon feedback that Ferroque has received from Citrix regarding the product roadmap and discussions at PTEC in November 2023, I believe we will see additional highly relevant reporting becoming available in future releases. Comparatively, FSLogix features that will be deprecated in the coming months include the FRXtray, which more times than not has been my FSLogix troubleshooting tool of choice.
• Hybrid Profile Storage. CPM now appears to exceed the functionality offered by FSLogix, while maintaining the functionality the product is previously known to provide with file-based profile management, offering administrators a flexible option for accommodating various layers of a profile within a single product/solution.
• Supported avenues for migration. Customers will not be necessarily thrilled at the prospect of re-architecting their straightforward FSLogix profile management solution, but there are compelling reasons to complete this migration for more advanced configurations that (for example) are using Cloud Cache. For these environments, Citrix offers a fully supported migration tool to migrate profiles from FSLogix, Windows, and earlier versions of CPM into CPM User containers.
• DR events under User Store Selection. It is hard to imagine a scenario wherein DR events and recovery from them can be handled worse than FSLogix currently does with Cloud Cache. In container-based profile management, the ability for a product to write to multiple storage locations is a must-have, as is the ability for the profile management product to be able to respond to changes in the availability of defined storage locations. If CPM offers customers any marked improvement over what they are experiencing with FSLogix, this in itself could justify a migration to CPM.
• Initial experience deploying updated CPM features in the field. Ferroque has already actively engaged with customers to deploy these bits in production environments. I’d like to take a moment to give a special shout-out to my team member Steve Szuster, who in particular is actively engaged with customers and assisting them with realizing performance gains in CPM container-based solutions.

Finally, thanks very much for reading. If you have any thoughts or opinions you would like to share on profile management in general, FSLogix, CPM, or any additional third-party profile management tools, I would love to hear your comments. More importantly, your comments will be particularly visible to the CPM Product Team, who are particularly hungry for your feedback!

If you haven’t already, please feel free to check out our EUI Innovations in Container-Based Profiles – CPM 2203 and Beyond webinar, to be hosted on January 18, 2024, with contributions from the Citrix Profile Management Product Team!