My First Time: Citrix Machine Creation Services (MCS)

Introduction

So today is my very first time: After years of MCS virginity, I decided it’s finally time to ditch the little farms and try out good ole’ Citrix Machine Creation Services. Over the last ten years, I almost exclusively installed small deployments. The bigger ones have about 150 concurrent users. All are built upon XenApp 6.5 or XenApp 7.6+ with static persistent virtual machines. I always told myself that static persistent virtual machines, together with fully automated patch management (for example: PDQ) are enough. And this is still true, because the maintenance effort is virtually nonexistent. But, it really bugs me that I’m not equally familiar with at least one of the provisioning methods. You might ask why I don’t try to learn PVS instead. Well, the simple reason is that my stomach tells me not to. The more valid reason is that MCS is included in every XenApp license and doesn’t require additional infrastructure. And additional infrastructure is always a really big topic for the customer.

Twitter

This blog post won’t be a classical "HowTo" guide, but more of a report on my journey to help me keep track of what I do. Maybe others suffer the same knowledge gap and are interested in my findings and the path I take.

Upsides and Downsides from a Beginner's Perspective

Before this project, when I had no real MCS experience, I always thought about the following points. Even if they bring no real content to this blog post, I would still like to write them down before we start. You may skip this part if you like ;-)

Upsides:

  • I could easily change the count of the XenApp VDA by adding more clones.
  • The cloned VMs would be 100% identical, compared to the static VMs I use today, which get unequal over time.
  • I get some kind of VM or generation versioning through snapshot management, if done correctly.
  • MCS RAM cache can help with write IO.

Downsides:

  • I have no real plan for how to handle Windows, Office and Application Updates. Since MCS-cloned machines are read-only, I think each change I have now fully automated through PDQ Deploy would be lost–after a weekly reboot, for example.
  • Does this mean, instead of (nearly) no work today, I would have to roll-out new updated snapshots 2-4 times a month manually, to keep the same level I’m used to?
  • What about updates on weekdays? Every now and then, customers decide to change things on weekdays. Or, a new Java version breaks a browser application and requires an immediate update. This will be a lot harder with non-persistent VMs + a catalog update always requires a reboot, I think.

Initial Situation

My company’s test environment has the following pre-condition:

  • VMware vSphere 6.5
  • Two XenApp 7.15 Delivery Controllers
  • Different static persistent Windows Server 2012 R2 & 2016 XenApp VDA
  • An up and running, fully functional Active Directory Environment based on Windows Server 2016

Getting started

Create a Master Image

  1. Install a fresh Windows Server 2016 Datacenter virtual machine.
  2. Install Hypervisor Tools -> VMware Tools:
     setup64.exe /S /v"/qn REBOOT=R"
  3. Fully patch the OS (for example with ABC-Update and automated reboots):
     ABC-Update.exe /A:Install /S:MSUpdate /R:10 /Log_Append:c:\WSUS.log /Exit:Restart
  4. Install the Citrix XenApp Virtual Delivery Agent, for example:
    Desktop-Experience
    import-module servermanager
    Add-WindowsFeature Desktop-Experience
    Reboot
    RDS
    import-module servermanager
    Add-WindowsFeature RDS-RD-Server
    Reboot
    VDA Setup
    VDAServerSetup_7.18.exe /quiet /components vda /controllers xa-wms2016-01.int.anaxco.de,xa-wms2016-02.int.anaxco.de /enable_hdx_ports /enable_hdx_udp_ports /enable_framehawk_port /enable_real_time_transport /enable_remote_assistance /virtualmachine /optimize /noreboot /logpath C:\
  5. Install the Citrix Workspace Environment Management Agent:
    WEM Agent
    Citrix Workspace Environment Management Agent v4.04.00.00 Setup.exe /s /v"/qn /norestart"
    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe update
    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe eqi 3
  6. Install the Citrix Connection Quality Indicator (CTX220774):
     msiexec.exe /i "CitrixCQI.msi" ALLUSERS=1 /qn /norestart /log output.log
  7. Install Base Image Script Framework (BIS-F) 6.1.0 -> Documentation
    setup.exe /SILENT

     

    1. Import and configure ADMX; example (click on images to enlarge):
      BIS-F GPO
    2. Provide 3rd party dependencies–I decided on:
      1. Delprof2
      2. Citrix Optimizer
  8. Apply additional OS optimizations based on your personal style.
    An excellent source for for this topic is: Carl Stalhood
    I will name a few examples:
    1. Disable Hotplug of VMware VMXNET3
    2. Remove VMware Tray Icon
    3. Black Screen on Windows Server 2016 / CTX225819
    4. Set UseProfilePathExtensionVersion registry value to 1
    5. Server 2016 desktop icons flickering @ discussions.citrix.com
    6. Edit pagefile.sys size
    7. Check NTFS permissions on C:\
    8. etc.
  9. Install some software, for example:
    1. 7-Zip
    2. Adobe Acrobe Reader DC Classic
    3. ASG-Remote Desktop
    4. (ControlUp Agent)
    5. (ESET AntiVirus + Agent)
    6. ESTOS ProCall CTO Client
    7. (fslogix)
    8. Google Chrome
    9. Greenshot
    10. MailStore
    11. Microsoft Office 365 ProPlus
    12. Mozilla Firefox
    13. Notepad++
    14. PutTTY + KiTTY
    15. TeamViewer
    16. VMware Remote Console
    17. VMware vSphere Client
    18. WinSCP
    19. (Zabbix Agent)
  10. Get your Computer and User Group Policy Objects in place with everything you need. For example, remove typical lock-down policies from the Master during creation and later maintenance:

     GPO

  11. Execute BIS-F to seal and shutdown your Master Image:
    Execute BIS-F

    Hint: I had several problems with that step because of very strict Anti-Virus policies. I had a great discussion at the BIS-F Slack with Matthias Schlimm about this problem. We narrowed it down to some AppLocker-like behavior, but it turned out we had some ESET policy in place, which prevented child processes from PowerShell.

    Troubleshooting with Matthias Schlimm

The Citrix Studio Hosting Connection

  1. Start Citrix Studio and create a new Hosting connection. I will refer to this example for a VMware vSphere 6.5 Cluster:
    Citrix Studio Hosting Connection
  2. Trust the self-signed certificate if necessary:

    Trust the self-signed certificate

  3. Set up your storage as desired, consult the storage administrator if you need help. IMHO this decision is not a Citrix Administrators Task:

    Storage Management

  4. Select storage LUNs as desired:
    Storage Selection

    Hint: As I had to learn the hard way, selecting all LUNs available wasn’t such a good idea, and your storage administrator won’t be very pleased. As it turns out, MCS deploys a clone of your master image on every LUN selected. In my case, I deployed the VMDK to 17 LUNs while I only intended to get used to MCS with two VMs:

    A hint from Jarian Gibson

    Turns out MCS needs a read-only disk on every selected LUN, independent from the VM count. Plan and size accordingly:
    Source: Citrix Product Documentation – XenApp Current Release

    MCS creates a full copy of the snapshot and places the copy on each storage location defined in the host connection.

    In the last chapter of this blog post, is a description about how I handled my error.

  5. Select the VLANs that will be available when you create the machine catalog. This setting won’t be used until the catalog is be created. For example:
    If you select three VLANs and plan to later provision VMs with one NIC, you have to select one of these three VLANs for the single NIC. The other two won’t be used. Consult the network administrator if you need help:

    Network

  6. Confirm the summary dialogue.

The MCS Wizard

  1. Login to your Hypervisor and create a snapshot manually. The VM should already be powered off, after you executed the BIS-F script. In my case, this is the Flash Player based vSphere WebClient 6.5:

    Create a VMware Snapshot

  2. Create a new Machine Catalog in Citrix Studio:

    Create Machine Catalog

  3. Because I mainly work with XenApp / RDSH / Server OS or whatever you might call it these days, I select Server OS:

    Operating System

  4. We set up a VMware virtual machine as our master image, so we have a power managed virtual machine. And as this guide is about MCS, we won’t select PVS. Always keeping things simple:

    Machine Management – Power managed – Machine Creation Services

  5. From the list of VMs, we search our powered off master machine.
    Important: In step one we created a snapshot manually! You could select the virtual machine itself as a source, and this will work without any problems. But you will lose control over your image management! If you let Citrix Studio do the work for you, all the snapshots created will have the same name and description.
    Source: Citrix Product Documentation – XenApp Current Release

     

    If you selected a master image (rather than a snapshot), MCS creates a snapshot.

    If you do it manually (or maybe scripted sometime later?) you will have control over the name and the description, which brings you to some form of MCS golden master image versioning. Always keep track of those versions and the changes made, so you know exactly where to go, if you ever have to roll back a catalog!
    Select Golden Master Image Snapshot

  6. Select one of the VLANs you configured in the Hosting connection for the cloned virtual machines:

    Select Network

  7. The next page is very important, you have four values:
    Configure a cache for temporary data on each machine
    1. Virtual machine count
      The machine count is self-explanatory, just set the number of XenApp workers you need and you’re set!
    2. Total memory for each machine
      This depends on your workload. There is no recommendation here other than: Test! Test! Test!
      Either do it manually, or use tools. For example, you could ask your monitoring administrator (#Zabbix) to provide you with hard facts about your workload. Or use some industrial standard tools like LoginVSI to simulate a real workload and get values for a proper sizing. But I can at least provide you with a link to an interesting blog post about sizing:
      Daniel Feller: Sizing Windows 2016, Windows 2012 and Windows 10 Virtual Machines
    3. A temporary RAM cache for writing operations
      With the release of XenApp 7.9, Citrix introduced a new feature for MCS, previously only known to PVS: A memory writing cache. This cache handles all writing operation prior to writing them to the temporary disk. This can deliver a serious IO reduction to your storage. The amount of RAM you configure here, gets cut from the “Total memory for each machine” you defined in step 2. So add more total RAM according to the RAM cache disk size.
      You can find a lot more information about this topic in the links I collected in step 5.
    4. A temporary Disk as overflow for the RAM cache
      The memory RAM cache disk is limited (in most cases). So lets say you allocate about 2-6 GB memory cache. Depending on your workload, this could fill up sooner or later. With a small or no overflow disk you will get a Bluescreen #BSOD very fast. The OS tries to write, runs out of space and crashes. So your overflow disk is there to step in, if the RAM cache can’t handle the write requests anymore. So if you would like to use the RAM cache feature, never use it without an overflow disk. A typical suggestion is, that the disk should be at least the size of the free disk space of your VDA worker.
      You can find a lot more information about this topic in the links I collected in step 5.
    5. Additional Information about MCS storage optimization
      https://www.citrix.com/blogs/2016/08/03/introducing-mcs-storage-optimisation/
      https://www.controlup.com/blog/everything-you-need-to-know-about-the-new-citrix-mcs-io-acceleration/
      https://www.jgspiers.com/machine-creation-services-storage-ram-disk-cache/
  8. Now the MCS cloned machines need to have computer accounts in Active Directory. Here you can select if new or existing accounts should be used. Additionally, you choose the OU where the computer accounts will be created. Important here is the naming scheme. You can see my example choice in the screenshot:

    Active Directory Computer Accounts

  9. Finally, you choose a name for the Machine Catalog. As soon as you click finish, MCS will start its work:

    Summary - Machine Catalog name

  10. MCS at work in Citrix Studio:

    https://www.mycugc.org/media/vlsztbvl.png

  11. Screenshots from VMware vSphere Client 6.5 (HTML5) during the Machine Catalog creation:

    VMware Machine creation progress

    VMware Machine creation progress
  12. After the process finishes, additional offline VMs will be available for use.

    New virtual machines in vSphere Web Client

  13. The machines will stay offline, until Citrix Studio tells them to start up. For example, after they are assigned to a Delivery Group.

Delivery Group

  1. Now that we have machines available, it’s time to create a Delivery Group, so we can access our work from a user's perspective for the first time. Select Create Delivery Group from within Citrix Studio:

    Create Delivery Group

  2. Select the previously created MCS catalog and enter the number of machines you want to use. Normally this is equal to the number of virtual machines you created:

    Select Machine Catalog

  3. Select a User or a security group for the Delivery Group if desired:

    Select Users

  4. Publish different applications if desired:
    (I presume you know what a published application is, how and why to use it etc.)

    Create published applications

  5. Create a published Desktop if desired:

    Create published Desktop

  6. Give the Delivery Group an appropriate name:

    Summary - Delivery Group name

  7. And done! By now your previously offline MCS VMs should boot up. After they run through sysprep and BIS-F (check the log files in the UNC path you configured!), the Citrix VDA will register itself with the Delivery Controller(s), and your newly-created Apps and Desktops will be available through Citrix Storefront!

Update Machine Catalog

After our first successful MCS roll-out, it’s time to talk about updates. I personally have no real practical experience with Machine Catalog Updates. My knowledge here is more theoretical, based on a lot of reading and many discussions on Slack and community meet-ups. So there are several challenges to solve in my opinion, just to list a few that come to my mind:

Cons

  1. Compared to static persistent Server OS VDA RDSH VMs, all changes that happen during the VMs up-time will be “lost” after a reboot. That is because the cloned MCS disks are read-only. Your classical Software-Deployment Auto Schedules would still work in some way, like a Java update for example, but as soon as you reboot, that will be reset.
  2. Without Microsoft System Center Configuration Manager (SCCM), no scheduled roll-out seems to be possible. This is a big disadvantage in my opinion. It seems like PowerShell would be the way to go. The Citrix PowerShell CMDlets seem to offer advanced features to control MCS catalog updates. If this could be solved, any Task Scheduler could solve the timing problem. 
    Rollout Strategy

    EDIT: I got informed by Jarian Gibson, that the wording in the Rollout Strategy is not the best, or may be just plain wrong:
    Troubleshooting with Jarian Gibson

    This seems to be true, so my initial description is wrong. As detailed in the Citrix Discussions post, you can schedule a rollout without SCCM. The trick is that the next VDA reboot has to be initiated through Citrix Studio. This can also be achieved through a scheduled reboot inside the Delivery Group. More on this later.
    Source: https://discussions.citrix.com/topic/368238-mcs-can-i-schedule-a-machine-catalog-update/

Pros

  1. All changes that happened during the VM's uptime, will be lost. From another perspective, this can also be considered very positive. As VMs will stay very clean and will never differentiate from each other.
  2. In addition to Point 2 from the Cons: You can work on your master image during work hours, without harming anybody. You can then rollout the image update, but decide for yourself when to initiate the reboot through Citrix Studio.

The Update Itself

After the Delivery Group was used for some time with success, we have to bring all those pesky updates that queued up into the master image.

  1. Boot up your master VM, if it’s offline.
  2. Log in to your master machine via console or RDP.
  3. Apply changes as desired. For example: Windows Updates, Office Updates, Application Updates, Runtime Updates, Citrix VDA Updates and so on.
  4. Run BIS-F to seal and shutdown the master image.
  5. Login to your hypervisor and manually create a snapshot. Give it a meaningful description or some kind of versioning.
  6. Select “Update Machines” inside Citrix Studio:

    Update Machines

  7. Select the Machine Catalog you want to update:

    Affected Delivery Groups

  8. Select the Snapshot we manually created and prepared beforehand:

    Select updated Master Image Snapshot

  9. Select a Rollout Strategy. As I mentioned before, this is currently a huge bummer for me, as I have no environment that makes use of Microsoft System Center Configuration Manager (SCCM).
    This means the only Rollout Strategy I have is: Immediately.
    In my understanding, this would translate into: Work each and every Sunday. And I think we can agree that this I a no-go.
    For now, I have to leave this one unsolved, because the purpose of this blog post is to test, try, learn and study. The obvious answer would be to utilize Citrix PowerShell cmdlets, scripting skills and a Task Scheduler. But this won’t be covered in this blog post.

    Rollout Strategy

    Important: As I mentioned earlier in this blog post: This is wrong!
    The description from Citrix in this Wizard is not good. If you select “On next shutdown” the whole rollout process will start and work out. The trick is, after the rollout on the hypervisor finished, the reboot has to be initiated through Citrix Studio. More on this topic in a separate upcoming blog post.

  10. Finish the wizard:

    Summary


My personal Key findings

  1. Use Citrix Optimizer, not VMware Optimizer, or take your time to study VMware Optimizer real good.
  2. Use BIS-F all the time, it won’t hurt and there is no reason not to.
  3. You may want to create an OU structure in your Active Directory, that lets you assign different GPOs to your Golden Master Images, then to your active VDA Worker. For example you might want to handle Microsoft Office and Windows Updates different on those machines.
  4. Don’t let Citrix Studio create the VM snapshots for you. Make snapshots yourself and give them meaningful names and a description.
  5. Keep an eye on your snapshot chain. Don’t let it grow too long. Some time in the future you will have to delete old versions.
  6. Make yourself familiar with the concept of MCS rolling catalog updates. Keep two catalogs up and running and roll out catalog updates rotationally. This is the fastest way back, if you discover business critical errors in your updated machines.
  7. Maybe find a workaround for MCS scheduled machine roll out (no SCCM available). <- Proved wrong. Scheduled rollout is possible via Citrix Studio and PowerShell.
  8. I really need to take a close look at Microsoft Deployment Toolkit (MDT) -> Check out @xenappblog Automation Framework for example.

Upcoming content

Thank you for reading one of the biggest blog posts I've ever created. This guide covers the basics to get started with Citrix Machine Creation Services. But there are future topics to get covered.
Blog posts that might follow could be:

  • Citrix Machine Creation Services: Rolling catalog updates
  • Citrix Machine Creation Services: Rollback Machine Catalog
  • Citrix Machine Creation Services: Schedule Machine Catalog updates
  • Citrix Machine Creation Services: Automate the master image update process
  • Citrix Machine Creation Services: Automate the catalog update

Acknowledgments

A very special Thank You to Jarian Gibson, who helped me a lot with this blog post through the Citrix CTP mentoring program for Citrix CTA. Without him, the information in this blog post would be of less quality.

I would also like to thank René Bigler for proofreading this blog post as he has used MCS a lot longer than me!

2 Comments
4 Likes

Nice compilation of experiences

September 22, 2018 01:07 AM by Tobias Kreidl

This journal that elucidates your "foray" into MCS is really nice, not only because of the wealth of content, but because it points out various stumbling blocks and how those were overcome! :-) Mission accomplished. Thereby, it also points out how the user community can be such a tremendous help in the process. We truly all learn so much from each other and the sharing of experiences enriches us all.

There is a Flaw in MCSIO that you need to be aware of, causing Really bad preformance

October 3, 2018 11:52 AM by Eric Hosmer

 

FYI - Passing on an info from a Citrix forum thread I have been following for a while.    And can back up with my own discovery the hard way.  

“XD MCS I/O Is Crazy Slow!”

https://discussions.citrix.com/topic/395312-xd-mcs-io-is-crazy-slow/

When I migrated from my old SAN to the Pure SAN in June of 2018.  Post migration I still saw erratic Performance with all flash storage using MCSIO.   MCSIO should make even VM's on Flash Storage Better.    But I discovered that MCSIO that “Memory Allocated to Cache (MB):” and “Disk Cache Size (GB):” make life very bad for every one users and admin's frustrated by slow login and anything else affected by bad disk IO due to MCSIO bug.

 

 I was able to identify that use of MCSIO “Memory Allocated to Cache (MB):” and “Disk Cache Size (GB):” were the issue.   And unchecking these items fixed the bad preformance.    Later a user in metioned Citirx he had confirmed the bug with them and was given a private fix. 

What really burns is this bug has been there for almost 2 years affecting my users.     Even though the Citirx Article specifies versions XenDestop 7.15, this issues is found in all versions of XenDesktop 7.12 to 7.18.    

 

vazques29 |  Enthusiast |  18 | Members | 104 posts

 

Performance issue with Machine Creation Services Storage Optimization (MCS I/O)

http://support.citrix.com/article/CTX234814?_ga=2.103984064.1149560099.1538506080-1934021296.1525786457

 

Do not use Citrix MCS Storage I/O Optimization for now

https://www.linkedin.com/pulse/do-use-citrix-mcs-storage-io-optimization-now-sina-torabi

 

 

 

Please login to add your comments.

Recent Stories
My Favourite Overlooked vs. Hidden Citrix Workspace App Features

How Citrix Supports Digital Transformation In Education

Be a #CitrixHero – DJ's TOP Tip: OS Optimization