Citrix ADC–Tips From The Field
As a consultant, I get the opportunity to work with companies of all sizes and across many verticals. The needs of customers can vary greatly, from simple ICA Proxy implementations of Citrix Gateway to complex multi-datacenter, cross-continent geo-load balancing. Despite this, I thought it would be helpful to share some tips that come up often working in the field.
How to Tell If a Policy Is Being Hit
One of the more frequently asked questions for troubleshooting configuration I receive is “Why isn’t my policy working?” Of course, this question can have many answers, but more often than not there are a few steps that can quickly answer that question for most cases.
To check for policy hits, connect to the NetScaler via SSH, then enter the shell by typing “shell”, followed by the command below.
nsconmsg -d current | egrep -I responder
Repeat your test and you should be able to see the output when the policy is hit. If you receive no output, the policy did not match.
“Why isn’t my [responder/rewrite] working on this Content Switching vServer?”
This question often comes up when users try to transform URLs or want to implement some sort of redirection via responder. For the answer, we turn to “Processing Order of Features.”
This graphic, which can be found on the “Getting Started with NetScaler” page, depicts the processing order of a request and response from a client. In the center, we can see the flow that starts from the client request. If we follow it down, we can see that Content Switching gets matched before Responder. The action sends it to a Load Balancing back vServer. In those cases, you can bind the policy to the Load Balancing vServer instead to achieve the desired result.
If you need to capture traffic on the NetScaler for troubleshooting, you can use the nstrace utility. This is available from the GUI or command line. To access the trace from the GUI, navigate under “System” to “Diagnostics.” You will find “Start New Trace” under Technical Support Tools. If your firmware version is 11.0-66.11 or higher, you can capture SSL keys so that SSL traffic can be decrypted and inspected. You may need to disable SSL session reuse on the vServer.
start nstrace -size 0 -filter "CONNECTION.LB_VSERVER.NAME.EQ(\"myVserver\") || CONNECTION.SVCNAME.EQ(\"serviceName\")" -link ENABLED
“My Config Is Gone on My VPX!”
This scenario usually plays out in a similar way. By the time the call for help comes in, things usually aren’t in great shape, unfolding like this:
- Noticed service(s) are not working after some outage or maintenance that resulted in a reboot
- Log into NetScaler to Check
- See things are down or that the config doesn’t look right
- Try to change a few things
- Instinctively save config more a couple of times
- Decide to reboot
- Forget to uncheck “Save configuration” during reboot
- Call for help
By this point, often the ns.conf and ns.conf rotations have been overwritten a few times with a config that is missing a lot of configuration around SSL. It’s not a fun situation to be in, and it's something that I hope Citrix can improve in the future. But why did much of the config “vanish?”
VPX licensing is based on MAC address. If the machine did not have the MAC address for the licensing NIC set to Static MAC allocation, there exists a possibility where it can change for a variety of reasons (usually related to migration technologies such as vMotion/XenMotion). When the NetScaler reboots, it will fail to load the license and disable unlicensed features, which include SSL. If an admin panics and saves the config a few times, they’ve overwritten their local backups of the ns.conf. You can use NetScaler MAS to ensure you have configuration backups taken at regular intervals to restore.
If find yourself in this situation, the best advice is as old as IT itself: Don’t panic and remain calm. Shutdown the VPX without saving the config, then edit the NetScaler MAC address in the Hypervisor. When the VPX completes booting, the license is loaded successfully with the full configuration.
These are just a few of the most common issues I help customers with, but this is by no means an exhaustive list. Citrix ADC is an extremely versatile solution capable of solving very complex business challenges for delivering applications for your organization. Feel free to share any tips you may have in the comments.