Slow Citrix Web Interface 5.4 on ASPX pages (or how to load test your WI)

We've been having issues with our Citrix Web Interface (5.4.2) with it cratering to the point that you cannot even get to the explicit logon page, stuck on SilentDetection.aspx or another aspx page.

One Moment Please...  This screen takes minutes to resolve

So what could be causing this?  More often than not, it's a broken XML broker that isn't responding in a reasonable amount of time.  But XML brokers don't come into play with 'Explicit Logon', especially if you can't even get to the logon page.  In our scenario, we have two production Citrix Web Interface servers which are load-balanced in a round-robin scenario.

So I theorized that what's happening is we are experiencing a load issue.  That is our server cannot handle the number of connections that we our clients and PNA agents are producing.  How can we prove this out?  How many connections can our web interface accept?

To test out what the maximum number of connections our web interface servers can sustain I came up with a plan.  We need to thoroughly exercise the ASP.NET scripts that Citrix has put together for the web interface.  To do that we need I decided that logging in via explicit logon, skipping client installation (so client detection occurs) and then application enumeration.  This sequence hits the following ASPX pages:

The timeline of processing the aspx pages of a unloaded server looks like this:

The initial silentDetection.aspx's take the majority of the processing time, about 4.3 seconds.  To create the ability to simulate this load, Microsoft has a tool called "WCAT", or the Web Capacity Analysis Tool.  This tool allows you to setup "scenario's" which are a sequence of HTTP actions (GET/POST/etc) which simulates each step of the sequence above.  To record this sequence for WCAT, you need to install Fiddler 2 then install the WCAT scenario creator.  There are numerous WCAT scenario creators, but I used this one.  To create the scenario do the following:

1) Launch Fiddler 2 and click 'Launch'

2) Go through your login sequence on your web server until you hit the page with application enumerations.

3) Go back to Fiddler 2 and at the bottom of the screen on the left in the black bar, type "wcat reset"

4) Go Edit > Select All.  Then in the black bar, type 'wcat addtrans'

5) Lastly, type 'wcat save'.

This will create a fiddler.wcat file in your C:\Program Files (x86)\Fiddler2 directory.

You can now open this file as a text file to see the scenario you just created.  One of the issues with this WCAT scenario creator you will need to fix up some cookie values.  Some cookie values are captured exactly as described, which has some issues as double quotes need to be escaped in the scenario file.

So, for example, this line:

value = "WIUser="CTX_ForcedClient#Off~CTX_LaunchMethod#Ica-Local~CTX_AuthMethod#Explicit~CTX_CurrentFolder#%7bAllResources%3d%5c%7d~CTX_ViewStyle#%7bAllResources%3dIcons%7d"; WINGDevice="CTX_DeviceId#WI_YxJYRT6LZxvCnDWTF"";

Won't work.  You need to escape out the double-quotes with a back-slash so it looks like so (the backslashes are bolded and underlined):

value = "WIUser=\"CTX_ForcedClient#Off~CTX_LaunchMethod#Ica-Local~CTX_AuthMethod#Explicit~CTX_CurrentFolder#%7bAllResources%3d%5c%7d~CTX_ViewStyle#%7bAllResources%3dIcons%7d\"; WINGDevice=\"CTX_DeviceId#WI_YxJYRT6LZxvCnDWTF\"";

Once you've escape'd the double quotes, save the file as 'scenario.ubr'; you can now setup WCAT.  

Install WCAT on some test boxes or a single box (for the purposes of this demo I'm doing just a single box).

Copy over your scenario.ubr to the C:\Program Files\wcat folder.  I took the settings.ubr file from the Samples directory in the wcat folder and put it in the root of the wcat folder as well.  

You then need to start the controller and attach clients to this server.  Open a command prompt to the wcat directory.  The command line I used for the controller is:

wcctl -t scenario.ubr -f settings.ubr -s wsctxwi01t -c 20 -v 20 -w 300 -u 300 -n 100

the values are:

-c = # of clients to connect to this controller.

-v = # of virtual clients to launch per client

-w = warm up time before all clients are connected

-u = duration of testing

-n = cooldown period for clients to disconnect.

Since I'm testing from one computer I then used the following command to start all my clients:

for /l %A IN (1,1,20) do (start "" wcclient.exe localhost)

The 20 clients connect and start loading the server, following the sequence we recorded earlier.

If you want to monitor to see where your maximum user load is vs. maximum performance (e.g., near instant speed), go to your web interface server and create a data collector set with the following counters:

Without any clients connected, start your data collector, then start the WCAT load testing.

This is what I see with regards to our load:

You can see around 220 current connections the Requests in Application Queue shoots up.  At this point any additional requests to ASPX files goes into a queue that makes the website appear 'frozen' which it waits for the ASPX file to be processed and sent down to you.  Citrix attempts to 'hide' this page by loading the 'loading.htm' during ASPX processing.  This page is your 'One Moment Please' page.  If you show the Request Execution Time counter, we can see the longest time taken to process a ASPX page:

In my example here, the longest time it took to process a ASPX page was is the lighter pink line.  The longest time was 14 seconds.

The Request Wait Time is how long it takes of while you wait in the queue for the ASPX page to be processed.  So the dark blue line with the spike in the bottom right was a 16 second wait time, waiting for my page to start processing, and with the request execution time being 14 seconds this particular request may have taken 30 seconds in total.

With this information we can fairly accurately determine that we would want to keep the number of current connections to around or under 150 just to keep a safe buffer.  To do that, we could increase the number of web interface servers or break out our web server to different Application Pools.  Our Citrix web interface was originally designed with a monolithic application pool and our PNA website and other websites all run under one process.  In our situation, we could exceed our connection limit during times when Citrix Receiver checks in to enumerate applications at the same time as users trying to use the web site.  If we break the PNA website to a new application pool then we can actually double the number of connections we can process at any one time.  The limit appears to be the w3wp.exe ability to process ASPX pages.  By adding another application pool it actually creates another w3wp.exe process and if we max out the connections on one of the websites the other website/process will continue to work without issue.