Friday, December 18, 2015

Help! My Citrix application is running slow (Meditech, Citrix, Imprivata)

We have an application (Meditech) that users have reported as performing poorly in Citrix.  They were able to confirm the poor performance by comparing it to a desktop that had the same software installed.  I was brought on to try and understand these performance differences and why Citrix would be performing so much worse.  To measure the performance, the user took me to a screen where they held down a key on the keyboard to increment a counter in the software.  When holding the key down on the desktop the counter was quick and incremented at a steady pace, on Citrix it seemed OK at first but then slowed as time went on.   To illustrate these performance differences, I made a video:

So we definitely have a problem.  My first steps on troubleshooting this problem was to compare the differences between the desktop and the VM.  The VM had server processors but they were older, and at less GHz at 2.6GHz for the server.  The desktop had 3.4GHz processors with a newer generation.  If the processing is NOT latent sensitive then the faster processor's can make a difference, and the CPUMark had them at 1500 and 2100 (~40% difference).  At first glance, this seems like it could be the difference in our timings but it's still too drastic at ~5s vs ~20s, a 400% difference.  To try and narrow the difference I took the application to a completely bare server with no software on it what-so-ever and reran the test.  It completed in about ~5-6s.  The processor it ran on was comparable to the original server processor but ran at 2.7GHz instead.  The processor was not running at nearly enough speed to make up the difference, something else must be consuming those cycles.

At this point I procmon'ed the benchmark and came up with the following:

(PID 6660 is the process that the benchmark runs against)

Very obviously, the pattern of each key stroke on the Citrix server is present with the initial pattern highlighted:

About 400ms from each key stroke to when the next one is registered.  So what is delaying the 400ms?  From my experience, unexplained delays are CPU related.  I then looked at the process activity summary to see if I can find the bottleneck:

Again, very obviously we are seeing the CPU ramp up, and it can also explain the faster, initial iterations of the GUI as the CPU 'ramps' up and doesn't spike to the it's maximum.  None of the other graphs show any activity at the time of the benchmark so the suspect is highly on the CPU.  When hovering over the graph we see the CPU percentage.

This is a 4 core box, so ~25% equals one full core for a single threaded application.  Again, this points to the application being bottlenecked by the processor, but again, the difference is too large to consider just the CPU at this point.  We need to find what part of the process is consuming these resources.  Fortunately, ProcExp (process explorer) can help us determine what is going on within a process.

I started a new run and got properties of the process:

MGUI.DLL is consuming all of the CPU.  That is a DLL utilized by the application, clicking on 'Stack' gives us the hierarchy of commands being utilized by that thread.

From this, I can understand that ntoskrnl.exe, ntdll.dll are native Microsoft Windows functions, MGUI.DLL is utilized by Meditech, but what is ISXHook.dll?

Doing a search within the process shows that it's utilized by Imprivata, a single sign-on solution we utilize to try and increase user efficiency.  It works by 'screen scraping' to determine fields that it needs to populate with user credentials to try and speed up user logins.  Logically, this sounds like it could be causing the delay by screen scraping every time a key is stroked or a change is registered.  To confirm this, we need to remove Imprivata from the application.  Fortunately, it's hooked in by services that can easily be terminated.

I'm going to terminate everything that says 'Imprivata Inc.'

With the processes terminated I reran my test.  Using Process Explorer and getting properties on the process, immediately CPU usage went from 25% down to 15% at a peak:

And getting Stack information showed a much cleaner stack:

In conclusion, I may need to investigate into Imprivata to determine if I can reduce its polling rate or find some way of allowing it to 'stop' polling the CSMagic process *after* the accelerated login.  Its current settings (which I'm not familiar with, sadly) is not acceptable and causes a significant slow down.  Fortunately, the root cause has been determined and we can work towards a full resolution.

Friday, December 04, 2015

AppV5 - Sequencing an application that has Com+ Applications

AppV has a few weaknesses, one of them is Com+ Objects.  It doesn't like Com+ Objects and sequencing an application with them generally results in this error message:

The Resolution seem easy enough:
1) Try extracting and installing the COM+ component natively on both sequencing station and the client
2) Thoroughly test your virtual application package to verify functionality.

But how do you extract the COM+ components?

1) Open 'Component Services'

2) Expand until you find your Com+ Applications, right-click and select 'Export'

3) Click 'Next'

4) Enter a path to save your MSI then click 'Next'

5) Click 'Finish'

Since this application had two Com+ Applications I exported my second one as well:

To ensure they work, I took them to our XenApp Test server and installed them.

Both *appeared* to have installed correctly.  But this application requires the Com+ Application to run under a service account.  Checking the original install:

I can see under the 'Identity' tab that it is configured for 'This user:'.  I then checked the MSI install on the XenApp Test server and looked at the Identity tab:

It appears it defaults to 'System Account' even though I selected 'Export user identities with roles'.  To change the user account silently, I used this script supplied by the vendor (as a encrypted vbe):

This script changes the user account on the Com+ applications to use the proper account.  When browsing the properties of the components of the SFSQL object I saw it referenced the dll to it:

Fortunately, it seems these files are captured by the MSI export of the Com+ Application.

To test everything now, I must now create my AppV package.  Before doing the install I installed my prerequisites (the Com+ Applications) into the Sequencer so they are not captured by the package.

I ran this script to do my prereq install:

Now I startup the sequencer and do my sequencing first steps.

I selected a batch file I created for the install (per the vendor's initial instructions):

Once the install completed I saved my package and took it to my XenApp Test server and published the application and ran it.

No, this is not correct

For some reason my application was failing.  I double checked the COM+ Applications are installed and appear to be working, so I opened Procmon.exe to see if I can find where it's going wrong:

I've used procmon enough now to practically pull the old' Matrix, "blonde, brunette, red head".  Working from bottom to top, the message box is generated from the 'Text' (LanguagePack) back to the KernelBase.dll.  So we can exclude that has the cause, this is telling us *of* the issue.  So if we look at the lines preceding it, one of them must be causing our message.

Looking at the first keys, the OLE keys, I can compare them to my sequencer and they match exactly, so the likelihood of them causing our issues is none.

So we go back a little further to the AppID key and jumping to the *SUCCESS* location to look at it's value...

I am on a server called WSCTXAPP301T but the registry key for this value is pointing to the system I did my sequencing on.  I changed the value to the actual server name and did a registry search for "WSAPVSEQ07" to see if it appeared anywhere else.  It did in one other location so I modified the value there as well to point to the local server:

And tested my application again:

Success!  The login screen.  This is as far as I can take this app until I get credentials or a user to login and try testing for me.  So, for now, I can create a preflight-script to setup some prerequisites when the application is published.  I modified my preflight script to include the two registry values being updated:

Then I updated the Deployment_Config.xml file for this application:

Lastly, I logged onto a new, fresh, system and manually tested my XML file by publishing from the command line:

Using Procmon to monitor to ensure the prerequisites get installed:

"Exit Status: 0" --> Usually a good thing :)
Testing the application:
Looks good

Then testing removal and monitoring with Procmon:

Com+ Application are removed cleanly :)
So everything looks good and I can now add this application to our AppV Management Server and deploy it to the machines that need it.

Friday, November 27, 2015

AppV5 - When Application Compatibility Toolkit doesn't work

I have an application that embeds an IE window and generates a small HTML file for some content.  This application has some.....   interesting....  configuration settings that require a folder with "EVERYONE:F" permissions.  This folder cannot be in a path that contains spaces, cannot be a environment variable and the vendor recommends to put the folder on the root of the C: drive as C:\TMP.  To make it more crazy, when launching the application it will generate a folder under the C:\TMP that is locked to the %CLIENTNAME% (e.g., C:\TMP\ComputerName) and every login to the application generates unique files to that session, so technically, each folder underneath TMP is unique to the user.  The vendor actually recommends completely deleting the contents of the TMP folder DAILY.

So why not just store these files in %TEMP% instead?  Great question.  If the vendor made that happen I wouldn't have this lovely blog post.

(Just for a note this application utilized multiple folders on the root of C: as well, TMP being one, since we move our PackageInstallationRoot to a different drive, TMP isn't available on C:)

With that all said, what am I looking to solve?

Well, I don't want to be creating folders on the root of the C: drive as this application will be on our 'generic' PVS Citrix servers and minimizing the potential for pollution on the master image is a goal, having to create a folder and configure permissions is possible and I've done it before but it's not very clean or elegant; really if we're going to do that then we may as well pollute the C: drive.  We want to maintain portability so creating a C:\TMP folder on the master image prevents us from publishing this package on desktops or other systems in the future unless this is well documented requirement.

We've just learned a new trick though, we can try using Microsoft ACT to create a file path shim that redirects the path to a different one.  So let's do that...

The problem.
 I created the shim using the steps from this post.  (C:\TMP;C:\ProgramData\Microsoft\AppV\Client\Integration\D8E3DB68-4E48-4409-8E95-4354CC6E664B\Root\VFS\ProgramFilesX64\dlc11.2\TMP) I then launched the application and...

Huh.  That doesn't look right.
And it didn't work.  I then used procmon.exe to examine to see if it was reading the file correctly:
I see SUCCESS...
Procmon.exe is reporting that it IS reading that HTM file correctly.  So why isn't it being displayed?

I ran across a similar issue on another application and what I found was that the embedded component appears to be running *outside* the bubble.  Since the shim targets applications (in this case prowin32.exe) all reads from prowin32.exe are being redirected by other processes are not.  I suspect (though I have no proof), in this case, that somehow the IE component is breaking outside the bubble and so it's NOT getting redirected.

Can we force all programs to get redirected to the proper path?  Yes, we can.

Using symbolic links we can force any access to the directory to be redirected to the AppV package path.  AppV will then see this is a path within the 'bubble' and redirect *again* to your local profile where the AppV5 writes will take place.

I removed the shim and then executed:
Symbolic Link
mklink /d C:\TMP C:\ProgramData\Microsoft\AppV\Client\Integration\D8E3DB68-4E48-4409-8E95-4354CC6E664B\Root\VFS\ProgramFilesX64\dlc11.2\TMP

Tracing with procmon.exe now showed us this:

And the application now displayed this:

So the application is now working without any issues, all of these 'temp' files are being redirected to a location where they will not be saved between sessions so no cleanup is ever needed.  We need to add the mklink.exe to the DeploymentConfig.xml.


And we are now done.

Thursday, November 26, 2015

AppV5 - Using Application Compatibility Toolkit to solve issues

We have several applications that install folders in the root of the C:\ drive.  For most AppV5 implementations, this wouldn't be an issue but we modify our PackageInstallationRoot folder so the token {AppVPackageDrive} turns into the drive letter specified in the PackageInstallationRoot registry key.  For us, that's the D:\ drive.  This causes an issue because when you sequence an application to C:\ the application has an expectation for it to be there.  Ideally, AppV takes care of that by the use of the tokens, but this breaks down when applications are hardcoded.

So far, we've been able to work around these issues by using junctions or setting the PVAD to the folder as the PVAD acts literally on the value specified, essentially becoming a hard coded, custom, token.

But now we have an application with THREE folders that are installed on the root of the C:\ and the application is hard-coded to look for two of them.

Hello my nemesis's

If we do nothing this is the error we get after attempting to login to the program:

Looking at the appvve we can see the D:\ is coming up with those folders.

So this isn't going to work.  How can we make it work?

After publishing this application on the server, we install ACT on the Citrix server.

Launch the "Compatibility Administrator (32-bit)"

Right-click on "New Database" and select "Create New > Application Fix"

Enter the details and browse to the application and click "Next"

Click 'Next' on the Compatibility Mode screen

Select 'CorrectFilePaths' and then click 'Parameters'

In command line, enter the path that should exist, then a semi-colon divider ";" and then the target path:
and click "OK"

Try a Test Run.

And the application launches without any error messages!

Click 'Next'

Then Finish.

Click 'Save' then name your database and click "OK":

Save your fix now:

At this stage you now need to put your SDB file somewhere accessible for when the package is published.  We put it on a fileshare.

Now, all we need to is install the fix.

Since we publish our application globally, I added the fix to the DeploymentConfig.xml:

And we are done! The application now works.