Citrix Desktop Director "The system is currently unavailable. Please try again or contact your administrator."

We are attempting to add Citrix Desktop Director to our helpdesk.  We have a geographically spread organization across three cities connected by a huge ring network.  We setup one Desktop Director server and pointed it at two servers in Calgary (WSCTXZDC301 and 302).  We were able to login and all appears good.  We then setup our second Desktop Director server and pointed it to two servers in Edmonton (WSCTXZDC303 and 304).  With this server we were unable to login.  We were getting this error message:

"The system is currently unavailable. Please try again or contact your administrator."

Well...  That helps.

Step 1 was to turn on logging for Desktop Director.

Step 2 was the reproduce the issue and analyze the log.

"15:13:37.2515 : [t:5, s:unknown] TimeoutException caught: This request operation sent to net.tcp://wsctxzdc303.healthy.bewell.ca:2513/Citrix/XenAppCommands did not receive a reply within the configured timeout (00:00:10).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client."

So.  It appears there is a timeout and the XenAppCommands is not completing in time.  Is this because something was wrong with our ZDC?  To test this I used Remote-XAPSSession to test the command above in the log (Get-XAServer -OnlineOnly -ZoneName A,B,C).  With powershell remoting I connected to the ZDC 301 and 302 and ran the command and it completed within a second.  I then tried it against 303 and 304 and the command completed in....   17 seconds and 22 seconds.  Far exceeding the 10 second timeout.  Why was it taking so long?  It turns out that the ZDC is forced to contact the database with that command as opposed to utilize the LHC (local host cache).  Procmon shows lots of database communication and our two ZDC's 303 and 304 are in Edmonton and the database resides in Calgary; communication round-trip makes those ZDC's exceed 10 seconds.

So...  It turns out that Desktop Director should be pointed at a data collector that sits closest to the database.  If it is too far away you may exceed this timeout and your users will be unable to login.  Unforunately, I am unable to find a way to manipulate this timeout to increase it (even though that will cause poor performance, at least users can actually operate).

Updating AppV 5 package files without resequencing

With AppV 5.0 HF4 you can update the files inside a package without having to go through resequencing.  The only catch is you have to prestage the file you want to replace/update first in the location you want, and I don't think you can update files in the PVAD, only VFS.

To update a file in a .appv package, open the .appv in the sequencer and select the "Edit Package" option and click "Next".

Click "Edit"

Expand the path to the file you want to update and right-click and select "Delete".

Right-click the folder above and select "Add"

Browse to the file, select it, click "Open".

Click "OK" to make a new VFS mapping.  Choose File > Save and save your package.

If you have the original package and rename it to .zip from .appv and get properties on your file, and do the same on your new version of the package (after making a copy of it) you should see that they are different and you have a new file in the new version without having to "Update" the package; avoiding capturing additional garbage that happens when you update an application.

AppV 5 - Formatted volume allocation size matters

After discovering that AppV 5 configures an allocation size independent of the file system beneath it, I explored the impact of different formatted allocation sizes on AppV packages; specifically mounting AppV packages.

I took our AppV setup and set the PackageInstallationRoot to D:\AppVData\PackageInstallationRoot and then formatted the D:\ to different allocation sizes and then mounted the AppV package.  I timed how long it took to mount the package over 4 runs per allocation choice, took the AppV file size allocation, and the total, actual size of the package on the drive.  Package Details:

Package size: 1838658907 (1.71GB)

Files: 9106

Folders: 357

 The results:

Different Allocation Sizes

Actual Used Space vs. Allocation size

Actual AppV 5 file allocation size vs. formatted allocation size

AppV 5 Mount Time vs. formatted allocation size

I would propose using 64KB allocation sizes for the AppV volume if possible (note, this only applies to fully mounted packages).  There does appear to be some benefit to using larger allocation sizes, one of the main ones is the properties of the PackageInstallationRoot now reflects more accurately the actual consumed filesystem on Windows 2008 / Windows 7.  Another is there is a speed improvement around the 4KB allocation size then minor increases afterwards.

AppV 5 - changing PackageInstallationRoot impacts AppVPackageDrive variable

I just sequenced an application (Epic) and it puts two folders on the root of the C drive.  C:\Epic and C:\Crashdumps.  When I published the application to our Citrix servers the application failed to launch.  Procmon'ing it I saw that Epic was unable to find C:\Epic.  I opened a command prompt and tried to cd /d C:\Epic which failed.  On a hunch I tried cd /d D:\Epic and lo and behold it was successful and I was put in that folder.  I investigated this and created a package with two folders on the root of the C:\.  They were C:\test and C:\test_happy_folder.  When I published the package on our Citrix servers I could not cd into them on the C:\ drive but I could because they were on the D:\ drive.  Examining the package it showed:

Why was it going to D:\?  We have our servers setup so that the PackageInstallationRoot registry key points to the D:\.

When I changed PackageInstallationRoot to be C:\AppVData\PackageInstallationRoot I could then cd /d C:\Epic without issue.  This is unfortunate but I do not think this was the expected behaviour because we sequence to C:.  To fix this I set the PVAD to C:\Crashdumps (which does expand out to the C: when specified in the sequencer, not the D:) and moved the C:\Epic folder to within a tokenized folder (programfiles\Epic) which correctly expands out to C:\Program Files\Epic.  I then needed to hunt down all config files and scripts that point to C:\Epic and update them to point to C:\Program Files\Epic.  Unfortunately, I do not know of a way to modify AppVPackageDrive to a drive letter without changing PackageInstallationRoot because I would love for my sequenced applications to stick to C:\ so I don't need to do any workarounds.

Essentially, I guess what I'm saying is I hope AppV 5, sometime in the future, allows you to either hard code the AppVPackageDrive variable to a specific drive letter, or use some other tokenized variable that points to a specific drive letter ({CDrive},{DDrive},{EDrive},{FDrive}, etc.)

AppV 5 - minimum mounted file size is 64KB, size of packages is misrepresented on 2008R2/Windows 7

Our current configuration for our Citrix environment utilizes AppV 5 and mounts the packages to a different drive letter.  Our Citrix servers have the OS on C:\ and we changed the packageinstallationroot to D:\AppVData\PackageInstallationRoot and do full mounts of our AppV packages.  What we found is we have some packages consuming significantly more space then their extracted size would dictate.  I worked with Microsoft premier support to try and determine why, finally, after flaying for several months on why I finally got a reason and how we can avoid package bloat.

First the problem.  We have several packages that their total size should be less than the disk space it is consuming on our D:\.  We started running out of disk space earlier than anticipated so we investigated.

We packaged VMWare vSphere Client and included multiple versions of vSphere that maximizes the ability to connect to the servers/vCenter versions while minimizing the number of vSphere clients in the package.  In total, we sequenced 3 vSphere installs in one package.  The total package size is:

0.98 GB (1,054,613,510 bytes)

When we rename the .appv to .zip and extract the package the total size is:

1.71GB on the AppVData for the extracted file.  1.83GB used space on disk.

But, when we mount the .appv package using AppV the total size is:

1.71GB on the AppVData folder, but 2.25GB used space on disk.  A discrepancy of ~400MB.  With 10 packages that could be 4GB of wasted space across 60 Citrix servers it adds up fast.

So...  Where is this missing space and why does AppVData not report it correctly?  Well... it turns out it does report the used space correctly in Windows 8 / 2012.  The AppVData folder for the mounted application actually shows that it's 2.2GB when mounted on a Windows 8 / 2012 box.  Why the difference?  It turns out that when Appv 5 mounts applications it is mounting them using a 64KB allocation size.  In Windows 8 / 2012 when you get properties on a file in the mounted package it shows that it's 64KB used on disk.  On Windows 2008 / 7 it shows the file size used on disk, then it goes to the allocation size:

Windows 2008 / 7 file sizes are incorrect for mounted packages

If we delete a file under 64KB we will actually free up 64KB of space and Windows 2008 reports that correctly:

19,045,400,576 bytes free before the 2,951 byte file is deleted, 19,045,466,112 bytes after.

19045400576 - 19045466112 = 65536 bytes.

So, in the end, it turns out that AppV 5 utilizes 64KB cluster size even if the allocation size on the disk is 4KB.  I'm not sure if having mismatched cluster sizes impacts the performance of AppV but the Microsoft support personnel implied that it could.  For AppV 5 you'll want to reduce the number of small files in your AppV packages to as few as possible to limit this space sucking overhead.  The missing space is "sparse" files, but fully mounted I wouldn't expect to have any sparse files.  So AppV is using sparse files to ensure a 64kb allocation size.

Allocation Report:

Within these files there are:
    Compressed              : 0
        Total allocated     : 0 bytes
        Total size          : 0 bytes.
        Savings             : 0.00 %
    Sparse                  : 9106
        Total allocated     : 2318139392 bytes
        Total size          : 1838549977 bytes.
        Savings             : -26.09 %
    Encrypted               : 0
        Total allocated     : 0 bytes

AppV 5 - Short file names are not created after publishing a new version of an application

Our AppV packages are loaded on Citrix PVS servers.  The PVS server is clean, and on each restart (restarts once a day) it will contact a AppV publishing server and pull down and mount the packages.

We have an application that makes file system calls in the 8.3 space (awesome, I know).  When launching the application I can see a trace of it hitting 8.3 name spaces (via Procmon) and the application works without issue.  So, we required a tweak to the package, I added to the management server, published the application and refreshed on the AppV Client, saw the new package get loaded and ran the application.  Only now when I trace via Procmon.exe I can see that it's using Long File Names instead of the 8.3 name space and the program errors out because it's trying to find a .exe with the short name.  I can confirm the original published package has the 8.3 namespace via dir /x and the new version does NOT.

8 dot 3 exists for folders but not files...

In the image above, the version 1E65...... was published to the server first and EE168.... is the new version published later in the day.

I have VFS folders in this package and 8.3 names are not created for the new package either.  :/

While attempting to create a 8.3 file name manually I got a NAME COLLISION and a failure to create it(?)

So it appears if you have an application that does not properly support LFN (Long File Names) then it may not operate correctly if you update an AppV package, at least until the next restart (for some reason for us it creates the LFN after that).

AppV 5 - .NetFramework applications that only work with 32bit .NetFramework architecture

I encountered an issue with an application of ours that refused to launch on Windows 2008 R2 SP2 but worked on Windows 2003 R2.  I did a procmon.exe trace and noticed some discrepancies in the trace results:

HKLM\SOFTWARE\Microsoft\Fusion\NativeImagesIndex\v2.0.50727_64
^^ 2008 R2 SP2 (non-working application launches)

HKLM\SOFTWARE\Microsoft\Fusion\NativeImagesIndex\v2.0.50727_32
^^ 2003 R2 (working application launches)

So, my application was defaulting to the .NetFramework64 architecture and no launching.  To correct this I was able to run this command:

If you have a 64 bit machine and want to run a .NET application that only works with the 32 bit CLR you would have to make changes to the .NET framework on your machine

Set the .NET framework to load the CLR in WOW mode through this command

Open up command prompt and type this command

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Ldr64.exe SetWow

Now you should be able to run apps that use only the .NET 32 bit CLR.

To revert back to the default 64 bit framework run

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Ldr64.exe Set64

(source)

It appears all this does is modify one registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework
REG_DWORD Enable64bit
DATA 0x0 (for SetWow) 0x1 (for Set64)

Alternatively, run corflags.exe against your executable to force it only utilize the 32bit .NetFramework.  Corflags.exe can be gotten from the ".NET Framework 2.0 Software Development Kit (SDK) (x64)"

Execute the application with the following:

“c:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\x64\CorFlags.exe” MyApplication.exe /32bit+

After doing so your application should only use the 32bit .NetFramework.  And, if you are in my scenario, the app now works without issue.



:::::::::::::::::::::::EDIT::::::::::::::::::::::::::::::

It appears the following registry key is for status only:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework
REG_DWORD Enable64bit
DATA 0x0 (for SetWow) 0x1 (for Set64)

It does not actually do anything configuration-wise. Changing this key does not manipulate the ability to .NetFramework to use 32bit or 64bit.