Use if you want, or not. This virtualization template is to be applied against the sequencer. I've found it removes a lot of useless captured information that can get caught in a sequence.
Friday, February 20, 2015
Wednesday, February 18, 2015
Using Process Monitor to make an application installer for 16bit .exe's
http://blogs.msdn.com/b/aaron_margosis/archive/2014/09/05/the-case-of-the-app-install-recorder.aspx
This is a great blog utilizing Process Monitor to create an application installer.
This is a great blog utilizing Process Monitor to create an application installer.
Thursday, February 05, 2015
Becareful of your AVHD to VHD chains with Citrix PVS with multiple sites
Citrix PVS is a great product. With a single VHD file you can boot multiple machines and with the new RAM caching technology introduced in PVS 7.1 it's super fast.
We have Citrix PVS 7.1 setup across 5 data centres in 3 geographically disperse regions with 2 primary sites each having a primary datacenter and a DR datacenter. Each datacenter has high-speed local share for our PVS images. Our Active Directory architecture is 2003 level. Our PVS setup is configured to stream the vDisks from a local file share. This information is important to add context for our process.
I found an issue in one of our datacenters when we had an issue and had to reboot some VM's. Essentially the issue is our target devices were booting slow. Like really, really, really slow. In addition to that, once they did boot; logging into them was slow. Doing any action once you were logged in was slow. But it did seem to get faster the more you click'ed and prodded your way around the system. For instance, clicking the 'Start' button took 15 seconds to open the first time but was normal afterwards, but clicking on any sub-folders was slow with the same symptoms.
Conversely, connecting to a target device at the other city showed these statistics:
So we have an issue. These two separate cities are configured nearly identically but one is suffering severely in performance. We looked at the VMHost that it the VM was hosted at, but moving it to different hosts didn't seem to make any difference. I then RDP'ed to our VM and launched ProcessMonitor so I could watch the bootup process. This is what I saw:
chfs04 is the local file server hosting our vDisk. citrixnas01 is a remote file server on a WAN connection in the datacenter in the other city. For some reason, PVS is reading and sharing the *versioned* avhd file that resides on the local file share, but the base VHD file it is reading from citrixnas01. This is, obviously, a huge issue. Reading the base image over a WAN probably will result in the poor performance we are experiencing and the high retry counts for packets.
But why is it going over the WAN? It turns out that the avhd file contains a section in it's header that describes the location of the parent VHD file. PVS is simply using the native sequence built in to the VHD spec for the chain'ed disks.
(Un)Fortunately for us, our PVS servers can read the file share across the WAN and pull and cache data locally so the speedups tended to gain the longer the VM was in use. In order to fix this issue immediately, we edited our hosts file on the PVS server to point to the local file server for citrixnas01.
After executing that change I rebooted one of the target devices in the *slow* site.
Now, the tricky thing about this issue is we thought we had it covered because we configured each site to have it's own store:
Our thoughts were that this path is what the PVS service would look for when trying to find VHD's and AVHD's. So when it would look for XenAppPn01.6.vhd it would look in that store. Obviously, this line of thinking is not correct. So it is important if you have sites that are distant that the path you use to create your version will correspond to the fastest, local, share in all your sites. For us, our folder structure is identical in all sites so creating a host file entry pointing citrixnas01 to the local file share works in our scenario to start with.
EDIT - I should also note that you can see the incorrect file paths in process explorer as well. Changing the host file wasn't enough, we also needed to restart the streaming service as the streaming service held cached data on how to reach the files across the WAN. Process Explorer can show you the VHD files that the stream service has access to and where (under files and handles):
After a streaming service restart:
We have Citrix PVS 7.1 setup across 5 data centres in 3 geographically disperse regions with 2 primary sites each having a primary datacenter and a DR datacenter. Each datacenter has high-speed local share for our PVS images. Our Active Directory architecture is 2003 level. Our PVS setup is configured to stream the vDisks from a local file share. This information is important to add context for our process.
I found an issue in one of our datacenters when we had an issue and had to reboot some VM's. Essentially the issue is our target devices were booting slow. Like really, really, really slow. In addition to that, once they did boot; logging into them was slow. Doing any action once you were logged in was slow. But it did seem to get faster the more you click'ed and prodded your way around the system. For instance, clicking the 'Start' button took 15 seconds to open the first time but was normal afterwards, but clicking on any sub-folders was slow with the same symptoms.
 |
| This is a poor performing VM. Notice the high number of retries, slow Throughput and long Boot Time |
Conversely, connecting to a target device at the other city showed these statistics:
 |
| Example of a much, much better performing VM. 14 second boot time, throughput about 10x faster than the slower VM and ZERO retries. |
So we have an issue. These two separate cities are configured nearly identically but one is suffering severely in performance. We looked at the VMHost that it the VM was hosted at, but moving it to different hosts didn't seem to make any difference. I then RDP'ed to our VM and launched ProcessMonitor so I could watch the bootup process. This is what I saw:
chfs04 is the local file server hosting our vDisk. citrixnas01 is a remote file server on a WAN connection in the datacenter in the other city. For some reason, PVS is reading and sharing the *versioned* avhd file that resides on the local file share, but the base VHD file it is reading from citrixnas01. This is, obviously, a huge issue. Reading the base image over a WAN probably will result in the poor performance we are experiencing and the high retry counts for packets.
But why is it going over the WAN? It turns out that the avhd file contains a section in it's header that describes the location of the parent VHD file. PVS is simply using the native sequence built in to the VHD spec for the chain'ed disks.
 |
| Hex editing the avhd file reveals the chain path |
(Un)Fortunately for us, our PVS servers can read the file share across the WAN and pull and cache data locally so the speedups tended to gain the longer the VM was in use. In order to fix this issue immediately, we edited our hosts file on the PVS server to point to the local file server for citrixnas01.
After executing that change I rebooted one of the target devices in the *slow* site.
 |
| Throughput is now *much* better, but the number of retries is still concerning |
Now, the tricky thing about this issue is we thought we had it covered because we configured each site to have it's own store:
Our thoughts were that this path is what the PVS service would look for when trying to find VHD's and AVHD's. So when it would look for XenAppPn01.6.vhd it would look in that store. Obviously, this line of thinking is not correct. So it is important if you have sites that are distant that the path you use to create your version will correspond to the fastest, local, share in all your sites. For us, our folder structure is identical in all sites so creating a host file entry pointing citrixnas01 to the local file share works in our scenario to start with.
EDIT - I should also note that you can see the incorrect file paths in process explorer as well. Changing the host file wasn't enough, we also needed to restart the streaming service as the streaming service held cached data on how to reach the files across the WAN. Process Explorer can show you the VHD files that the stream service has access to and where (under files and handles):
 |
| Citrixnas01 shouldn't be in there... Host file has been changed... |
After a streaming service restart:
 |
| Muuuuch better. |
Thursday, January 22, 2015
How important is it to launch your application in a AppV package while sequencing?
For a bit I was of the mindset that sequencing a AppV package should be as clean as humanely possible. This would include finding all configuration tweaks to files/registry keys ahead of time and implementing them so any registry keys generated would be unique to the user. I've seen some applications generate a unique GUID that vendor's would use to lock it so that the application was tied to one machine. Since this key would be generated in HKLM so all users would be able to see the key, it prevented new launches.
But if you didn't launch the application while sequencing, the key wouldn't get generated until the user launched it in their bubble. This effectively allowed multiple users to use the same application on one server. With this new information in mind, and a new outlook on keeping the AppV registry hive to a bare minimum; forward I strode. And hard into a wall I ran.
What I eventually ended up finding was applications that error'ed on first launch:
But would launch just fine the second time:
So what would cause it to fail the first time? I imagine for most cases the error message is caused by missing file(s) or registry key(s) or values. So how do you find these newly generated profile? Well, the nice thing about AppV is it stores all these things in two places. Your registry hive or your user profile.
Before launching the application I did a "dir /s /b C:\Users\Adtest91 >>Clean.txt" and saved that to a text file. I then launched the program twice and ran this command "dir /s /b C:\Users\Adtest91 >> Working.txt" I then compared the files and found the following new paths generated:
C:\Users\adtest91\AppData\Local\Microsoft\AppV\Client\VFS\ADB25534-3FE9-44BD-9FC8-D5AAD8C0E728
C:\Users\adtest91\DesktopFax
To see if these two paths caused my issue I deleted them and relaunched the application. The application launched just fine. I then backed up those two folders. To rule out the file system with some more finality, I deleted my user profile then copied my two backup folders to their recorded paths and tried launching the program and got the error message again. With that I felt confident I could rule out files/folders as the cause.
The beautiful thing about AppV registry changes is they are recorded to your user profile. This is stored here:
HKEY_USERS\!GUID!_Classes\AppV\Client\Packages
Export this key prior to launching your application, launch your application, and export the key again and do a difference. Any created or modified registry keys will reside in this location for you to examine.
For this issue, this was the key that was generated:
[HKEY_USERS\S-1-5-21-38857442-2693285798-3636612711-15053136_Classes\AppV\Client\Packages\ADB25534-3FE9-44BD-9FC8-D5AAD8C0E728\REGISTRY\MACHINE\SOFTWARE\Classes\TypeLib\{3B7C8863-D78F-101B-B9B5-04021C009402}\1.2\0\win32]
@="C:\\Windows\\system32\\Richtx32.ocx"
Deciphering the key results in the following missing value:
[HKLM\SOFTWARE\Classes\TypeLib\{3B7C8863-D78F-101B-B9B5-04021C009402}\1.2\0\win32]
Looking locally on the server, this is what I saw in the registry:
There was nothing in the Data field! AppV5 'integrates' the Classes key in your AppV package when you publish it. I resequenced and launched the application after the install and checked the key again:
Surprise, surprise. And now the application launches without issue. So it appears that some application installers don't completely register all files (OCX files are two instances of this issue happening that I noticed) until the application is launched. So now, our policy will always to, at a minimum, launch the application while sequencing.
But if you didn't launch the application while sequencing, the key wouldn't get generated until the user launched it in their bubble. This effectively allowed multiple users to use the same application on one server. With this new information in mind, and a new outlook on keeping the AppV registry hive to a bare minimum; forward I strode. And hard into a wall I ran.
What I eventually ended up finding was applications that error'ed on first launch:
But would launch just fine the second time:
So what would cause it to fail the first time? I imagine for most cases the error message is caused by missing file(s) or registry key(s) or values. So how do you find these newly generated profile? Well, the nice thing about AppV is it stores all these things in two places. Your registry hive or your user profile.
Before launching the application I did a "dir /s /b C:\Users\Adtest91 >>Clean.txt" and saved that to a text file. I then launched the program twice and ran this command "dir /s /b C:\Users\Adtest91 >> Working.txt" I then compared the files and found the following new paths generated:
C:\Users\adtest91\AppData\Local\Microsoft\AppV\Client\VFS\ADB25534-3FE9-44BD-9FC8-D5AAD8C0E728
C:\Users\adtest91\DesktopFax
To see if these two paths caused my issue I deleted them and relaunched the application. The application launched just fine. I then backed up those two folders. To rule out the file system with some more finality, I deleted my user profile then copied my two backup folders to their recorded paths and tried launching the program and got the error message again. With that I felt confident I could rule out files/folders as the cause.
The beautiful thing about AppV registry changes is they are recorded to your user profile. This is stored here:
HKEY_USERS\!GUID!
Export this key prior to launching your application, launch your application, and export the key again and do a difference. Any created or modified registry keys will reside in this location for you to examine.
For this issue, this was the key that was generated:
[HKEY_USERS\S-1-5-21-38857442-2693285798-3636612711-15053136_Classes\AppV\Client\Packages\ADB25534-3FE9-44BD-9FC8-D5AAD8C0E728\REGISTRY\MACHINE\SOFTWARE\Classes\TypeLib\{3B7C8863-D78F-101B-B9B5-04021C009402}\1.2\0\win32]
@="C:\\Windows\\system32\\Richtx32.ocx"
Deciphering the key results in the following missing value:
[HKLM\SOFTWARE\Classes\TypeLib\{3B7C8863-D78F-101B-B9B5-04021C009402}\1.2\0\win32]
Looking locally on the server, this is what I saw in the registry:
There was nothing in the Data field! AppV5 'integrates' the Classes key in your AppV package when you publish it. I resequenced and launched the application after the install and checked the key again:
Surprise, surprise. And now the application launches without issue. So it appears that some application installers don't completely register all files (OCX files are two instances of this issue happening that I noticed) until the application is launched. So now, our policy will always to, at a minimum, launch the application while sequencing.
Sunday, January 04, 2015
Awesome blog to follow
I'm learning a ton from reading http://www.remkoweijnen.nl/. Very good blog with good step-by-step on how he executes his troubleshooting. One thing I've learned from it is to love Ida Pro.
Subscribe to:
Posts (Atom)