SCVMM: Install VM components Failed

I'm attempting to deploy a virtual machine from a template but I get this error: Install VM components: Failed.

Error (2940)
VMM is unable to complete the requested file transfer. The connection to the HTTP server 2012-SCVMM.bottheory.local could not be established.
Unknown error (0x80072ee2)

Recommended Action
Ensure that the HTTP service and/or the agent on the machine 2012-SCVMM.bottheory.local are installed and running and that a firewall is not blocking HTTP/HTTPS traffic on the configured port.

TL;DR
SO...  Long story short; if you are encountering this error, I would suggest booting your VHD file in a VM and re-sysprep /generalize it.  If you've maxed out on sysprep's I have a post earlier in my blog on how to get around the 3-times limit and rerun sysprep.  Alternatively, you can try what I did, but I can't gaurantee your success and replace the BCD file in your Library VHD with a BCD you *know* has been sysprepp'ed and try redeploying it.
=------------------------------------------------------------------------------------------------=

I've ensured that HTTP and HTTPS is not blocked (firewall is disabled) and the agent on my SCVMM machine is installed and running.  So this error message is somewhat useless.

So I took my two boxes, 2012-SCVMM (the SCVMM server) and S5000VXN-SERVER (the Hyper-V host) and procmon'ed them while it was attempting to "Install VM Components" to try and understand what it's trying to do.

After reinitating the task, we can see that the vmmAgent.exe on the Hyper-V host accesses the file about 2 seconds after I submit the command to retry the job.

A few seconds after this, it appears to "CreateFile" in the Windows\Temp directory; but this is not actually correct.

What it is really doing is *mounting* the VHD into a folder in the Windows\Temp directory.  You can actually watch this in action if you view Disk Management on the Hyper-V host while you execute the task.

So now that we know it's mounting the file as a volume this helps us narrow down on what Hyper-V is attempting to do...  And I suspect what it is attempting to do is "offline servicing" of the attached vdisk/vhd.

After attaching the vdisk the next thing it does it query the BCD file on the system.  Maybe it needs to be in a certain mode to operate?  I'm not sure...

Continuing on we can see that events are written to the Microsoft-Windows-FileShareShadowCopyProvider Operational.evtx, System.evtx, and Microsoft-Windows-Bits-CompactServer Operational.evtx event logs.  Examining each log at the time stamp showed the FileShareShadowCopyProvider and System log were just noticed of volume shadow copy starting, but the BITS event log was interesting.

It showed that it was doing something with the BCD file.  The Hyper-V host was *serving* it out.  I suspect it was serving it to the SCVMM.
Looking back at the SCVMM server we see that was executing some WinRM commands.  Sadly, we do not know what commands it was trying to send.

If I mount the vhd file and check out the BCD file I can see that it appears to be corrupted in that it doesn't know what the proper boot device should be.

C:\Windows\system32>bcdedit /store "K:\boot\bcd" /enum all

Windows Boot Manager
--------------------
identifier              {bootmgr}
device                  unknown
description             Windows Boot Manager
locale                  en-US
inherit                 {globalsettings}
bootshutdowndisabled    Yes
default                 {default}
resumeobject            {b520e13c-48da-11e2-9a8b-00155d011700}
displayorder            {default}
                        {7619dcc9-fafe-11d9-b411-000476eba25f}
toolsdisplayorder       {memdiag}
timeout                 3

Windows Boot Loader
-------------------
identifier              {7619dcc9-fafe-11d9-b411-000476eba25f}
device                  ramdisk=[boot]\sources\boot.wim,{7619dcc8-fafe-11d9-b411
-000476eba25f}
path                    \windows\system32\boot\winload.exe
description             Windows Setup
locale                  en-US
inherit                 {bootloadersettings}
osdevice                ramdisk=[boot]\sources\boot.wim,{7619dcc8-fafe-11d9-b411
-000476eba25f}
systemroot              \windows
detecthal               Yes
winpe                   Yes
ems                     Yes

Windows Boot Loader
-------------------
identifier              {default}
device                  unknown
path                    \Windows\system32\winload.exe
description             Windows Server 2012
locale                  en-US
inherit                 {bootloadersettings}
allowedinmemorysettings 0x15000075
osdevice                unknown
systemroot              \Windows
resumeobject            {b520e13c-48da-11e2-9a8b-00155d011700}
nx                      OptOut
detecthal               Yes

Resume from Hibernate
---------------------
identifier              {b520e13c-48da-11e2-9a8b-00155d011700}
device                  unknown
path                    \Windows\system32\winresume.exe
description             Windows Resume Application
locale                  en-US
inherit                 {resumeloadersettings}
allowedinmemorysettings 0x15000075
filepath                \hiberfil.sys

Windows Memory Tester
---------------------
identifier              {memdiag}
device                  unknown
path                    \boot\memtest.exe
description             Windows Memory Diagnostic
locale                  en-US
inherit                 {globalsettings}
badmemoryaccess         Yes

EMS Settings
------------
identifier              {emssettings}
bootems                 Yes

Debugger Settings
-----------------
identifier              {dbgsettings}
debugtype               Serial
debugport               1
baudrate                115200

RAM Defects
-----------
identifier              {badmemory}

Global Settings
---------------
identifier              {globalsettings}
inherit                 {dbgsettings}
                        {emssettings}
                        {badmemory}

Boot Loader Settings
--------------------
identifier              {bootloadersettings}
inherit                 {globalsettings}
                        {hypervisorsettings}

Hypervisor Settings
-------------------
identifier              {hypervisorsettings}
hypervisordebugtype     Serial
hypervisordebugport     1
hypervisorbaudrate      115200

Resume Loader Settings
----------------------
identifier              {resumeloadersettings}
inherit                 {globalsettings}

Device options
--------------
identifier              {7619dcc8-fafe-11d9-b411-000476eba25f}
ramdisksdidevice        boot
ramdisksdipath          \boot\boot.sdi

It's not actually corrupted though; the reason why the devices are unknown is because bcdedit isn't finding the disk signature of the volume I mounted.  But it has the disk signature because I can boot with it without issue.

Continuing on...

In order to try and find out what commands WSMAN was sending I enabled debugview on the SCVMM server:
http://support.microsoft.com/kb/970066?wa=wsignin1.0

I then reproduced the error by rerunning the Create Virtual Machine job.

DebugView gave me more information to narrow down what was happening.  It appears that the process is failing with:
[4276] 10B4.000C::05/31-12:52:00.894#16BcdUtil.cs(1987): bootDevice UnknownDevice
[4276] 10B4.000C::05/31-12:52:01.334#16MountedVhd.cs(91): MountedVhd windows volume not found
[4276] 10B4.000C::05/31-12:52:01.342#16VMAdditions.cs(874): VMAdditions install failed at OS detection phase for vm AirVid
[4276] 10B4.000C::05/31-12:52:01.388#16VMAdditions.cs(874): Microsoft.VirtualManager.Engine.VmOperations.MountedSystem+BootOrSystemVolumeNotFoundException: Virtual Machine Manager cannot locate the boot or system volume on virtual machine NO_PARAM. The resulting virtual machine might not start or operate properly. 

Doing some googling on this took me to a Korean Microsoft page where the following was stated:
http://social.technet.microsoft.com/Forums/ko-KR/momsmsmofko/thread/fedbc514-1fc0-4b55-979b-7d07babb074b/

When creating a new VM from template, and that template contains a blank VHD or a non-Windows OS or a Windows OS that has not been generalized (i.e. the OS is not sysprepped), then the job will fail because VMM expects a VM from template to go through the sysprep customization process (hence why we ask for an OS profile). VMM will crack open the VHD and check of the OS is in a sysprep state. If not, then the job will fail. To create a proper template, you can use an existing Vm with a running Windows OS (right click on it and select New Template… this will kick off sysprep in the OS and then store the VM to Library as template… this removes the original VM). Or… if you already have VHDs that have been sysprepped, simply import them to the Library and then when you create a new template… attach that VHD instead of the blank one. Last… you can create a new template that does not require a sysprepped OS by selecting ‘Customization Not Required’ from the Guest OS profile dropdown:
http://blogs.technet.com/b/hectorl/archive/2008/08/21/digging-deeper-into-error-13206-virtual-machine-manager-cannot-locate-the-boot-or-system-volume-on-virtual-machine.aspx

Thinking about it, I do not think my VHD was SysPrep'ed.  I find it interesting that sysprep appears to do something to the BCD file.  To find out what sysprep does to the BCD file I booted up my 2012 VHD into Windows and ran sysprep, generalize and shutdown.

With the VM now generalized I can crack open the VHD and see what's added to the BCD to make it so special...

C:\Users\amttye\Desktop>fc before-sysprep.txt after-sysprep.txt
Comparing files before-sysprep.txt and AFTER-SYSPREP.TXT
***** before-sysprep.txt
identifier              {bootmgr}
device                  partition=K:
description             Windows Boot Manager
***** AFTER-SYSPREP.TXT
identifier              {bootmgr}
device                  locate=unknown
description             Windows Boot Manager
*****

***** before-sysprep.txt
toolsdisplayorder       {memdiag}
timeout                 3

***** AFTER-SYSPREP.TXT
toolsdisplayorder       {memdiag}
timeout                 30

*****

***** before-sysprep.txt
identifier              {default}
device                  partition=K:
path                    \Windows\system32\winload.exe
***** AFTER-SYSPREP.TXT
identifier              {default}
device                  locate=\Windows\system32\winload.exe
path                    \Windows\system32\winload.exe
*****

***** before-sysprep.txt
inherit                 {bootloadersettings}
allowedinmemorysettings 0x15000075
osdevice                partition=K:
systemroot              \Windows
***** AFTER-SYSPREP.TXT
inherit                 {bootloadersettings}
recoveryenabled         No
allowedinmemorysettings 0x15000075
osdevice                locate=\Windows
systemroot              \Windows
*****

***** before-sysprep.txt
nx                      OptOut
detecthal               Yes

***** AFTER-SYSPREP.TXT
nx                      OptOut

*****

***** before-sysprep.txt
identifier              {b520e13c-48da-11e2-9a8b-00155d011700}
device                  partition=K:
path                    \Windows\system32\winresume.exe
***** AFTER-SYSPREP.TXT
identifier              {b520e13c-48da-11e2-9a8b-00155d011700}
device                  locate=\Windows\system32\winresume.exe
path                    \Windows\system32\winresume.exe
*****

***** before-sysprep.txt
inherit                 {resumeloadersettings}
allowedinmemorysettings 0x15000075
filepath                \hiberfil.sys

***** AFTER-SYSPREP.TXT
inherit                 {resumeloadersettings}
recoveryenabled         No
allowedinmemorysettings 0x15000075
filedevice              locate=\hiberfil.sys
filepath                \hiberfil.sys
debugoptionenabled      No

*****

***** before-sysprep.txt
identifier              {memdiag}
device                  unknown
path                    \boot\memtest.exe
***** AFTER-SYSPREP.TXT
identifier              {memdiag}
device                  locate=\boot\memtest.exe
path                    \boot\memtest.exe
*****

Interestinginly the differences appear to be mostly "locate=%%".  I guess this would make sense as the assumption is the BCD is being moved to a new disk with a new disk signature and so it can't lock on to the existing signature.  Some other oddities is the removal of "detecthal" and explicit declarations of debugoptionenabled and recoveryenabled.  I suspect that a generalized BCD file is portable, so I'm going to extract it from this image and inject it into my previously "failing" image.  I then edited the template and removed the old VHD and added the new one.

And..........?  Lets go to DebugView:

[4276] 10B4.000C::05/31-13:57:40.193#16SystemInformation.cs(192): SYSTEM: :\ Version=0.0 HALType=  Memory=2MB, Procs=1 Is64s=False . OSLanguage=0
[4276] 10B4.000C::05/31-13:57:40.193#16SystemInformation.cs(970): Lookup for '\??\Volume{b42a3001-c871-11e2-93f4-0015172fc019}' in MountedDevices
[4276] 10B4.000C::05/31-13:57:40.193#16SystemInformation.cs(764): DISK: 10, signature=a9083c0a #partitions 1
[4276] 10B4.000C::05/31-13:57:40.194#16SystemInformation.cs(768):  PARTITION: 10.0 Bootable=True, #LDs 1
[4276] 10B4.000C::05/31-13:57:40.194#16SystemInformation.cs(775):   LOGICALDRIVE: \\?\Volume{b42a3001-c871-11e2-93f4-0015172fc019}\(\\?\Volume{b42a3001-c871-11e2-93f4-0015172fc019}\), WindowsDrive=True, IsBoot=False, BootSectorType=Bootmgr, FullSize=136363114496, [1048576-136364163072]
[4276] 10B4.000C::05/31-13:57:40.194#16MountedVhd.cs(623): FindBootVol \\?\Volume{b42a3001-c871-11e2-93f4-0015172fc019}\, found boot drive candidate
[4276] 10B4.000C::05/31-13:57:40.194#16MountedVhd.cs(308): Bootmgr boot loader
[4276] 10B4.000C::05/31-13:57:40.194#16MountedVhd.cs(1053): Trying to get the mounted point for volume \\?\Volume{b42a3001-c871-11e2-93f4-0015172fc019}\.
[4276] 10B4.000C::05/31-13:57:40.196#16CommonUtils.cs(171): Fixup & Copy 'C:\Windows\TEMP\tmp2E1.tmp\boot\bcd' to C:\Users\svc_scvmm\AppData\Local\Temp\tmpA2F4.tmp
[4276] 10B4.000C::05/31-13:57:40.196#04BitsDeployer.cs(1392): Deploy file C:\Windows\TEMP\tmp2E1.tmp\boot\bcd from s5000vxn-server.bottheory.local to C:\Users\svc_scvmm\AppData\Local\Temp\tmpA2F4.tmp on 2012-SCVMM.bottheory.local

Voila!  It appears much better than before.  It found the drive correctly and checked the BCD file and found it is the "Generalize" state.  The *actual* image I originally made was NOT sysprep'ed and all I did was replace the BCD file, but it allowed it to continue beyond and the machine actually completed the imaging process properly.  It joined the domain and whatever else the answer file was I gave it in SCVMM.

It appears I was correct in my earlier assumption about what SCVMM is doing.  It goes and grabs the BCD file and transfers it from the target machine to itself, "fixes it up" (not sure what it's doing at this stage precisely), then sends it back for injection.  Certainly a bit of a complicated process with a fair bit that can go wrong, but shame on Microsoft for having such a poor error message.  I suspect it wouldn't have required much effort to push out a real message; something to the effect of, "The BCD of this vDisk does not appear to have been through the sysprep /generalize process.  Please rerun sysprep against the image and try again".

SO...  Long story short; if you are encountering this error, I would suggest booting your VHD file in a VM and re-sysprep /generalize it.  If you've maxed out on sysprep's I have a post earlier in my blog on how to get around the 3-times limit and rerun sysprep.  Alternatively, you can try what I did, but I can't gaurantee your success and replace the BCD file in your Library VHD with a BCD you *know* has been sysprepp'ed and try redeploying it.

Manually add Windows startup scripts (or inject startup scripts into an offline image)

Due to the CGP issue, our solution is to add a startup script to each vDisk.  Since I don't want to make a version of each vDisk than attach it to a server than boot it up than gpedit.msc...  We have around 10 vDisks and that process would be annoying and take a while.  So I decided to investigate doing it offline as mounting a VHD using cvhdmonut.exe and then injecting the startup script would be a lot easier.

To do that, one simply needs to browse to:

"C:\windows\system32\GroupPolicy\Machine\Scripts\Startup" (for machine startup script, aka, a script that starts when your computer starts up) or "C:\windows\system32\GroupPolicyUsers\Machine\Scripts\Startup" (for all users startup script [I'm assuming since I actually didn't go through and test the user portion]) and copy your script file there.  

Then, back out one level to C:\windows\system32\GroupPolicy\Machine\Scripts and edit Scripts.ini to include your new script file; incrementing the last line.

To:

(OS 10061)No connection could be made because the target machine actively refused it. : Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

 This has been an ongoing problem for us (Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

I may have found out why it was happening in our environment.  We are using Provisioning Services and with it we are using two NIC's, one for the Provisioning Services and one for Standard networking.

It appears the XTE service became configured to use the Provisioning Services NIC.  This was verified in the httpd.conf in the C:\Program Files (x86)\Citrix\XTE\conf folder.

Provisioning NIC and Production (network) NIC

httpd.conf as was when the system booted (and non-functional)

When I traced the XTE service using procmon.exe and wireshark with this non-functional conf this is what I saw when I attempted to launch the application:

You can see it attempt to connect to itself via 1494 but then nothing else happens

Wireshark shows virtually nothing on the network and nothing related to IMA

When I edited the file to have the Production NIC...

then restarted the XTE service and retraced via Procmon and Wireshark...

We now see tons of activity and the application now launches without issues.

================EDIT===============

We have now found why we are getting this error, and why we are getting it intermittently.  The issue is we are using PVS with multi-homed NIC's.  One NIC (LanAdapter 1) is the "Provisioning" network, and the second NIC (LanAdapter 2) is the "Production" network.  The Provisioning network is on a completely seperate vLan and sees no traffic outside of it's little network.  The ICA Listener was attaching itself to the Provisioning network instead of the production network, so when we tried to connect to the server it would fail with the CGP tunnel error because the outside network cannot talk to the Provisioning network.  To attempt to resolve this issue one of our techs (Saman) created a group policy preference registry key that set the following value (HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Terminal Server\WinStations\ICA-TCP - LanAdapter):

By setting it to "2" we could ensure the ICA listener is always listening on LanAdapter 2, our production network.  Unfortunately, a Windows Update appears to have caused either Group Policy Registry Preferences to execute (sometimes) *after* the IMAService service started, or allowed the IMAService service to start *before* Group Policy Registry Preferences.  IMAService will recreate that file every second restart.  To resolve this issue I created a startup script that executes after 65 seconds, deleting the httpd.conf file and restarting the appropriate services until the httpd.conf file is recreated.

In my testing it appears you need to restart the "IMAService" service twice to get it to recreate the httpd.conf file.  Because of this, I created the script to retry up to 3 times to try and regenerate the file.

:: ===========================================================================================================
::
:: Created by: Trentent Tye
::
::
::
:: Creation Date: May 28, 2013
:: Modified Date: May 28, 2013
::
:: File Name: Citrix_Restart_IMASrv_Delayed.cmd
::
:: Description: This script fixes the CGP Tunnel issue where the IMASrv.exe starts before
:: group policy prefencess start.  This causes the httpd.conf file in the
:: XTE folder to have the wrong IP and the ICA client listener to actually
:: be listening on the wrong NIC.  To resolve this issue we will ensure 
:: group policy is applied then restart the IMASrv service *twice*.
:: We have to do it twice because the IMASrv won't recreate the httpd.conf
:: file on the first restart.
::
:: ===========================================================================================================

@ECHO OFF
CLS

SET COUNT=0

eventcreate /ID 1 /L APPLICATION /T INFORMATION /SO "Local GP Startup Script" /D "Starting Citrix_Restart_IMASrv_Delayed.cmd script"

del /q "C:\Program Files (x86)\Citrix\XTE\conf\httpd.conf"

ECHO N | GPUPDATE /FORCE >NUL
ping 127.0.0.1 -n 65 >NUL

:RetryCreate
net stop CitrixWMIService
net stop IMAService
net stop CitrixXTEServer

ping 127.0.0.1 -n 5 >NUL
net start IMAService
net start CitrixWMIService

ping 127.0.0.1 -n 5 >NUL
net stop CitrixWMIService
net stop IMAService

ping 127.0.0.1 -n 5 >NUL
net start IMAService
net start CitrixWMIService
net start CitrixXTEServer

IF %COUNT% GEQ 3 EXIT
SET /A COUNT=%COUNT%+1

IF NOT EXIST "C:\Program Files (x86)\Citrix\XTE\conf\httpd.conf" GOTO RetryCreate
eventcreate /ID 1 /L APPLICATION /T INFORMATION /SO "Local GP Startup Script" /D "Completed Citrix_Restart_IMASrv_Delayed.cmd script"

Enable advanced logging on the XTE service of a XenApp server

Back in February or March we did Windows updates on our PVS XenApp servers and then sometime after the servers do not allow anyone to login.

The issue can crop up as a CGP Tunnel error message, a protocol driver error message or something along those lines.  We do not know why it's happening or why it only happens after we apply Windows updates from that time period.  The odd thing is it's intermittent as well, we can launch 20 systems from 1 vDisk that has the updates applied and everything will be fine for 2 weeks then, suddenly, 5 of the systems won't allow logins via ICA with the errors.  Rebooting the servers sometimes fixes it, sometimes not.  Very intermittent and very weird.  So I attempted to troubleshoot this issue again by adding:

# Log Level
loglevel debug

to the httpd.conf in the XTE folder of a server that was exhibiting these issues.  The log levels are listed here:
http://support.citrix.com/article/CTX114680

After adding those lines and restarting the XTE service the issue resolved itself!  Frustrating to be sure, and I will look at adding this line to our vDisk image so that when it crops up we'll have more diagnostic data to look at.

Citrix Provisioning Services (PVS) 6.1 - Automatic vDisk Update "Update device failed to shutdown within the timeout period."

I have setup our PVS environment to execute the vDisk Automatic Update feature utilizing a custom script (update.bat).  This script does a bunch of things, resync's the time with NTP (to avoid daylight savings issues), refreshs GPO's, executes Windows Update, cleans up temp files, runs the PVS optimizer, etc.

Unfortunately this can take longer than 30 minutes.  For some reason, when executing the ESD client as "None" (aka, so a script runs) the "Update timeout" doesn't seem to take effect, instead the 30 minute shutdown timeout is on the clock.

ESD client is set to none

You cannot increase the shutdown timeout beyond 30 minutes

Total time for the updates was 49 minutes (10:11AM to 11:02AM)

The only solution I have been able to come up with so far is set to run the updates in less than 30 minutes.  I think I'll attempt changing the "Update.bat" to "Preupdate.bat" and see if that avoids the "Shutdown timeout".

Unfortunately, I do not know when the shutdown timeout period starts or why it starts.  I was hoping the "Update timeout" was started when running the "Update.bat" file.  This does not appear to be so, sadly.

Citrix documentation implies that you should work hard to keeping the timeout below 30 minutes.

"Citrix recommends to only apply updates that can be downloaded and installed in 30 minutes or less."

======================EDIT==================
Preupdate.bat does not appear to operate under the Update Timeout either.

======================EDIT 2=================
To increase the limit you need to create a registry key called "DiskProvider" and create a dword with the decimal value of the length of time you want the *total* time called "deviceShutdownTimeout".

NOTE: This does NOT change the value in the GUI and will override the  value in the GUI, regardless of what it is set to.  You need to restart the SOAP service after making this change.  This registry key must exist on the PVS server that the site designates as the "vDisk Update Server"

ERROR 0x8024402c Windows Update

Recently, I was applying Windows Update to a 2008 system and it failed on 4 updates after being successful for months.  I was unsure why, but the updates were Office updates.  I don't think that the fact they were Office updates are important, but it's something to mention anyways.

Symptoms of the issues I found and the resolution for this issue.

1) Getting "ERROR 8024402C" when running Windows Update.
2) Checking %WINDIR%\WindowsUpdate.log reveals lines like:

2013-05-16 10:41:01:577 1404 820 DnldMgr BITS job {97BB86BA-69EA-4091-91E6-DBD1EE012652} hit a transient error, updateId = {E6EC40C4-CD27-4D9C-A8C2-CE2B8A31E903}.201, error = 0x80072EE7
2013-05-16 10:41:01:577 1404 820 DnldMgr BITS job {97BB86BA-69EA-4091-91E6-DBD1EE012652} hit a transient error, updateId = {E6EC40C4-CD27-4D9C-A8C2-CE2B8A31E903}.201, error = 0x80072EE7
2013-05-16 10:41:01:577 1404 10a0 AU  # WARNING: Download failed, error = 0x8024402C
2013-05-16 10:41:01:577 1404 10a0 AU  # WARNING: Download failed, error = 0x8024402C

To determine the cause of the issue, I used the nicely revamped Event Viewer and looked at the BITS-Client logs.  Which was a waste, nothing showed up there.  I checked the WindowsUpdateClient log and nothing was there either.  I then learned BITS uses WinHTTP when I was googling for this issue and saw there was a WinHTTP log file.  (You may have to enable analytics and debug logs).  I enabled the Diagnostics Log.

When reattempting to execute Windows Update I went back into the log and scanned through it.  I found the following:

Windows update was going to the wrong server!  The event viewer said it was going to wswsus02.XXXX.ab.ca.  This was our old server address and we since replaced it with going directly to the IP of that server via GPO.

Checking regedit for the WU preferences showed it was pointing to the correct server, but for some reason Windows Update wasn't picking up the new server.  Rebooting the machine and refreshing the GPO did not resolve the issue.

This is the correct settings

Saman suggested some fixes:


net stop wuauserv
net stop BITS
net start wuauserv
net start BITS
wuauclt /resetauthorization /detectnow
wuauclt /reportnow

These did not appear to work however.  But, we did try the following:
esentutl /p %windir%\security\database\secedit.sdb /o
Gpupdate /force

And I believe this worked.  After running this command, WinHTTP reported that it was pulling the Windows Update from the Microsoft servers, not our WSUS server:

au.download.windowsupdate.com is not our WSUS server

At this point I could have probably ran the net stop and net start commands and it may have worked, but I rebooted the server instead.

Upon the server coming back up I reran Windows Update and confirmed it was pulling from our WSUS server:

Success!

Windows Update then downloaded and installed the updates successfully!

Corrupted VHD files

Corrupt VHD files...  We utilize PVS and when we were attempting to update the target device software utilizing Hyper-V we got several error messages saying the VHD is corrupted.  The actual root of the issue is that PVS makes 16MB block size VHD files (thanks SAMAN!) and Windows 2008 only reads VHD files that are 512KB or 2MB block sized files.

Attempting to mount the VHD file via disk management fails as well.  The only utility that would mount the VHD file that was generating the error messages at the end of this post is the Citrix CVHDMount utility available via PVS (C:\Program Files\Citrix\Provisioning Services\CVHDMount.exe)

Since our Hyper-V server was separate from the PVS server, we needed to install the drivers to allow CVHDMount.exe to work.  In order to do this you need to install the drivers in this folder:
C:\Program Files\Citrix\Provisioning Services\drivers

You can right-click "Install" on the cfsdep2.inf file.

For the other driver you need to open Device Manager, go through "Add new hardware...", "Add legacy hardware" and then browse to the drivers folder and add the cvhdbusp6.inf.

Now we would mount the disk as a drive letter using this command line:

C:\Users\trententtye>"C:\Program Files\Citrix\Provisioning Services\CVhdMount.exe" -p 1 "\\wsctxapp824\d$\XenApp65Pn02.2.vhd"
Opening \\?\root#scsiadapter#0000#{c97efdbd-972c-4fe8-8279-9ce6c19fa260}
Bus interface opened.
Plugin device with SerialNumber: 1
filename: \Global??\UNC\wsctxapp824\d$\XenApp65Pn02.2.vhd

At this point you need to set the disk to "Offline" in Disk Management.

From here we can now add the disk as a physical disk to Hyper-V.

And we can now do the Target Device Update.

"---------------------------
Virtual Disk Manager
---------------------------
The file or directory is corrupted and unreadable. 
---------------------------
OK   
---------------------------

"

"[Window Title]
Settings

[Main Instruction]
Error Applying Hard Drive Changes

[Content]
Failed to modify device 'Microsoft Virtual Hard Disk'.

Cannot open attachment 'D:\XenApp65Pn02.2.vhd'. Error: 'The file or directory is corrupted and unreadable.'

Cannot get information for attachment 'D:\XenApp65Pn02.2.vhd'

Cannot open attachment 'D:\XenApp65Pn02.2.vhd'. Error: 'The file or directory is corrupted and unreadable.'

[Expanded Information]
'NewMachine' failed to modify device 'Microsoft Virtual Hard Disk'. (Virtual machine ID D8D73511-6D6B-4604-A09B-BA4F5CD35206)

'NewMachine': Cannot open attachment 'D:\XenApp65Pn02.2.vhd'. Error: 'The file or directory is corrupted and unreadable.' (0x80070570). (Virtual machine ID D8D73511-6D6B-4604-A09B-BA4F5CD35206)

'NewMachine': Cannot get information for attachment 'D:\XenApp65Pn02.2.vhd'. (Virtual machine ID D8D73511-6D6B-4604-A09B-BA4F5CD35206)

'NewMachine': Cannot open attachment 'D:\XenApp65Pn02.2.vhd'. Error: 'The file or directory is corrupted and unreadable.' (0x80070570). (Virtual machine ID D8D73511-6D6B-4604-A09B-BA4F5CD35206)

[V] See details  [Close]"

asdf