Manually add Windows startup scripts (or inject startup scripts into an offline image)

Due to the CGP issue, our solution is to add a startup script to each vDisk.  Since I don't want to make a version of each vDisk than attach it to a server than boot it up than gpedit.msc...  We have around 10 vDisks and that process would be annoying and take a while.  So I decided to investigate doing it offline as mounting a VHD using cvhdmonut.exe and then injecting the startup script would be a lot easier.

To do that, one simply needs to browse to:

"C:\windows\system32\GroupPolicy\Machine\Scripts\Startup" (for machine startup script, aka, a script that starts when your computer starts up) or "C:\windows\system32\GroupPolicyUsers\Machine\Scripts\Startup" (for all users startup script [I'm assuming since I actually didn't go through and test the user portion]) and copy your script file there.  

Then, back out one level to C:\windows\system32\GroupPolicy\Machine\Scripts and edit Scripts.ini to include your new script file; incrementing the last line.

To:

(OS 10061)No connection could be made because the target machine actively refused it. : Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

 This has been an ongoing problem for us (Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

I may have found out why it was happening in our environment.  We are using Provisioning Services and with it we are using two NIC's, one for the Provisioning Services and one for Standard networking.

It appears the XTE service became configured to use the Provisioning Services NIC.  This was verified in the httpd.conf in the C:\Program Files (x86)\Citrix\XTE\conf folder.

Provisioning NIC and Production (network) NIC

httpd.conf as was when the system booted (and non-functional)

When I traced the XTE service using procmon.exe and wireshark with this non-functional conf this is what I saw when I attempted to launch the application:

You can see it attempt to connect to itself via 1494 but then nothing else happens

Wireshark shows virtually nothing on the network and nothing related to IMA

When I edited the file to have the Production NIC...

then restarted the XTE service and retraced via Procmon and Wireshark...

We now see tons of activity and the application now launches without issues.

================EDIT===============

We have now found why we are getting this error, and why we are getting it intermittently.  The issue is we are using PVS with multi-homed NIC's.  One NIC (LanAdapter 1) is the "Provisioning" network, and the second NIC (LanAdapter 2) is the "Production" network.  The Provisioning network is on a completely seperate vLan and sees no traffic outside of it's little network.  The ICA Listener was attaching itself to the Provisioning network instead of the production network, so when we tried to connect to the server it would fail with the CGP tunnel error because the outside network cannot talk to the Provisioning network.  To attempt to resolve this issue one of our techs (Saman) created a group policy preference registry key that set the following value (HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Terminal Server\WinStations\ICA-TCP - LanAdapter):

By setting it to "2" we could ensure the ICA listener is always listening on LanAdapter 2, our production network.  Unfortunately, a Windows Update appears to have caused either Group Policy Registry Preferences to execute (sometimes) *after* the IMAService service started, or allowed the IMAService service to start *before* Group Policy Registry Preferences.  IMAService will recreate that file every second restart.  To resolve this issue I created a startup script that executes after 65 seconds, deleting the httpd.conf file and restarting the appropriate services until the httpd.conf file is recreated.

In my testing it appears you need to restart the "IMAService" service twice to get it to recreate the httpd.conf file.  Because of this, I created the script to retry up to 3 times to try and regenerate the file.

:: ===========================================================================================================
::
:: Created by: Trentent Tye
::
::
::
:: Creation Date: May 28, 2013
:: Modified Date: May 28, 2013
::
:: File Name: Citrix_Restart_IMASrv_Delayed.cmd
::
:: Description: This script fixes the CGP Tunnel issue where the IMASrv.exe starts before
:: group policy prefencess start.  This causes the httpd.conf file in the
:: XTE folder to have the wrong IP and the ICA client listener to actually
:: be listening on the wrong NIC.  To resolve this issue we will ensure 
:: group policy is applied then restart the IMASrv service *twice*.
:: We have to do it twice because the IMASrv won't recreate the httpd.conf
:: file on the first restart.
::
:: ===========================================================================================================

@ECHO OFF
CLS

SET COUNT=0

eventcreate /ID 1 /L APPLICATION /T INFORMATION /SO "Local GP Startup Script" /D "Starting Citrix_Restart_IMASrv_Delayed.cmd script"

del /q "C:\Program Files (x86)\Citrix\XTE\conf\httpd.conf"

ECHO N | GPUPDATE /FORCE >NUL
ping 127.0.0.1 -n 65 >NUL

:RetryCreate
net stop CitrixWMIService
net stop IMAService
net stop CitrixXTEServer

ping 127.0.0.1 -n 5 >NUL
net start IMAService
net start CitrixWMIService

ping 127.0.0.1 -n 5 >NUL
net stop CitrixWMIService
net stop IMAService

ping 127.0.0.1 -n 5 >NUL
net start IMAService
net start CitrixWMIService
net start CitrixXTEServer

IF %COUNT% GEQ 3 EXIT
SET /A COUNT=%COUNT%+1

IF NOT EXIST "C:\Program Files (x86)\Citrix\XTE\conf\httpd.conf" GOTO RetryCreate
eventcreate /ID 1 /L APPLICATION /T INFORMATION /SO "Local GP Startup Script" /D "Completed Citrix_Restart_IMASrv_Delayed.cmd script"

Enable advanced logging on the XTE service of a XenApp server

Back in February or March we did Windows updates on our PVS XenApp servers and then sometime after the servers do not allow anyone to login.

The issue can crop up as a CGP Tunnel error message, a protocol driver error message or something along those lines.  We do not know why it's happening or why it only happens after we apply Windows updates from that time period.  The odd thing is it's intermittent as well, we can launch 20 systems from 1 vDisk that has the updates applied and everything will be fine for 2 weeks then, suddenly, 5 of the systems won't allow logins via ICA with the errors.  Rebooting the servers sometimes fixes it, sometimes not.  Very intermittent and very weird.  So I attempted to troubleshoot this issue again by adding:

# Log Level
loglevel debug

to the httpd.conf in the XTE folder of a server that was exhibiting these issues.  The log levels are listed here:
http://support.citrix.com/article/CTX114680

After adding those lines and restarting the XTE service the issue resolved itself!  Frustrating to be sure, and I will look at adding this line to our vDisk image so that when it crops up we'll have more diagnostic data to look at.

Citrix Provisioning Services (PVS) 6.1 - Automatic vDisk Update "Update device failed to shutdown within the timeout period."

I have setup our PVS environment to execute the vDisk Automatic Update feature utilizing a custom script (update.bat).  This script does a bunch of things, resync's the time with NTP (to avoid daylight savings issues), refreshs GPO's, executes Windows Update, cleans up temp files, runs the PVS optimizer, etc.

Unfortunately this can take longer than 30 minutes.  For some reason, when executing the ESD client as "None" (aka, so a script runs) the "Update timeout" doesn't seem to take effect, instead the 30 minute shutdown timeout is on the clock.

ESD client is set to none

You cannot increase the shutdown timeout beyond 30 minutes

Total time for the updates was 49 minutes (10:11AM to 11:02AM)

The only solution I have been able to come up with so far is set to run the updates in less than 30 minutes.  I think I'll attempt changing the "Update.bat" to "Preupdate.bat" and see if that avoids the "Shutdown timeout".

Unfortunately, I do not know when the shutdown timeout period starts or why it starts.  I was hoping the "Update timeout" was started when running the "Update.bat" file.  This does not appear to be so, sadly.

Citrix documentation implies that you should work hard to keeping the timeout below 30 minutes.

"Citrix recommends to only apply updates that can be downloaded and installed in 30 minutes or less."

======================EDIT==================
Preupdate.bat does not appear to operate under the Update Timeout either.

======================EDIT 2=================
To increase the limit you need to create a registry key called "DiskProvider" and create a dword with the decimal value of the length of time you want the *total* time called "deviceShutdownTimeout".

NOTE: This does NOT change the value in the GUI and will override the  value in the GUI, regardless of what it is set to.  You need to restart the SOAP service after making this change.  This registry key must exist on the PVS server that the site designates as the "vDisk Update Server"

ERROR 0x8024402c Windows Update

Recently, I was applying Windows Update to a 2008 system and it failed on 4 updates after being successful for months.  I was unsure why, but the updates were Office updates.  I don't think that the fact they were Office updates are important, but it's something to mention anyways.

Symptoms of the issues I found and the resolution for this issue.

1) Getting "ERROR 8024402C" when running Windows Update.
2) Checking %WINDIR%\WindowsUpdate.log reveals lines like:

2013-05-16 10:41:01:577 1404 820 DnldMgr BITS job {97BB86BA-69EA-4091-91E6-DBD1EE012652} hit a transient error, updateId = {E6EC40C4-CD27-4D9C-A8C2-CE2B8A31E903}.201, error = 0x80072EE7
2013-05-16 10:41:01:577 1404 820 DnldMgr BITS job {97BB86BA-69EA-4091-91E6-DBD1EE012652} hit a transient error, updateId = {E6EC40C4-CD27-4D9C-A8C2-CE2B8A31E903}.201, error = 0x80072EE7
2013-05-16 10:41:01:577 1404 10a0 AU  # WARNING: Download failed, error = 0x8024402C
2013-05-16 10:41:01:577 1404 10a0 AU  # WARNING: Download failed, error = 0x8024402C

To determine the cause of the issue, I used the nicely revamped Event Viewer and looked at the BITS-Client logs.  Which was a waste, nothing showed up there.  I checked the WindowsUpdateClient log and nothing was there either.  I then learned BITS uses WinHTTP when I was googling for this issue and saw there was a WinHTTP log file.  (You may have to enable analytics and debug logs).  I enabled the Diagnostics Log.

When reattempting to execute Windows Update I went back into the log and scanned through it.  I found the following:

Windows update was going to the wrong server!  The event viewer said it was going to wswsus02.XXXX.ab.ca.  This was our old server address and we since replaced it with going directly to the IP of that server via GPO.

Checking regedit for the WU preferences showed it was pointing to the correct server, but for some reason Windows Update wasn't picking up the new server.  Rebooting the machine and refreshing the GPO did not resolve the issue.

This is the correct settings

Saman suggested some fixes:


net stop wuauserv
net stop BITS
net start wuauserv
net start BITS
wuauclt /resetauthorization /detectnow
wuauclt /reportnow

These did not appear to work however.  But, we did try the following:
esentutl /p %windir%\security\database\secedit.sdb /o
Gpupdate /force

And I believe this worked.  After running this command, WinHTTP reported that it was pulling the Windows Update from the Microsoft servers, not our WSUS server:

au.download.windowsupdate.com is not our WSUS server

At this point I could have probably ran the net stop and net start commands and it may have worked, but I rebooted the server instead.

Upon the server coming back up I reran Windows Update and confirmed it was pulling from our WSUS server:

Success!

Windows Update then downloaded and installed the updates successfully!