Friday, January 15, 2016

Citrix Director 7.7 & XenApp 6.5

Citrix has 'replaced' Edgesight with a new tool called Citrix DesktopDirector. It's a monitoring tool that provides help desk or other type of technicians with data on what's going on in user sessions (when referring to XenApp). Although vastly inferior to ControlUp, it is free.

But it's not without poor documentation and its own tales of woe. We have Citrix DesktopDirector 2.1 currently live in our environment. Our environment is purely XenApp 6.5 (at the moment).

For version 2.1, logins are incredibly slow. Enumerating any sort of information on DesktopDirector is horribly slow. It's just incredi-bad.

Enter Citrix Director 7.6 7.7; lots of promises to overcome the failings of the previous version.

First, installing Citrix Director 7.7 for XenApp 6.5:

1) Launch the AutoRun (AutoSelect.exe) installation file

Select ‘Start’ for XenApp

Select ‘Citrix Director

Agree to the licensing agreement and click ‘Next

Select ‘Next

Without entering any information, select ‘Next

Select ‘Next

Select ‘Next’ for the firewall rules

Select ‘Install

Wait for the install to complete

Open IIS Manager on the Director Server. Select the Director site under Default Web Site and double-click on ‘Application Settings

Under ‘Actions’ click ‘Add'

For ‘Name’ enter ‘Service.AutoDiscoveryAddressesXA’ and for value put the IP of the local ZDC server

If you are not setting up certificates on the server for SSL, you can disable the SSL verification by changing the UI.EnableSslCheck to false.

15) Reset IIS from the command line.

On EACH XenApp server, install the Director WMI Provider (DirectorWMIProvider_x64.msi for 64bit OS's and DirectorWMIProvider_x86.msi for 32bit OS's). No reboot is needed, it takes effect immediately.

17) Enable WinRM on each XenApp server. In a command prompt run:
'winrm qc'

18) Configure WinRM with the appropriate permissions:
ConfigRemoteMgmt.exe /configwinrmuser HEALTHY\DesktopDirector.TST /all


Ensure you use the latest version you can find. For XenApp/XenDesktop 7.7 the version that comes with it is:

It has been revised as time has gone on and numerous bug fixes have been implemented. We implemented a script to update our WinRM so all of our XenApp servers would have the user configured. We didn't include checking, we just assumed if a group was added it would not be added again. With an older version of ConfigRemoteMgmt.exe this was an incorrect assumption and our WinRM SDDL became so large that the command line that ConfigRemoteMgmt.exe generates for winrm.cmd breaks the command line. You see, ConfigRemoteMgmt.exe is also a front end to "C:\Windows\system32\winrm.cmd". Here is the command it generates:

C:\Windows\system32\cmd.exe /c ""C:\Windows\system32\winrm.cmd" set winrm/config/service @{RootSDDL="O:NSG:BAD:P(A;;GA;;;S-1-5-21-38857442-2693285798-3636612711-99999999)(A;;GA;;;BA)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)"}"

The part I bolded is the GUID of the object you pass in the /configwinrmuser. The ConfigRemoteMgmt.exe takes your object and converts it to a GUID and then sets the SDDL using the native Windows tools. The bug in a pervious version of this tool is that it would add another GUID, even if one already existed. So in the buggy version the next command would be this:

C:\Windows\system32\cmd.exe /c ""C:\Windows\system32\winrm.cmd" set winrm/config/service @{RootSDDL="O:NSG:BAD:P(A;;GA;;;S-1-5-21-38857442-2693285798-3636612711-99999999)(A;;GA;;;S-1-5-21-38857442-2693285798-3636612711-99999999)(A;;GA;;;BA)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)"}"

And so on. Eventually, this causes WinRM to fail and it will just hang at this point:

To check your WinRM Root SDDL, execute this command:

"C:\Windows\system32\winrm.cmd" get winrm/config/service

If you have an output similar to this, then you have a problem:

Fortunately, it's a super easy fix. If you run the command with a single GUID it all gets reset:

C:\Windows\system32\cmd.exe /c ""C:\Windows\system32\winrm.cmd" set winrm/config/service @{RootSDDL="O:NSG:BAD:P(A;;GA;;;BA)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)"}"

And now you can execute the ConfigRemoteMgmt.exe command again. The new version of ConfigRemoteMgmt.exe doesn't seem to just 'stack' the new version on top.


Now that you have WinRM configured, Director installed, you login and it works. So you try with a HelpDesk account and it fails:

In order to trouble shoot this, we need to turn logging on for Director. To do so, go into IIS Manager Console, to the 'Director' Site, and double click on Application Settings. Turn on the following "Log." settings:

Now, create the folder you specified the log will be captured in and set the following permissions on it:

Note: The permissions are %COMPUTERNAME%\IIS_USERS

Restart IIS and attempt to login again. You should get a log file generated with content. A failed login had this output:

And a succesful login had this output:

I highlighted the lines in the successful output that showed the command succeeding and the output from that. So why did this fail? The one account "adtest91" is setup as a 'help desk' style account and has 'custom permissions'; essentially 'View-only' but with some categories removed altogether. Could this be permissions based? The odd thing is DesktopDirector 2.1 works without issue with this exact same configuration.

Fortunately, Citrix tries to make this easy to troubleshoot. I will login to the ZDC and start a Citrix PowerShell session with "adtest91" and run the command it supposedly fails on:

Well, at least it appears consistent. So, apparently, we need to get this command working. My first try at troubleshooting was to login to the AppCenter console, go to Administrators and get properties on this account, and change it's 'Privileges' from Custom to Full:

When I attempted to login it worked! So it's definitely a permission issue. I went back to Powershell and executed the same command:


So if the Powershell command specified in the log file works then 'Director' should work, and so far that is what it looks like. Now I wanted to narrow down *which* permission does this as we do not want our Help Desk to have full control over the farm. The, very obvious, first thing that stood out was:

"Edit Zone Settings." We are having issues enumerating Zones, perhaps this setting enables you to *read* zones? I enabled this setting and retested:

Success! And our 'Help Desk' account can now login to Director.

We don't really want Help Desk to have the ability to 'Edit' Zones but I don't see a 'Read-only' permission so we may have to live with it.

So now we've logged into Director and we enumerate an account but we get this message:

Cannot retrieve the data. Cannot find the referenced object. View Director server event logs for further information.

The IIS logging shows:

Citrix.Dmc.Common.NotFoundException: Exception of type 'Citrix.Dmc.Common.NotFoundException' was thrown. at Citrix.Dmc.Connector.WinRMConnector.WinRMWqlQueryExecute(String query, String wqlUri) at Citrix.Dmc.Connector.WinRMQueryThrottle.ExecuteQuery(String query, String wqlUri, Int32 timeout, Nullable`1 refreshTimeoutInSec, WinRMQueryInvoker invoker) at Citrix.Dmc.Connector.WinRMConnector.WinRMWqlQuery(String query, String wqlUri, Nullable`1 refreshIntervalInSec) at Citrix.Dmc.Connector.WinRMConnector.WinRMWqlQuery(String query, String wqlUri) at Citrix.Dmc.Common.ConnectorWrapper`1.Invoke[TReturn](Func`2 method) --- End of inner exception stack trace --- at Citrix.Dmc.Common.ConnectorWrapper`1.Invoke[TReturn](Func`2 method) at Citrix.Dmc.WebService.Wmi.WinRMConnectorQuery.ExecuteSelectQuery(String resourceUri, String className, String[] attributes, String whereClause, Nullable`1 timeoutInSec) at Citrix.Dmc.WebService.Wmi.VDAMachineDao.ExecuteSelectQuery(String resourceUri, String className, String[] attributes, String whereClause, Nullable`1 timeoutInSec) at Citrix.Dmc.WebService.Wmi.VDAMachineDao.GetRunningApplicationsData(String sessionId) at Citrix.Dmc.WebService.XAConsoleDao.GetRunningApplicationsData(String farmId, String machineId, String sessionId, Boolean detailed) at Citrix.Dmc.WebService.ConsoleService.GetRunningIMAApplicationsData(String farmId, String machineId, String sessionId, Boolean detailed) 01/14/2016 08:42:30.5628 : [t:9, s:tnccy45w1lfhxwkzaq0m0geh] Fault mapper Convert method called 01/14/2016 08:42:30.5628 : [t:9, s:tnccy45w1lfhxwkzaq0m0geh] Service exception is mapped to fault error code 104 01/14/2016 08:42:30.5628 : [t:9, s:tnccy45w1lfhxwkzaq0m0geh] Fault mapper Convert method returning 01/14/2016 08:42:30.5784 : [t:9, s:tnccy45w1lfhxwkzaq0m0geh] DmcWebErrorHandler channel level error System.ServiceModel.FaultException`1[Citrix.Dmc.WebService.DmcServiceFault]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Citrix.Dmc.WebService.DmcServiceFault). ]]>

And the event log shows:

The requested data could not be found in the data 'The virtual desktop via WinRM service reported an exception. See the event log for more information.' ('').

User: 'HEALTHY\adtest91'

Console operation: 'Retrieving running application details for IMA Session…'

Additional information:

'Exception of type 'Citrix.Dmc.Common.NotFoundException' was thrown.'

When I got this message with my 'Administrator' account I installed the Director WMI Provider and it started working. But with this 'limited' user account I was getting an this error. Amazingly, this error goes away as soon as a user logs in to Citrix Director webpage, who would also be a local administrator on the Director server. After searching and searching and troubleshooting and scanning and trying various things to get this working, the only thing that would work was to grant our Director group admin privileges on the Director web server. I finally stumbled across another article that reaffirmed that the only solution is to have the users be local admins on the web server.

To test WinRM the following command can be used:
winrm get winrm/config -r:%server%      -- Non-administrator user gets access is denied making WinRM queries

This error, though, is only if you set Connector.WinRM.Identity as 'User' and those users are not Administrators.  The fix, is to put your users into a group and make them a local admin on the Director server.


Alex said...

Exactly what I was looking for!! Thanks!! Do you know why it's not showing the Trends View? I'm only able to see the Help Desk view.

Trentent said...

Director 7.7 only supports HelpDesk view with XenApp 6.5 :(