Citrix XenApp, OpenGL pass-through and Nvidia GRID cards on Amazon EC2 (G2 Instances)

I'm attempting to do a Proof of Concept (POC) for a client and one of the ideas was to utilize the Amazon EC2 cloud to provide GPU instances to the users for their applications (Maya, SolidWorks, etc.).  In order to understand how GPU sharing works, I setup my home lab to take advantage of these features first, in order to understand how it operates.

Citrix provides documentation on setting up the GPU sharing.  For my test, I'm doing this on a bare metal Citrix server.  Essentially, the notes state that OpenGL is automatically shared and enabled and special steps must be taken for DirectX, OpenCL, CUDA and Windows Server 2012.  To enable GPU sharing for XenApp for these features, the following registry file will enable these:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\Graphics Helper]
"DirectX"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper]
"DirectX"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\Multiple Monitor Hook]
"EnableWPFHook"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Multiple Monitor Hook]
"EnableWPFHook"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\Graphics Helper]
"CUDA"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper]
"CUDA"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\Graphics Helper]
"OpenCL"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper]
"OpenCL"=dword:00000001

In addition to this registry file, for Server 2012, the following Group Policy object is required:

• On Windows Server 2012, Remote Desktop Services (RDS) sessions on the RD Session Host server use the Microsoft Basic Render Driver as the default adapter. To use the GPU in RDS sessions on Windows Server 2012, enable the Use the hardware default graphics adapter for all Remote Desktop Services sessions setting in the group policy Local Computer Policy > Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Remote Session Environment.

My initial setup is a Q87M-E system with a Intel 4771 and onboard graphics.  My system is setup with Windows 2012 R2 with Citrix XenApp 7.6.

Launching an ICA session to the XenApp 7.6 server results in:

 We have OpenGL working, DirectX 11, and OpenCL (the onboard Intel GPU's do not support CUDA).  So we have a full, working implementation of GPU sharing in a ICA session on a XenApp server.

But the onboard Intel graphics card will not get the me the performance I want.  I had a Nvidia GTX 670 video card on hand to see if I can get better 3D performance.  I installed that card in the system, installed the video drivers and checked the results.

Where did my OpenGL go?  Everything else is working correctly; Direct3D, CUDA, OpenCL, but not OpenGL.  My understanding from Nvidia is that OpenGL should just be 'passed through' by Citrix.  I know that it *does* pass-through because we, literally, just saw it with the onboard Intel GPU and the Intel drivers.

My next thought is maybe it had to do with the drivers?  Maybe if I tried the Quadro drivers?  It turns out Nvidia has released special Quadro drivers that enable OpenGL in a RDP session. Maybe if I modified the INF to add my GTX670 to these special drivers I could get OpenGL to work?

It did not work.  OpenGL remained disabled in RDP/ICA sessions.

Suspecting Nvidia is doing some form of detection that is disabling OpenGL (it's probably considered a 'pro'-feature) I acquired a Quadro FX5800 and using the *same* modified Quadro drivers, these were my results:

OpenGL is now working!!

Ok, so, at this point I know how to enable GPU sharing for Citrix XenApp, I know how to check and verify it's functionality, and I know that different Nvidia cards can have OpenGL enabled or disabled but am not sure if it's the driver that matters or the hardware.  If it's the hardware I'm a bit surprised Intel would incorporate hardware accelerated OpenGL into ICA sessions for their consumer pieces but Nvidia would not for their discrete cards.  To *attempt* to test this I went and got the oldest driver I could find that would support a FX5800:

Sure enough, it works.

My last thought is maybe Nvidia has it hard coded somewhere to check for a string or a specific 'type' of video card and, if found, enable OpenGL?

My thinking is that the Nvidia drivers are doing some kind of detection and making a determination between a console session and all others.  If I'm lucky, maybe they only implemented this in their *newer* drivers, maybe after they started the RDS OpenGL acceleration...

To test this theory I went and grabbed the oldest driver I could find for my GTX 670 that would work on Windows 2012R2.  327.23.

Well now...  OpenGL is working.  This is interesting.  And leads evidence that OpenGL is being disabled in ICA via the driver.  I attempted to find when OpenGL *stopped* working.

331.82 --> Works, and now with OpenGL 4.4

337.88  -> Works

340.52 -> No OpenGL.  This driver (340.52) is now the first gaming driver *After* the "OpenGL on RDS release" (340.43).  It appears something on or after the 340.XX branch is disabling OpenGL in ICA sessions.

At the same time I was testing my Nvidia gaming GPU on my home lab, I was testing Amazon.  The GPU instance that Amazon provides utilize the Nvidia GRID K520 card as a vGPU.  This card is marketed as a 'GRID Gaming' card.  I setup this instance with Citrix XenApp and, at the time, used the latest driver (347.70).  At the time of this testing, this was my 3rd rebuild of this instance so I went with Server 2008 because my previous 2 builds were 2012 and I was convinced I was doing something wrong.  The OS shouldn't matter, but I'm noting it here.

347.70 --> No OpenGL (just like the gaming card):

Knowing that downgrading the gaming card's driver worked, I installed the oldest driver I could for the K520:

320.59 --> OpenGL Works!

Just like the gaming card.  I suspect the K520 will have the same issue as the GTX 670, and that any driver after 340.XX will disable OpenGL in a ICA session.  Unfortunately, the Grid K520 appears to only have 3 drivers to chose from, 320.59, 335.35, and 347.70.  To finish this testing I will test with 335.35:

OpenGL Works!  So it appears driver 340 and newer will disable OpenGL for ICA sessions across various types of Nvidia GPU's, but not Quadro's..

If you want OpenGL to work on Amazon EC2 instances, you must (at the time of this writing...  hopefully Nvidia corrects this over sight for all cards - consumer and not) you must use a driver older than 340.