RemoteFX is great, seconded.
I have not tried this yet, I will setup a Server 2012 tonight to test that. I will try it 2 ways, natively and then through xen. Thats right, Server 2012 with GPU passthrough, running vms using the GPU. How cool would that be? I am not so familiar with Hyper-V to know if that will work inside of a Xen domU, I suspect not. Maybe it won't whine to much about the extensions it doesnt have access to.
According to
an nVidia blog post about remoteFX, Quadro cards support remoteFX. This post was a few years prior to GRID cards, but I believe they should work too.
Will get back to the community on this.
PS: Through some testing and config verification, the offloading physx and poor GPU performance in passthrough was a problem with my mobo not truly supporting xen. Upgraded and everything is great. It is truly wonderful. (Plus I have an onboard iGPU to passthrough now, yay!)