[MOVED] Hacking NVidia Cards into their Professional Counterparts

[mosmo]

Update:
As some of you that have been following this thread are aware, I've been having a rather bizzare problem with only DL-DVI working on my modifed GK104 based cards (680 and 690). It just occurred to me that the two cards I have happen to have on thing in common - they are both Gainward cards. Has anyone else successfully managed to get DL-DVI to work in a VM with a Gainward GTX680 Phantom 4GB or Gainward GTX690? Gainward 690s have a few strapping resistors in different places to the EVGA and other 690s, so it is plausible they made some tweaks that cause the problem. Does anyone have either of those cards working with DL-DVI outputs successfully?

On a separate note, I just learned that GTX780Ti has device ID 0x100A. K6000 is 0x103A. We only need to change the 3rd nibble (via oguz286's awesome mod). His mod is to use a 33K resistor to jack up the 3rd nibble from 0 to 2. On my Titan using an 18K resistor instead boosts the ID to 3. So to make a 780Ti to a K6000, simply apply an 18K resistor between VCC and SCLK and voila, job done. For extra points, solder a couple of wires between those pins, solder an 18K resistor on one of them, and a switch to connect them. Break the switch out somewhere accessible (extra extra points for making the switch easily and neatly accessible from the back of the card without obstructing the airflow too badly). Now you can switch between a 780Ti and a K6000 at a flip of a single switch.

Needless to say, the neatness of this means that I'll be acquiring a 780Ti as soon as I've ebayed my Titan.

That has made my day! Looking forward to a mod on a Palit, Zotac or Asus non ref 780ti. If this works, I will defo be buying one of those.

[Macker]

Hey guys, I'm new here and I know there is probably a place/thread to introduce myself, but I couldn't keep myself away from this one.

I work in 3D graphics and have been looking at moving into GPU rendering, but the cost of the quadro's is just so damn much! I am currently using a k4000 at work, but on my home rig that I am speccing, I am seriously considering this kind of mod. Whilst I'm sure this has been covered in this thread, it's already at 52 pages long(!) so I guess I'll have to ask again;

1) what are the potential pitfalls of this?
2) I am no electrical technician/engineer, so how much will I need to learn to do this? I am willing to put a bit of time into it.

Thanks in advance, and well done on the hack!

[gordan]

I work in 3D graphics and have been looking at moving into GPU rendering, but the cost of the quadro's is just so damn much! I am currently using a k4000 at work, but on my home rig that I am speccing, I am seriously considering this kind of mod. Whilst I'm sure this has been covered in this thread, it's already at 52 pages long(!) so I guess I'll have to ask again;

1) what are the potential pitfalls of this?
2) I am no electrical technician/engineer, so how much will I need to learn to do this? I am willing to put a bit of time into it.

1) Your application may not befit. Modified cards do not have the full feature set of the real Quadros. For example, the SPECviewperf scores are no different from a normal GeForce card, and stereo 3D remains unavailable.

2) For Tesla and Fermi series cards (GeForce 2xx/3xx/4xx) no hardware modification is required - you just need to modify the BIOS by about half a byte. See the link I posted above (or website link under my profile) for more details on that. What cards does your application support for rendering? If it supports GeForce cards, you probably don't need to modify them. If it only supports Quadros, see if it supports Quadro 5000 or 6000. If so, I would suggest you get yourself a relatively cheap 4xx series card (GTX470 or GTX480), and modify them into a corresponding Quadro (470 -> 5000, 480 -> 6000), and see how that fares.

Almost all of us here are mainly interested in the modding to get virtualization features and TCC working (those are the big wins), so if you decide to go down this route, please report your findings/results/before+after results - it would be nice to get some feedback from someone using this for 3D rendering purposes.

[Macker]

I work in 3D graphics and have been looking at moving into GPU rendering, but the cost of the quadro's is just so damn much! I am currently using a k4000 at work, but on my home rig that I am speccing, I am seriously considering this kind of mod. Whilst I'm sure this has been covered in this thread, it's already at 52 pages long(!) so I guess I'll have to ask again;

1) what are the potential pitfalls of this?
2) I am no electrical technician/engineer, so how much will I need to learn to do this? I am willing to put a bit of time into it.

1) Your application may not befit. Modified cards do not have the full feature set of the real Quadros. For example, the SPECviewperf scores are no different from a normal GeForce card, and stereo 3D remains unavailable.

2) For Tesla and Fermi series cards (GeForce 2xx/3xx/4xx) no hardware modification is required - you just need to modify the BIOS by about half a byte. See the link I posted above (or website link under my profile) for more details on that. What cards does your application support for rendering? If it supports GeForce cards, you probably don't need to modify them. If it only supports Quadros, see if it supports Quadro 5000 or 6000. If so, I would suggest you get yourself a relatively cheap 4xx series card (GTX470 or GTX480), and modify them into a corresponding Quadro (470 -> 5000, 480 -> 6000), and see how that fares.

Almost all of us here are mainly interested in the modding to get virtualization features and TCC working (those are the big wins), so if you decide to go down this route, please report your findings/results/before+after results - it would be nice to get some feedback from someone using this for 3D rendering purposes.

Well basically I run 3DS Max, in which the viewport performance gains between quadro & gtx are pretty huge; but as for 3D rendering the GTX cards are on a par with the high end quadro's to be honest, so that's not really where I'd be looking at getting any benefits. All CUDA enabled cards are supported in the rendering package (VRay) afaik.

I'd purely be doing it so that I can get higher FPS whilst modelling things. I work in architectural visualisation, so (due to high res foliage usually!) I often end up with 10~20 million polygon models, which would grind to a halt on a GTX. I am looking to set up a 3D PC at home, but don't have the kind of budget for a quadro - hence being so interested in these mods.

[Edit]I'm still not entirely convinced that GPU rendering is quite there yet; the main reason being the ram limitations of GPU. The K6000 is the first card I've seen that I've thought would be able to load my scenes without any issues. Sadly, whilst adding more and more GPU's to your rig will speed up the rendering process (sometimes almost realtime), it doesn't have the same effect with the vRAM - no matter how many 4GB cards you stick in, you will only ever have 4GB available because it isn't shared - each card must load all of the assets required to render...

Meaning the k6000's biggest selling point for me (and many others) is the ram available, not the CUDA cores.

[gordan]

I work in 3D graphics and have been looking at moving into GPU rendering, but the cost of the quadro's is just so damn much! I am currently using a k4000 at work, but on my home rig that I am speccing, I am seriously considering this kind of mod. Whilst I'm sure this has been covered in this thread, it's already at 52 pages long(!) so I guess I'll have to ask again;

1) what are the potential pitfalls of this?
2) I am no electrical technician/engineer, so how much will I need to learn to do this? I am willing to put a bit of time into it.

1) Your application may not befit. Modified cards do not have the full feature set of the real Quadros. For example, the SPECviewperf scores are no different from a normal GeForce card, and stereo 3D remains unavailable.

2) For Tesla and Fermi series cards (GeForce 2xx/3xx/4xx) no hardware modification is required - you just need to modify the BIOS by about half a byte. See the link I posted above (or website link under my profile) for more details on that. What cards does your application support for rendering? If it supports GeForce cards, you probably don't need to modify them. If it only supports Quadros, see if it supports Quadro 5000 or 6000. If so, I would suggest you get yourself a relatively cheap 4xx series card (GTX470 or GTX480), and modify them into a corresponding Quadro (470 -> 5000, 480 -> 6000), and see how that fares.

Almost all of us here are mainly interested in the modding to get virtualization features and TCC working (those are the big wins), so if you decide to go down this route, please report your findings/results/before+after results - it would be nice to get some feedback from someone using this for 3D rendering purposes.

Well basically I run 3DS Max, in which the viewport performance gains between quadro & gtx are pretty huge; but as for 3D rendering the GTX cards are on a par with the high end quadro's to be honest, so that's not really where I'd be looking at getting any benefits. All CUDA enabled cards are supported in the rendering package (VRay) afaik.

I'd purely be doing it so that I can get higher FPS whilst modelling things. I work in architectural visualisation, so (due to high res foliage usually!) I often end up with 10~20 million polygon models, which would grind to a halt on a GTX. I am looking to set up a 3D PC at home, but don't have the kind of budget for a quadro - hence being so interested in these mods.

My 3DSMax viewport performance experiments showed no difference at all between a GTS450, GTS450 modified into a Quadro 2000, and a genuine Quadro 2000. I then tested it against a GTX470/GTX480/GTX580 and each of those modified into the corresponding Quadro (5000, 6000, 7000), and found no change again - they all managed about 7fps viewport performance, on a C2Q@3.2GHz, which implies it is software rendered and CPU bound (bearing in mind that a C2Q@3.2GHz is not by any sane measure an underpowered CPU).

As for being RAM limited - you are pretty much out of luck on that one. GeForce cards usually (there are exceptions) come with a lot less RAM than the corresponding Quadros, e.g. 6GB on Q6000 vs. 1.5GB on a GTX480, other things being very similar). There are 4GB variants of GTX680, which makes it near identical to the K5000. I'm still working on a Titan-K6000 mod, but that is still only 6GB of VRAM. If 6GB isn't enough I'd say you are well and truly out of luck.

[Macker]

Really? Because everybody that I've spoken to that has had the chance to compare (non hacked) GTX & quadro cards has said the opposite.

What version of Max were you using? The nitrous viewport in later versions has improved version on version when it comes to performance, though whether this is taking more advantage of the GPU or not is up for debate. I don't think that it is CPU bound, because the performance in the viewport literally grinds to a halt when I start rendering a scene on the GPU and trying to work in the viewport alongside it, despite low CPU load.

[gordan]

Really? Because everybody that I've spoken to that has had the chance to compare (non hacked) GTX & quadro cards has said the opposite.

You should probably do your own testing. You can do so on the cheap with a Quadro 2000 (genuine ones go for so little on ebay there is no point on modifying GTS450s) and a GTS450 (DDR5 variant!). Try them in turn and see what you get. Then see if modifying the GTS450 into a Q2000 makes any difference.

I don't think anyone should take any 3rd party benchmarks and reports at face value - do your own, with your expected workload, and see what you get.

And when you are done you can probably ebay them and recover most of your investment.

What version of Max were you using?

The latest downloadable demo/trial version about 2-3 months ago.

The nitrous viewport in later versions has improved version on version when it comes to performance, though whether this is taking more advantage of the GPU or not is up for debate. I don't think that it is CPU bound, because the performance in the viewport literally grinds to a halt when I start rendering a scene on the GPU and trying to work in the viewport alongside it, despite low CPU load.

That slowdown could be explained by all sorts of other things, such as extra DMA I/O to/from the card (this is where bidirectional async DMA on the modified 470/480 might help you) and extra context switching. I never tested the performance impact of rendering on viewports speed, I just ran a viewport benchmark test I found on a forum, and found it hovered between 5fps and 7fps regardless of which GPU I swapped into the machine.

[vacaloca]

For anyone that wants to play with a 780Ti, TigerDirect has them for $649.99 after $50 rebate 11/29 only:
http://slickdeals.net/f/6500748-nvidia-geforce-gtx-780-ti-649-99-fs-w-3-free-games-on-tigerdirect

I don't think the Visa.me $30 off stacks with $30 AMEX Sync offer, so doing either of them would make it $619.99.

[mrkrad]

He's probably the dude that flashes cards on ebay for mac users. I think before recently, many apps would unlock some speed, or would work better if the OSX saw two "quadro-like" cards. IE gtx 780 modded uefi rom plus a real quadro for viewport.

The situation is vastly different for (linux,esxi,xen) and osx/windows as far as ease of tricking "the application". No trickery is being done to the card really. more the app

[gordan]

Update:
1) GTX680 -> Tesla K10 mod by simply deleting the 4th nibble resistor works fine for VGA passthrough. It seems to yield all the advantages of modding to Tesla/Quadro/Grid while keeping things trivially simple. And since the resistor to be removed is on the back of the PCB, you don't even have to remove the heatsink or break any "warranty void" stickers, if you are concerned about such things. This really does reduce the mod complexity to a completely trivial level.

2) The DL-DVI issue I mentioned a few times before is a Gainward specific issue on both the 680 and 690. I just got a MSI GTX680, and the problem does not manifest on that card. So if you are planning to do this, avoid Gainward cards.

[dredok]

Update:
1) GTX680 -> Tesla K10 mod by simply deleting the 4th nibble resistor works fine for VGA passthrough. It seems to yield all the advantages of modding to Tesla/Quadro/Grid while keeping things trivially simple. And since the resistor to be removed is on the back of the PCB, you don't even have to remove the heatsink or break any "warranty void" stickers, if you are concerned about such things. This really does reduce the mod complexity to the level of completely trivial.

2) The DL-DVI issue I mentioned a few times before is a Gainward specific issue on both the 680 and 690. I just got a MSI GTX680, and the problem does not manifest on that card. So if you are planning to do this, avoid Gainward cards.

so according to summary table of verybigbadboy (https://www.eevblog.com/forum/chat/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550) it says the second resistor should be a 40k one. Actually he says he puts a 100k one because ran some stability issues ... what am I missing?

[gordan]

Update:
1) GTX680 -> Tesla K10 mod by simply deleting the 4th nibble resistor works fine for VGA passthrough. It seems to yield all the advantages of modding to Tesla/Quadro/Grid while keeping things trivially simple. And since the resistor to be removed is on the back of the PCB, you don't even have to remove the heatsink or break any "warranty void" stickers, if you are concerned about such things. This really does reduce the mod complexity to the level of completely trivial.

2) The DL-DVI issue I mentioned a few times before is a Gainward specific issue on both the 680 and 690. I just got a MSI GTX680, and the problem does not manifest on that card. So if you are planning to do this, avoid Gainward cards.

so according to summary table of verybigbadboy (https://www.eevblog.com/forum/chat/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550) it says the second resistor should be a 40k one. Actually he says he puts a 100k one because ran some stability issues ... what am I missing?

I think what you are missing is that not all cards are the same. Every manufacturer adds little tweaks that cause minor issues like this. My findings are:

1) Deleting both resistors for 680 -> Grid K2 works on at least on Gainward 680 and Gigabyte 680
2) Deleting the 4th nibble resistor works fine for converting a MSI 680 into a Tesla K10. Have not tried removing the 3rd nibble resistor on that card.
3) Deleting the 4th nibble resistor works for converting EVGA and Gainward 690s to Tesla K10
4) Replacing the 3rd nibble resistor with alternate location 20KO one on the Gainward 690 renders the card unbootable - it attenuates the BIOS lines too much and even though the device ID reads correct (0x11BF), the BIOS can not be read (but it can be erased, so be careful).
5) On both EVGA and Gainward 690 deleting (without replacing) the 3rd nibble resistor causes instability in the 3rd nibble - it will flap between A and B

[mosmo]

Ok, so modding an EVGA 690 is not that difficult then. So, by modding it, can you still take advantage of the Dual GPU which is akin to SLI? I might buy one and do the mod, will tie me over until the 780ti is 100% modded.

[gordan]

Ok, so modding an EVGA 690 is not that difficult then. So, by modding it, can you still take advantage of the Dual GPU which is akin to SLI? I might buy one and do the mod, will tie me over until the 780ti is 100% modded.

690 doesn't have XP64 drivers (only Windows I use), and I don't see a SLI option in the nvidia control panel when it's running as a Grid K2 / Tesla K10. I never tried passing more than one GPU to a VM.

I'm more than a little annoyed than my 690 is a Gainward with the bizzare DL-DVI issue, because I really wanted to run with a 690 split between two VMs, due to a slot shortage and an uncrippled GK104 is good enough to run anything (including Crysis at max settings) but the worst programmed games (*cough*Metro Last Light*cough*). But now I've got a Titan on a workbench and a 780Ti in the post, it's hard to not justify using those instead - if the mod goes as planned.

I'll have the 780Ti mod tested within a few days (however long it takes for mine to arrive in the post, which is going to be days rather than weeks), so I'd advise you don't bother buying a 690 for those few days (by the time a 690 you buy today arrives, the 780Ti mod will be tested). Unless you want to trade a tested Teslified 690 for a tested Quadrified 780Ti afterwards (what part of the world are you in?)? Just make sure that if you get a 690 it isn't a Gainward, or you'll be limited to 1280x800 when virtualized.

[mosmo]

Ok, so modding an EVGA 690 is not that difficult then. So, by modding it, can you still take advantage of the Dual GPU which is akin to SLI? I might buy one and do the mod, will tie me over until the 780ti is 100% modded.

690 doesn't have XP64 drivers (only Windows I use), and I don't see a SLI option in the nvidia control panel when it's running as a Grid K2 / Tesla K10. I never tried passing more than one GPU to a VM.

I'm more than a little annoyed than my 690 is a Gainward with the bizzare DL-DVI issue, because I really wanted to run with a 690 split between two VMs, due to a slot shortage and an uncrippled GK104 is good enough to run anything (including Crysis at max settings) but the worst programmed games (*cough*Metro Last Light*cough*). But now I've got a Titan on a workbench and a 780Ti in the post, it's hard to not justify using those instead - if the mod goes as planned.

I'll have the 780Ti mod tested within a few days (however long it takes for mine to arrive in the post, which is going to be days rather than weeks), so I'd advise you don't bother buying a 690 for those few days (by the time a 690 you buy today arrives, the 780Ti mod will be tested). Unless you want to trade a tested Teslified 690 for a tested Quadrified 780Ti afterwards (what part of the world are you in?)? Just make sure that if you get a 690 it isn't a Gainward, or you'll be limited to 1280x800 when virtualized.

wow, I like your desire to risk bricking those very expensive cards  See PM.

[gordan]

wow, I like your desire to risk bricking those very expensive cards  See PM.

The risk of permanent damage is pretty close to 0.

[mosmo]

wow, I like your desire to risk bricking those very expensive cards  See PM.

The risk of permanent damage is pretty close to 0.

You haven't seen me with a scalpel, knife or any sharp instrument for that matter 

[beckend]

The i got questions about the "easy, remove 4th nimble gtx680 trick", it would result in a Tesla K10.
Would the DVI/HDMI outputs actually work if used on host OS, not VM?
Very tempted to do it on my Palit GeForce GTX 680 Jetstream 4GB.

[gordan]

The i got questions about the "easy, remove 4th nimble gtx680 trick", it would result in a Tesla K10.
Would the DVI/HDMI outputs actually work if used on host OS, not VM?

The output ports continue to work just fine, VM or bare metal, it makes no difference. What controls the configuration of the output ports is the BIOS. Since we are not changing the output port configuration in the BIOS, they will continue to work just fine. If you were to flash a K10 BIOS onto the card, the output ports would almost certainly stop working since the BIOS on the K10 assumes there aren't any. The driver just checks what the BIOS set up in terms of output ports and runs with it. I first test my modified cards on bare metal before trying them in the VM, and it works the same regardless of whether it's on bare metal or virtualized.

Note that Grid K2 also doesn't have any output ports, and a lot of people here have modified their 680s into those, and they continue to work just fine while providing video output.

[WillV]

Has anyone used their 640/K1 or 680/K2 cards for the following:

http://www.nvidia.com/object/virtual-gpus.html


More specifically as it applies to this part:

PREPARING FOR AND INSTALLING NVIDIA GRID VGPU

Step 1: Make sure you have configured XenServer 6.2.0, XenServer vGPU Tech Preview Hotfix, and XenDesktop 7.1 Tech Preview. Visit www.citrix.com/go/vgpu for more information.
Step 2: Download the NVIDIA GRID vGPU Software by visiting www.nvidia.com/drivers and selecting:
Product Type: GRID.
Product Series: NVIDIA GRID vGPU.
Product: GRID K1 or K2.

The GRID vGPU software package includes:
Release Notes
GRID vGPU User Guide
GRID vGPU Manager
Driver for GRID K1 or K2

[mrkrad]

well that was my problem. I'd be glad to help with my K2 but I need to get a "demo" or "trial" of the xen server preview to do vgpu. I only have esxi 5.5 (api-intercept) and server 2012R2 (api-intercept).

I know "trials" and "demo use" keys exist, I'm not even considering a "pir8" copy.

Anyone have any help here? I think that by shadowing each step and comparing output there would be a more likely chance to get it going with 680/690 but there are a myriad of bugs with vgpu right now. the readme's show tons of issues (screens going black over 1920x1080, iommu mapping to the wrong area due to 64bit and 4gb vram issues, not doing the right command in the right order=fail,rdp disables hardware acceleration period).

If you check the forums on xen for getting VGPU working its literally a minefield of this gpu bios and that server bios and being on the right 2nd tuesday of the month to get full hardware vgpu going.

API intercept does not use an nvidia bios or driver in the VM.

vgpu shared is what i'd love to try out. alas no software

[mosmo]

well that was my problem. I'd be glad to help with my K2 but I need to get a "demo" or "trial" of the xen server preview to do vgpu. I only have esxi 5.5 (api-intercept) and server 2012R2 (api-intercept).

I know "trials" and "demo use" keys exist, I'm not even considering a "pir8" copy.

Anyone have any help here? I think that by shadowing each step and comparing output there would be a more likely chance to get it going with 680/690 but there are a myriad of bugs with vgpu right now. the readme's show tons of issues (screens going black over 1920x1080, iommu mapping to the wrong area due to 64bit and 4gb vram issues, not doing the right command in the right order=fail,rdp disables hardware acceleration period).

If you check the forums on xen for getting VGPU working its literally a minefield of this gpu bios and that server bios and being on the right 2nd tuesday of the month to get full hardware vgpu going.

API intercept does not use an nvidia bios or driver in the VM.

vgpu shared is what i'd love to try out. alas no software

Isnt latest Xen Server 6.2.0 Open Source? If you want support you need to buy commercial version...

[gordan]

Passthrough and vGPU have different issues.

With passthrough the problem is always with the motherboard (system BIOS and PCIe bridges). There is a reason why both Citrix and VMware only have about 2 servers/workstations from HP and Dell certified for VGA passthrough / vDGA.

The vast majority of motherboards out there have BIOSes that were written by the lowest bidding monkeys and were never tested for operation with anything but bare metal Windows (and even then they have to issue a dozen or so BIOS updates to fix various problems during the lifetime of the product). Prime example being that the vast majority of BIOSes don't map 64-bit IOMEM regions above the 4GB range - which means that if your total BAR sizes add up to more than whatever the maximum amount is that the BIOS is capable of mapping (usually not more than 3GB on such broken implementations), it all breaks spectacularly and doesn't work at all. This is particularly bad if you want to modify your Nvidia card to make sure the BAR region that allows direct access to VRAM is at least equal to VRAM size (so you can use the whole of VRAM as a block device, for example). The card rightly advertises the BAR in question as 64-bit addressable, but the BIOS will fail to map it if the BAR is too big to fit into low memory.

If the motherboard also contains chips like the NF200 PCIe brodges designed with too-smart-by-half ideas (clever I/O boost by doing DMA directly and bypassing the upstream PCIe hub, which unfortunately breaks IOMMU operation because for IOMMU to work all DMA traffic must pass through it, or else the virtual-to-physical IOMEM remapping cannot happen). Sometimes it is possible to work around this by ensuring that virtual BARs are at the same addresses as the physical BARs, but that only currently works with PV Xen containers (e820_host=1 option). Hopefully it will also work with PVH guests so that guest OS-es other than Linux can benefit from it. My gross temporary hack (pending PVH availability in Xen 4.4) to get it working was to patch hvmloader to mark the whole of the  1GB-4GB range as reserved to avoid the domU stomping all over the physical BARs of various hardware and crashing the host in various potentially data corrupting ways.

vGPU is completely different and in theory should be a lot more forgiving toward bad hardware implementations. It should also be a lot less picky about what physicalGPU is used on the host to accelerate the rendering. The key problem here is that ESX vSGA documentation is rather lacking, and the similar Xen solution is still rather immature.

XenServer is open source, but to get access to the vGPU stuff you have to at least sign up for an account on the site. I'm not sure if there is extra licensing involved - I haven't looked at it particularly closely because I'm having enough issues with the supposedly much more mature VGA passthrough operation (and I have no immediate use for vGPU stuff).

[mrkrad]

well if you happen to find yourself with the vgpu shared stuff - pick it up. Perhaps there are answers in all of that. IE if we get true GRID VGX to work, maybe it will take care of the basic problems. (or not), but i'm offering to mirror your setup.

The BAR problem might be. Since I have only older Westmere workstations and servers (L5639 $70 6-core lower power westmere). I think my most expensive server is a $599 DL380 with dual L5639 lol. with a video card that is several times more valuable.

I'm starting to think video and networking are both doing price fixing.

When street value is $1500 for a $7000 item.. folks are yanking your chain. same thing exists in the 10gbe switch arena about 1 year ago it was $1250 used for 24-port 10gbe and $7000 new. [now you can get 48 port 10gbe + 4 40gbe for $9999 new]..

New dual port 10gbe nic $1000 - used $75. New DAC or SFP+ $250 - used $40.

Same thing with pro video... which is why I am determined to work this grid thing out.

I'm pretty sure you need xenserver 6.2 plus xendesktop 7.x preview and the desktop portion is the big money remote graphics VGPU portion and definitely not free.

If you think about it, the hyper-visor only needs to run the VGX software, the desktop "tools" - vmware tools/drivers - that's where the nvidia USM bios/driver has to work its magic inside the vm.

I think ESXi consolidates all of this (free), where as XEN can sell both parts individually.

If the OS is 64bit - why does it matter where the BAR maps? Would you want to map to top of 64gb addressable minus VRAM? I understand many videocards with 4gb run into the old problem of only having 3.6gb usable like system ram. weird. I will have to find out if any of my Z400/HP Gen6-7 servers have this option given how old they are.

[WillV]

I'm pretty sure you need xenserver 6.2 plus xendesktop 7.x preview and the desktop portion is the big money remote graphics VGPU portion and definitely not free.

If you think about it, the hyper-visor only needs to run the VGX software, the desktop "tools" - vmware tools/drivers - that's where the nvidia USM bios/driver has to work its magic inside the vm.

I think ESXi consolidates all of this (free), where as XEN can sell both parts individually.

This is not entirely correct.  I have not seen anywhere that VMWare is doing VGX as of yet, instead opting to stick with their vSGA.  There are some similarities but they are definitely not the same thing.  Citrix is the first to release NVidia VGX integration, likely stemmed from their partnership work with NVidia to create this and providing software for NVidias demo servers.  At the very least, when comparing VMWare vSGA with Citrix and VGX, you are leaving out the required VMWare Horizon View package which is definitely not free.