[MOVED] Hacking NVidia Cards into their Professional Counterparts

[gordan]

Really? Nvidia have modified their drivers to allow passthrough with GeForce cards?

[combsbj]

No, VMware ESXi allows it to be passed through.  It shows in the VM as a gtx 650 as it should.  It shows correctly in GPU-Z as well. But, I cannot do anything with it. View client will only use it if the quadro driver is loaded.  I'm not aware of any other remote desktop client that can use it either.  I thought Splashtop maybe, but so far no luck.

Alternatively, I have a Quadro FX 4400 and ESXi will not even let me enable PCI passthrough for it at all.

[gordan]

Ah, I see. Yes, this is indeed the normal symptom. You can pass the card through as a normal PCI device, but the driver inside the VM will refuse to initialize it if it's device ID isn't in the whitelist.

[The_Moves]

So has anyone done a GTX 760? From reading most of this thread, it is basically a no - is it just because no one has tried? I have an EVGA 760 that I can't use as i'm messing with ESXi and GPU Pass-through. I ended up buying a GTX 465 and converting it to a Quadro 4000 - works great in Windows 7 with NVIDIA drivers installed and I can play Crysis 2 on moderately high settings. Not nearly as smooth as running my system bare metal with my GTX760. Originally, the reason I converted my GTX 465 was to use it with OS X as a VM, but havn't been able to get the Display to initalize - here I will attempt to get the Q4000 passed through to a linux VM and working, then maybe i'll understand how to do it in OS X.

Should I try to get some 10/15x pictures of my GTX 760 and splice them together? Would that help people looking at the card? Does removing the heatsink void the warranty?

I did find this picture, and it appears that the required resistors are near the top right heatsink screw:

http://images.anandtech.com/doci/7103/NVIDIA_GeForce_GTX_760_Back.jpg

[gordan]

GTX760 is a rebadged GTX660, which is a GK104 same as 670 and 680. There is a good chance the PCB is similar enough that the resistor locations are the same. I'd say you should compare it with the photos already posted about modifying the 670 and 680 cards, the chances are the relevant parts will be the same. If your card's device ID is 0x1185 or 0x1187 you should be able to just remove the resistor for the 4th nibble and turn it into a K10.

[The_Moves]

My goal is to use it for GPU Passthrough, so I was under the impression I would need a GRID K1 or GRID K2? Or even a K500 since it is a GK104 - per wikipedia.

Is the TESLA K10 capable of GPU Passthrough? I don't really need or have a desire for vSGA Shared Graphics, just dedictated GPU to a VM which I will connnect to a monitor

[gordan]

Tesla, Quadro, Grid, they are all (except the cheap low end NVS models) white-listed in the driver to let it initialize the card despite detecting it is running in a VM. I'm using a GTX680 modified into a K10 on one of my machines.

[Raim]

I set microswitch with resistor 18k in 780ti and changed memory size in BIOS from Quadro.
I parse BIOS dump with NVBIOS( Thanks to Gordan for the hint with Gainward 2Gb and 4Gb. ) and found a difference between 780ti and Quadro in R[0x11e200] and R[0x11e204].
One can eliminate it and get 3Gb with Quadro BIOS.

But double precision performance in CUDA-Z not changed. May be it hard strapped, or cut from crystal, or as in Titan case: "Full speed FP64 is disabled by default on GTX Titan. FP64 can be enabled in NVIDIA control panel.", but Quadro not show this option...


 

[gordan]

Cool, any chance of a a hex diff for the Q6000 BIOS to adjust the memory to 3GB?

[Raim]

Here is HexDiff image.

And parsed ROM from 780ti
0xa8ab: 8f 00 e2 11 00 04 02                           RAM_RESTRICT_ZM_REG_GROUP   R[0x11e200] 0x04 0x02
0xa8b2:                                                   R[0x11e200] = {
0xa8b2: 00 90 cf 01                                          0x01cf9000
0xa8b6: 00 90 cf 01                                          0x01cf9000
0xa8ba: 00 90 cf 01                                          0x01cf9000
0xa8be: 00 90 cf 01                                          0x01cf9000
0xa8c2: 00 90 8f 03                                          0x038f9000
0xa8c6: 00 90 8f 03                                          0x038f9000
0xa8ca: 00 90 8f 03                                          0x038f9000
0xa8ce: 00 90 8f 03                                          0x038f9000
0xa8d2:                                                   }
0xa8d2:                                                   R[0x11e204] = {
0xa8d2: 00 90 55 02                                          0x02559000
0xa8d6: 00 90 55 02                                          0x02559000
0xa8da: 00 90 44 02                                          0x02449000
0xa8de: 00 90 44 02                                          0x02449000
0xa8e2: 00 90 55 02                                          0x02559000
0xa8e6: 00 90 55 02                                          0x02559000
0xa8ea: 00 90 55 02                                          0x02559000
0xa8ee: 00 90 55 02                                          0x02559000
0xa8f2:                                                   }

and not modified  Quadro
0xa8c6: 8f 00 e2 11 00 04 02                           RAM_RESTRICT_ZM_REG_GROUP   R[0x11e200] 0x04 0x02
0xa8cd:                                                   R[0x11e200] = {
0xa8cd: 00 90 cf 01                                          0x01cf9000
0xa8d1: 00 90 cf 01                                          0x01cf9000
0xa8d5: 00 90 cf 03                                          0x03cf9000
0xa8d9: 00 90 cf 03                                          0x03cf9000
0xa8dd: 00 90 cf 01                                          0x01cf9000
0xa8e1: 00 90 cf 01                                          0x01cf9000
0xa8e5: 00 90 cf 03                                          0x03cf9000
0xa8e9: 00 90 cf 03                                          0x03cf9000
0xa8ed:                                                   }
0xa8ed:                                                   R[0x11e204] = {
0xa8ed: 00 90 55 02                                         0x02559000
0xa8f1: 00 90 55 02                                       0x02559000
0xa8f5: 00 a0 66 02                                          0x0266a000
0xa8f9: 00 a0 66 02                                          0x0266a000
0xa8fd: 00 a0 55 02                                          0x0255a000
0xa901: 00 a0 55 02                                          0x0255a000
0xa905: 00 a0 55 02                                          0x0255a000
0xa909: 00 a0 55 02                                          0x0255a000
0xa90d:                                                   }

[Raim]

Compare the hard strap resistor position on Quadro and 780. As you can see minimum two resistors position changed. May be 18k mod not fully convert 780ti to Quadro.

[gordan]

Now that is quite interesting. I don't suppose you actually have a K6000 to measure the resistances on it?

[Raim]

This photo is part of
http://www.coolingconfigurator.com/upload/pictures/Nvidia-Quadro-K6000-12GB-GDDR5---PCB.jpg

[vacaloca]

is anyone going to mod a titan black? it has a pci id of 10DE 100C. is this easier to mod than titan?

I'd be interested to know as well. Will have a Titan Black in my hands sometime next week.

[Raim]

I'd be interested to know as well. Will have a Titan Black in my hands sometime next week.

Can anyone read the Titan card's strap.
Getting the card's strap is done by peeking at 0x00101000. The strap is an hardwired register that gives some knowledge upon the specific card. It is set by the gpu manufacturer and cannot be extracted from the vbios.
The instruction is from https://github.com/pathscale/pscnv/wiki/TestingTimings#get-your-cards-strap

Ubuntu terminal

Code: [Select]

sudo apt-get install git libpci-dev libpciaccess-dev libxml2-dev cmake flex bison libx11-dev libvdpau-dev libxext-dev
git clone http://github.com/envytools/envytools/
cd envytools
cmake .
make
cd ..
sudo ./envytools/nva/nvapeek 0x00101000 > nv_strap_peek.txt
Please post the generated file nv_strap_peek.txt

[gordan]

I am about 90% certain the strap you speak of is set by the resistors on the PCB - what we have been hacking here. The soft strap also exists which can be set by vbios, but that is nowdays used by the likes if Grid cards to present a different personality to the driver in the VM than what is given by the hard strapped device ID.

Still, it is perhaps worth an extra check to see if this is the same as the hard strap set by the potential divider (resistors on the PCB).

Either way, I think the way forward in the medium term is patching Xen/KVM/QEMU to neuter the method the Nvidia driver uses to check whether it is running in a VM. That would make modding deprecated and irrelevant.

[Raim]

I am about 90% certain the strap you speak of is set by the resistors on the PCB - what we have been hacking here. The soft strap also exists which can be set by vbios, but that is nowdays used by the likes if Grid cards to present a different personality to the driver in the VM than what is given by the hard strapped device ID.

Yes, when I added the pull up resistor,  the strap register changed.
bit 28 DEVICE_ID bit 4
bit 30: DEVICE_ID bit 5

My aim is to see what the difference in this register between a Titan and 780ti.

The link to document about strap register
https://github.com/envytools/envytools/blob/master/hwdocs/io/pstraps.rst

[gordan]

I think you'll find the value of the register in both cases is the same as the card's PCI device ID (unless you changed the device ID via the soft strap). The driver checks the value of the register, rather than the PCI device ID, hence why soft-modding no longer works on Kepler class cards.

[mosmo]

Just wanted to report back that I've been running an EVGA Superclocked 780ti with resistor for a few months now. Absolutely no issues passed through under 5.5/5.1 ESXi.
I went one step further and bought the back-plate and ACX cooler for the Titan. I had to dremmel the cooler heatsink and plate to fit the 780ti, but I managed it :-)
Now all we need is some clever sod to either mod ESXi or the Nvidia drivers so that we can use non-ref 780ti with our hypervisors.

Thanks to this forum and Gordon in particular, I have learn't a fair bit. Keep it going chaps!

[gordan]

I have to say I am unimpressed by just about all the non-reference coolers like the ACX. The reference design is actually really, really good - not only does it ensure good cooling of the card, it also makes all of the hot air get straight out of the case via the grille in the back plate.

In contrast, the coolers like the ACX dump all the hot air straight back into the case, which both reduces the cooling effectiveness (you are cooling with pre-heated recycled air) and makes everything else inside the case run hotter. This is a problem both with running multiple GPUs and power hungry overclocked CPUs.

Oh, and the resistor-across-the-EEPROM mod should work on non-reference 780Tis as well.

[mosmo]

I have to say I am unimpressed by just about all the non-reference coolers like the ACX. The reference design is actually really, really good - not only does it ensure good cooling of the card, it also makes all of the hot air get straight out of the case via the grille in the back plate.

In contrast, the coolers like the ACX dump all the hot air straight back into the case, which both reduces the cooling effectiveness (you are cooling with pre-heated recycled air) and makes everything else inside the case run hotter. This is a problem both with running multiple GPUs and power hungry overclocked CPUs.

Oh, and the resistor-across-the-EEPROM mod should work on non-reference 780Tis as well.

I agree with the hot air inside the case issue, but with adequate exhaust, it should not be an issue for most. I have two top, on bottom and one rear all 140mm Noctua fans. They shift the hot air out of the case brilliantly. Also, whilst the ref cooler has improved, it's still very noise (to me) on high GPU load.

I did forget to mentio though that my EVGA 780ti is now producing a louder coil whine. I am not sure when this started and it may have always been there but nonetheless, it's starting to seriously annoy me.

[gordan]

I am switching to watercooling for the CPUs, but leaving my GPUs with Titan type reference coolers as they are. The GPUs are easily staying under their (95C) thermal limits (subject to not pushing the TDP past 125%) at full load (e.g. litecoin mining), but under similarly full load with only a 25% OC, my X5650 Xeons are incapable of staying below their 81C TCase limit.

[isaiah]

 i read some post..

 so....

 can moded 680 to k5000.. and can unlock some feature only quadro have..

 but no adventage in open gl performance...

 am i understand right?



 and what is exactly 'some feature quadro only have'??

[gordan]

You understand right. The features that become unlocked are the ones that are purely up to the driver to allow, such as correct functioning when used virtualised (e.g. Xen PCI passthrough or Xen VDGA/VSGA). It also supposedly unlocks some multi-monitor features (this was OPs original requirement).

[isaiah]

You understand right. The features that become unlocked are the ones that are purely up to the driver to allow, such as correct functioning when used virtualised (e.g. Xen PCI passthrough or Xen VDGA/VSGA). It also supposedly unlocks some multi-monitor features (this was OPs original requirement).

 thanks a lot..

  to me most imortent thing is open gl.. so i am dispointment....