[MOVED] Hacking NVidia Cards into their Professional Counterparts

[gordan]

There is no such thing as "Quadro mode". Some of the functionality of the non-Quadro cards is disabled by some other means, most probably a combination of the implementation of some of the GL primitives being cut out of the silicon (or not programmed into an FPGA, or something along those lines), plus a deliberately crippled software implementation of those GL primitives.

Modifying the card's device ID gets you the features the driver deliberately disables on GeForce cards, such as virtualization and TCC support (both of which are 100% purely driver features, there is no "hardware" part of the implementation for those features it's just driver deciding based on the device ID whether it'll let you have them).

Mosaic support is almost certainly a driver feature but it probably also covers vsynced output from multiple combined cards, syncing over SLI, which might actually require some extra hardware to do. If that is missing on GeForce cards, then the driver might just wholesale disable all Mosaic options.

In terms of stitching together multiple monitors, I have certainly run 3+ monitors stitched together under Linux into a single desktop using Xinerama (you have to disable randr extensions) - e.g. to run a T221 at full refresh rate, so there is no hardware limitation preventing that from happening per se. Perhaps gnif might shed some more info on the subject of Mosaic support, as I never quite fully understood what he was talking about when he was saying that modifying his 690 enabled extra multi-monitor flexibility - I just figured it was about some other feature that I never needed. As for Windows, I'm afraid I can't help much - to me Windows is just a somewhat thick bootloader for games that haven't been ported to Linux yet.

[beckend]

Strange. Had you verified that it is coming up with the correct device ID in GPU-Z and GPU Caps Viewer?

Yes, I always do.
Maybe I'm speaking from my ass here but the tesla mod might be flaky/unstable in my case, but I only tried only one version of the drivers.

[gordan]

Strange. Had you verified that it is coming up with the correct device ID in GPU-Z and GPU Caps Viewer?

Yes, I always do.
Maybe I'm speaking from my ass here but the tesla mod might be flaky/unstable in my case, but I only tried only one version of the drivers.

It is plausible - as has already been mentioned several times, different cards exhibit different problems under various circumstances. Gainward cards have issues outputting DL-DVI when virtualized. IIRC verybigbadboy reported that one of his cards had issues with the default mod and he had to use 100K resistor in position 2 and no resistors in other positions. My Gainward 690 suffers from the EEPROM effectively becoming unreadable if the 3rd nibble is modified as per the original instructions. Other people have reported that just removing resistors causes an instability, while it works for others.

There are quirks, so perhaps we should start collating information on what works on which specific model, it looks like there are a lot of possible combinations.

[mrkrad]

I'd bet $50 running OSX the same hacked card would perform better than (win/linux)

throw that quadro as video1 and the modded card as #2 btw

[blackice504]

Just some advice.

Its one thing to hack the ID but I am sure that the video ram is ECC on Quadro where on the Geforce cards are not ECC, if this is the case those errors that can happen in ram or on the way to and from ram will not be corrected thus the data is not 100% accurate compared to a system with ECC ram, just like a Desktop can be used as a Basic Server but at the end of the day its just a desktop with server software, the real servers have ECC and all those other features that Servers come with.

Just the heads up but as I said I am not 100% on this.

what this means for rendering, I guess you could render something and it might have errors in the image and may spoil your work.

Notes, there also could be other things as well like different memory bus example 256bit vs 384bit this basically means the bus is wider thus more memory bandwidth. "if the clock and timings are the same"

Instruction sets can be different as well and that will have an impact depending on what your doing.

Me personally I would only try this on a SPARE card and research to make sure everything else hardware wise is the same specs.

There is also the possibility that this could work 100%.

[gordan]

Its one thing to hack the ID but I am sure that the video ram is ECC on Quadro where on the Geforce cards are not ECC, if this is the case those errors that can happen in ram or on the way to and from ram will not be corrected thus the data is not 100% accurate compared to a system with ECC ram, just like a Desktop can be used as a Basic Server but at the end of the day its just a desktop with server software, the real servers have ECC and all those other features that Servers come with.

ECC on Nvidia GPUs is done in software - you get 8/10 storage overhead for ECC, but there is no difference in the actual RAM. You enable the ECC feature in the MC, reboot and your VRAM reduces and gets a bit slower. On modified GeForce cards this doesn't work. If you mod a 4GB 680 to a K5000 and flash the K5000 BIOS onto it, the option will appear in the control panel, but it won't actually do anything (RAM size doesn't shrink, RAM I/O doesn't slow down).

what this means for rendering, I guess you could render something and it might have errors in the image and may spoil your work.

For aircraft component design or nuclear simulations, that might be an issue. For most uses it's not really an issue.

Notes, there also could be other things as well like different memory bus example 256bit vs 384bit this basically means the bus is wider thus more memory bandwidth. "if the clock and timings are the same"

Quadro/Grid/Tesla parts generally have equivalent GeForce parts, and the spec of the GeForce part is at least as good as the pro counterpart, e.g. GTX480 beats the Quadro 6000 on everything but RAM size (1.5GB vs. 6GB), but it is clocked higher, has the same width memory bus, and it has more shaders (480 vs. 448)

Instruction sets can be different as well and that will have an impact depending on what your doing.

If that is the case, the driver glazes over it. While there is a difference in the exposed GL primitives (even after modification), it is also very evident that the software implementation of the missing GL primitives on GeForce cards is deliberately massively crippled.

Me personally I would only try this on a SPARE card and research to make sure everything else hardware wise is the same specs.

There is also the possibility that this could work 100%.

I guess you haven't noticed (size of the thread being a hint) that there are hundreds of people who have been modifying cards like this for things like virtualization with great success.

[blackice504]

Its one thing to hack the ID but I am sure that the video ram is ECC on Quadro where on the Geforce cards are not ECC, if this is the case those errors that can happen in ram or on the way to and from ram will not be corrected thus the data is not 100% accurate compared to a system with ECC ram, just like a Desktop can be used as a Basic Server but at the end of the day its just a desktop with server software, the real servers have ECC and all those other features that Servers come with.

ECC on Nvidia GPUs is done in software - you get 8/10 storage overhead for ECC, but there is no difference in the actual RAM. You enable the ECC feature in the MC, reboot and your VRAM reduces and gets a bit slower. On modified GeForce cards this doesn't work. If you mod a 4GB 680 to a K5000 and flash the K5000 BIOS onto it, the option will appear in the control panel, but it won't actually do anything (RAM size doesn't shrink, RAM I/O doesn't slow down).

what this means for rendering, I guess you could render something and it might have errors in the image and may spoil your work.

For aircraft component design or nuclear simulations, that might be an issue. For most uses it's not really an issue.

Notes, there also could be other things as well like different memory bus example 256bit vs 384bit this basically means the bus is wider thus more memory bandwidth. "if the clock and timings are the same"

Quadro/Grid/Tesla parts generally have equivalent GeForce parts, and the spec of the GeForce part is at least as good as the pro counterpart, e.g. GTX480 beats the Quadro 6000 on everything but RAM size (1.5GB vs. 6GB), but it is clocked higher, has the same width memory bus, and it has more shaders (480 vs. 448)

Instruction sets can be different as well and that will have an impact depending on what your doing.

If that is the case, the driver glazes over it. While there is a difference in the exposed GL primitives (even after modification), it is also very evident that the software implementation of the missing GL primitives on GeForce cards is deliberately massively crippled.

Me personally I would only try this on a SPARE card and research to make sure everything else hardware wise is the same specs.

There is also the possibility that this could work 100%.

I guess you haven't noticed (size of the thread being a hint) that there are hundreds of people who have been modifying cards like this for things like virtualization with great success.

as stated I just wanted to outline things to look out for as in many cases people miss differences especially in hardware as sometimes firmware or ID may work but not often.
But also stated I have not looked into this.

[snotz]

Excellent work on this and thanks to everyone who provided information and got these hacks to work!

I have a few questions.  I have successfully modded the DeviceID of my MSI GTX770 to a Tesla K10 (but could not get DeviceID to change to 0x11BF for a Grid K2 using the 40k resistor at the 2nd resistor, even tried 100k.  My current mod only removed the 25k at 3rd resistor location, so its setup as NONE, 25k, NONE, NONE, respectively) and flashed it with my own modified BIOS (changing the DeviceID and fixing the checksum on a copy of my backed up BIOS)

It gets detected in GPU-Z properly:

But, it says its running @x16 1.1.  Is there a fix for that?

When I try to run nvidia-smi it says its in TCC mode (-dm 0 and -fdm 0 dont change it), but I'd like to get it into WDDM mode.  Is this possible?
I'd like to use WDDM mode to test the card's performance on the local Windows installation before I start setting up gpu passthrough, but the card doesnt seem to be working properly.  When I go into Screen Resolution, Advanced, it says the adapter type is Microsoft Basic Display Driver (I'm assuming because its in TCC mode), but my device manager says that I only have a Tesla K10 installed.

Also, has anyone else figured out what resistor to put to get it to change the third byte to B from 8 on the MSI cards?

[gordan]

Your card isn't working properly - that much is obvious from GPU-Z output. The RAM size is wrong, and the clocks are reading as 0. I suspect if you check Device Manager it'll say there's a problem with the device. A common cause of this is that your BIOS checksum is wrong. Did you fix the checksum using Kepler BIOS Tweaker? NiBiTor doesn't understand UEFI header/footer on the BIOS so will probably get the checksums wrong. But you would be best off just flashing the original unmodified BIOS back onto the card without any modifications (apply appropriate overrides to get it to stick).

I have a MSI 680 (i.e. the exact same card) modified to a K10, and it works just fine for me.

Also note that there is nothing to be gained to modifying to anything else. The features enablable by the mod are TCC and virtualization, which you will get regardless of whether you mod to K10, K5000 or Grid K2. Stick with K10 as it is the simplest and most painless mod on the 680 that gets you all the benefits.

[snotz]

The only reason I flashed the modified BIOS onto the card was because it was doing the same thing with the original BIOS and I thought that would fix it.  In Device Manager, there's no issues.  Yes I fixed the checksum using Kepler's BIOS Tweaker.  I just reflashed the BIOS and its doing the same thing, 0 clocks.

Any suggestions?

Thanks in advance

[gordan]

Which driver are you using? Is the card showing up as Tesla K10 in Device Manager, without any warnings on it? DM says it's working properly?

It is vaguely plausible that Nvidia might have updated their driver to make modified cards not work. Possibly a PR disaster waiting to happen (again - last time was when SLI magically started working on non-Nvidia chipset motherboards for Core i series CPUs when Intel unceremoniously refused to licence them for a QPI implementation because Nvidia didn't allow SLI to work on Intel's chipsets back in the X38/X48 days).

[snotz]

I am using the latest drivers (331.82).  A funny thing just happened... The card came up as a GTX760 0x1187 and surprise... it worked... as a GTX760.  I took the card out and checked my solder joints and it came back up as a K10, and same thing, nothing works.

[gordan]

Hmm... Can you check what GPU caps viewer says about the device ID?

[snotz]

GPU caps viewer is acting strange:


Update:  I tried 320.49 drivers... same results.  0s across the board

[gordan]

Is the card the primary card in your system? Does the machine POST with it as the primary?
Two things to check:
1) your solder joints
2) UEFI header for device ID straps (discussed on this thread about 10 pages back)

Also, can you dump the BIOS out of the card in this state?

Can you try the Quadro/Tesla driver version 320.92?

[snotz]

Yes, its the only card in the system

Checked...
1) Solder joints are good
2) My Straps are correct: FF FF FF F7 00 00 00 80 at 0x1C... its unlocked, or are you talking a different part of the UEFI header?

I did dump the bios in this state, and it compares equally to my original backup.

When dumping the BIOS, it has the right DeviceID, but check this info in the text file.  Device Name still says GTX 770.


And still not correct info in GPUz

[snotz]

Gordan, THANK YOU SO MUCH!!

SUCCESS!!  I have successfully modified MSI GTX 770 to an Nvidia Grid K2:


The process was to remove both 25k resistors at 1 and 3 spots AND doing a BIOS modification, changing the OR section of the strap to 00 3C 00 90
thanks to Gordan's post:
https://www.eevblog.com/forum/chat/hacking-nvidia-cards-into-their-professional-counterparts/msg316134/#msg316134



My only question is that I noticed in Caps viewer, it says the DeviceID is 0x11AF.  With the BIOS modification forcing the bit to be high, will that be a problem with vga passthrough?

[gordan]

Unfortunately, that won't work. The driver checks the hard strap (what caps viewer shows). If this isn't on the driver's approved whitelist for virtualization, it won't enable the card from the VM. You need to unmod the 3rd nibble. If you are having random issues with 4th nibble resistors removed (and you have double checked your 3rd nibble soldering) you need the 40k resistor there to hard set the 4th nibble to 0xF.

[alex870]

So has anyone found a suitable K6000 bios for their 780TI-18K yet?  I got the hardware strap to work fine according to the guztech recipe, and I can report adding 4" of wire wrap wire to an external switch causes no problems (i.e. parasitic) with the device ID either.  I'll post a few pics if ppl want - the soldering is a bit embarrassing though.

I can only find one K6000 bios so far and when I tried nvflash --version against it, it reports "PCI block chain corrupted".  So not sure about that.  But also, the reported memory size needs tweaking down to 3GB too.

--update

Looks like you don't need a bios update unless you want to enable TCC.  I'm now successfully running the card under ESXi 5.5 using pass through.  The quadro drivers are loaded in the guest OS.  (thanks to mosmo for cluing me in on this)

[snotz]

The issue is that I haven't been able to get the 3rd byte to B on the hard strap at all. I would like it to be a grid because I will be testing shared vgpu using xen and the k1 or k2 are the only cards supported.  If I put the 25k resistor back it changes it to 8 and if I put a 40k or 100k it changes to A.  Any ideas how to get it to B?

Thanks again for all your help gordan

[gordan]

IIRC it should also work on Tesla cards - probably worth a shot before you waste any more of your time.

[snotz]

All the documentation I've read says K1 or K2 only, so I'd rather try to figure out how to get the 3rd byte to a B in the hard strap.

Also, you noticed what happened in the drivers when I set it to a Tesla, so I'm not sure if there's a way for the drivers to detect the hacked cards, or not, but the K2, the way the card is currently set, seems to work OK on native Windows.

Is there anything you can suggest I can try with the resistors, or should I experiment?

Update: I noticed my card has two extra resistors by the EEPROM that aren't in the pics shown for the 680 mod.  I then noticed pics online for this board are version 3.0 which matched the pics, mine is version 4.0.  They changed the board!  The two resistors are very low... 6ohm (not even 1k) and are labeled R779 and R780. Should I try to jump one of them and remove the resistor?

I will try to take a pic but my camera is poor quality.

Update2: I added the 25k ohm resistor back and made it a Tesla K10, having the same issues as before.  Now, I modified the BIOS AND strap to make the DeviceID 0x1184, with success... it installed the driver for the GTX 770, and is fully operational on bare metal Windows 8.1.  This finding tells me that the K10 (which doesnt have any display outputs) was changed in the driver to not link the DVIs to the GPU (The DVIs came up as Microsoft Displays, see above).  Changing the soft strap allowed the GTX 770 driver link the DVI to the GPU, fully enabling the card.

[ijsf]

Just wanted to give you guys a heads up on the modifications you're doing on these cards. I see the knowledge I tried to share in my earlier thread on the NVIDIA forums has been put to good use. Honestly, I did expect them to nuke the thread instantly though, but it still seems to be there.

Either way, as oguz286 from Guztech has probably already mentioned, we've successfully done yet another GTX780 to K20 modification. I was primarily involved with the reverse engineering of a series of BIOS ROMs, this was an unexpected but mandatory step due to the PCB configuration differences between the two cards (RAM chip configuration, notably).

Part of the success was in decompiling the ROM script opcodes into readable form, which we did for a number of ROMs, and figuring out which opcodes were responsible for setting up the RAM block configuration. Lots of thanks to the guys over at the open-source nouveau NVIDIA driver there. I'm pretty certain other features such as PCIe 3.0, ECC, double precision floating point and copy engines are also contained inside the opcode values, but our mission was accomplished as soon as our CUDA kernels started working.

Can't help but say though that sooner or later these guys will be switching to built-in fuses..

[gordan]

ijsf, good to see that you have made an appearance.

I am not as confident as you that some of the features are in fact purely BIOS driven. For example, if you flash a K5000 BIOS onto a suitably modified 4GB 680, the ECC settings become selectable, but verifiably don't do anything. Secondary copy engine also doesn't become available, which implies that both of those features are already disabled in the silicon. As for PCIe 3.0, I'm reasonably confident the reason your modified 780 isn't working in 3.0 mode is because it's hardware device ID doesn't match the actual device ID (check what GPU caps viewer reports the device ID as - it will show the hard-strapped ID, not the soft-strapped one). It is the hard-strap that makes the driver decide whether to make certain features work, such as PCIe 3.0.

On my MSI 680 I had a similar problem when soft-modding into a K10. Soft-modded, the Tesla driver "worked" fine, but PCI passthrough wasn't working and the card was always running in PCIe 1.1 mode. When I switched from soft-strap mod to hard-strap mod, passthrough started working and PCIe went into 3.0 mode.

[ijsf]

Note that the GTX780 to K20 simply isn't possible without hardware strap/resistor modifications, so the card is a combination of hard and soft strap modifications.

By the way, we never did get to flash a genuine or even modified K20 BIOS into the GTX780 because the board configurations were simply too different. We ended up modifying the original GTX780 BIOS, specifically the soft straps, and the opcodes writing to the corresponding board configuration registers, in order to change the RAM block.

As far as I remember, the K20 doesn't ship with PCIe 3.0 enabled. It would make sense that one or more GPU registers would control this functionality in terms of cost effectiveness, but I didn't really look for it while I was at it.

Anyway, research from Nouveau suggests that there are number of fuses on these cards that indeed (permanently) disable an array of these features. These features do have corresponding control bits in GPU registers, as long as the fuses aren't gone. Not sure if these are irreversible or efuses.