[MOVED] Hacking NVidia Cards into their Professional Counterparts

[dongsterlicious]

Hi, I've got an evga 670 2gb card that i'm interesting in modding to a k5000.

It seems that everyone is modding their cards to be used in virtual machines, but i'm wondering if modding it opens up the hardware opengl or if it will give me any performance gain for that matter.

I'm mainly using maya, mari, adobe and etc, so i'm wondering if this mod will uncripple the geforce cards for these applications.

Thanks.

[gordan]

Hi, I've got an evga 670 2gb card that i'm interesting in modding to a k5000.

It seems that everyone is modding their cards to be used in virtual machines, but i'm wondering if modding it opens up the hardware opengl or if it will give me any performance gain for that matter.

Read the whole thread. Yes, I know there's 39 pages. No, that doesn't excuse you being lazy. :p
Uncrippling the missing GL functionality has not been achieved (at least not yet). Neither has ECC support or stereo 3D support.

On a separate note, I've been working on my GTX690. Inexplicably, it behaves very differently compared to the GTX680 under soft-modding. I removed resistor controlling the 3rd byte, and tried to soft-mod from there and it looks like the UEFI part of the BIOS is the only way to control this. Modifying the straps in the old places doesn't seem to do it. Removing the UEFI headers renders the card unbootable (not bricked, just no BIOS init). Removing the UEFI footer seems to not quite be doable - nvflash complains about a size mismatch. nvflash versions old enough to have the --eraseeprom don't recognize the card. Padding the BIOS out to 64KB like on the Gigabyte GTX680 that has a non-UEFI BIOS allows flashing onto the card, but the VBIOS fails to initialize. This is distinctly different from the GTX680 on which the soft-modding worked just fine.

[cubansite]

How is this accomplish with an EVGA 670 FTW ?

Its pcb is the same as 680 so would this mean the same resistor change ?

anyone ?

[gordan]

Read the rest of the thread. I'm sure I remember people posting about having successfully modded the GTX670.

[wukoje]

Hi guys this is my first post, but I have been reading this thread for quite some time, would you mind tell me what would the benefits be in modding the gtx 580 to a tesla?
-Did anyone actually test it, does it make any difference in double precision (since this is supposed use case)?
-Until now did anyone manage to run multiple VM instances on the same quadrified/gridfied GPU? if not what's the best option, going for a real k5000,k6000 or just doing it the "as supposed to be way" with a grid card?
-Gordan would you mind sharing what 4gb gtx 680 did you manage to quadrfie to k5000?
-Did anyway manage to get any card (including ATIs consumer cards) to work with multiple instances?

[gordan]

Hi guys this is my first post, but I have been reading this thread for quite some time, would you mind tell me what would the benefits be in modding the gtx 580 to a tesla?
-Did anyone actually test it, does it make any difference in double precision (since this is supposed use case)?
-Until now did anyone manage to run multiple VM instances on the same quadrified/gridfied GPU? if not what's the best option, going for a real k5000,k6000 or just doing it the "as supposed to be way" with a grid card?
-Gordan would you mind sharing what 4gb gtx 680 did you manage to quadrfie to k5000?
-Did anyway manage to get any card (including ATIs consumer cards) to work with multiple instances?

1) The only benefit of modifying the GTX580 Into a Tesla is that you can put the card into TCC mode and avoid the windows WDM driver overhead. If you do your number-crunching on Linux it won't gain you anything.

2) None of the modification discussed in this thread make any difference to DP performance. The only performance improvement from modding is on GTX470/GTX480 where you get the 2nd DMA channel enabled when you modify to Quadro 5000/6000. GTX580 only has one DMA channel.

3) The only system that supports GPU sharing in the way you describe is recent VMware ESX and that only supprots Grid GPUs. You should be able to use a GTX680/690 modified to Grid K2, but most of us here use Xen, and that only supports GPU virtualization by passing the whole GPU to a VM. If you want to try VMware with GPU sharing, go for it, but this is sufficiently obscure that you're bound to run into issues if you run on anything but their supported reference hardware. Also note that you will need a VMware specific client on the workstation where you are outputting the video stream of the 3D rendering. You cannot just plug monitors into different outputs and have each be a separate VM sharing a GPU. Grid GPUs have no video outputs at all. I suggest you go and read through all the VMware and Xen documentation on the subject before you ask questions like this here.

4) The GTX680 I used is a Gainward Phantom 4GB model, but I would expect any 4GB GTX680 to work fine when flashed with a K5000 BIOS (you will of course also need to hard-mod the resistors).

[wukoje]

You cannot just plug monitors into different outputs and have each be a separate VM sharing a GPU. Grid GPUs have no video outputs at all. I suggest you go and read through all the VMware and Xen documentation on the subject before you ask questions like this here.

I don't see where you understood that, since all the time I was talking about instances (maybe that has a different meaning for us) in any case shared gpu (particularly Nvidia Grid) is supported on all 3 major player in the virtualization field , MS with RemoteFX, Citrix and WMware.
The nice thing about using nvidia GPUs is the hardware support for H.264 Encoding in Kepler GPUs witch allows to encode the rendered streams fast and CPU free. Did anyone manage to use this hardware acceleration feature for virtualization? Anyway I wonder why do you insist so much on nvidia cards for Dedicated GPU (GPU passthrough) virtualization even by quadrifieing them, while amd consumer cards support it by default?

[gamezr2ez]

Anyway I wonder why do you insist so much on nvidia cards for Dedicated GPU (GPU passthrough) virtualization even by quadrifieing them, while amd consumer cards support it by default?

AMD cards passthrough, sure. But they are buggy and you cannot reboot the VM. Lots of stability issues.

Modding my GTX680 allows me to reboot the VM as many times as I would like without any stability issues. I have shutdown the vm, passed the GPU back to my host, then passed it back to a different VM all without any reboots or performances issues.

That being said, I am not sure I would buy Nvidia again since I have completely converted to using virtualization for everything. The hardware mod is nice, but future generations will probably make this method obsolete. AMD contributed some of the initial GPU passthrough patches, I doubt they are going to block it like Nvidia does.

[verybigbadboy]

The nice thing about using nvidia GPUs is the hardware support for H.264 Encoding in Kepler GPUs witch allows to encode the rendered streams fast and CPU free. Did anyone manage to use this hardware acceleration feature for virtualization?

Hi I tried to use NVENC sdk but seems it is not posible to use hardware encoder on GRIDfied 680 card for me. I have error about not valid licence key. But It should works fine on any non geforce card. May be I did something wrong.

[rboxeur]

To verybigbadboy

wowza.com is a project which supports NVIDIA NVENC. This project is supported on Linux.

If you have a look at http://www.wowza.com/forums/content.php?512 explaining how to install NVIDIA NVENC on Ubuntu 12.04 then it is clearly written

Note: Accelerated encoding isn't available when running on a virtual hardware environment such as VMware or Xen.

If I am not mistaken then it should work on bare metal with a quadrified card. I didn't test it.

[gordan]

You cannot just plug monitors into different outputs and have each be a separate VM sharing a GPU. Grid GPUs have no video outputs at all. I suggest you go and read through all the VMware and Xen documentation on the subject before you ask questions like this here.

I don't see where you understood that, since all the time I was talking about instances (maybe that has a different meaning for us) in any case shared gpu (particularly Nvidia Grid) is supported on all 3 major player in the virtualization field , MS with RemoteFX, Citrix and WMware.
The nice thing about using nvidia GPUs is the hardware support for H.264 Encoding in Kepler GPUs witch allows to encode the rendered streams fast and CPU free. Did anyone manage to use this hardware acceleration feature for virtualization?

The real question here is what is the use-case? You still have to have enough GPU on the hardware to do high-res H264 decoding in realtime to make this worthwhile. It's only useful for dedicated terminal hardware that comes with hardware video decoding yet is cheap. The only problem is that such hardware doesn't really exist because the client software is Windows-only AFAIK (and certainly x86 only). So the cost benefit is somewhat questionable, although it makes it a little easier to throw away workstations without the hassle of reintalling and data copying.

Anyway I wonder why do you insist so much on nvidia cards for Dedicated GPU (GPU passthrough) virtualization even by quadrifieing them, while amd consumer cards support it by default?

Because ATI cards utterly suck. Have you actually tried using an ATI card with VGA passthrough? The drivers are unusably crap, especially if you try to use multiple monitors, and doubly so on XP. They randomly blue-screen, and on a reboot they tend to either slow down to a crawl, blue-screen the guest, or even crash the host. Having battled an ATI card for days, I invested in a Quadro 2000 (lowest spec Quadro supported for GPU passthrough), and the experience was quite eyeopening.

[CDeLorme]

I just finished reading all the contents of this thread, totally awesome.  gnif I donated to your cause, I hope you reach your goal soon and can show off an amazing video of the GTX 690 modification.  I just ordered a GTX 670 to try this mod with, and will probably get around to it this weekend.

I am also a Xen user and have spent months tinkering with AMD, and can affirm that gordan speaks the truth.

At most I would say amd cards make it "possible", to say they "support" it feels like a misnomer.  It is far from ideal, and not streamlined in the least.  Success still varies on model/make and surrounding hardware.  For my card it took days of nonstop experimenting just to figure out what exactly caused BSoD's, and then driver corruption which would lead to live degradation.

Even after all that work, manual ejection is required unless you want to reboot the physical machine, and my Windows 7/8 installs tend to have a maximum lifespan of about 6 months, after which point they become corrupt and the degradation happens within a day of runtime (not a reboot), and it isn't "fixable" except by a reinstall (or restoring from a pre-driver image).

Honestly, if I had the money and time I would have been experimenting with alternatives sooner.  I look forward to posting back whether I have success with the GTX 670 grid k2 mod.

[mrkrad]

someone just sold a grid k2 on ebay for $1525

with prices like that, maybe not a bad idea to just deal hunt these

[gordan]

Indeed, that sounds like a deal good enough to not bother with modding - only there is a distinct discrepancy between what a Grid K2 is for and what most of us here want to use it for. A real Grid K2 has no video outputs. From that point of view, the use-case here is more along the lines of a Quadro than a Grid (passing a whole dedicated card to the VM).

On a separate note, I've been doing a bit more digging on soft-modding a the 6xx series cards. From what I can tell, the reason why things weren't working as expected on my first GTX690 was because it was not working properly (finally gave up the ghost the other day, most annoyingly). I got another one, I can soft-mod it to a Tesla K10. Except get this - modding the ID the old way in the main BIOS body has no effect. The old straps that used to start at 0x58 (0x458 on the UEFI BIOS) no longer have any effect when the UEFI wrappers are in place - even when running on a non-UEFI motherboard. Instead the strap at 0x0C is what controls the device ID. The bit layout, at least for the device ID control bits, is the same.

So for those of you that only want to mod to a Tesla K10, you can soft-mod, no need to take a soldering iron to he card. It also means that you can soft mod the 4th nibble, and only hard-mod the resistor pair for the 3rd nibble.

[oguz286]

First of all, a big thanks to anyone who contributed: gnif, verybigbadboy, and all the others I forgot to mention

I'm trying to convert a GTX780 to a Tesla K20 which have the following device IDs:

GTX780	0x1004
K20	0x1022

According to the resistor values that were discovered so far, this would suggest that I would have to find a 5K and a 25K resistor and change them both to 15K since they are both digits are in the 0-7 range. I found the EEPROM and measured the values of the resistors around it. You can find the results below:


As you can see, I found a 5K resistor which I removed and replaced it with a multi-turn 50K pot which I set to 15K. Unfortunately, this did nothing as the device ID still remains 0x1004 whereas I expected it to be 0x1024. There are two 4.7K resistors at the back of the board and other than that there are no 5K resistors on the board. Either NVIDIA changed the way the device ID is determined, or they changed the values, or there is a simple resistor divider action going on.

Before I go and change the 25K resistor, I want to make sure that I can change the 3rd digit from 0 to 2. I did try to flash a K20 ROM onto the EEPROM, but it is still recognized as a GTX780. Strangely enough, with the K20 ROM, the nvidia-smi tool reports that the board supposedly has 6GB of RAM instead of the actual 3GB. Any ideas or suggestions?

EDIT: I will be updating the image when I discover any new values/paths.

[verybigbadboy]

First of all, a big thanks to anyone who contributed: gnif, verybigbadboy, and all the others I forgot to mention

I'm trying to convert a GTX780 to a Tesla K20 which have the following device IDs:

GTX780	0x1004
K20	0x1022

According to the resistor values that were discovered so far, this would suggest that I would have to find a 5K and a 25K resistor and change them both to 15K since they are both digits are in the 0-7 range. I found the EEPROM and measured the values of the resistors around it. You can find the results below and in the attached photo:

As you can see, I found a 5K resistor which I removed and replaced it with a multi-turn 50K pot which I set to 15K. Unfortunately, this did nothing as the device ID still remains 0x1004 whereas I expected it to be 0x1024. There are two 4.7K resistors at the back of the board and other than that there are no 5K resistors on the board. Either NVIDIA changed the way the device ID is determined, or they changed the values, or there is a simple resistor divider action going on.

Before I go and change the 25K resistor, I want to make sure that I can change the 3rd digit from 0 to 2. I did try to flash a K20 ROM onto the EEPROM, but it is still recognized as a GTX780. Strangely enough, with the K20 ROM, the nvidia-smi tool reports that the board supposedly has 6GB of RAM instead of the actual 3GB. Any ideas or suggestions?

Hello oguz286,

Bios contain information about memory and memory type. It is absolutely normal that nvidia smi shows 6gb of ram with K20 bios.

Also I looked at gtx 780 bios, and have a question. What gtx780 model do you use? I think you need to do bios modification like a for GT 640

Thank you.

[oguz286]

Hello verybigbadboy (thanks for the many mods ).

I have a Palit GTX780, which I believe is the standard NVIDIA reference design (http://www.palit.biz/palit/vgapro.php?id=2132).

Ok so what I understand from your post that you linked is that I need to modify the GTX780 BIOS on the card to unlock it.
My BIOS has the following values:
00000010: 08 E2 00 00 00 06 00 00 02 10 10 82 FF 3F FC 7F
00000020: 00 50 00 80 0E 10 10 82 FF FF FF 73 00 00 00 8C

So that means my BIOS is locked and I need to change the FF 3F FC 7F to FF FF FF 7F. Is that correct?

I'm going to try it right now

EDIT: NiBiTor cannot read the ROM file correctly. I saved the modified BIOS using NiBiTor 6.06 but I do not know if the checksum is now correct. I guess I'll find out soon enough.

[verybigbadboy]

Hello verybigbadboy (thanks for the many mods ).

I have a Palit GTX780, which I believe is the standard NVIDIA reference design (http://www.palit.biz/palit/vgapro.php?id=2132).

Ok so what I understand from your post that you linked is that I need to modify the GTX780 BIOS on the card to unlock it.
My BIOS has the following values:
00000010: 08 E2 00 00 00 06 00 00 02 10 10 82 FF 3F FC 7F
00000020: 00 50 00 80 0E 10 10 82 FF FF FF 73 00 00 00 8C

So that means my BIOS is locked and I need to change the FF 3F FC 7F to FF FF FF 7F. Is that correct?

I'm going to try it right now

yes, and next line too
00000020: 00 50 00 80 to 00 00 00 80

also please update  checksum. without it card won't start at all

4. Change values to be equal values from 4.
5. Update checksum. I do it by nibitor tool. just open bios rom and save it. It produces lot of warnings, but it is ok.
6. Upload bios back to card.

[oguz286]

Hello verybigbadboy (thanks for the many mods ).

I have a Palit GTX780, which I believe is the standard NVIDIA reference design (http://www.palit.biz/palit/vgapro.php?id=2132).

Ok so what I understand from your post that you linked is that I need to modify the GTX780 BIOS on the card to unlock it.
My BIOS has the following values:
00000010: 08 E2 00 00 00 06 00 00 02 10 10 82 FF 3F FC 7F
00000020: 00 50 00 80 0E 10 10 82 FF FF FF 73 00 00 00 8C

So that means my BIOS is locked and I need to change the FF 3F FC 7F to FF FF FF 7F. Is that correct?

I'm going to try it right now

yes, and next line too
00000020: 00 50 00 80 to 00 00 00 80

also please update  checksum. without it card won't start at all

4. Change values to be equal values from 4.
5. Update checksum. I do it by nibitor tool. just open bios rom and save it. It produces lot of warnings, but it is ok.
6. Upload bios back to card.

Aha, I forgot the second line Anyway, I changed both lines and saved it to a file (FILE_A). I opened that file in NiBiTor (which complained), and saved it to a different file (FILE_B). Then I flashed the card with FILE_B and rebooted.

With the pot set at 5K (which is the original value), nvidia-smi now reports that it cannot determine the device handle and gives an unknown error Did I miss something?

EDIT: I do see that FILE_A and FILE_B are different:
one byte is different at 0x8DFF
one byte is different at 0x391FF

So I guess the checksum has been updated correctly.

[verybigbadboy]

With the pot set at 5K (which is the original value), nvidia-smi now reports that it cannot determine the device handle and gives an unknown error Did I miss something?

EDIT: I do see that FILE_A and FILE_B are different: one byte is different at 0x8DFF so I guess the checksum has been updated correctly.

yes checksum looks like corrected correctly.
can you check lspci for videocard id?

or
boot via dos flash drive and
nvflash --list ?

are you trying to flash it with gtx780 or k20 bios? please make changes with original bios first.

EDIT:
Also I suggest to change resistor near flash first. I am not sure that it is correct one are you trying to change now.

[oguz286]

With the pot set at 5K (which is the original value), nvidia-smi now reports that it cannot determine the device handle and gives an unknown error Did I miss something?

EDIT: I do see that FILE_A and FILE_B are different: one byte is different at 0x8DFF so I guess the checksum has been updated correctly.

yes checksum looks like corrected correctly.
can you check lspci for videocard id?

or
boot via dos flash drive and
nvflash --list ?

are you trying to flash it with gtx780 or k20 bios? please make changes with original bios first.

I used the original BIOS which I first saved to a file and then modified and saved it with the correct checksum. With the modified BIOS:

5K pot:

nvflash --list gives GK11x (10DE,1004,10DE,104B)
lscpi gives device ID 0x1004

15K pot:

nvflash --list gives GK11x (10DE,1004,10DE,104B)
lspci gives device ID 0x1004

So there are no changes

[gordan]

A few points:

1) The values for the resistors for the 4th nibble are the ones that were documented. Resistor values for the 3rd nibble are NOT the same. For example, on my GTX690, the 3rd nibble resistor is 25K (24.8K) and the value is 8.

2) You cannot measure the value of the resistor while it is attached to the board. What you will end up measuring is the resistance of the resistor in parallel with the resistance of the rest of the circuit (if it is connected - which in most cases it will be).

3) The 3rd nibble isn't fully adjustable. On 6xx series cards it tops out at 0xB. It doesn't matter what you set it to past 40K, I suspect you'll find it will not go past that value. This may be different on 7xx series cards.

4) Be careful when blanking out the strap values at 0x0C - the card could plausibly be partially soft-strapped, which means that editing the strap value can brick the card - hard. Normally, unbricking is reliant on the card being fully hard-strapped. You can then ground the EEPROM power pin, and the card will boot EEPROM-less and and show up again for nvflash (I have a GTS450 modified this way for easy unbricking when BIOS-modding). If the card relies on partial soft-strapping and you break the soft-strap, the only way of unbricking it may well be to find how the important other bits are hard-strapped and modify them for the correct hard-strap - much harder considering that nobody has yet reverse engineered anything other than the device ID resistor locations.

5) Cross-flashing a ROM from a similar card with a different amount of RAM will not work. At best you will end up with garbled/corrupted screen output, even if text mode boot-up works (and/or the card shows up as a secondary card). The only way you will get a Quadro/Tesla/Grid ROM working on a GeForce card is if you use a card with the same GPU with the same amount of VRAM. The only cross-flashes I have managed to get working are Q2000 1GB -> GTS450 1GB works, and QK5000 4GB -> GTX680 4GB works. And if you are doing that, you will also want to edit the BIOS to adjust the clocks and fan speeds back to where they were on the GeForce card.

[oguz286]

A few points:

1) The values for the resistors for the 4th nibble are the ones that were documented. Resistor values for the 3rd nibble are NOT the same. For example, on my GTX690, the 3rd nibble resistor is 25K (24.8K) and the value is 8.

Ok, that was my misunderstanding then. I need to go through the topic then and see if there is any logic to be found concerning the 3rd nibble resistor.

2) You cannot measure the value of the resistor while it is attached to the board. What you will end up measuring is the resistance of the resistor in parallel with the resistance of the rest of the circuit (if it is connected - which in most cases it will be).

You are right. I just got lucky in that the 5K resistor I desoldered is in fact a 5K resistor. Unfortunately I cannot 'randomly' desolder parts as they are very fragile, so I want to be as sure as possible that I got the right resistor.

3) The 3rd nibble isn't fully adjustable. On 6xx series cards it tops out at 0xB. It doesn't matter what you set it to past 40K, I suspect you'll find it will not go past that value. This may be different on 7xx series cards.

Good to know. I soldered 50K multi-turn pots, so I can test with different values and see if it makes any difference. For the third nibble, I need to go from 0 to 2 so I hope that it is possible.

4) Be careful when blanking out the strap values at 0x0C - the card could plausibly be partially soft-strapped, which means that editing the strap value can brick the card - hard. Normally, unbricking is reliant on the card being fully hard-strapped. You can then ground the EEPROM power pin, and the card will boot EEPROM-less and and show up again for nvflash (I have a GTS450 modified this way for easy unbricking when BIOS-modding). If the card relies on partial soft-strapping and you break the soft-strap, the only way of unbricking it may well be to find how the important other bits are hard-strapped and modify them for the correct hard-strap - much harder considering that nobody has yet reverse engineered anything other than the device ID resistor locations.

I see. I thought that if I adjust the value of the 5K resistor to its original value and potentially mess with the power pin of the EEPROM, that I can simply reflash it with the original BIOS. I'll take that into consideration next time I try to adjust values in the BIOS.

5) Cross-flashing a ROM from a similar card with a different amount of RAM will not work. At best you will end up with garbled/corrupted screen output, even if text mode boot-up works (and/or the card shows up as a secondary card). The only way you will get a Quadro/Tesla/Grid ROM working on a GeForce card is if you use a card with the same GPU with the same amount of VRAM. The only cross-flashes I have managed to get working are Q2000 1GB -> GTS450 1GB works, and QK5000 4GB -> GTX680 4GB works. And if you are doing that, you will also want to edit the BIOS to adjust the clocks and fan speeds back to where they were on the GeForce card.

Yes, apparently the size of the RAM is also stored in the BIOS, so the chances of cross flashing to work are very slim if the hardware differs.

[oguz286]

I found something interesting. After I changed the original BIOS like verybigbadboy mentioned, I opened it in NiBiTor and saved it to a different file as NiBiTor recalculates the checksum. I opened this supposedly correct file in the Kepler Bios Tweaker 1.25 tool and found that the checksum of this new file is incorrect and that the original BIOS and the BIOS file which I edited (but not opened/saved with NiBiTor) both have correct checksums.

So I flashed the modified BIOS onto my card and now the nvidia-smi tool does detect the card properly as a GTX780. It seems that NiBiTor breaks the checksum of the BIOS file. Unfortunately, with the pot at 15K, there is no change in the device ID.

[amyk]

4) Be careful when blanking out the strap values at 0x0C - the card could plausibly be partially soft-strapped, which means that editing the strap value can brick the card - hard. Normally, unbricking is reliant on the card being fully hard-strapped. You can then ground the EEPROM power pin, and the card will boot EEPROM-less and and show up again for nvflash (I have a GTS450 modified this way for easy unbricking when BIOS-modding). If the card relies on partial soft-strapping and you break the soft-strap, the only way of unbricking it may well be to find how the important other bits are hard-strapped and modify them for the correct hard-strap - much harder considering that nobody has yet reverse engineered anything other than the device ID resistor locations.

Is everything stored in the EEPROM or is there some configuration stored in nonvolatile memory in the GPU itself? Otherwise it seems that a failsafe way to unbrick would be to rewrite the EEPROM out-of-system using something like a buspirate.