[MOVED] Hacking NVidia Cards into their Professional Counterparts

[krutav]

Hello everyone!

I have recently obtained Tesla K10 (converted as K2) from eBay. Unfortunately, K2 drivers (non-vGPU) are not supported by modern Linux, so I decided to convert it back to K10 for now. I installed resistors and I got it displayed as K10, however I cannot find a correct BIOS dump for those GPUs.
I used nvflash with override to flash one of the chips using this one: https://www.techpowerup.com/vgabios/213266/213266

Does anyone have full BIOS dump from original K10 for both vBIOS chips? From what I understand, they are not the same.
 
P.S. I plan to work on some interesting project with final goal to convert dual GPU Tesla K10 to dual K5000. K10/K2 and K5000 share similar GPUs (GK104) of different part#, however number of CUDA cores, TMUs and ROPs is the same. (same story as GK104 on GTX690 VS GK104 on K5000 or GK104 on GTX680).
Tesla K10:        10de:118F
Grid K2:           10de:11BF
Quadro K5000: 10de:11BA

So far. all the values of resistors mentioned on this topic are valid for real K10 as well. Picture below shows resistors used for the K10 BIOS chip (rear one, near power connectors). R2 and R3 are responsible for the byte change from 8 to B while R4 and R5 are responsible for the GDDR5 manufacturer (Samsung VS Hynix). R1 and some supporting resistors around are the part of the ROM circuit that is identical to GTX7xx lineup. I compared values and reverse engineered schematic of the ROM. Schematic is attached on the second picture. vBIOS model is the following: http://ww1.microchip.com/downloads/en/devicedoc/doc0606.pdf

From my understanding, 4th byte difference comes not from the vBIOS circuit (because it affects 3rd byte only), but from the strap #2 on the GPU die itself. GTX780ti schematic is attached for your reference. I have found it on some Russian electronics repair forum. I have seen it was told that I can change 4th byte using BIOS straps, however I did not understand how to do it. That would be great if someone can elaborate on this. Thank you.

I could only find GPU2 vBIOS on TechPowerUp. One thing you might be able to do is convert the card to Quadro K5000 by doing the resistor method. You may not have to flash vBIOS for this, but if you do have to, flash the K5000 vBIOS onto GPU1, and flash Tesla K10 on GPU2. Someone here might have K10 GPU1 vBIOS...

[dgusev]

That's exactly what I am looking for, GPU1 vBIOS from K10. If someone has it, it would help me a lot.
Also, I wonder if community here figured out where 4th byte resistor is connected to on GK104 chips. I understand it goes to the GPU die, however I cannot reverse engineer it without disassembling GPU itself.

[krutav]

That's exactly what I am looking for, GPU1 vBIOS from K10. If someone has it, it would help me a lot.
Also, I wonder if community here figured out where 4th byte resistor is connected to on GK104 chips. I understand it goes to the GPU die, however I cannot reverse engineer it without disassembling GPU itself.

Here is GPU1:
https://www.techpowerup.com/vgabios/213893/213893
Here is GPU2:
https://www.techpowerup.com/vgabios/213266/213266

Give it a shot, may or may not work, hoping it does.

[dgusev]

Cannot flash vBIOS #1 because chip has device ID ending with eight zeros. How to resolve this? Thank you.

[krutav]

Cannot flash vBIOS #1 because chip has device ID ending with eight zeros. How to resolve this? Thank you.

Those last 8 numbers are the Subsystem ID. GPU1 is supposed to be Subsystem 10DE 0970. So you will need to change that first. What you can do is force the GPU1 flash by entering the force arguments for NVFLASH. There is a modded a version of NVFLASH that bypasses these IDs. Please try that and see if it will let you flash.

[dgusev]

Cannot flash vBIOS #1 because chip has device ID ending with eight zeros. How to resolve this? Thank you.

Those last 8 numbers are the Subsystem ID. GPU1 is supposed to be Subsystem 10DE 0970. So you will need to change that first. What you can do is force the GPU1 flash by entering the force arguments for NVFLASH. There is a modded a version of NVFLASH that bypasses these IDs. Please try that and see if it will let you flash.

Yeah, I have tried this already. Seems like EEPROM was not accessible and it refused to flash it. I am currently playing with Raspberry Pi flashrom software to flash this chip outside of the GPU. I will see how it goes 

P.S. still no success. My plan now is to get CH341A from Amazon and try to flash it using that tool.

P.P.S. GOT IT WORKING! 
Patched "NVflash with mismatched ID" does not work at all for PCI subsystem ID mismatch

I used NVFlash 5.670 with the following command: nvflash.exe --index=0 --overridesub TeslaK10-1.rom
Proof below: I use it my Proxmox Ubuntu 20.04 server VM

[user567876]

There is a post in this thread with working hacked vgpu grid driver. Why you not ask him for a working solution?

There's 2 options. Make an add-on to DualCoder's script that takes out the licensing requirement. OR, hack the licensing server because it runs FlexNet license server which is apparently very easy to crack.

Still waiting for a working noob version. 😊

Same for me, I’ve tried to find out how to extend the trial on the licensing server, but I looks like I’ll just have to keep signing up for trial licenses...

[krutav]

Same for me, I’ve tried to find out how to extend the trial on the licensing server, but I looks like I’ll just have to keep signing up for trial licenses...

Well actually I called their enterprise support and they said that they do not provide more trial licenses. Based on my testing, the internal timer to license requirement starts at 20 minutes of guest VM Nvidia driver running. If you can mask that, you get the features forever. But that is harder to do. The easier solution that I have figured out is to just use a vGPU profile that doesn't have a licensing requirement. Inconveniently, this is only possible on the Grid K1/K2 cards which means that you have to create your own profile and sign it with a key. OR, you could add the Grid K2 profile to the list of profiles in VgpuConfig.xml and then modify DualCoder's script to spoof your GPU to the PCI ID of GRID K2. Unfortunately, CUDA has not been enabled on the GRID K2 profiles for some ridiculous reason so you won't get the benefit of that. Happy modding, let us all know if you have any updates or successes!

[DualCoder]

Based on my testing, the internal timer to license requirement starts at 20 minutes of guest VM Nvidia driver running. If you can mask that, you get the features forever. But that is harder to do.

The guest's driver? On a Windows guest? Then here is a crazy idea: Attach CheatEngine to the driver, enable speedhack, speed = 0, done. I have no idea if that could work, it will probably cause other problem though.

[dgusev]

krutav's method works and I managed to split Tesla K10 into two Quadro K5000s for two Windows VMs under Proxmox (screen attached).

What's working:
1. 3D acceleration benchmarks like FurMark run perfectly
2. 1st Gen NVENC movie transcoding for MKV containers using XMedia Recode (according to CraftComputing vGPU does not support NVENC)
3. Converting Tesla K10 to other GK104 GPUs (e.g. GTX770) with all the drivers installed properly
4. MSI afterburner temperature monitoring and even overclocking

What's not working:
1. When I try to run, for example, a game (not a Steam cuz Steam detects VMs), 3D acceleration is not working.
2. Parsec refuses to load whatsoever. So, I guess, RDP or VNC is the only option.
3. I am also working to make SolidWorks run as well as other CADs to use Quadro K5000 for what it is intended for, however SolidWorks with my university license just refuses to run under KVM virtualization whatsoever.

Interesting notes:
1. Fake Quadro K5000 inside Windows VM can be detected as Tesla K10 by Quadro Experience that offers to install Tesla drivers instead of Quadro drivers, however nothing prevents you from downloading drivers from Nvidia website and install them yourself. Quadro Experience will indicate your graphics card as Quadro K5000 after it.
2. After I changed config to turn K10 into GTX770 and ran my VM with K5000 drivers, MSI afterburner indicated it as Tesla K10, while Device Manager and Nvidia drivers show it as GTX770. I guess this method allows to trick Nvidia to install drivers and partially work with 3D acceleration, however I believe it is not fully enabled, unless your host GPU is 3D capable by itself (here is why my next step is to rebuild K10 back to Grid K2).

Next step:
Convert K10 back to Grid K2 with resistor and BIOS mod and try Windows VM again using krutav's method. I previously bumped into the issue that Grid K2 gives Error 43 inside Windows VM and neither of Proxmox tricks help this.
1. Grid K2 (actual) -> krutav's method -> Grid K2 in Windows and see if drivers work.
2. Grid K2 (actual) -> krutav's method -> Quadro K5000 (I want to see if 3D acceleration issue can be resolved by physicall enabling 3D acceleration on the host card).

For those who would like to repeat this experiment, I attach my Proxmox VMID.conf line. BIOS is available at TechPowerUp.

Code: [Select]

args: -device 'vfio-pci,host=06:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=quadrok5000.rom,x-pci-vendor-id=0x10de,x-pci-device-id=0x11BA,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x0965'
UPD: Unigine Heaven 4.0 works perfectly. Screen attached. Stable 60-80 FPS in 720p with DirectX 9 with Tesla K10 tricked to appear as K5000.
UPD2: Games do not work whatsoever on Tesla K10 faked as K5000.
UPD3: Solidworks does not work on Tesla K10 faked as K5000. It runs, but system becomes incredibly slow. Next test is to try both Grid K2 and Grid K2 (faked as K5000).
UPD4: DirectX games do not work at all. Vulkan/OpenGL games work on Tesla K10 faked as Quadro K5000. Older games (e.g. HL2, S.T.A.L.K.E.R) refuse to run either in Steam versions or pirated versions. Steam Doom (2016) runs perfectly with Steam Remote Play.

UPD5 (March 22nd, 2021): SolidWorks actually works but I guess it is slow because of Windows RDP. Quadro K5000 is properly utilized.

[krutav]

Based on my testing, the internal timer to license requirement starts at 20 minutes of guest VM Nvidia driver running. If you can mask that, you get the features forever. But that is harder to do.

The guest's driver? On a Windows guest? Then here is a crazy idea: Attach CheatEngine to the driver, enable speedhack, speed = 0, done. I have no idea if that could work, it will probably cause other problem though.

CheatEngine won't work simply because the host system driver starts an internal 20-minute timer the moment the windows vGPU driver turns on. I have tried everything from killing the NV Container process to deleting it entirely, but host is smart and can see if the vGPU has any activity on it. The solution I found out is to create a new vGPU profile with <license> set to none and get it signed by the private key. That part I don't know how to do but maybe it's possible.

[krutav]

mcerveny: "470.05_gameready_win10-dch_64bit_international.exe" allow passthrou of GTX card without any virtualiaztion quirks (and without error code 43) ?

however SolidWorks with my university license just refuses to run under KVM virtualization whatsoever.

If you enable Hyper-V on the guest system, for whatever reason the VM becomes a regular computer and doesn't register as a VM, allowing you to play pretty much any game with VM detection anticheat. I'm sure Solidworks will work on VMs because that's the main selling point of vGPU in the first place. If it doesn't work, make sure you install Hyper-V, on guest, enable nested virtualization on host, and set CPU model to 'host' or 'passthrough.'

Edit 1: Also make sure to install QEMU guest driver or it wont work. I will also test out 470.05 drivers and see if that bypasses the checks.
Edit 2: You probably don't even need to enable HyperV in the first place. With CPU set to 'host' and Nested VT enabled, my VM just works with no Error 43 at all!

1. Fake Quadro K5000 inside Windows VM can be detected as Tesla K10

I should have probably mentioned earlier that all that this PCI ID spoof does is change the PCI ID and name of the GPU, but it still recognizes as whatever the GPU actually is. That's why it's not a very effective method. My recommendation for you is to convert each chip on the K10 to a Quadro K5000, or a Grid K2 and it pass it through either as Grid K2 OR K5000 and hopefully all display technologies can activate, such as DirectX. Let me know what works for you and what doesn't.

[kevinshane]

Massive thanks to @DualCoder for making the vGPU Unlock program. It works fairly well!
I tested it out with PROXMOX hypervisor (KVM) and a Windows 10 Virtual Machine. I was even able to game at 60 FPS! Only problem is that after 10 minutes it capped me at 3 FPS because I can't afford a license 

In all seriousness though, this is awesome work and it is actually very well done. I suggest all of you try it out and contribute to the project!

Next step: trying to figure out how to bypass licensing...

Hi dear krutay, does the whole "gtx1080 fake to a p4-4q" method can run multiple win10 vm at the same time? say for example, 4 of the "fake p4-4q" vm running on proxmox? Thanks

edit1: Oh sorry I just double check the p4-4q profile and it's a 4G card, so if I understand it correctly, this type of profile can run 2 win VMs at the same time theoretically? so how about the "P4-1Q" profile? as the nvidia doc said it can support up to 8 vgpus per board, so theoretically your gtx1080 can at least run 8 VMs simultaneously?

edit2: for the nvidia license price, I don't get it why you are saying it's freaking expensive, because I just asking a guy who sell those licenses for like 150 US$/8vDWS or around 150US$/32vPC, I don't know what exactly the difference between vPC and vDWS tho.. but the price sounds good to me(if the license is legal, which again have no idea).. May I ask you what was the price for your location that nvidia charge you?

[user567876]

Who quoted you that price? According to Nvidia’s website and what I see on Dell.com it’s $50 per year per user which has to be bought in 4-5 year increments. Check out this link. https://www.nvidia.com/en-us/data-center/buy-grid/] [url]https://www.nvidia.com/en-us/data-center/buy-grid/[/url]

[dgusev]

mcerveny: "470.05_gameready_win10-dch_64bit_international.exe" allow passthrou of GTX card without any virtualiaztion quirks (and without error code 43) ?

however SolidWorks with my university license just refuses to run under KVM virtualization whatsoever.

If you enable Hyper-V on the guest system, for whatever reason the VM becomes a regular computer and doesn't register as a VM, allowing you to play pretty much any game with VM detection anticheat. I'm sure Solidworks will work on VMs because that's the main selling point of vGPU in the first place. If it doesn't work, make sure you install Hyper-V, on guest, enable nested virtualization on host, and set CPU model to 'host' or 'passthrough.'

Edit 1: Also make sure to install QEMU guest driver or it wont work. I will also test out 470.05 drivers and see if that bypasses the checks.
Edit 2: You probably don't even need to enable HyperV in the first place. With CPU set to 'host' and Nested VT enabled, my VM just works with no Error 43 at all!

1. Fake Quadro K5000 inside Windows VM can be detected as Tesla K10

I should have probably mentioned earlier that all that this PCI ID spoof does is change the PCI ID and name of the GPU, but it still recognizes as whatever the GPU actually is. That's why it's not a very effective method. My recommendation for you is to convert each chip on the K10 to a Quadro K5000, or a Grid K2 and it pass it through either as Grid K2 OR K5000 and hopefully all display technologies can activate, such as DirectX. Let me know what works for you and what doesn't.

UPD #1: burned vBIOS #2 chip while soldering with fan ( ). I will buy a replacement one. Model is GD25Q20B.

UPD #2: vBIOS #1 was flashed as Grid K2 and I still have driver issues with K2 under Windows VM. No idea why.

UPD #3: K2 cannot be PCI ID spoofed into K5000 to resolve this. K5000 drivers can be installed but they fail after reboot in the same way as K2 drivers. K10 can be spoofed into K5000 and work with SolidWorks and some OpenGL/Vulkan games.

UPD #4: I have found a set of resistors that might help to turn K10 into dual K5000. Values are ranging between 5K and 45K just like GTX780ti schematic I sent before. I guess this might help. I will do more tests and research and make an update. Picture #1 shows GPU die straps for #1 (PCIe side), Picture #2 is for GPU #2 (power connector side)

P.S. GPU#2 straps will be in 2nd post, because 5000kb max per post.

[dgusev]

GPU die straps for GPU#2 (power side) 

Edit 1: These are NOT vBIOS straps. Those are related to GPU die straps and responsible for the various settings such as VGA enable etc.

mcerveny: "470.05_gameready_win10-dch_64bit_international.exe" allow passthrou of GTX card without any virtualiaztion quirks (and without error code 43) ?

however SolidWorks with my university license just refuses to run under KVM virtualization whatsoever.

If you enable Hyper-V on the guest system, for whatever reason the VM becomes a regular computer and doesn't register as a VM, allowing you to play pretty much any game with VM detection anticheat. I'm sure Solidworks will work on VMs because that's the main selling point of vGPU in the first place. If it doesn't work, make sure you install Hyper-V, on guest, enable nested virtualization on host, and set CPU model to 'host' or 'passthrough.'

I tried again today with single K2 chip. Windows 10 does not detect VM after Hyper-V installation (sounds like another microsoft bug), but Grid K2 still gives Error 43.

My server is based on Dell T3600 that uses C600 chipset just like on R720 and all IOMMUs are correct. Again, Tesla K10 (either by itself or K5000) works perfectly as well as Radeon Vega cards work inside VMs. I have issues only with Grid K2 or lower Quadro cards (I tested with P400 before) and only under Windows. Linux has no issues with VMs and Nvidia. I previously got Grid K2 to work under Linux and even run some gears OpenGL benchmarks.

[dgusev]

3D acceleration fully enabled for "physically" unmodded Tesla K10 under QEMU KVM Proxmox 6.3 with Windows 10 VM.

Both DirectX 9-11 with finally full support for both games and benchmarks as well as full support under Vulkan.

Will post my software solution a little later.

Edit #1: Parsec remote gaming works with Nvidia-based video encoding
Edit #2: Renders in Blender work, 4K render on-going

[krutav]

GPU die straps for GPU#2 (power side)

Personally, I own a GTX 660 Ti which I have done the PCI strap mod on and got it to a GRID K2. But that did not unlock vGPU so it was useless.

There is a video made by YouTuber Craft Computing where he does the Tesla K10 to K2 mod, and there are also instructions for that in this thread posted a few years ago. Were you able to get it converted into a Quadro K5000? I've been thinking about doing some sort of mod to get better workstation performance... (potato server)

In the meantime, I am going to do some experimentation on AMD cards because that seems like a pathway that nobody is really exploring right now. Let me know if any of you are interested!

[dgusev]

GPU die straps for GPU#2 (power side)

There is a video made by YouTuber Craft Computing where he does the Tesla K10 to K2 mod

Craft Computing YouTube recommendation is where my entire Tesla K10 journey has started. I got Tesla K10 from eBay that was already converted to Grid K2 by someone.

It works in bare-metal windows machine in a PC tower, however my goal was to use it inside of the Proxmox potato server I have. vGPU does not work under KVM (unless it is RHEL) and I have never succeeded in passthrough of the Grid K2 to the VM, unresolvable code 43. I basically gave up on K2 and my next goal is to get full hardware mode of the K10 into K5000, however I don't think it is necessary anymore.

I have created another way to do it without hardware mode.

====SOLUTION BELOW====
Solution is still under development and tested on GPU #1 ("video port" side) only on Tesla K10. GPU #2 (power connector side) and new rom test is required, however my GPU#2 requires repairs, thus I will test it only later in April when I get my components.

Disclaimer: I am not an Assembly programmer nor GPU designer (yet? still studying to become one). Whatever is written below was just my final most dumb approach (what if it just works?).

0. GPU differences:
GPU: ------- CUDA? ------- 3D enabled ------- Video out? ------- QEMU KVM?
K5000 ------ yes ------------- yes -------------- yes -------------- yes
Tesla K10 --- yes ------------ no ---------------- no --------------- yes
Grid K2 ----- no ------------- yes --------------- no ---------------- no (RHEL KVM works)

So, the goal is to bring 3D-acceleration to the Tesla K10 in one form or another...

1. Project was initially inspired by krutav's PCIe ID spoofing method, because, in my opinion, it was not tested enough.
What do we know?
PCIe ID change using hardware mode enables some of the features, but not all of them... 660Ti or some other cards converted to Grid K2 does not enable all features of the Grid K2.
PCIe ID spoofing using purely Proxmox enables some of the features, but again, not all of them... Probably, KVM limitation, probably something else.

My PCIe ID spoofing results of softstraping Tesla K10 to K5000:
DirectX: NO
Vulkan: YES (partially)
NVENC: yes
CUDA: yes
CAD: yes

What is my goal?
Make it work as true Quadro K5000.

What prevents other GPUs (including K10) from being a true hacked target GPU (e.g. K5000)?
If you check GTX780ti schematic I have sent before and you check page 28 (ROM and straps) you will see that GPU die straps are responsible for the bunch of secondary GPU settings that are not PCIe ID related. VGA enable and many other ones. I believe, some of them may affect the final GPU performance.

Additionally, my assumption that GPU's vBIOS also heavily affects final performance, because it is obviously not just VRAM config and CLK config. My assumption. that vBIOS also controls GPU functionality and allows to enable or disable features.
This assumption comes from some old Plex Media Server NVENC hack from habr.com written by some Russian user, who hacked nvidia to run multiple NVENC sessions on GeForce.

While Grid K2 and Tesla K10 share 99.99% same PCB (except for 3rd nibble resistor), Tesla K10 and VGA-enabled GPUs (GTX680, K5000 etc) do not share same strap resistor config pattern (GTX780ti, I can also drop other Kepler schematics, I have GTX770 that uses GK104), thus configuration of the Tesla K10 to K5000 may not be that simple as I initially thought.

2. What is my idea?
What if I can flash K5000 vBIOS to K10 without doing a hardware mod?
Long time ago in this thread, some users mentioned rewriting vBIOS and replacing PCIe IDs and PCIe subsystem IDs using hex editor to change values and KeplerBiosTweak to verify the checksum. That's exactly what I decided to do!
I took K5000 vBIOS and replaced PCIe and subsystem IDs of the K5000 with Tesla K10 IDs. Why? Because nvflash tool cannot even force-flash a card if hardware IDs are incorrect.

So, my solution is to flash K5000 BIOS with K10 IDs and fixed checksum. ROM is attached below for anyone who is willing to play with this. Only Tesla K10 is supported and ONLY GPU #1 is tested right now.

Use nvflash tool with mismatched ID patch.
Command:

Code: [Select]

nvflash.exe --index=0 -6 k10-1-5000.rom
Install your card under KVM based VM using krutav's method.

Code: [Select]

args: -cpu 'host,hv_time,kvm=off' -device 'vfio-pci,host=06:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=quadrok5000.rom,x-pci-vendor-id=0x10de,x-pci-device-id=0x11BA,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x0965'
Remember to change the host-0X:00.0 to your GPU IDs using lspci -v command in terminal.
Also, quadrok5000.rom is the real K5000 vBIOS from TechPowerUp, not a modified version below.

Install drivers for K5000 either using default .exe (Windows) / .run (Linux) or Quadro Experience. It just works.

3. Did it work?
Kinda. More yes than no.

DirectX: supported now both under games and benchmarks (DirectX < 12), older games written in DirectX9 tested and work
Vulkan: works (as much as it can)
OpenGL: works
3D rendering: works (Blender and PhotoView360 tested)
NVENC: fully enabled (including cloud streaming e.g. Parsec Remote Gaming)
CUDA: works
CADs: work (SolidWorks 2020)

4. Conclusion:
I plan to do same mod to the second vBIOS of the K10 and run two VMs under Proxmox with full GPU passthrough.
Whatever bug inside drivers/Proxmox/vBIOS compare to original krutav's method actually encloses your true GPU to the system and Quadro Experience, MSI Afterburner and games/benchmarks see it as Tesla K10, while Windows sees it as K5000.

vBIOS is available below:
https://github.com/amidg/teslak10-3d-enabled

[user567876]

GPU die straps for GPU#2 (power side)

Personally, I own a GTX 660 Ti which I have done the PCI strap mod on and got it to a GRID K2. But that did not unlock vGPU so it was useless.

There is a video made by YouTuber Craft Computing where he does the Tesla K10 to K2 mod, and there are also instructions for that in this thread posted a few years ago. Were you able to get it converted into a Quadro K5000? I've been thinking about doing some sort of mod to get better workstation performance... (potato server)

In the meantime, I am going to do some experimentation on AMD cards because that seems like a pathway that nobody is really exploring right now. Let me know if any of you are interested!

I tried the AMD FirePro S1750 and the card was cheap but I was never able to get the MXGPU to function properly with esxi 7 as they only support up to 6.5. (So I did get it working but it would crash under load)

[krutav]

====SOLUTION BELOW====
Solution is still under development and tested on GPU #1 ("video port" side) only on Tesla K10. GPU #2 (power connector side) and new rom test is required, however my GPU#2 requires repairs, thus I will test it only later in April when I get my components.

Disclaimer: I am not an Assembly programmer nor GPU designer (yet? still studying to become one). Whatever is written below was just my final most dumb approach (what if it just works?).

0. GPU differences:
GPU: ------- CUDA? ------- 3D enabled ------- Video out? ------- QEMU KVM?
K5000 ------ yes ------------- yes -------------- yes -------------- yes
Tesla K10 --- yes ------------ no ---------------- no --------------- yes
Grid K2 ----- no ------------- yes --------------- no ---------------- no (RHEL KVM works)

Wow @dgusev thank you so much for the writeup! Can't wait to get it it working on the 660 Ti perhaps.

As far as vGPU is concerned on K10's and GRID K2's, I'm thinking of building the XEN Grid 350 drivers for RHEL so that all of us linux users can enjoy Grid K2 on KVM! The drivers across the 2 operating systems are very similar, the only difference being that XEN does not use VFIO-MDEV for VFIO mediated devices, which is what we need for RHEL/linux KVM support. Because the XEN driver uses RPM, it will be pretty easy to work with on RHEL, not sure about other operating systems though. Because I don't own a GRID K2 or Tesla K10, I have no way of testing this and may have to resort to modding DualCoder's vGPU unlock script to support GTX 660Ti. There is also no NVENC or CUDA support on GRID K2 vGPU (Not that I know of)

[dgusev]

====SOLUTION BELOW====
Solution is still under development and tested on GPU #1 ("video port" side) only on Tesla K10. GPU #2 (power connector side) and new rom test is required, however my GPU#2 requires repairs, thus I will test it only later in April when I get my components.

Disclaimer: I am not an Assembly programmer nor GPU designer (yet? still studying to become one). Whatever is written below was just my final most dumb approach (what if it just works?).

0. GPU differences:
GPU: ------- CUDA? ------- 3D enabled ------- Video out? ------- QEMU KVM?
K5000 ------ yes ------------- yes -------------- yes -------------- yes
Tesla K10 --- yes ------------ no ---------------- no --------------- yes
Grid K2 ----- no ------------- yes --------------- no ---------------- no (RHEL KVM works)

Wow @dgusev thank you so much for the writeup! Can't wait to get it it working on the 660 Ti perhaps.

As far as vGPU is concerned on K10's and GRID K2's, I'm thinking of building the XEN Grid 350 drivers for RHEL so that all of us linux users can enjoy Grid K2 on KVM! The drivers across the 2 operating systems are very similar, the only difference being that XEN does not use VFIO-MDEV for VFIO mediated devices, which is what we need for RHEL/linux KVM support. Because the XEN driver uses RPM, it will be pretty easy to work with on RHEL, not sure about other operating systems though. Because I don't own a GRID K2 or Tesla K10, I have no way of testing this and may have to resort to modding DualCoder's vGPU unlock script to support GTX 660Ti. There is also no NVENC or CUDA support on GRID K2 vGPU (Not that I know of)

I would be happy if we can find a way to run vGPU under Proxmox on Kepler-based cards, however I don't think we can do it. Even with @DualCoder's solution, because Kepler generation does not have a Debian KVM compatible Grid driver, so we basically need to write our own. My server based on Dell T3600 runs everything including TrueNAS, Plex (K2000 in the same server for NVENC) and Windows VMs with GPUs, thus I believe the solution above is pretty much what I have been looking for.
K2000 occupies nvidia driver in proxmox kernel, while K10 uses vfio-pci driver.

Ultimately, this method allows to get two K5000 without vGPU and other struggles and everything is pretty much enabled compare to Grid K2 that has CUDA and NVENC disabled. I think this is probably the only solution for Kepler-based Tesla cards.

[krutav]

Ultimately, this method allows to get two K5000 without vGPU and other struggles and everything is pretty much enabled compare to Grid K2 that has CUDA and NVENC disabled. I think this is probably the only solution for Kepler-based Tesla cards.

I think your solution is awesome and it will definitely come in handy for many others. I should mention that this requires a KVM host for the PCI ID spoof, and unfortunately my server won't enable passthrough for KVM so I have to use ESXI. Because of this, everyone who is not using a KVM hypervisor or is using as a bare-metal graphics device will need to do PCI ID straps resistors, something covered with great detail on this forum.

Only problem is that I have GTX 660 Ti which has the lowest binned silicon compared to the Tesla and Quadros, which means that your method has a low chance of working on the low-end GK104 cards. I'd love to be wrong though, so if I have time I will try soldering some new resistors on.

And one more thing for all of you, I have the schematics/boardviews for Kepler series graphics cards like the GTX 760 (GK104) and GTX 780 (GK110) which you can reference if you want to do a mod or repair your graphics card. Not sure if I can link them here, but they are available on a BadCaps forum if you need.

[dgusev]

Ultimately, this method allows to get two K5000 without vGPU and other struggles and everything is pretty much enabled compare to Grid K2 that has CUDA and NVENC disabled. I think this is probably the only solution for Kepler-based Tesla cards.

I think your solution is awesome and it will definitely come in handy for many others. I should mention that this requires a KVM host for the PCI ID spoof, and unfortunately my server won't enable passthrough for KVM so I have to use ESXI. Because of this, everyone who is not using a KVM hypervisor or is using as a bare-metal graphics device will need to do PCI ID straps resistors, something covered with great detail on this forum.

Oh, yeah, as stated on GitHub that's basically tested only for K10 and I targeted my Proxmox server with KVM.
I probably need to clearly state this on GitHub that this solution is only for compatible cards that share same GK104 variant (or at least very similar) with same VRAM etc. I am sure it will not help 660ti or any other lower-end GeForce card because vBIOS/drivers will have to deal with non-existing CUDA cores, non-existing VRAM etc.

That solution is super niche.

[krutav]

Yes, it is official now. GTX/RTX is enabled for passthough without modding from R465 beta drivers.

Awesome! Personally I never had code 43 issues, GTX passes through fine and driver doesn't care at all about VM. 

I think we'll definitely be looking at a more open future of consumer graphics cards. Also, for those of you are working remotely with GeForce cards, Nvidia released a patch you can download called "nvidiaopenglrdp.exe" that enables remote desktop OpenGL for CAD users working remotely. You'll need to create an Nvidia developer account to download it.

Edit: @dgusev my P106 is broken so I cannot try the 3D enable mod similar to how you did with the Tesla K10. It may work... but for some reason the +5V line is going straight to GND.