Hello everyone!
I have recently obtained Tesla K10 (converted as K2) from eBay. Unfortunately, K2 drivers (non-vGPU) are not supported by modern Linux, so I decided to convert it back to K10 for now. I installed resistors and I got it displayed as K10, however I cannot find a correct BIOS dump for those GPUs.
I used nvflash with override to flash one of the chips using this one: https://www.techpowerup.com/vgabios/213266/213266
Does anyone have full BIOS dump from original K10 for both vBIOS chips? From what I understand, they are not the same.
P.S. I plan to work on some interesting project with final goal to convert dual GPU Tesla K10 to dual K5000. K10/K2 and K5000 share similar GPUs (GK104) of different part#, however number of CUDA cores, TMUs and ROPs is the same. (same story as GK104 on GTX690 VS GK104 on K5000 or GK104 on GTX680).
Tesla K10: 10de:118F
Grid K2: 10de:11BF
Quadro K5000: 10de:11BA
So far. all the values of resistors mentioned on this topic are valid for real K10 as well. Picture below shows resistors used for the K10 BIOS chip (rear one, near power connectors). R2 and R3 are responsible for the byte change from 8 to B while R4 and R5 are responsible for the GDDR5 manufacturer (Samsung VS Hynix). R1 and some supporting resistors around are the part of the ROM circuit that is identical to GTX7xx lineup. I compared values and reverse engineered schematic of the ROM. Schematic is attached on the second picture. vBIOS model is the following: http://ww1.microchip.com/downloads/en/devicedoc/doc0606.pdf
From my understanding, 4th byte difference comes not from the vBIOS circuit (because it affects 3rd byte only), but from the strap #2 on the GPU die itself. GTX780ti schematic is attached for your reference. I have found it on some Russian electronics repair forum. I have seen it was told that I can change 4th byte using BIOS straps, however I did not understand how to do it. That would be great if someone can elaborate on this. Thank you.
That's exactly what I am looking for, GPU1 vBIOS from K10. If someone has it, it would help me a lot.
Also, I wonder if community here figured out where 4th byte resistor is connected to on GK104 chips. I understand it goes to the GPU die, however I cannot reverse engineer it without disassembling GPU itself.
Cannot flash vBIOS #1 because chip has device ID ending with eight zeros. How to resolve this? Thank you.
Cannot flash vBIOS #1 because chip has device ID ending with eight zeros. How to resolve this? Thank you.
Those last 8 numbers are the Subsystem ID. GPU1 is supposed to be Subsystem 10DE 0970. So you will need to change that first. What you can do is force the GPU1 flash by entering the force arguments for NVFLASH. There is a modded a version of NVFLASH that bypasses these IDs. Please try that and see if it will let you flash.
QuoteThere is a post in this thread with working hacked vgpu grid driver. Why you not ask him for a working solution?
There's 2 options. Make an add-on to DualCoder's script that takes out the licensing requirement. OR, hack the licensing server because it runs FlexNet license server which is apparently very easy to crack.
Still waiting for a working noob version. 😊
Same for me, I’ve tried to find out how to extend the trial on the licensing server, but I looks like I’ll just have to keep signing up for trial licenses...
Based on my testing, the internal timer to license requirement starts at 20 minutes of guest VM Nvidia driver running. If you can mask that, you get the features forever. But that is harder to do.
args: -device 'vfio-pci,host=06:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=quadrok5000.rom,x-pci-vendor-id=0x10de,x-pci-device-id=0x11BA,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x0965'
Based on my testing, the internal timer to license requirement starts at 20 minutes of guest VM Nvidia driver running. If you can mask that, you get the features forever. But that is harder to do.
The guest's driver? On a Windows guest? Then here is a crazy idea: Attach CheatEngine to the driver, enable speedhack, speed = 0, done. I have no idea if that could work, it will probably cause other problem though.
mcerveny: "470.05_gameready_win10-dch_64bit_international.exe" allow passthrou of GTX card without any virtualiaztion quirks (and without error code 43) ?
however SolidWorks with my university license just refuses to run under KVM virtualization whatsoever.
1. Fake Quadro K5000 inside Windows VM can be detected as Tesla K10
Massive thanks to @DualCoder for making the vGPU Unlock program. It works fairly well!
I tested it out with PROXMOX hypervisor (KVM) and a Windows 10 Virtual Machine. I was even able to game at 60 FPS! Only problem is that after 10 minutes it capped me at 3 FPS because I can't afford a license
In all seriousness though, this is awesome work and it is actually very well done. I suggest all of you try it out and contribute to the project!
Next step: trying to figure out how to bypass licensing...
Quotemcerveny: "470.05_gameready_win10-dch_64bit_international.exe" allow passthrou of GTX card without any virtualiaztion quirks (and without error code 43) ?Quotehowever SolidWorks with my university license just refuses to run under KVM virtualization whatsoever.
If you enable Hyper-V on the guest system, for whatever reason the VM becomes a regular computer and doesn't register as a VM, allowing you to play pretty much any game with VM detection anticheat. I'm sure Solidworks will work on VMs because that's the main selling point of vGPU in the first place. If it doesn't work, make sure you install Hyper-V, on guest, enable nested virtualization on host, and set CPU model to 'host' or 'passthrough.'
Edit 1: Also make sure to install QEMU guest driver or it wont work. I will also test out 470.05 drivers and see if that bypasses the checks.
Edit 2: You probably don't even need to enable HyperV in the first place. With CPU set to 'host' and Nested VT enabled, my VM just works with no Error 43 at all!Quote1. Fake Quadro K5000 inside Windows VM can be detected as Tesla K10
I should have probably mentioned earlier that all that this PCI ID spoof does is change the PCI ID and name of the GPU, but it still recognizes as whatever the GPU actually is. That's why it's not a very effective method. My recommendation for you is to convert each chip on the K10 to a Quadro K5000, or a Grid K2 and it pass it through either as Grid K2 OR K5000 and hopefully all display technologies can activate, such as DirectX. Let me know what works for you and what doesn't.
Quotemcerveny: "470.05_gameready_win10-dch_64bit_international.exe" allow passthrou of GTX card without any virtualiaztion quirks (and without error code 43) ?Quotehowever SolidWorks with my university license just refuses to run under KVM virtualization whatsoever.
If you enable Hyper-V on the guest system, for whatever reason the VM becomes a regular computer and doesn't register as a VM, allowing you to play pretty much any game with VM detection anticheat. I'm sure Solidworks will work on VMs because that's the main selling point of vGPU in the first place. If it doesn't work, make sure you install Hyper-V, on guest, enable nested virtualization on host, and set CPU model to 'host' or 'passthrough.'
GPU die straps for GPU#2 (power side)
QuoteGPU die straps for GPU#2 (power side)
There is a video made by YouTuber Craft Computing where he does the Tesla K10 to K2 mod
nvflash.exe --index=0 -6 k10-1-5000.rom
args: -cpu 'host,hv_time,kvm=off' -device 'vfio-pci,host=06:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=quadrok5000.rom,x-pci-vendor-id=0x10de,x-pci-device-id=0x11BA,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x0965'
QuoteGPU die straps for GPU#2 (power side)
Personally, I own a GTX 660 Ti which I have done the PCI strap mod on and got it to a GRID K2. But that did not unlock vGPU so it was useless.
There is a video made by YouTuber Craft Computing where he does the Tesla K10 to K2 mod, and there are also instructions for that in this thread posted a few years ago. Were you able to get it converted into a Quadro K5000? I've been thinking about doing some sort of mod to get better workstation performance... (potato server)
In the meantime, I am going to do some experimentation on AMD cards because that seems like a pathway that nobody is really exploring right now. Let me know if any of you are interested!
====SOLUTION BELOW====
Solution is still under development and tested on GPU #1 ("video port" side) only on Tesla K10. GPU #2 (power connector side) and new rom test is required, however my GPU#2 requires repairs, thus I will test it only later in April when I get my components.
Disclaimer: I am not an Assembly programmer nor GPU designer (yet? still studying to become one). Whatever is written below was just my final most dumb approach (what if it just works?).
0. GPU differences:
GPU: ------- CUDA? ------- 3D enabled ------- Video out? ------- QEMU KVM?
K5000 ------ yes ------------- yes -------------- yes -------------- yes
Tesla K10 --- yes ------------ no ---------------- no --------------- yes
Grid K2 ----- no ------------- yes --------------- no ---------------- no (RHEL KVM works)
Quote====SOLUTION BELOW====
Solution is still under development and tested on GPU #1 ("video port" side) only on Tesla K10. GPU #2 (power connector side) and new rom test is required, however my GPU#2 requires repairs, thus I will test it only later in April when I get my components.
Disclaimer: I am not an Assembly programmer nor GPU designer (yet? still studying to become one). Whatever is written below was just my final most dumb approach (what if it just works?).
0. GPU differences:
GPU: ------- CUDA? ------- 3D enabled ------- Video out? ------- QEMU KVM?
K5000 ------ yes ------------- yes -------------- yes -------------- yes
Tesla K10 --- yes ------------ no ---------------- no --------------- yes
Grid K2 ----- no ------------- yes --------------- no ---------------- no (RHEL KVM works)
Wow @dgusev thank you so much for the writeup! Can't wait to get it it working on the 660 Ti perhaps.
As far as vGPU is concerned on K10's and GRID K2's, I'm thinking of building the XEN Grid 350 drivers for RHEL so that all of us linux users can enjoy Grid K2 on KVM! The drivers across the 2 operating systems are very similar, the only difference being that XEN does not use VFIO-MDEV for VFIO mediated devices, which is what we need for RHEL/linux KVM support. Because the XEN driver uses RPM, it will be pretty easy to work with on RHEL, not sure about other operating systems though. Because I don't own a GRID K2 or Tesla K10, I have no way of testing this and may have to resort to modding DualCoder's vGPU unlock script to support GTX 660Ti. There is also no NVENC or CUDA support on GRID K2 vGPU (Not that I know of)
Ultimately, this method allows to get two K5000 without vGPU and other struggles and everything is pretty much enabled compare to Grid K2 that has CUDA and NVENC disabled. I think this is probably the only solution for Kepler-based Tesla cards.
QuoteUltimately, this method allows to get two K5000 without vGPU and other struggles and everything is pretty much enabled compare to Grid K2 that has CUDA and NVENC disabled. I think this is probably the only solution for Kepler-based Tesla cards.
I think your solution is awesome and it will definitely come in handy for many others. I should mention that this requires a KVM host for the PCI ID spoof, and unfortunately my server won't enable passthrough for KVM so I have to use ESXI. Because of this, everyone who is not using a KVM hypervisor or is using as a bare-metal graphics device will need to do PCI ID straps resistors, something covered with great detail on this forum.
Yes, it is official now. GTX/RTX is enabled for passthough without modding from R465 beta drivers.