[MOVED] Hacking NVidia Cards into their Professional Counterparts

[krutav]

Dear Krutav,
thx for your idea, only for understanding i can install a KVM (Kernelbased Virtual Machine) where i can
Install a vbios from any device (example quadro P4000) and configure it to to start a vm with windows 10 and i have a quadro k4000 after in system, right?

Hello Bayx, I recommend you install proxmox VE because it implements KVM into a relatively low resource using OS. You can then use the arguments code that I put in the previous message and modify it according to your need. You can also go to techpowerup.com and download the VBIOS for the card, and upload it to your Proxmox machine according to the wiki guide that they have made for that. If it doesn't work for you, you can always try GPU-P that I detailed in the previous message. Thanks and let me know the results.
If more people want instructions for this, I can write a small guide possibly.

[bayx]

"If more people want instructions for this, I can write a small guide possibly"

I spend 10 beers 😊

[RoGeorge]

If more people want instructions for this, I can write a small guide possibly.

Yes, please do!   

[krutav]

"If more people want instructions for this, I can write a small guide possibly"

I spend 10 beers 😊

Don't worry friends I am working on the experiment right now and you will all have a guide in no time! Currently working: all Kepler cards work with PCI software spoofing on KVM. Pascal architecture is going to be tested next.

Update: My VM is running extremely slow for some reason (I may have tried to enable nested virtualization) so I am getting code 43 regardless of what is passed through. Will fix it first.

[bayx]

I have no idea what about you talk )) but i believe in you... Still waiting for a step by step method.

[krutav]

I have no idea what about you talk )) but i believe in you... Still waiting for a step by step method.

Alright, so there isn't too much to it, but you just need to make sure your system is capable of passthrough. You can search how to configure your KVM software for it. I am using Proxmox VE which is Debian linux.

For the QEMU configuration file, you first need to figure out what PCI device you want to passthrough. Let's bring up the remote SSH shell from Proxmox web interface.
Now the expectation with this guide is that you are already familiar with PCI passthrough on Proxmox and have a basic understanding of all this, if not, there is a great guide on the Proxmox Wiki.
Make sure your machine has been configured for passthrough before proceeding.

Code: [Select]

lspici -nn

Code: [Select]

02:00.0 VGA compatible controller (0300): NVIDIA Corporation GP104 (GeForce GTX 1080)
Okay, so we now know that our Nvidia GPU is on PCI root port 02:00.0.
Next, I created a Linux virtual machine and gave it ID of 100. I then proceeded to install Linux and set it up. To pass the GPU, I needed to shut down. To edit the configuration of the VM, we need to run the next line.

Code: [Select]

nano /etc/pve/local/qemu-server/100.confNow that you are inside your VM configuration, you will need to add this one line that lets us spoof our graphics card.

Code: [Select]

args: -device 'vfio-pci,host=02:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=NVP4000,x-pci-vendor-id=0x10de,x-pci-device-id=0x1BB1,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x11A3'In this code, there are a few things you will need to edit for yourself.

Code: [Select]

host=02:00.0Change this value according to what you got from running lspci -nn

Code: [Select]

romfile=YOURROM.romYou need to obtain the ROM for the graphics card that you choose to spoof as, so I went to techpowerup.com and downloaded Quadro P4000 BIOS. On techpowerup.com, you can also obtain the PCI device ID and the subsystem ID.

Code: [Select]

x-pci-device-id=0x1BB1Here, I have chosen my new Device ID to match that of the Quadro P4000.

Code: [Select]

x-pci-sub-device-id=0x11A3Here, I have chosen my new SubSystem ID to match that of the Quadro P4000.

Alright, we are ready to power on VM. Once VM is on, my graphics card automatically outputted to a monitor. Running lspci -nn on the guest VM revealed that it was indeed functioning as a Quadro P4000 instead of GTX1080, so we have successfully spoofed the graphics card!

[bayx]

Nice. Thx. I test it today first on my laptop.
i5 7300HQ (Vt-d=yes) Geforce1050ti (QUADRO P1000)

edit ON:
NOW I UNDERSTAND :) i hope my laptop is alive after this night
edit OFF

[bayx]

Proxmox= done
vm= done (Windows 10 is running)
change to pcie Passthrough = failed (No IOMMU detected, please activate it.See Documentation for further information.) 
In bios is only one place to change to vt-d on and off thats all. I think it does not work with my laptop HP Pavilion 17-ab303ng

i test it one more time i think i do something wrong....!

[krutav]

Proxmox= done
vm= done (Windows 10 is running)
change to pcie Passthrough = failed (No IOMMU detected, please activate it.See Documentation for further information.) 
In bios is only one place to change to vt-d on and off thats all. I think it does not work with my laptop HP Pavilion 17-ab303ng

i test it one more time i think i do something wrong....!

Laptops don't have a very high success rate with VT-D so I recommend using a desktop. If you still want GPU virtualization, you can try out the new GPU partitioning on windows 10 with the reddit guide that I linked in the earlier post.

[bayx]

Proxmox= done
vm= done (Windows 10 is running)
change to pcie Passthrough = failed (No IOMMU detected, please activate it.See Documentation for further information.) 
In bios is only one place to change to vt-d on and off thats all. I think it does not work with my laptop HP Pavilion 17-ab303ng

i test it one more time i think i do something wrong....!

Laptops don't have a very high success rate with VT-D so I recommend using a desktop. If you still want GPU virtualization, you can try out the new GPU partitioning on windows 10 with the reddit guide that I linked in the earlier post.

Ok. After Proxmox dismis then i test windows variant.... but i have good news

root@pve:~# dmesg |grep -e DMAR -e IOMMU -e AMD-Vi
[    0.007631] ACPI: DMAR 0x000000007AFA9000 0000CC (v01 HPQOEM 845E     00000001 HP   00040000)
[    0.040937] DMAR: IOMMU enabled
[    0.084938] DMAR: Host address width 39
[    0.084939] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.084943] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.084944] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.084947] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.084947] DMAR: RMRR base: 0x0000007a734000 end: 0x0000007a753fff
[    0.084948] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fffffff
[    0.084949] DMAR: ANDD device: 1 name: \_SB.PCI0.I2C0
[    0.084950] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.084951] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.084951] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.084952] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.086408] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.939698] DMAR: ACPI device "device:7e" under DMAR at fed91000 as 00:15.0
[    0.939708] DMAR: No ATSR found
[    0.939733] DMAR: dmar0: Using Queued invalidation
[    0.939735] DMAR: dmar1: Using Queued invalidation
[    0.948245] DMAR: Intel(R) Virtualization Technology for Directed I/O

iommu is running... stay tuned

"If you still want GPU virtualization" i want that my graficscard gf1070 transfors to a quadro p1000 vbios

[bayx]

Passthrough is working but nvidia driver "cant find any driver for you system" and when i try to install it manually, says me error 43 !!!

[krutav]

Passthrough is working but nvidia driver "cant find any driver for you system" and when i try to install it manually, says me error 43 !!!

Same problem here, code 43 on windows. Keep in mind this is because of a few virtualization quirks, not that the nvidia driver isn't working. This means that there is further configuration required on the Proxmox KVM hypervisor to allow windows to work. Because I use Red Hat linux, I was able to do this and even get full legitimate Nvidia GRID vGPU drivers to install. Currently, I am working on getting my Red Hat VM to create a VM inside of it to passthrough this vGPU. Otherwise, it works just fine on linux. You will also need to set your CPU type to "host" in order to expose hardware features to your VM (Very important).

So far: I converted my GTX 1080 to a Tesla P4 with this VFIO modification, and have gotten vGPU driver to initialize. Quadro P4000 spoof also works too. I can also confirm this is fully working with Kepler cards. Currently verified working on Linux guest VM. Windows will require further configuration, including disable MSIs and nested Virtualization.

[bayx]

I dismissed with my laptop...(1050ti problem card) I test it with my PC.

[krutav]

Just an update everyone. I have tried to use the VFIO spoofing trick that I showed earlier. The problem is that I was unable to get the vGPU software on my Red Hat VM to work. The driver initializes, but because of my hardware being 11 years old, the vGPUs refused to bind to the guest VM. The same thing happened without VFIO. So my new plan is actually to reverse engineer the process of determining whether the installed graphics card is vGPU capable. To do this, a magic script is required, which is what user Mcerveny showed off earlier. Because he will not release the magic script to us, we will have to develop it ourselves. With that being said, I have already thought out a way to spoof the PCI IDs of the created vGPUs so that licensing is not necessary, but that is somewhat against the EULA. I had to sign up for the Nvidia GRID 90 day trial to obtain these special drivers.

So the magic script that we need will enable us to use any consumer graphics card with the same chip as a tesla product to act as a supported tesla-based vGPU card by tricking the vGPU-mgr service into thinking that there is a tesla so it can initialize. It shouldn't be too hard I hope. If you want to take a shot at tricking the driver, you can pm me to obtain the open-source Linux driver. I can also post some logs here from my testing.

Fake Tesla P4:

Code: [Select]

Nov 29 16:51:41 L1Proxmox kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 238
Nov 29 16:51:41 L1Proxmox kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module  450.89  Thu Oct 22 20:49:26 UTC 2020
Nov 29 16:51:42 L1Proxmox nvidia-vgpud[702]: Verbose syslog connection opened
Nov 29 16:51:42 L1Proxmox nvidia-vgpud[702]: Started (702)
Nov 29 16:51:42 L1Proxmox nvidia-vgpud[702]: Global settings:
Nov 29 16:51:42 L1Proxmox nvidia-vgpud[702]: Size: 16
Nov 29 16:51:42 L1Proxmox nvidia-vgpud[702]: Homogeneous vGPUs: 1
Nov 29 16:51:42 L1Proxmox nvidia-vgpud[702]: vGPU types: 401
Nov 29 16:51:42 L1Proxmox nvidia-vgpud[702]:
Nov 29 16:51:42 L1Proxmox kernel: NVRM: GPU at 0000:01:00.0 has software scheduler DISABLED with policy NONE.
Nov 29 16:51:43 L1Proxmox nvidia-vgpud[702]: pciId of gpu [0]: 0:1:0:0
Nov 29 16:51:43 L1Proxmox nvidia-vgpu-mgr[715]: notice: vmiop_env_log: nvidia-vgpu-mgr daemon started
Nov 29 16:51:43 L1Proxmox nvidia-vgpud[702]:
Nov 29 16:51:43 L1Proxmox nvidia-vgpud[702]: Physical GPU:
Nov 29 16:51:43 L1Proxmox nvidia-vgpud[702]: PciID: 0x0000 / 0x0001 / 0x0000 / 0x0000
Nov 29 16:51:43 L1Proxmox nvidia-vgpud[702]: Size: 52
Nov 29 16:51:43 L1Proxmox nvidia-vgpud[702]: DevID: 0x10de / 0x1bb3 / 0x10de / 0x0000
Nov 29 16:51:43 L1Proxmox nvidia-vgpud[702]: Supported vGPUs count: 14
Real GTX 1080 (Same GP104 chip as Tesla P4)

Code: [Select]

Nov 29 10:00:56 DellT3500-PVE kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 239
Nov 29 10:00:56 DellT3500-PVE kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module  450.89  Thu Oct 22 20:49:26 UTC 2020
Nov 29 10:00:58 DellT3500-PVE nvidia-vgpud[753]: Verbose syslog connection opened
Nov 29 10:00:58 DellT3500-PVE nvidia-vgpud[753]: Started (753)
Nov 29 10:00:59 DellT3500-PVE kernel: NVRM: GPU at 0000:02:00.0 has software scheduler DISABLED with policy NONE.
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: Global settings:
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: Size: 16
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: Homogeneous vGPUs: 1
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: vGPU types: 401
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]:
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: pciId of gpu [0]: 0:2:0:0
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpu-mgr[762]: notice: vmiop_env_log: nvidia-vgpu-mgr daemon started
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: GPU not supported by vGPU at PCI Id: 0:2:0:0 DevID: 0x10de / 0x1b80 / 0x10de / 0x0000
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: error: failed to send vGPU configuration info to RM: 6
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: PID file unlocked.
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: PID file closed.
Nov 29 10:00:59 DellT3500-PVE nvidia-vgpud[753]: Shutdown (753)
Nov 29 10:00:59 DellT3500-PVE systemd[1]: nvidia-vgpud.service: Main process exited, code=exited, status=6/NOTCONFIGURED
Nov 29 10:00:59 DellT3500-PVE systemd[1]: nvidia-vgpud.service: Failed with result 'exit-code'.

I hope this helps you all out in your discoveries!

[bayx]

Hey Krutav,
Can you write me a complete solution without
"Error 43" Problem. For me is importand how can i spoof it as QUADRO.
I missed one more time i have no idea why... :/
Cant find working howto in the forums or in youtube... Everytime come the f...* Error 43 failure.

Best regards

[krutav]

Hey Krutav,
Can you write me a complete solution without
"Error 43" Problem. For me is importand how can i spoof it as QUADRO.
I missed one more time i have no idea why... :/
Cant find working howto in the forums or in youtube... Everytime come the f...* Error 43 failure.

Best regards

I tested all this on Linux, which works perfectly. On windows, you need to make sure that the CPU type is set to whatever CPU architecture you have. Because my CPU is Kaby Lake, I set that. I will not set it to host because there is some bug with the nested virtualization that is causing it to break. I actually found a solution to this on the proxmox forum somwhere, and while i dont have the link, it should be something about running Hyper V on proxmox with nested virtualization. I'd like to point out that the spoofing trick is intended for loading alternate drivers for something. It will NOT improve SPECviewperf scores or CAD performance on consumer hardawre. Quadro GPU uses far higher quality chips compared to geforce. I'll send some info later on as it comes up.

[bayx]

It will NOT improve SPECviewperf scores or CAD performance on consumer hardawre. Quadro GPU uses far higher quality chips compared to geforce.

I know. Only Solidworks must see it as prof.card to open some features like Anti-Aliasing, i dont need more performance its enough power what GTX1070 brings.
I buy me a second machine to make experiments. This machine is a workstation i cant make experiments with it. January maybe can i test it.
Is Ryzen/nvidia combo good for this project?

[krutav]

It will NOT improve SPECviewperf scores or CAD performance on consumer hardawre. Quadro GPU uses far higher quality chips compared to geforce.

I know. Only Solidworks must see it as prof.card to open some features like Anti-Aliasing, i dont need more performance its enough power what GTX1070 brings.
I buy me a second machine to make experiments. This machine is a workstation i cant make experiments with it. January maybe can i test it.
Is Ryzen/nvidia combo good for this project?

Get an AMD GPU like RX480. It can be flashed with the vBIOS of a FirePRO and Radeon Pro Instinct GPUs. They are also far cheaper compared to Nvidia, which I wouldn't bother with because of its ridiculous price. I use Solidworks on my GTX 1060 and it works perfectly fine, haven't had any issues with it. Ryzen is great for the project because of the price to performance value.

[bayx]

Any link ? which firepro vbios?

Edit: I find some how to´s but only for RX480 to change it in a RX580 nothing about firepro w7100 or something like that.

[krutav]

Any link ? which firepro vbios?

Edit: I find some how to´s but only for RX480 to change it in a RX580 nothing about firepro w7100 or something like that.

You won't find any, but it shares the same chip as the Radeon instinct MI6. "Polaris 10 XT." This means that you can also flash it with the vBIOS for the Radeon Pro WX7100 which is a pretty good card. All you need is ATIFlash and the vBIOS file from techpowerup website. there is no tutorial for it right now. Here is the link for the vBIOS. https://www.techpowerup.com/vgabios/196721/196721
I remember converting a junk FirePro to a consumer card so I could use a newer driver with it by flashing the vBIOS. Because the RX 480 8GB is so cheap now, this is not a very expensive mod to do, which I could recommend if you want to improve your CAD experience or unlock SR-IOV features.

[bayx]

This is to tricky... Buy me ne hw for 2nd pc (server) Ryzen 3900X + RTX2080ti

[krutav]

This is to tricky... Buy me ne hw for 2nd pc (server) Ryzen 3900X + RTX2080ti

Yes... it is. That's why we have new technologies coming out all the time. Notably, GPU-P on windows allows you to partition any GPU for the VM for absolutely free on Hyper-V. It's far easier compared to all this hardware and software modding because it "just works."

[bayx]

This is totaly new. No thx
My goal is now buy a new gpu (2080ti)
Install ubuntu 20/10
Install kvm
Configure a VM with gpu pathtrough
Spoof it as Quadro RTX4000 or 5000 <--- this point is very importand for me.
I find a cool video there everything explained how to install it on ubuntu.

 https://youtu.be/ID3dlVHDl0c

Only thing is can't find videos for spoofing gpus.

[krutav]

This is totaly new. No thx
My goal is now buy a new gpu (2080ti)
Install ubuntu 20/10
Install kvm
Configure a VM with gpu pathtrough
Spoof it as RTX6000 or 8000 <--- this point is very importand for me.
I find a cool video there everything explained how to install it on ubuntu.

 https://youtu.be/ID3dlVHDl0c

Only thing is can't find videos for spoofing gpus.

You can use the VFIO PCI spoof arguments that I posted earlier. It works flawlessly.

[bayx]

This is totaly new. No thx
My goal is now buy a new gpu (2080ti)
Install ubuntu 20/10
Install kvm
Configure a VM with gpu pathtrough
Spoof it as RTX6000 or 8000 <--- this point is very importand for me.
I find a cool video there everything explained how to install it on ubuntu.

 https://youtu.be/ID3dlVHDl0c

Only thing is can't find videos for spoofing gpus.

You can use the VFIO PCI spoof arguments that I posted earlier. It works flawlessly.

Perfect thxxxx

Last questen is there any "error 43" problems with amd radeon cards?

PS Status : Sold my 2 GTX1070 now looking for 2080ti on Ebay 🤪