Author Topic: [MOVED] Hacking NVidia Cards into their Professional Counterparts  (Read 1652088 times)

0 Members and 1 Guest are viewing this topic.

Offline gordan

  • Frequent Contributor
  • **
  • Posts: 277
  • Country: 00
    • Tech Articles
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #625 on: November 01, 2013, 01:35:56 pm »
Budget isn't an issue (within reason), I'm looking to set up a virtualised gaming rig much like yourself. There'll be a Windows 7 (64-bit ent) VM that'll need as much 3D gaming grunt as possible and a couple of other VMs that need some acceleration to be responsive and usable (possibly 3D too). Naturally I want to go for as much power as possible, I plan to use the machine for current and next gen gaming.

Depends on your intended resolution. If all you want is 1080 resolution capability, GTX480/Q6000 will deliver. Granted, my eyes don't seem to see things the way most people's do (more pixels, not as many frames per second, or so it seems), but I happily completed Crysis+Warhead on a T221 3840x2400 in a VM on my Quadrified GTX480. On the same physical host, my wife was finding Borderlands 2 unplayably bad at 2560x1600 (so I temporarily put a HD7970 in her VM (and yes, it crashed the host when you try to reboot the VM - I'm hoping the ATI pollution is going to be temporary) and kept the 480 for my VM).

If you need more than 2 VMs with 3D acceleration on that motherboard, you are going to have to use something like a GTX690, given it only has 2 PCIe x16 slots.

I thought about converting a GTX 480 into a Q6000 as they're fairly cheap to pick up used on ebay, however I'm not sure which brand would follow reference design to make the modification less of a headache.

They all do / it doesn't matter.

What wasn't made clear earlier was that a Q6000 clone (GTX 470 / 480) can be used as vSGA with 6 guests, that's a great piece of information for those looking to accelerate 3D on multiple VMs on a budget  :-+

Quick googling some of the terms in the article from memoring yields this:
http://www.simonlong.co.uk/blog/2012/10/25/vmware-view-3d-gaming-experience/

Note that this is not exactly high-performance gaming - it may be 6 clients, but they are 800x600@25fps.

I should also point out that I don't use ESXi for this stuff, so you might want to start on the cheap with a modified 470 or 480 (performance difference between them is pretty negligible, and a 470 will likely outperform a Q6000 as long as you don't run out of VRAM) as a proof of concept before you commit to a more expensive piece of kit like a 680/770 or a 690. I should perhaps also point out (hint:nudge) that there is currently a quadrified GTX470 on eBay at the moment. ;)

What would also be handy to know, and again I assume this is probably beyond scope of this thread, would be if multiple Q6000 clones can be added to a system, one passthrough'd to one VM directly for as much acceleration as possible, and a second Q6000 clone distributing as vSGA between remaining VMs? GTX 480s can be picked up second hand for around £100 each on eBay so they'd make a great price vs benefit starting point.

I see no reason why you couldn't use one card for vSGA and one for vDGA. Just bear in mind that you won't be getting video directly out of your vSGA cards - those VMs will feed you a compressed video stream of the desktop that you will have to decode on another machine. The problem with this being that you need another machine as a terminal (unless you use your vDGA machine as a terminal for it, which would work I suppose, but it gets a bit recursive).

But as I said before - I am not an ESXi user, and while I would expect their solution to be a little more polished than Xen, I suspect you will also have a lot less community support to fall back on if it doesn't work out of the box. Also, last time I checked vDGA was treated as an experimental feature.

The GTX 680 I was secretly hoping you would have found a solution by now, but as with all things of this nature, it wouldn't be too easy or everyone would be doing this to their cards :)

It could be that I just have a weird GTX680, or there is some OS/environmental issue that is manifesting as the problem I mentioned. One of the guys on the Xen list modified a completely standard GTX680, and his works fine in all modes, so my DVI issues are most likely just a bizzare quirk of my system configuration.
 

Offline foxdie

  • Newbie
  • Posts: 4
  • Country: gb
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #626 on: November 01, 2013, 02:02:59 pm »
Depends on your intended resolution. If all you want is 1080 resolution capability, GTX480/Q6000 will deliver. Granted, my eyes don't seem to see things the way most people's do (more pixels, not as many frames per second, or so it seems), but I happily completed Crysis+Warhead on a T221 3840x2400 in a VM on my Quadrified GTX480. On the same physical host, my wife was finding Borderlands 2 unplayably bad at 2560x1600 (so I temporarily put a HD7970 in her VM (and yes, it crashed the host when you try to reboot the VM - I'm hoping the ATI pollution is going to be temporary) and kept the 480 for my VM).

If you need more than 2 VMs with 3D acceleration on that motherboard, you are going to have to use something like a GTX690, given it only has 2 PCIe x16 slots.

I'm admittedly not an UHD junkie, 1080p would meet my needs (and avoid having to buy a new monitor). Framerate is however a potential issue, I'd like to be able to play Crysis 3 at 1080p60 with no slowdown.

I should perhaps also point out (hint:nudge) that there is currently a quadrified GTX470 on eBay at the moment. ;)

That's funny, I can't find it, think someone might have snapped it up O0

I see no reason why you couldn't use one card for vSGA and one for vDGA. Just bear in mind that you won't be getting video directly out of your vSGA cards - those VMs will feed you a compressed video stream of the desktop that you will have to decode on another machine. The problem with this being that you need another machine as a terminal (unless you use your vDGA machine as a terminal for it, which would work I suppose, but it gets a bit recursive).

Recursion can be fun! That's probably what I'd do anyway ;)

I have an Asus Radeon HD 6450 bouncing around on my desk anyway so I'll first try passing that through as vDGA and use the GTX 470 as vSGA.

But as I said before - I am not an ESXi user, and while I would expect their solution to be a little more polished than Xen, I suspect you will also have a lot less community support to fall back on if it doesn't work out of the box. Also, last time I checked vDGA was treated as an experimental feature.

True but someone needs to take the plunge to see if this'll work right? I'm in the same boat with the X9SRA, I haven't found any solid documentation this will work but progress isn't made on repetition ;)

It could be that I just have a weird GTX680, or there is some OS/environmental issue that is manifesting as the problem I mentioned. One of the guys on the Xen list modified a completely standard GTX680, and his works fine in all modes, so my DVI issues are most likely just a bizzare quirk of my system configuration.

If you want, I'm happy to test the card in my new system for you, confirm if its working, then post it back in the same condition it was received in. I'm only up the road (in relative terms given a global community, still 120 miles away :-DD) so RM Special Delivery would be fast.
 

Offline gordan

  • Frequent Contributor
  • **
  • Posts: 277
  • Country: 00
    • Tech Articles
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #627 on: November 01, 2013, 03:08:36 pm »
But as I said before - I am not an ESXi user, and while I would expect their solution to be a little more polished than Xen, I suspect you will also have a lot less community support to fall back on if it doesn't work out of the box. Also, last time I checked vDGA was treated as an experimental feature.

True but someone needs to take the plunge to see if this'll work right? I'm in the same boat with the X9SRA, I haven't found any solid documentation this will work but progress isn't made on repetition ;)

That is, indeed, my view as well. If we'd stuck with repetition the fastest way to travel would still involve staring at an ox's backside for excruciatingly prolonged periods of time.

It could be that I just have a weird GTX680, or there is some OS/environmental issue that is manifesting as the problem I mentioned. One of the guys on the Xen list modified a completely standard GTX680, and his works fine in all modes, so my DVI issues are most likely just a bizzare quirk of my system configuration.

If you want, I'm happy to test the card in my new system for you, confirm if its working, then post it back in the same condition it was received in. I'm only up the road (in relative terms given a global community, still 120 miles away :-DD) so RM Special Delivery would be fast.

Interesting thought. Unfortunately, I'm using it at the moment, and, ironically, I'm fresh out of spares I could drop in.
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #628 on: November 02, 2013, 12:29:59 pm »
Servus,

I changed the marked resistor in the pic from 25k to 40k. Now the pci-id is 1025 instead of 1005 for a Titan. The aim is to get 1020 (K20X). I changed already the resistors near by. But it doesn't changed anything. Somebody any ideas?
 

Offline gordan

  • Frequent Contributor
  • **
  • Posts: 277
  • Country: 00
    • Tech Articles
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #629 on: November 02, 2013, 01:25:47 pm »
I changed the marked resistor in the pic from 25k to 40k. Now the pci-id is 1025 instead of 1005 for a Titan. The aim is to get 1020 (K20X). I changed already the resistors near by. But it doesn't changed anything. Somebody any ideas?

Holy crap, you did it!
There is no need to change any further resistors - you can soft-mod the last nibble. See the info on page 41.

* gordan goes to acquire a Titan.
« Last Edit: November 02, 2013, 02:19:07 pm by gordan »
 

Offline foxdie

  • Newbie
  • Posts: 4
  • Country: gb
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #630 on: November 02, 2013, 03:04:45 pm »
Yeah I was just thinking he could soft-mod it, great progress :) Wonder what the potential of the titan is? Extreme virtualised gaming? :D

(Might treat myself for xmas lol)
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #631 on: November 02, 2013, 04:34:13 pm »
Haha, I was JUST going to post that I managed to convert my GTX780 to a Tesla K20 and I saw that johndoe beat me to it :-DD I know I should have posted here before I wrote an article on my site about how I figured it out.

The first 5 bits of the device ID are softstrapped so just change the appropriate bits and reflash your card.

But I can also tell you that modding the GTX TITAN BIOS with the right straps will not turn it into a working Tesla K20. For example, you cannot disable the TCC mode, and you cannot run any CUDA code. :( What you CAN do however is go to TechPowerup and download the only K20 BIOS they have, and change the soft straps on that BIOS ;)

Btw, it would be awesome if you run some CUDA samples to see if everything runs fine (and report back of course ;) ).
« Last Edit: November 02, 2013, 05:03:28 pm by oguz286 »
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #632 on: November 02, 2013, 05:08:38 pm »
Ok, can someone explain me in a little bit more detail how to work with the softstraps, I have some sim software to test cuda.
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #633 on: November 02, 2013, 05:53:35 pm »
Ok, this is untested and I'm doing this from the top of my head, so please check how this works!

First download the K20c BIOS from techpowerup.com (or if you have a Tesla K20X BIOS somewhere, use that). Open up this file with a hex editor (for example HxD). You will see something like this:

00000000: 4E 56 47 49 92 01 10 00 60 04 00 00 DE 10 82 09
00000010: 08 E2 00 00 00 06 00 00 02 10 10 82 FF FF FF 7F
00000020: 00 00 00 80 0E 10 10 82 EF FF FE 73 00 00 00 8C
00000030: 36 10 10 02 FF FF FF FF 00 00 00 00 02 87 08 02

Now you have to change the values at 0x1C (mine contains FFFFFF7F as you can see above) to FFC3FF7F, and make sure the values at 0x20 are 00000080.
Make sure you do this correctly! If you make a mistake then you could brick your card. It should be something like this but beware, I DO NOT TAKE ANY RESPONSIBILITY IF YOU BRICK YOUR CARD.

00000000: 4E 56 47 49 92 01 10 00 60 04 00 00 DE 10 82 09
00000010: 08 E2 00 00 00 06 00 00 02 10 10 82 FF C3 FF 7F
00000020: 00 00 00 80 0E 10 10 82 EF FF FE 73 00 00 00 8C
00000030: 36 10 10 02 FF FF FF FF 00 00 00 00 02 87 08 02
« Last Edit: November 02, 2013, 06:06:47 pm by oguz286 »
 

Offline gordan

  • Frequent Contributor
  • **
  • Posts: 277
  • Country: 00
    • Tech Articles
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #634 on: November 02, 2013, 08:59:06 pm »
Thanks for the diagram for figuring out the hard-straps. It more or less provides a template for figuring out the hard-straps in the future.

It is both interesting and concerning that you had to flash a Tesla BIOS onto the card to make it work.

Please keep us posted on the progress of figuring out the VRAM size.

My Titan should be arriving on Tuesday, so with a bit of luck I'll have a pseudo-K6000 by the end of next week. The use case for a Titan is 2-fold:
1) Extreme virtualized gaming, as foxdie says
2) Number crunching that required double precision floating point - IIRC Titan, unlike the GTX780, doesn't have crippled DP performance.

Now that I mention that - can you check the before/after DP performance on the Teslified 780? Is there an improvement?
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #635 on: November 02, 2013, 09:20:44 pm »
No problem, sharing is caring after all :)

I cannot test the DP performance right now as I have taken the card apart again. When it's put together I'll test it although I'm pretty sure that it will stay the same.

Could you please convert your Titan into a Tesla and check if CUDA samples run correctly? Because I found out that my card does not run all CUDA programs correctly, and I assume that the incorrect VRAM size is the culprit. It is weird as I have a Tesla K20c BIOS, and the official Tesla K20 has 5GB of VRAM, not 6GB. I can also allocate almost 4GB of memory in CUDA even though my card has 3GB.

Since the Titan is the same a Tesla K20X, I would like to buy a Titan if I know that the 'Teslalized' Titan runs CUDA apps correctly.
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #636 on: November 03, 2013, 12:36:45 am »
Haha, I was JUST going to post that I managed to convert my GTX780 to a Tesla K20 and I saw that johndoe beat me to it :-DD I know I should have posted here before I wrote an article on my site about how I figured it out.

The first 5 bits of the device ID are softstrapped so just change the appropriate bits and reflash your card.

But I can also tell you that modding the GTX TITAN BIOS with the right straps will not turn it into a working Tesla K20. For example, you cannot disable the TCC mode, and you cannot run any CUDA code. :( What you CAN do however is go to TechPowerup and download the only K20 BIOS they have, and change the soft straps on that BIOS ;)

Btw, it would be awesome if you run some CUDA samples to see if everything runs fine (and report back of course ;) ).

Ok, first I changed everything according to your hints in the TITAN bios. You are right: nvflash recognize now K20X but it is not possible to use the card as Tesla card. I just wanted to try  ;). Honestly, I don't want to load the K20c bios. Anybody an idea where to get a K20x bios?
------------
Maybe I found something...
« Last Edit: November 03, 2013, 12:52:57 am by johnjoe »
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #637 on: November 03, 2013, 01:08:40 am »
As far as I know, you can simply use the K20c BIOS with the soft straps set correctly, and it should work fine. I've done it as well and it works (kind of, I just need to modify the BIOS to report 3GB of VRAM instead of 6GB like you have).
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #638 on: November 03, 2013, 01:09:20 am »
I found an update file for linux for k20xm (1021). The bios is included but quite small, 128k, but i think try and error. I saved the extracted part and read in with bios tweaker. Checksum ok, parameters ok and bios is read as k20xm, as it should.

how i have to edit the softstraps to get 1021 instead of 1020?
-----------
You mean, I use the bios of k20c but it looks like k20x, right? But bios of k20c is smaller than TITAN bios, problem?
« Last Edit: November 03, 2013, 01:14:22 am by johnjoe »
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #639 on: November 03, 2013, 01:31:06 am »
Yes, if you use the K20c BIOS but change the soft straps like you did for your Titan BIOS, it should get recognized as a K20X.
And it's not a problem if the BIOS is smaller. However, I don't know about the other BIOS you found.
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #640 on: November 03, 2013, 01:44:02 am »
Ok, i updated the bios, but still not possible to install tesla driver, also manually (driver is not for this windows version). strange...
 

Offline mrkrad

  • Contributor
  • Posts: 37
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #641 on: November 03, 2013, 01:48:23 am »
you guys notice the GTX780 is a GK110 chip? !!?!

GeForce GTX 760 192-bit [4]    Unknown    GK104    28    PCIe 3.0 x16    15363072    1152:96:24    823    888    5808    19.8    79    134    GDDR5    192    11    4.4    1.1    1896    79    14.6    130    OEM
GeForce GTX 760 [5]    June 25, 2013    GK104-225-A2    28    PCIe 3.0 x16    20484096    1152:96:32    980    1033    6008    31.4    94.1    192    GDDR5    256    11    4.4    1.1    2258    94.1    13.3    170    $249
GeForce GTX 760 Ti [6]    Unknown    GK104    28    PCIe 3.0 x16    2048    1344:112:32    915    980    6008    29.3    103    192    GDDR5    256    11    4.4    1.1    2460    103    14.5    170    OEM
GeForce GTX 770 [7]    May 30, 2013    GK104    28    PCIe 3.0 x16    20484096   1536:128:32    1046    1085    7008    33.5    134    224    GDDR5    256    11    4.4    1.1    3213    134    14.0    230    $399
GeForce GTX 780 [8]    May 23, 2013    GK110    28    PCIe 3.0 x16    3072    2304:192:48    863    900    6008    41.4    166    288    GDDR5    384    11    4.4    1.1    3977    166    15.9    250    $649
GeForce GTX 780 Ti    Nov 7th, 2013    GK110    28    PCIe 3.0 x16    3072    2880:240:48    876    928    7000    42.048    210.24    336    GDDR5    384    11    4.4    1.1    5045.76    Unknown    Unknown    Unknown    $699 (TBC)
GeForce GTX Titan    February 19, 2013    GK110    28    PCIe 3.0 x16    6144    2688:224:48    837    876    6008    40.2    188    288    GDDR5    384    11    4.4    1.1    4500    1500    18.0    250    $999
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #642 on: November 03, 2013, 01:49:58 am »
And?
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #643 on: November 03, 2013, 01:51:03 am »
Ok, i updated the bios, but still not possible to install tesla driver, also manually (driver is not for this windows version). strange...

Is it the only card in your computer? It was in mine and then the Tesla did not work. When I put in another NVIDIA graphics card as the primary card, then I could install the driver and use the card.

you guys notice the GTX780 is a GK110 chip? !!?!

Did you notice that I modified my GTX780 into a Tesla? :P That wouldn't be possible if it was not a GK110 chip ;)
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #644 on: November 03, 2013, 01:52:36 am »
I'm working with a second card, titan is not connected to screen. very strange...
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #645 on: November 03, 2013, 01:56:02 am »
But is your primary card an NVIDIA card? It does not work if it is an AMD card.

Very strange indeed. You used the K20c BIOS right? For me it works just fine. What does your OS say about the card?
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #646 on: November 03, 2013, 01:57:24 am »
primary card is also nvidia gt9500. Windows reports this card as 3d-game controller.
---------
Did you used the automatic driver installer or manually?
« Last Edit: November 03, 2013, 01:59:52 am by johnjoe »
 

Offline oguz286

  • Contributor
  • Posts: 39
  • Country: nl
  • Make, break, hack, tweak
    • GuzTech
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #647 on: November 03, 2013, 02:02:39 am »
If you look at the photos on my site (http://www.guztech.nl/wordpress/index.php/2013/11/researching-nvidia-gpus-geforce-gtx780-and-gtx-titan-to-tesla-k20-and-tesla-k20x/) you will see that nvflash recognizes my card as a Tesla K20(X), but also the last two numbers are non-zero. nvflash reports four numbers (0x10DE, 0x1020, 0x10DE, 0x104B), and when I was researching the values of the resistors, I once got (0x10DE, 0x1020, 0x0000, 0x0000).

You could check if the last two numbers are correct. I don't know what else could be wrong.

I just downloaded the newest Geforce driver and it installed it automatically for me. Took a couple of minutes after rebooting for the card to be recognized (because Windows was installing the driver).

 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #648 on: November 03, 2013, 02:06:18 am »
strange, i have different sub id: <0> Tesla K20X           (10DE,1020,10DE,0982)
did you changed something more?
 

Offline johnjoe

  • Contributor
  • Posts: 18
Re: [MOVED] Hacking NVidia Cards into their Professional Counterparts
« Reply #649 on: November 03, 2013, 02:57:33 am »
I changed now also sub-id and bios to k20xm but it doesn't changed anything. Nvflash recognize card as k20x but windows as 3d-video controller (yellow marked) and it is impossible to install manually or automatically any tesla driver. If somebody has an idea you are welcome!
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf