Deadbugging PCIE 3.0 GPU 8GHz

[invalid_syntax]

First time poster, if wrong board, please recommend (RF ?)

I am trying to see the possibility of deadbug style connection to this GPU.

https://i.ebayimg.com/images/g/p3IAAOSwz2plQqT9/s-l1600.png

- Pins are 1.67mm aparts. Soldering doable (* would help is thin wire can be used)
- Total of 32 Differential Pairs ( Or 16 if gpu supports x8 downgrade).
- Connecting to a PCIE Male connector going to motherboard.
- Using 3M Twinax for the cable - https://www.digikey.com/en/products/detail/3m/SL8802-08-20DN5-00/3837412
- Wanted to test it before/while working on the PCB with proper socket.

While i wait for the gpu to arrive or begin testing. I have following two major concerns.

1. Instead of connecting the twinax directly to the pins, is it possible to use a twisted pair, or twisted pair + several gnds thin wire like magnet wire 1-2 CM to a landing pad with many pins from where the Twinax will continue.

2. The twinax does not have a drain wire per differential pair. It has two GND wires at opposite ends of 4 diff pairs. Waiting for the cable to arrive as well. If the shield is not grounded, do i need to ground it or not necessary for diff pair as long as return line is present ?

Edit for additional details.
- Data sheet for cable: https://multimedia.3m.com/mws/media/673519O/3mtm-twin-axial-cable-sl8800-series.pdf
- GPU is SXM2 P100

[inse]

I have no specific knowledge about PCIe but I expect signal skew to be absolutely relevant here.
The proposed cable has solid conductors, they would break off already during soldering.
Good luck on that…

[ajb]

Seems like it would be a LOT less work to design a simple adapter PCB, on top of being more useful.  There's a lot that can go wrong here, not just with signal connections but you also need to somehow get 300W into the thing.  Hard enough to get all of that right on a PCB, let alone hand soldering flying leads onto a 1.6mm pitch grid.

Do you have experience designing things like this? 

[tooki]

First time poster, if wrong board, please recommend (RF ?)

I am trying to see the possibility of deadbug style connection to this GPU.

https://i.ebayimg.com/images/g/p3IAAOSwz2plQqT9/s-l1600.png

- Pins are 1.67mm aparts. Soldering doable (* would help is thin wire can be used)
- Total of 32 Differential Pairs ( Or 16 if gpu supports x8 downgrade).
- Connecting to a PCIE Male connector going to motherboard.
- Using 3M Twinax for the cable - https://www.digikey.com/en/products/detail/3m/SL8802-08-20DN5-00/3837412
- Wanted to test it before/while working on the PCB with proper socket.

While i wait for the gpu to arrive or begin testing. I have following two major concerns.

1. Instead of connecting the twinax directly to the pins, is it possible to use a twisted pair, or twisted pair + several gnds thin wire like magnet wire 1-2 CM to a landing pad with many pins from where the Twinax will continue.

2. The twinax does not have a drain wire per differential pair. It has two GND wires at opposite ends of 4 diff pairs. Waiting for the cable to arrive as well. If the shield is not grounded, do i need to ground it or not necessary for diff pair as long as return line is present ?

Edit for additional details.
- Data sheet for cable: https://multimedia.3m.com/mws/media/673519O/3mtm-twin-axial-cable-sl8800-series.pdf
- GPU is SXM2 P100

This seems like a fool’s errand to me that will just result in a dead GPU and a bunch of wasted cable and time.

Either design your PCB now, or contact this dude who is willing to share his design files for the same thing: https://www.reddit.com/r/nvidia/comments/l7gon4/is_it_possible_to_convert_from_the_sxm2_interface/

[nctnico]

I'd design a test PCB for sure. Using wires is going to end up nowhere; PCIe (and similar high speed signals) really need tight impedance control.

[langwadt]

I'd design a test PCB for sure. Using wires is going to end up nowhere; PCIe (and similar high speed signals) really need tight impedance control.

https://hackaday.com/2023/11/08/getting-pcie-working-on-the-new-pi-5/   

[langwadt]

I have no specific knowledge about PCIe but I expect signal skew to be absolutely relevant here.
The proposed cable has solid conductors, they would break off already during soldering.
Good luck on that…

each lane of PCIe is separate, afair the allowed skew between lanes for PCIe 3.0 is 6ns, that is a lot of wire

[ejeffrey]

I'd design a test PCB for sure. Using wires is going to end up nowhere; PCIe (and similar high speed signals) really need tight impedance control.

That's not the problem really.  High quality  twinax is plenty good for pcie.  Actually implementing it is another thing. Soldering 32 twinax pairs (16 each direction) directly to pins sounds like a nightmare and likely to mess up and destroy it, but the biggest problem is power and bypassing.  Power planes in a PCB provide a low impedance path to the regulators and bypass capacitors.  That's going to be basically impossible to do with dead bug construction.

[invalid_syntax]

> .. slow 8 year old Volta GPU with only 16GB of RAM
My though process was to test with a $40 card with made up heat shink just to get connections validated. After that was planning to move to P100, still an old card but does job running quantized models or small fine tunings.

> The cost of spinning a board with an SXM2 interface conversion function is likely to massively exceed the cost of buying a modern GPU with 2-10x the performance of this card.

At worst, will write it off as learning experience .

Based on research so far (need to compile them and share it as well).
- SXM2 differential Pins are 1:1 with PCIE Lanes. 32 Diff pairs as well as SMBUS, REFCLK and other pcie houskeeping pins.
- Rest of the pins are power pins and NVLINK pins. NVLINK is optional can be left unconnected may be ?.
- For LLM inference, major bottleneck is the GPU memory bandwidth. Can take hit in CPU to GPU bandwidth, once models are loaded, the pcie link is used less. So link downgrade might be acceptible.

For PCB.
- Aiming as frugal overall solution as possible.
- The PCB will be bare bone SXM socket pins to PCIE male pins only.
- In addition to data traces, For power, either PCIE GPU Header or plain 12V pins.
- Looking at some of the PCIE extension cable's dimensions, 100x100 board should do it. PCBWAY or JLPCB's cheap 4 Layer should cover it.
- Biggest hurdle/cost sink is the BGA socket with 400 pins. Out of two, only one is required if not using NVLINK.
- My first experiment is going to be checking whether a PCB cutout can be used in place of socket. Basically a BGA pad placed on top of supporting structure that has screws to mount. And either soldering pins or high density headers.  This is need to research more.

[invalid_syntax]

That's not the problem really.  High quality  twinax is plenty good for pcie.  Actually implementing it is another thing. Soldering 32 twinax pairs (16 each direction) directly to pins sounds like a nightmare and likely to mess up and destroy it, but the biggest problem is power and bypassing.  Power planes in a PCB provide a low impedance path to the regulators and bypass capacitors.  That's going to be basically impossible to do with dead bug construction.

Looking at the on-board power controller. I was hoping, pins would be crude power source which gets properly distributed by onboard power controller.
This is the reference design i am following.
http://files.opencompute.org/oc/public.php?service=files&t=1b45195dfc4f7a8f204a246fb41f77b9

I think cadence files. I dont have access to it, so using gerbers and kicad. Files are large, takes long time to load.

[quince]

Ran out of steam, eh?

[Miyuki]

PCIe can work just fine over this crazy cable, probably the same as was used for IDE drives. Chinese shops are full of those and surprisingly they work just fine.

[amyk]

I suspect those ribbon cables are just barely adequate, but shielded ones perform much better. Here's an interesting video showing just how tolerant of delays PCIe is: