How does a PCIe slot work?

[dicky96]

Hi guys
I'm a bit puzzled by something and hopefully someone here can clear the fog.

Attached is a pinout of a PCIe x16 slot

I can see that there are sixteen LVDS lanes, each with two pairs of connections
EXP_A_TXP_0 / EXP_A_TXN_0 to EXP_A_TXP_15 / EXP_A_TXN_15
EXP_A_RXP_0 / EXP_A_RXN_0 to EXP_A_RXP_15 / EXP_A_RXN_15

So given TX means transmit and RX means receive in electronics, and I can google that a LVDS connection is mono directional this tells me each PCIe lane has a transmit and receive pair.  The attached pic is for a PCIe slot on a motherboard so I assume that the TX pair are sending data TO an inserted PCIe card and the RX pair are receiving data FROM the card

All good so far....

Now this is the bit that I can't get my head around.

If I insert a GPU into a mining adapter, there are only four connections going via a short USB type cable from the GPU to the adapter in the PCIe slot.  Plus of course the 12V power cable - the 3.3V is generated by a small buck converter from the 12V input on the mining adapter itself.

So there are only four connections other than power and ground.  I metered them out - I have
EXP_A_TXP_0 and EXP_A_TXN_0
REFCLK- and REFCLK+ which is a 17th pair of connections from the PCIe slot so is in itself another LVDS lane?

From the name REFCLK this sounds to me like a reference clock which suggests that the PCIe LVDS data is synchronous?  But googling for PCIe REFCLK I just can't find any info on what it actually does. Yet the GPU is still detected in the single channel mining adapter so somehow it receives a request from the PC BIOS and returns its ID/parameters. 

But if EXP_A_TXP is mono directional data from the PC and REFCLK is a clock, how on earth does the GPU send data to the PC???

Oh and if you wondered about the signal PCIRST - on my adapter this is just held high via a resistor, so the reset is not used.

Which suggests I am making some basic mistake in my assumptions here 

Help!!! 

[Berni]

The PCIe over a USB cable trick uses USB 3.0 cables.

Those USB 3.0 cables have 2 extra diff pairs inside for the 5Gbit USB RX and TX lines. So this brings the total number of diff pairs available to 3 plus power. So this way you have enugh pairs to run a 1x PCIe slot by using the USB RX for PCIe RX using USB TX for PCIe TX and then using the USB 2.0 data pair for the refclock since that thing is just a slow 100MHz.

Some of Keysight PC based scopes use a similar trick of running PCIe via a SATA cable. However since SATA only has 2 pairs they had to use 2 cables. One carries the RX and TX while the other carries refclk and  some control signals like reset.

[dicky96]

Thanks Berni you are correct.  I can see the three pairs of tracks on the mining adapter when I look under my microscope.

So that explains it.  So just one other thing, out of interest, what does REFCLK do?  Does a PCIe lane actually run at 100 MHz synchronous?

[Berni]

The PCIe device uses the 100MHz clock as a reference for its PLL in order to generate the actual 2.5Gbit 5Gbit 10Git...etc clock used to move the data. This makes it easier on the RX side, since clock recovery is not needed if the other side is sending the data at the exact correct clock speed.

The PCIe bus does not work anything like the old PCI bus where each edge of a clock moves data over a parallel bus. Instead in PCIe each lane is its own independent point to point channel between the device and host. This makes the high speeds easier to deal with because the clock signal doesn't have to make a full round trip for each bit, the speed of signals trough copper traces is too slow for that to even work. So since each lane is independent means that you don't need to length match the diff pairs to each other, intact you can even swap the positive and negative side of the pair and it still works (also makes PCB routing easier). On startup the pairs adjust themselves to whatever clock delay is seen by the RX side and just keep a continuous stream of data going after that. The higher up logic then just splits the data over however many lanes are available.

All of this is the reason why PCIe works over surprisingly long cables even tho it was never designed to go over a cable.

[TC]

When thinking about clocking methods for communication between two devices consider the phase and frequency relationships.

The two most common clocking methods are:

Asynchronous
- Different frequencies
- No phase relationship

Synchronous
- Same frequency
- Same phase

Two other important clocking methods are:

Mesosynchronous
- Same frequency
- Unknown phase relationship

Plesiosynchronous
- Frequencies are not required to be identical, but must be close to each other
- Unknown phase relationship

PCI Express uses mesosynchrous clocking. The reference clock supplies the clock frequency of the system to the device so that it can frequency-lock to the same frequency. There is a delay between the transmitter and the receiver and this results in a significant phase shift at high-speeds. So the receiver uses the reference clock to regenerate the same frequency as the transmitter but using a PLL is able to phase lock to the Rx bit stream and recover the bits.

With appropriate design methods it is possible to design a PCI Express device that will operate with plesiosynchronous clocking but that is harder and uncommon.

[ConKbot]

Just to add for sake of completeness, given others covered most the low level attributes already, PCI-e has a ton of forward/reverse compatibility.

The PCI-E link will work between all generations and widths.
Plug a PCI-E 2.0 video card into a 4.0 motherboard, the motherboard slows down to 2.0 link rates. Plug a 16x device into a 4x slot with an open back, and the link will work. Plug a PCI-E 4.0 graphics card into a 3.0 motherboard, and it operates still.

That's how the mining risers work. Mining isn't intensive on GPU/CPU bandwidth, so a 1x link is sufficient. SI is a bit dodgy on the adapter? It can enumerate at 1x 3.0 lane instead of 1x 4.0 lane. However, just because the link works doesn't mean the device has to work. Using a mining riser and then trying to play a game would most likely end up with garbage performance as it's choked for bandwidth to communicate with the CPU.

[Berni]

Yep PCIe has some excellent backwards compatibility features.

Tho in terms of signal integrity issues the enumeration is not that sure to detect that. But if you do have an excessively long riser cable that doesn't work reliably you can still typically force a PCIe slot to use a older slower standard.