PXIe instrument in a regular PC

[Marsupilami]

I started a thread in the RF section because the device I'm interested in is an RF test instrument.
https://www.eevblog.com/forum/rf-microwave/ni-vector-signal-transceiver/

The actual realization more belongs here I believe so I'm posting this for folks who care.

Here's where I'm at right now.
-Electronics is pretty much done.
-I'm trying to figure out if I can get a sheet metal cage going (probably sendcutsend.com)

Depending on my mood I'm certain it's going to work easy-peasy, but on lonely cold nights, while contemplating about the meaning of life and the universe and stuff I'm not sure if
-the PCIe extension cable won't be too long
-there is some sort of software check for the driver to check the chassis parameters (I know there is some on newer models)
-cooling will be adequate or efficient enough (even with modifications to the PC drive cage)

The PXIe standard is free, but it heavily references the CompactPCI Express standard which is not and I didn't feel like paying $750 for it, so I'm trying to piece this together from random internet resources.

The cage sketch around the device on the first picture has the dimensions of a double height 5.25" drive bay. I was thinking getting a refurbished Dell Optiplex for ~$150 to put it in.

[coromonadalix]

something like this ? 

https://www.sundancedsp.com/2562-2/smt580-pcie-to-pxie-adapter-card/

385 pound  i think ?  and yes  darn expensive  for that board


Is it only  pcb or cable lanes   possible / direct interfacing without dedicated ic's ??    really intrigued  by your project


https://www.scientific.net/AMR.718-720.1262

[Marsupilami]

Yes, totally like that. I haven't seen this one, so I'm glad as it serves a proof for the validity of the electrical concept.
I just want to make mine somewhat better integrated mechanically.

I was struggling with fitting the whole thing into double 5.25" bay and can't find a good solution.

I was looking at PC cases or old refurb PCs that have 3x 5.25" bays, but they are much rarer, and accessibility is a major point here.

Another idea that I might implement is to ditch the direct PCIe cable and use a Mini SAS HD adapter and cable. I'd have to redo layout completely on the board, but it'd fit a lot better and the cables are cheap. Even with the adapter it's below $50, while the 3M PCIe cable I was looking at before cost almost $100.

https://a.co/d/7kzoLJd
https://a.co/d/4mQAcyV
https://mou.sr/3IM0VR1

My only concern is signal integrity. I don't have a good feel for how much margin I have here. The PXIe-5644, that is the target device for me is Gen 1 PCIe and I found some contradicting info on how does it behave when cabled.

[Berni]

You can push PCIe pretty far in terms of signal integrity.

I ran PCIe trough some ribbon cable and a bunch of bodge wiring at one point (Agilent PC based scope with a proprietary pinout), worked perfectly fine.

I also seen some youtuber chain together a bunch of PCIe extenders to see how far it would work. The huge chain ended up something like 2 meters long before it stopped working.

[coromonadalix]

I'm no pci / pcie specialist, but i think you have differential pairs on some signals    it normally would not be a problem

[samofab]

How about this? Create an adapter for FFC cable to motherboard then mate this cable to connector on the PXIe end (as opposed to mating it to adapter). this should be much more compact on the PXIe side, perhaps enought for original plan of fitting into 2x 5.25" drive bay?

[coromonadalix]

problem would be power lines ??

[Marsupilami]

How about this? Create an adapter for FFC cable to motherboard then mate this cable to connector on the PXIe end (as opposed to mating it to adapter). this should be much more compact on the PXIe side, perhaps enought for original plan of fitting into 2x 5.25" drive bay?

I'm going for something similar I just want to choose something that's at least semi-available in the consumer space.
I like the MiniSAS HD, except it has a really irritating pinout when used for PCIe. The lanes are organized in a square pattern and if I break that out it becomes 0-1-3-2 it annoys the f out of me.

But going down this rabbit hole at least saved me a board spin, because I realized that in CompactPCI Express and thus PXI the peripheral lane directions are swapped on the backplane. (the system slot's TX pins go to peripheral slot's RX pins) I was doing TX to TX as that's how it's referenced in regular PCIe. So I totally had that messed up.

[Marsupilami]

problem would be power lines ??

I have to power it separately anyway. The full SATA power connector has all 3 voltages the PXIe module needs (3.3V/5V/12V) and I have terminals for 3 sets to make sure I can pull enough current through.

[Marsupilami]

Who wants to bet if I'm going to mess this one up too?

[jhenderson0107]

Here https://www.pxisa.org/pxi5_pxiexpresshw_r11/#dearflip-df_rand225093977/11/ is a link to the PXI Express standards document.  Be sure to incorporate all of the control signals so that all operational corner-cases are covered.  For example when rebooting the host or powering the target chassis after the host has already booted, the target cards must re-enumerate.  This process relies on sideband signals. 

[Marsupilami]

Luckily, at least if I understand correctly, hot swap is not a required feature, and that significantly simplifies all the control signals.
The PXIe spec is not much help though because for anything covered there it references the CompactPCI Express spec, which is unfortunately not free.
I'm not going to have any of the PXI specific signals either, e.g. triggers, but I'm only intending to connect one module anyway.

[jhenderson0107]

If you omit hot-swap, then the slave chassis must be powered prior to the host so that the targets are available during PCI bus enumeration.  You'll still need the ref clk, perst* and a few other critical signals, similar to the excerpt below (which is XMC, not PXIe). 

[Marsupilami]

There is no host chassis, it's all one computer. There is no PCIe switch or bridge in the way that has to init before the host does.
I have the reflclk, reset, present ,wake and so on.
I even added a spot for an XO to generate the 100MHz PXIe clock as I've seen a module before that refused to start up without that. I doubt it's needed though.

This is where I'm at right now:


Added a bunch of SATA plugs (P10,P11,P12) to use the 15 pin power connector from the ATX PSU. I couldn't find any that's 15 pin.
I'll keep the wire terminals too for flexibility. (P5,P6,P7)

Amazon meanwhile bough me this. I ordered it ahead to be able to check the pinout. It's definitely an easier form factor than the twinax ribbon cables and the card and the cable was less than $25 combined.


Routing the diff pairs isn't as bad from the SAS connector as I though it would be. There is one awkward loop, but nothing too bad. Still have to do length matching within pairs.

[Hydron]

RE the worries about cable length - if you use a good cable then for any reasonable length the losses in it will be WAY lower than what you get from the short traces running over the motherboard/card PCBs. Spend your time worrying about connectors and adapter PCBs, not about a half meter of (appropriate) PCIe cable. The SAS stuff should be a good choice I think.

[coromonadalix]

Does it need  signal traces the same length / balancing   from one connector to the other ?

[Marsupilami]

Spend your time worrying about connectors and adapter PCBs, not about a half meter of (appropriate) PCIe cable. The SAS stuff should be a good choice I think.

Thanks, that makes me feel good about it. I'm trying to impedance match the traces well and they will be only about 1.5in long max.
My only concern is about the stubs presented by the connector pins. On the ADF connector (CPCIe) the pins are longer than the board width, so the worst case is for signals that connect to the pin on the primary side of the board. I found a presentation detailing the use of "boomerang vias" but it was referring to Gen 4 16GT/s PCIe and for me the target device is only Gen 1 2.5G, so I'm hoping I can get away with it.
I could simulate it, but I'm not sure it's worth the effort and I don't really have an alternative anyway.

[Marsupilami]

Does it need  signal traces the same length / balancing   from one connector to the other ?

The PCIe spec says that the lane-to-lane output skew should be less than 1300ps, which is about 8 inches of trace, so even with the unknown parts of the signal chain I'm certain it's a no issue. The two conductors of each differential pair has to be matched pretty close, I try to get them within a few mils. It's not done yet on the screenshot I posted, but will add little wiggles to even them out where needed.

[Berni]

Yep PCIe actually synchronizes itself on a pair by pair basis, so the length matching between pairs is not really required (as long as you don't go out of the synchronization range)

You can even swap the P and N wire in a pair on PCIe randomly and it will still work due to the encoding used. This lets you route the PCIe traces around without having to use extra vias to swap pairs around.

Making sure you are getting full PCIe speed can also be tricky since even with broken pairs PCIe will still work, it just falls back to a more narrow bus.

[Hydron]

Yeah try very hard to match within a pair, but you're free to totally ignore matching between pairs or the P/N polarity as noted. I try and not leave the long stub from the pin by being careful about what side I route on, but I've also seen it work at gen4 speeds even with a connector stub. You'll almost certainly get away with it at gen1.

[Marsupilami]

Making sure you are getting full PCIe speed can also be tricky since even with broken pairs PCIe will still work, it just falls back to a more narrow bus.

On that note do you guys know of any software that can get diagnostic info like that?

[switchabl]

On Linux, I think lspci -vv gives a lot of diagnostic info.

[Hydron]

On Linux, I think lspci -vv gives a lot of diagnostic info.

Yep this is what you need (run as root), a warning though it's VERY spammy. Look for the LnkCap and LnkSta lines for the device in question, these show the card capabilities and then the actual speed/width. Much easier to get this sort of diagnostics info in linux than windows btw.

[Marsupilami]

More progress yaaay.
I'm getting close.

I decided to use proper subrack hardware to support the board. I will have to cut the rails, but I can live with that.

Cost estimate so far:
-Sheet metal top, bottom, front bezel and rear cover from SendCutSend ~$130
  (including bending, countersinking and threading the top and bottom and anodizing the front)
  (it would be much cheaper with quantity, and structurally neither the front nor the rear panel is needed, but it looks nice and won't short)
-Subrack hw - 1pc 19" horizontal rail with threaded insert, 10pc card guides ~$20
-PCB ~120x60mm 6 layer, JLBCB more expensive stackup to meet the trace impedance 10pcs. ~$140
-Press fit connectors - MiniSAS, CompactPCI 4pcs - $75
-Other PCB parts + screws ~$15
-------------------------------------------
$380 for the prototype (with PCBs to spare)

I was planning to put fan(s) in the cage. A 70x70x10mm would fit fine. I decided to ditch that though, as the PC case this would go into needs modification anyway to allow airflow through the drive bay and if I have to do that then fitting a much larger fan (120mm) is not a lot more hassle, promising better cooling.





Thanks for the PCIe diagnostics tip. I'm not too much of a linux person, but I can get a live image running or something I guess to check if I can't find a good Windows alternative.

[Marsupilami]

Orders placed.

Here's how much it is so far in case anyone interested: