Some thoughts on PCI on 3.3V FPGAs

[Forty-Bot]

The other thread about PCI reminded me of my recent thoughts on this subject. I'm going to write some of them down before I forget.

PCI has some nice features. It's slow enough that you don't need serdes or other fast I/Os, so it can be used on cheap/slow/FOSS FPGAs. You can use it on a regular PC, no need to add a soft-cpu to control things. And there's a ton of old motherboards with PCI slots. Unfortunately, there are some problems

• First, while there's a 3.3 V version of the bus, almost no motherboards implement it. So you're stuck using 5V. This means either using a 5 V FPGA (which don't get made any more and have outdated tooling) or level shifting. The main problem with level shifters is that the delay will affect your timing. You have 11 ns from when the clock goes high to when your output has to be valid, and 7 ns before the clock goes high when inputs are valid. Depending on your FPGA, you will lose several ns just waiting for your signals to get off-chip (or on-chip). This can be offset a little using PLLs, but then you start to eat into your input setup time. So ideally, you want very fast level shifters, preferably with sub-5 ns propagation delays. Fortunately, even faster level shifters exist. "FET Bus switches" like the CBT or CB3T families (and several others from different vendors) connect the in/outs directly to the source/drain of a FET, with the gate held at a constant voltage. By keeping the gate at 4.3 V or so, you can prevent either side from ever getting above 3.3 V, which is a logic high for 5 V TTL. These level shifters typically have propagation delays on the level of 250 ps, due to RC delay, which is effectively instant. We end up with around 10ns to decide what to do.
  
• Unfortunately, there's another problem with level shifting. You can have a maximum of 10 pF on any input pin. Around half of that is going to be eaten up by your FPGA. Unfortunately, those bus switches from earlier have a capacitance of 10-15 pF when on (aka functioning), which eats up the other three halves. On top of this, reflections on the bus can cause signals to spike briefly to 11 V or -5.5 V, requiring diodes for protection (the level shifters will typically have clamping diodes, but you need something more sturdy). You can find diodes with suitably low capacitance, like the 1PS302 or BAT63-07W, but they don't usually come in bulk packages, so you'd have to have one per signal. Fortunately, the capacitance is per-slot, so you can always use a multi-slot motherboard and leave the other slot empty.
• The last electrical problem is R_ON. Although not explicitly spelled out in the spec, you'll want to have reasonable R_ON in order to meet the driver requirements, especially since bus switches provide no amplification. Unfortunately, R_ON depends on V_GS, which is capped by the need to perform level shifting. The best I found was around 8 Ohms (e.g. TC7MPB9307FK).
• There are also some trace length limits (<1.5" for signals, and =2.5" for the clock). Presumably these are specified to ensure proper timing, but I'm not sure how everything interacts.
  
• In terms of timing, while you can take a few cycles for address decoding, and you can wait up to 8 cycles between transfers (and up to 16 on the first cycle), you still have to respond to inputs on the previous cycle for IRDY/TRDY handshaking. This isn't terribly onerous, but you'll certainly have better throughput if you can respond faster.

So you can do everything with level-shifting, but it won't completely be in-spec (due to capacitance). What are some other options?

• You can add a PCI-to-PCI bridge, or better yet a PCIe-to-PCI bridge. This lets you work at 3.3 V and skip the level shifter and protection diodes. This is probably the most practical way.
• You can use a PCIe PIPE phy, like the XIO1100. This implements around 2/3 of the PCIe physical layer, so you'd need to do the other 1/3 of the physical layer, the data link layer, and the transaction layer handling on the FPGA. There are some projects implementing the transaction layer, but I couldn't find anything doing the data link layer. PIPE is a 16 bit bus running at 125 MHz, which is much faster than PCI, but not much faster than the 10 ns (aka 100 MHz) you'd need to work at anyway. There aren't any back-to-back responses, and the XIO1100 will align the clock to the center of the data, so it's not all that bad.

[DiTBho]

while there's a 3.3 V version of the bus, almost no motherboards implement it. So you're stuck using 5V

on my HP RISC workstations

Code: [Select]

PCI-32/66 66 MHz 32-bit 266 MB/s 3.3 V
PCI-64/33 33 MHz 64-bit 266 MB/s 3.3 V/5 V
PCI-64/66 66 MHz 64-bit 533 MB/s 3.3 V
PCI-X 100 MHz 64-bit 800 MB/s 3.3 V
my MIPS router has miniPCI, 32-bit 3.3V data @33Mhz.
PCI-64 3.3V is perfect: it can be downgraded to 32-bit 

[DiTBho]

The other thread about PCI

my topic 

[berke]

You can add a PCI-to-PCI bridge, or better yet a PCIe-to-PCI bridge. This lets you work at 3.3 V and skip the level shifter and protection diodes. This is probably the most practical way.

That's quite interesting, do you have a link to a PCIe-to-PCI adapter that works at 3.3V?  It would be surprising if those are readily available if 3.3V PCI boards were a very tiny minority.

But it would be a very neat solution for all the cases where you want to pipe a data stream from a cheap/FOSS FPGA to something with enough RAM and with good reliability.  The signals are also slow enough that you have a chance at scoping or analyzing them without having to sell any kidneys.

[Forty-Bot]

while there's a 3.3 V version of the bus, almost no motherboards implement it. So you're stuck using 5V

on my HP RISC workstations

Code: [Select]

PCI-32/66 66 MHz 32-bit 266 MB/s 3.3 V
PCI-64/33 33 MHz 64-bit 266 MB/s 3.3 V/5 V
PCI-64/66 66 MHz 64-bit 533 MB/s 3.3 V
PCI-X 100 MHz 64-bit 800 MB/s 3.3 V
my MIPS router has miniPCI, 32-bit 3.3V data @33Mhz.
PCI-64 3.3V is perfect: it can be downgraded to 32-bit 

Yeah, that's certainly the way to go if you have it. But it's much less common on PCs, as they all want to support old 5V-only cards. For example, a quick search shows no motherboards on the first page with 3.3V PCI slots.

That's quite interesting, do you have a link to a PCIe-to-PCI adapter that works at 3.3V?  It would be surprising if those are readily available if 3.3V PCI boards were a very tiny minority.

Most bridges will have a dedicated input rail for the I/O supply, so it's pretty easy to select 3.3V or 5V as you please. For example, PEX8311-AA66BCF, PI7C9X111SL, and XIO2001 all support 3.3V PCI busses.

[dolbeau]

If you want to create a device for a PCI host (using a PCIe device), don't you need a PCI-to-PCIe bridge rather than PCIe-to-PCI? Some PCIe-to-PCI are 'reversible' like the PI7C9X110, but I don't think all of them are? Also the number of available SKUs is dwindling :-( For PCI-to-PCI, Mouser pretty much only has the PCI2250P left.

I can confirm CB3T works fine for at least some vintage bus - I use them (after some help from this very forum!) to put an Artix-7 (so 3.3V) on SBus (5V), Sun's expansion bus they moved to PCI. Works fine, but SBus has a capacitance limit of 20pF per signal (more for some) and is up to 25 MHz so it's got a bit more margin than PCI.

Though with this talk of reflections and spike (I'm a SW guy...), you now got me worried about long-term effects, I don't have any clamping on the bus - just the CB3T standing between the bus & the FPGA...

[berke]

If you want to create a device for a PCI host (using a PCIe device), don't you need a PCI-to-PCIe bridge rather than PCIe-to-PCI?

That's the thing, with a bridge you won't need a PCI host, you can use a modern device.

[dolbeau]

on my HP RISC workstations

Out of curiosity, which one? I recently 'inherited' a C240 from a friend, I investigated the GSC slots (publicly available information: none) but forgot to check the voltage on the PCI slots - you just made me realize it had 3V3-only slots (for the 64b/66MHz)! So in theory, they could interface directly with an Artix-7. Might be a solution to add some devices to the C240 like the SBusFPGA does to my SPARCstation 20.

Annoyingly, PowerMac G4 with 64b/66MHz slots (on any other) are 5V-only, so they won't take any 3V3-only card, which most PCI-X cards are (I have never seen a 5V SAS controller, for instance). I discovered that when I wanted to try a PCI-X LSI1064 device flashed with a Sun F-ROM to see if it would boot the G4. A PCIe LSI1068e flashed the same way did boot a PCIe G5 (I don't have a PCI-X G5 to test the theory further).

Incidentally and more on-topic, the 'bridge' solution is almost as old as PCI - early PCI Sun's (like the Ultra 5 or Ultra 10) had 5V-only PCI bus, but later ones like the Blade 100 were 3.3V-only, with a PCI-PCI bridge to create backward-compatible 5V slots.

And ultimately the real issue with a new PCI device will turn out to be how to support the thing in firmware and software...

[DiTBho]

C3600 

[asmi]

Out of curiosity, which one? I recently 'inherited' a C240 from a friend, I investigated the GSC slots (publicly available information: none) but forgot to check the voltage on the PCI slots - you just made me realize it had 3V3-only slots (for the 64b/66MHz)! So in theory, they could interface directly with an Artix-7. Might be a solution to add some devices to the C240 like the SBusFPGA does to my SPARCstation 20.

If you have Artix, you might as well simply implement a PCI Express, it's going to be much easier because that FPGA contains both transceivers and PCIE block, so all you need to do is to process packets, the rest is taken care of by the hardware. But if all you want is just to map a part (or whole) of device's memory into the host's address space and have some register interface in a PCI config space, there are free IPs from Xilinx which do just that.

And ultimately the real issue with a new PCI device will turn out to be how to support the thing in firmware and software...

That's actually the easier part, because PCI/PCIE devices typically look like any other memory-mapped device to a CPU once the BIOS/UEFI has done it's job and performed the mapping. Of course if your device is complex, so will be your driver, but in general I think complexity of PCI/PCIE is a bit overstated in many cases when it comes to mainsteam OSes like Linux and Windows, because they come with the code to handle a lot of low-level details for you.

[dolbeau]

If you have Artix, you might as well simply implement a PCI Express

But if the host only has PCI (C-class PA-RISC workstation like my C240 or DiTBho's C3600 are from the end of the 90's), then either you implement PCI or you need a PCI-to-PCIe bridge to the Artix. And DiTBho's original thread is about the MIPS Atlas, an old devboard for MIPS cores - also only PCI, but apparently with 3.3V support.

Of course is the FPGA itself was to be the primary host, then PCIe is the way to go on Artix. In fact I've been thinking about implementing a SBus <-> PCIe bridge, but that would require a PCIe root complex rather than an endpoint in the Artix. Xilinx' IP is an option, but open-source would be better; Litex already has PCIe endpoint, unfortunately not (yet?) the root complex. Also the Xilinx IP have driver in Linux, but not in NetBSD that I know of (lots of old hardware is not/no longer supported by Linux but still by NetBSD).

(PCI-not-express is likely of no interest to anyone not involved with vintage systems anymore. PCIe is the way to go in the modern world, but this thread started specifically about PCI-not-express, such as the 5V stuff).

That's actually the easier part, because PCI/PCIE devices typically look like any other memory-mapped device to a CPU once the BIOS/UEFI has done it's job and performed the mapping.

You do need some support in the firmware - in particular is they are required for booting like non-volatile storage or networking - which may not be BIOS/UEFI. That's not always easy. On my SPARC systems it's OpenBoot (later to become OpenFirmware, also available on a lot of PowerPC systems) using Forth. For PA-RISC workstation, no idea what is needed. And if the devices are not already supported in the OS (in which case you need just the bus glue, done that for USB OHCI on SBus), you also need some drivers there. In my experience, getting all those bits together is harder than the actual hardware, as a lot of the knowledge and expertise to program in obsolete environments is gone (again - for PCI-not-express and older busses).

C3600 

Nice toy, lucky you :-)

[asmi]

Of course is the FPGA itself was to be the primary host, then PCIe is the way to go on Artix. In fact I've been thinking about implementing a SBus <-> PCIe bridge, but that would require a PCIe root complex rather than an endpoint in the Artix.

PCIE block in Artix can be configured as a root complex as well as an endpoint.

Xilinx' IP is an option, but open-source would be better;

It's a hardware block, so it can't be open source unless you don't mind wasting a bunch of FPGA resources to reimplement functionality of that hardware.

[nctnico]

PCI works just fine with 3.3V signalling. 20+ years ago I have designed PCI interface cards using 3.3V FPGAs (IIRC Xilinx Spartan-II and Virtex 2). These have been produced in large quantities and have been used in many different PCs. These cards where keyed to fit both 3.3V and 5V slots. These FPGAs are 5V tolerant though.

[Forty-Bot]

If you want to create a device for a PCI host (using a PCIe device), don't you need a PCI-to-PCIe bridge rather than PCIe-to-PCI? Some PCIe-to-PCI are 'reversible' like the PI7C9X110, but I don't think all of them are? Also the number of available SKUs is dwindling :-( For PCI-to-PCI, Mouser pretty much only has the PCI2250P left.

Well, the whole point is to get away with using an FPGA that is too slow to have a serdes capable of PCIe.

I can confirm CB3T works fine for at least some vintage bus - I use them (after some help from this very forum!) to put an Artix-7 (so 3.3V) on SBus (5V), Sun's expansion bus they moved to PCI. Works fine, but SBus has a capacitance limit of 20pF per signal (more for some) and is up to 25 MHz so it's got a bit more margin than PCI.

It was your post on these guys that inspired me to look into them ))

Though with this talk of reflections and spike (I'm a SW guy...), you now got me worried about long-term effects, I don't have any clamping on the bus - just the CB3T standing between the bus & the FPGA...

The CB3T has internal clamping diodes, they just aren't spec'd for the PCI worst case. So if anything fails, it should be your level shifter and not your FPGA. That said, the PCI specs are definitely worst-case, and probably will never happen in the real world (especially if using newer PCI devices).

PCI works just fine with 3.3V signalling. 20+ years ago I have designed PCI interface cards using 3.3V FPGAs (IIRC Xilinx Spartan-II and Virtex 2). These have been produced in large quantities and have been used in many different PCs. These cards where keyed to fit both 3.3V and 5V slots. These FPGAs are 5V tolerant though.

Yeah, it's much easier to find 3.3V cards vs 3.3V hosts. The problem with using 5V-tolerant FPGAs of course is that just like PCI they are a dying breed.

You do need some support in the firmware - in particular is [sic] they are required for booting like non-volatile storage or networking - which may not be BIOS/UEFI.

Booting from a PCI device is a while level beyond what's necessary just to get something working.

[DiTBho]

PCI-X should be 3.3V-only.

Perhaps useful if you can slow down the clock to 33Mhz 

[dmendesf]

Two other avenues for work:

WCH has some generic glue pcie bridges (o mean pcie to some simple synchonous bus)

http://www.wch-ic.com/products/category/2.html#data

There are some ready made pcie-to-pci bus converters. As there's no 5V in the pcie slot i presume It gets 5V from the Power cable. Maybe you can hook It to 3V3 and be happy:


https://a.aliexpress.com/_m0zLdPs

[dolbeau]

PCIE block in Artix can be configured as a root complex as well as an endpoint.
(...)
It's a hardware block, so it can't be open source unless you don't mind wasting a bunch of FPGA resources to reimplement functionality of that hardware.

Thanks for the comments - they led me to investigate how Litex does it. It turns out for 7-series (and some others presumably) they just create a nice wrapper around the hard block. So it should be possible to do the same for root complex mode, and benefits from Litex ease-of-use & Xilinx' IP...

[Forty-Bot]

Two other avenues for work:

WCH has some generic glue pcie bridges (o mean pcie to some simple synchonous bus)

http://www.wch-ic.com/products/category/2.html#data

Ah, these are interesting. CH368 works out to 44 MiB/s (32 bits @ 11 MHz), and it's done all the bus decoding already. It says it supports DMA, but there's no mention of how it's implemented. A couple people have also wondered about this. I had a look at their driver, but even though they enable pci_master, they don't seem to use it.

While looking around, I also found this thread, which talks about PLX PCI9303/PCI9054 chips. These seem great, but they are quite expensive now that they are no longer produced.

[DiTBho]

PCI to ISA, but then it's 5V 

what about Cardbus? PCMCIA?

[DiTBho]

What about AGP?
AGP is a heavily modified PCI 3.3V slot.

Some UNIX workstations (e.g. HP C8K) have AGPs as well as some x86-PCs.

[DiTBho]

anyone?