Interfacing a high-speed DAC to an FPGA

[unikeyname]

Thanks, this actually inspired me to overclock that chip, I have some laying around just doing nothing there..

Anywhere to look deeper at this? hopefully a PCB which specifically pushing that to the extreme.

[booscrawl]

Thanks, this actually inspired me to overclock that chip, I have some laying around just doing nothing there..

Anywhere to look deeper at this? hopefully a PCB which specifically pushing that to the extreme.

Are you talking about RP2350? You don't need any special PCB, a Pico 2 can be overclocked to even 400 MHz without any hardware changes.

There is this: https://github.com/Altaflux/gb-rp2350

[hamster_nz]

i'm working on a 1GS/s 12-bit dual channel DAC system, for a PoC.

DAC CLK is driven by FPGA with the clock forwarded using the same SERDES setup as the data lines, (1:4 serialization, 250MHz fabric clock), but just with a fixed "0101" data pattern.

The clock output has an ODELAY to the clock edges to be adjusted relative to the data edges. We have not performed the full calibration of the delay primitive, but just established the usable valid data window on the test board, and centered the delay tap value.

It would be possible to calibrate the interface better, with additional per-pin delays to adjust for skew, but not worth the time at the moment - it works well enough for now.

[asmi]

DAC CLK is driven by FPGA with the clock forwarded using the same SERDES setup as the data lines, (1:4 serialization, 250MHz fabric clock), but just with a fixed "0101" data pattern.

This is how AMD/Xilinx recommends forwarding clock. Vivado recognizes this pattern as a forwarded clock if you run a clock setup wizard. That is a parallel LVSD bus, right?

The clock output has an ODELAY to the clock edges to be adjusted relative to the data edges. We have not performed the full calibration of the delay primitive, but just established the usable valid data window on the test board, and centered the delay tap value.

Why do you need ODELAY for clock? Is that a center-aligned clock? The "orthodox" solution for that is to use MCMM to create a phase-shifted clock. Advantage of that is that you can actually change that phase shift at run time if needed. But that usually required when layout was sloppy with poor/no length matching.

It would be possible to calibrate the interface better, with additional per-pin delays to adjust for skew, but not worth the time at the moment - it works well enough for now.

Typically it's receiver's job to do a deskew like this. But in practice it's only required when lanes were not properly length-matched.

[hamster_nz]

This is how AMD/Xilinx recommends forwarding clock. Vivado recognizes this pattern as a forwarded clock if you run a clock setup wizard. That is a parallel LVSD bus, right?

Yes

Why do you need ODELAY for clock? Is that a center-aligned clock? The "orthodox" solution for that is to use MCMM to create a phase-shifted clock. Advantage of that is that you can actually change that phase shift at run time if needed. But that usually required when layout was sloppy with poor/no length matching.

It is a center-aligned clock, and we are working at the device's top performance characteristic. We are doing it this way for consistency in the clocking of the SERDES blocks.

A (calibrated) output delay offers greater consistency.

[jaxon]

Try using OSERDES + IODELAY to fine-tune clock/data phase, or use differential clock routing with matched length traces.

[radiolistener]

I need a clean, low-noise output, so I want to keep clock jitter to a minimum. My current plan is to use a high-quality clock source and feed it directly to the DAC, with a copy going to the FPGA for data output clocking. The obvious problem with this approach is that unless I’m really lucky, the data won’t arrive at the DAC with the correct phase relationship to the clock.

What’s the usual technique for handling this? I’ve gone through both datasheets and haven’t found much guidance on how to meet the timing requirements. Ideally, I’d need something that works over the full temperature range of the chip.

Your approach of driving the DAC directly from a low-jitter clock source is the right one. The FPGA's internal clock generation (PLL) typically adds much more jitter than you want on the DAC sampling clock, so it's generally better to keep the DAC on the clean reference clock and use the FPGA only to synchronize its output to that clock.

I'd also pay attention to clock isolation. It's good practice to use proper clock buffers so that switching noise from the FPGA doesn't couple back into the DAC clock.

As for the clock/data phase relationship, that's usually not the hard part. Most modern FPGAs let you adjust the timing of their output pins (phase shifting, etc.), so it's normally straightforward to center the data eye at the DAC inputs during bring-up.

The bigger challenge is matching the delays of the individual data lines. If different bits arrive at noticeably different times, that can become much harder to fix than simply shifting all the data relative to the clock. Good PCB layout with matched trace lengths is well worth the effort.

[Someone]

The bigger challenge is matching the delays of the individual data lines. If different bits arrive at noticeably different times, that can become much harder to fix than simply shifting all the data relative to the clock. Good PCB layout with matched trace lengths is well worth the effort.

Are people trying to be like AI and just say things without calculating the basis behind them?

A high-speed DAC should do the job nicely. The AD9707 (175 MSPS) seems like a good fit for the task. I’m also considering the MAX5890 (600 MSPS) for more headroom with the sample rate. I’m leaning towards a Spartan 7 as the FPGA, though I’m open to other suggestions.

Do tell us the trace skew that you think matters in this instance and then compare that to the size of the chips involved!

90% of the content in this thread is "well meaning" regurgitation of "nice ideas", and wildly misleading for the OP's well communicated problem/question.

[radiolistener]

Are people trying to be like AI and just say things without calculating the basis behind them?

I don't know about people trying to imitate AI. As for the point itself, a common timing offset between the entire data bus and the clock is easier to adjust. Bit-to-bit skew isn't. And this is the real headache.

I learned that the hard way while bringing up a GMII (RTL8211) on a Cyclone IV (EP4CE15F23C8N). If I'd known beforehand how much trouble poorly matched data traces could cause, I wouldn't have bought one of those cheap Chinese boards with poor PCB layout. You spend days trying every trick you can think of. But getting reliable error-free operation near 1 Gbps became an endless game of "yeah, it finally works!"... until the next stress test proved otherwise.

Despite many attempts, I didn't manage to get a continuous real-time error-free 1 Gbps stream. That said, it was back in the pre-AI era. Maybe with AI the process would have been simpler and more successful, since I had relatively little experience with FPGA development at the time, and it would likely have helped me a lot.

That PCB had several questionable layout decisions and all of them related to the clock domain. Whether that was intentional or just poor design, I don't know. But it certainly didn't make timing closure any easier. For example, routing the clock trace to a regular GPIO pin right next to the dedicated clock input is the kind of design decision that leaves me wondering whether it was intentional sabotage. If you've familiar with Cyclone IV, you know that PLL inputs can only be driven from dedicated clock pins. There is no routing path from general-purpose GPIO to the PLL clock network, so a PLL cannot be driven from a GPIO pin.

I’d be interested to hear what practical techniques you’ve found effective for dealing with bit-to-bit skew on high-speed parallel buses in real designs with Altera FPGA? Especially when a PCB cannot be redesigned and you have to work with the existing layout.

[Someone]

I learned that the hard way while bringing up a GMII (RTL8211) on a Cyclone IV (EP4CE15F23C8N). If I'd known beforehand how much trouble poorly matched data traces could cause,

Sounds like you didn't ever calculate the required setup/hold/skew when you so conveniently avoid adding that detail that was specifically asked for.
https://www.ti.com/lit/ab/scea139/scea139.pdf
Oh look hundreds of ps, massive data-data skew possible before even compensating for in the FPGA. Trace matching is not realistically needed.

[Someone]

Since the forum seems to have eaten a reply. I'll put it here again.

I’d be interested to hear what practical techniques you’ve found effective for dealing with bit-to-bit skew on high-speed parallel buses in real designs with Altera FPGA? Especially when a PCB cannot be redesigned and you have to work with the existing layout.

Its called timing constraints, skew of the magnitude to be impractical to correct with FPGA outputs is implausible for a (R)GMII interface (more inconsistencies/mistakes in your claims).

[pcprogrammer]

Oh look hundreds of ps, massive data-data skew possible before even compensating for in the FPGA. Trace matching is not realistically needed.

Some numbers found on the web, for 2ns delay ~13 inch / ~33 cm of trace is needed. (It depends on which layer and how many vias are in the trace) This would be needed if the RGMII MAC and PHY do not support signal delay themselves. Quite a long trace for sure.

In application notes or datasheets of high speed devices the allowed skew can be mentioned, but with ~150ps / inch there is a lot of wiggle room.

RGMII for gigabit ehternet runs on a 125MHz clock, but uses double data rate technology, so on each clock edge the data has to be ready. This means data clocked every 4ns. Both the clock and data lines normally switch at the same time, but the RTL8211 allows for delaying the data lines of both the RX and TX side, reducing the need of delaying the clock signals. Based on this a design should still work, even if there is a trace length difference of an inch, or even more.

https://resources.pcb.cadence.com/blog/2019-guide-on-pcb-trace-length-matching-vs-frequency

But the ridiculous extend some designers go to when designing a board to be "conform" can be seen on a T113-S3 board I bought. Even the SD card present switch trace is length matched to the SD card clock and data signals. 

[radiolistener]

Oh look hundreds of ps, massive data-data skew possible before even compensating for in the FPGA. Trace matching is not realistically needed.

I wasn't claiming that PCB trace mismatch alone makes GMII impossible. My point was that, in practice, bringing up a bad PCB layout without proper clock routing turned into a debugging nightmare. As I already said, on that particular PCB, the clock from the PHY wasn't routed to a dedicated clock input on the Cyclone IV, so I couldn't use a PLL on it. That removed one of the tools normally used for timing adjustment, making timing closure significantly more challenging. At that point, the problem is no longer just matching the timing of the individual data lines. You also lose the ability to use the FPGA clocking infrastructure to manage clock phase and alignment. This is because Cyclone IV does not provide a routing path from general-purpose GPIO pins to the internal clock distribution network.

Maybe there was a clean and simple solution. I'm not an FPGA developer and I had nobody to ask for help, and AI assistants didn't exist at that time. I was left with the documentation and forum discussions, most of which assumed the reference clock was already connected to a clock pin.

It's also possible that part of the timing problem was in my Verilog code. However, I also tried alexforencich/verilog-ethernet core. It exhibited the same transfer errors on that PCB, but worked ok on another PCB layout. So I still can't say whether the problem was fixable at all.

Today, with AI, it would probably be much easier to explore different approaches and identify possible issues. Back then, though, I felt like a blind kitten.

It's easy to say that everything can be easy fixed. In practice, though, things often look very different. Once you've spent days chasing timing issues on real hardware, you start to appreciate just how important good PCB layout and proper clock routing really are.

[Someone]

Oh look hundreds of ps, massive data-data skew possible before even compensating for in the FPGA. Trace matching is not realistically needed.

I wasn't claiming that PCB trace mismatch alone makes GMII impossible. My point was that, in practice, bringing up a bad PCB layout without proper clock routing turned into a debugging nightmare. As I already said, on that particular PCB, the clock from the PHY wasn't routed to a dedicated clock input on the Cyclone IV, so I couldn't use a PLL on it. That removed one of the tools normally used for timing adjustment, making timing closure significantly more challenging. At that point, the problem is no longer just matching the timing of the individual data lines. You also lose the ability to use the FPGA clocking infrastructure to manage clock phase and alignment. This is because Cyclone IV does not provide a routing path from general-purpose GPIO pins to the internal clock distribution network.

You didn't say that until later, which is walking this off into some internet point scoring nonsense where you try to cling onto being "right" despite all the plainly false things you said/claimed.

Being true .... but only if all these other unstated conditions are present simultaneously.... is being false unless those other things are stated.

Maybe there was a clean and simple solution. I'm not an FPGA developer and I had nobody to ask for help, and AI assistants didn't exist at that time. I was left with the documentation and forum discussions, most of which assumed the reference clock was already connected to a clock pin.

Ok, so now you're poisoning the AI that you hope will solve your problems. Your actions are directly ruining others activities, and your hoped for future.  Why add your voice if it is incorrect and off topic? Taking your attitude and then asking for specific help, hilariously bold.

[radiolistener]

I've clarified what I meant. If you have practical suggestions for dealing with bit-to-bit skew on high-speed parallel buses, I'm happy to discuss them. I have no interest in discussing each other personally.

[nctnico]

Oh look hundreds of ps, massive data-data skew possible before even compensating for in the FPGA. Trace matching is not realistically needed.

Some numbers found on the web, for 2ns delay ~13 inch / ~33 cm of trace is needed. (It depends on which layer and how many vias are in the trace) This would be needed if the RGMII MAC and PHY do not support signal delay themselves. Quite a long trace for sure.

In application notes or datasheets of high speed devices the allowed skew can be mentioned, but with ~150ps / inch there is a lot of wiggle room.

RGMII for gigabit ehternet runs on a 125MHz clock, but uses double data rate technology, so on each clock edge the data has to be ready. This means data clocked every 4ns. Both the clock and data lines normally switch at the same time, but the RTL8211 allows for delaying the data lines of both the RX and TX side, reducing the need of delaying the clock signals. Based on this a design should still work, even if there is a trace length difference of an inch, or even more.

https://resources.pcb.cadence.com/blog/2019-guide-on-pcb-trace-length-matching-vs-frequency

But the ridiculous extend some designers go to when designing a board to be "conform" can be seen on a T113-S3 board I bought. Even the SD card present switch trace is length matched to the SD card clock and data signals. 

The latter is probably just autorouted with all the SD card traces thrown into the same length matching group. Don't attribute to malice what can be explained by laziness 

[radiolistener]

RGMII for gigabit ehternet runs on a 125MHz clock, but uses double data rate technology, so on each clock edge the data has to be ready. This means data clocked every 4ns. Both the clock and data lines normally switch at the same time, but the RTL8211 allows for delaying the data lines of both the RX and TX side, reducing the need of delaying the clock signals. Based on this a design should still work, even if there is a trace length difference of an inch, or even more.

Yes, RTL8211 provides configurable TXDLY and RXDLY options that enable or disable a fixed 2 ns clock delay relative to the data. They don't provide fine-grained phase adjustment, but rather implement the clock-to-data offset required by the RGMII specification. This allows the required timing relationship to be generated inside the PHY instead of having to create these delays in the FPGA.

The RTL8211 datasheet guarantees at least 0.8 ns of hold time. Typical values may be larger, but timing closure should always be based on the guaranteed minimum values rather than typical ones. A fixed 2 ns delay option can be used when the overall timing already falls within the available margin, but it cannot compensate for arbitrary clock/data phase errors. It is intended to provide the fixed clock-to-data offset required by the RGMII specification, not to replace fine-grained timing adjustment.

I can agree, that from a pure propagation-delay perspective, 0.8 ns certainly looks like a fairly comfortable margin. At roughly 150-180 ps/inch, it corresponds to several inches of trace length difference. I assume that's the point you're making.

However, that estimate only considers propagation delay. In a real design, the sampling margin is also affected by signal integrity: trace bends, vias, impedance changes, parasitic capacitance and inductance, reflections, crosstalk and jitter. Those effects change not only the propagation delay, but also the edge shape and the instant at which the FPGA input actually crosses its logic threshold. As a result, the effective sampling margin can differ from what a simple propagation-delay estimate would suggest.

Returning to the original topic, the topic starter concern was about FPGA interfacing in general. ADC/DAC interfaces typically don't provide the configurable clock/data delay options at all.

In one of my prototypes (parallel interface clocked at 100 MHz), I initially connected the ADC to the FPGA using a short (~10-12 cm) ribbon cable. I remember that even relatively small changes to the clock wire affected sampling stability. Making the clock wire slightly longer/shorter (± 1-2 cm) or slightly changing its position/geometry while keeping approximately the same length was enough to shift the sampling point so that the FPGA started capturing the bus during transitions, producing obvious noise.

I could observe this effect in real time by gently bending the clock wire with my fingers. As soon as I bent the wire beyond a certain point, the captured test signal immediately became distorted and noisy. Straightening the wire restored stable operation. That's one of the reasons I've become quite cautious about wire lengths and routing.

Therefore, I'm skeptical of generalizing the conclusion that "an inch, or even more difference should work" to high-speed parallel interfaces. In practice, I have observed that even relatively small changes in clock routing at 100 MHz can have a noticeable impact on sampling stability in real hardware.

[asmi]

An easy number to remember ("A Rule of Thumb") is 6 ps/mm - this propagation speed is between that of a microstrip and a stripline, so it gives a good initial estimate.

[Someone]

But the ridiculous extend some designers go to when designing a board to be "conform" can be seen on a T113-S3 board I bought. Even the SD card present switch trace is length matched to the SD card clock and data signals. 

The latter is probably just autorouted with all the SD card traces thrown into the same length matching group. Don't attribute to malice what can be explained by laziness 

Adding the constraints is (stupid make-busy) work not laziness. Even for the quoted example with a peak of SDR104, 1ns of slack available. Any plausible routing would be a tiny fraction of that without need to resort to matching.

[asmi]

Adding the constraints is (stupid make-busy) work not laziness. Even for the quoted example with a peak of SDR104, 1ns of slack available. Any plausible routing would be a tiny fraction of that without need to resort to matching.

SD bus has a different problem - it doesn't have a source-synchronous receive clock at the master end, and so once you go above legacy "high-speed" mode (3.3 V 50 MHz), depending on the total traces length your receiver might need to do link training (which is also supported by SD protocol). I've been recently working on an SD/MMC host controller capable of going all the way up to HS400 (though I ended up only going as far as HS200, but I'm hoping one day I will finish the task I initially planned), and receiver can't know in advance when data stream is going to start arriving, so it has to sample data lines looking for a start bit (data lines are idling at high signal level, so a 1 to 0 transition indicates arrival of a start bit). Also the way they utilize multiple data lines clearly indicate that they are meant to operate independently - kind of like multi-lane serial interfaces like PCI Express. Now, they ended up going for a different approach for UHS-II - instead of running existing data lines at even higher speeds they went with the USB 3-like approach of adding a pair of dedicated high-speed serial lanes, but UHS-II cards are still not very popular outside of high-end cameras which do actually require more bandwidth than what UHS-I cards can provide - but there they compete against other standards like CF Express - which is basically an NVMe drive in a different form factor.
Also MMC's HS400 interface (which uses the same physical and protocol layers as SD) runs at 200 MHz DDR and thus is encroaching into the territory where length matching can be important.
At the end of the day length matching traces which do not need it doesn't really hurt anything aside from requiring additional effort from PCB designer to actually do that matching, so I can totally understand a sentiment like "let's do a length matching just in case".