Author Topic: SerDes headache on Xilinx FPGA  (Read 13711 times)

0 Members and 1 Guest are viewing this topic.

Offline AlexTopic starter

  • Regular Contributor
  • *
  • Posts: 175
  • Country: gb
SerDes headache on Xilinx FPGA
« on: June 21, 2017, 08:14:09 pm »
Do you have any experience debugging SerDes inputs? Apologies for the lengthy post, this is proving a particularly difficult bug.  |O

This concerns a high speed parallel DAQ.  Right now we are getting erroneous output data when sampling a sinewave (attached image).

We have 14 bits coming in from the ADC, serial, DDR at 580 Mb/s. Xilinx SerDes IP cores implement 1:7 decoding, we then implement 7:14 demultiplexing. The data from this is often stable, outputting correct portions of the input sinusoid (see attached).

The system 'bitslips', until the framing signal is correctly locked, i.e. '01111111000000' is bitslipped by one to obtain '11111110000000'. This works and remains locked for greater than 10 mins.

The user demultiplexing appears to work fine, as we get portions of the sinusoid and the boundary between the top nibble (7 bits) and bottom nibble (7 bits) seems to also work fine.

In the output sinusoid, we get periodic amplitude jumps that happen to be exactly 2^10, i.e. 1024 from an erroneous bit toggle on the 10th bit. Rather than being a single sample glitch, this lasts for approx 50 samples. Meanwhile the other bits correctly switch, allowing the shape of the sinusoid to be correct within that 50 sample period. The output sinusoid therefore looks to have 2^10 amplitude jumps within it (see attached).

Now, the 10th bit is in the middle of the top 7bit nibble. As the other bits of that nibble are correct, it is clearly not an issue with 7 bit to 14 bit time demultiplexing. Likewise if it was PCB trace routing delays, it would be more random rather than periodic and would probably effect other bits. It would also change with different physical constraints and firmware re-builds as logic is moved around on the FPGA, which it doesn't. I've checked the bitslip functionality and it appears to be fine. The ADCs are definitely configured, and their clock (160 MHZ) is stable.

As many of the bits are correct, and indeed bit 10 is often correct for 100s of samples, I don't think it is SerDes singaling (timing or voltages). I'm using a test widget to pretend it is the synchronisation module, allowing the RAW FIFO to be setup and to grab some data. With test data this is perfect. The test widget starts RAW acquisition 800-ish clock cycles after the widget is enabled from the embedded software, and crucially grabs data into the FIFO just once, hence no problem with data writing when we are reading the FIFO etc.

Now, there are other things going on. For example the input sinusoid has an amplitude of 200mVac and frequency 200kHz with signal generator DC bias set to the same as the ADC's natural DC biasing point. However with no change in hardware, if I increase amplitude to 400mVac, I get almost a flat-line with +/-200 DN noise, no sinusoid!

The issue doesn't seem to go away if we reduce the ADC speed, however I suspect it is not LVDS SerDes signalling anyway, but somewhere in the 1:7:14 deserialisation process. I've re-coded the 7:17 user demultiplexers for more robust timing, demultiplexing in time rather than two 7bit registers with asynchronous multiplexers. I.e. increasing the demultiplexing action from tclk/2 for the case of asynchronous multiplexer CLBs to the full tclk for synchronous 7:14 demultiplexing.

Now, there are other issues, one being single sample glitches, the other being another periodic bit issue with the 14bit signing bit (MSB), hence the offsets that go from positive to negative etc. But I expect those are symptoms of the same  issue, i.e. fix the issue with bit 10 and bit 13 is also likely to be fixed.

Any pointers would be much appreciated. This is truly doing my head in...  :-//
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: SerDes headache on Xilinx FPGA
« Reply #1 on: June 21, 2017, 09:29:51 pm »
Any chance of seeing the raw data?

External causes:
- Signal integrity on the serial data
- Intersymbol interference - Too many '1's or zeros in a row cause the value to bleed into the following bit.

Maybe map out the raw data, as it is on the wire, and then 'corrected' raw data, and see what the differences look like.

Internal causes:
- Bitslip - does it really work as you are expecting. I never trust/understood bitslip, and have sometimes implemented my own logic to find the correct framing.
- Sampling phase - if not already doing so, try adding an IDELAY2 and some option to set the delay. That will allow you to fine-tune when the bits are sampled. This can look a lot like a signal integrity issue.

In 7-series parts you can sort of automatically detect and adjust sampling phase, but you can't use 'wide' Serdes. You use two 4-bit Serdes, one sampling on the inverted clock, and then tune the IDELAY so that for one of the serdes blocks the transitions are picked up 50% of the time.

At 580Mb/s You could also consider sampling twice as fast at 1:7, combining that into a oversampled 1:14 frame, and then tuning the delays so that you are sure that the transition occurs on close to the odd bits, then use the even bits as good data.
« Last Edit: June 21, 2017, 10:09:47 pm by hamster_nz »
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: SerDes headache on Xilinx FPGA
« Reply #2 on: June 21, 2017, 10:28:45 pm »
There also seems to be some sort of pattern of errors at a finer scale too.  maybe at the 256 or 128 counts.
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline langwadt

  • Super Contributor
  • ***
  • Posts: 4414
  • Country: dk
Re: SerDes headache on Xilinx FPGA
« Reply #3 on: June 21, 2017, 10:50:13 pm »
I'm thinking that the timing or signal integrity is marginal so occasionally an odd and even bit gets swapped/duplicated

can you setup the ADC to output a test pattern?
 

Online BrianHG

  • Super Contributor
  • ***
  • Posts: 7725
  • Country: ca
Re: SerDes headache on Xilinx FPGA
« Reply #4 on: June 21, 2017, 11:11:34 pm »
There also seems to be some sort of pattern of errors at a finer scale too.  maybe at the 256 or 128 counts.
If the serial bits are being shifted to the left or right by 1, the MSB may be mapped in the previous or next LSB bit, and you could see the type of waveform you are getting since you see only the 7 LSBs on bits 7 through 1, with what should have been the MSB bit 7 on the next or previous LSB bit 0.  This may create that type of chop in the waveform when your signal crosses the level which triggers the true MSB of the ADC.

Otherwise the output of your ADC is 2's complement AC and you are reading a straight linear signal, or, visa-versa.  This means your serdes is good, you just need to invert the 7 LSBs when the MSB is set, or change your graphing from an UNSIGNED BYTE to a SIGNED BYTE, or visa-versa.

« Last Edit: June 21, 2017, 11:14:37 pm by BrianHG »
 

Offline jefflieu

  • Contributor
  • Posts: 43
  • Country: au
Re: SerDes headache on Xilinx FPGA
« Reply #5 on: June 21, 2017, 11:21:29 pm »
Do you have any experience debugging SerDes inputs? Apologies for the lengthy post, this is proving a particularly difficult bug.  |O

This concerns a high speed parallel DAQ.  Right now we are getting erroneous output data when sampling a sinewave (attached image).

We have 14 bits coming in from the ADC, serial, DDR at 580 Mb/s. Xilinx SerDes IP cores implement 1:7 decoding, we then implement 7:14 demultiplexing. The data from this is often stable, outputting correct portions of the input sinusoid (see attached).

The system 'bitslips', until the framing signal is correctly locked, i.e. '01111111000000' is bitslipped by one to obtain '11111110000000'. This works and remains locked for greater than 10 mins.

The user demultiplexing appears to work fine, as we get portions of the sinusoid and the boundary between the top nibble (7 bits) and bottom nibble (7 bits) seems to also work fine.

In the output sinusoid, we get periodic amplitude jumps that happen to be exactly 2^10, i.e. 1024 from an erroneous bit toggle on the 10th bit. Rather than being a single sample glitch, this lasts for approx 50 samples. Meanwhile the other bits correctly switch, allowing the shape of the sinusoid to be correct within that 50 sample period. The output sinusoid therefore looks to have 2^10 amplitude jumps within it (see attached).

Now, the 10th bit is in the middle of the top 7bit nibble. As the other bits of that nibble are correct, it is clearly not an issue with 7 bit to 14 bit time demultiplexing. Likewise if it was PCB trace routing delays, it would be more random rather than periodic and would probably effect other bits. It would also change with different physical constraints and firmware re-builds as logic is moved around on the FPGA, which it doesn't. I've checked the bitslip functionality and it appears to be fine. The ADCs are definitely configured, and their clock (160 MHZ) is stable.

As many of the bits are correct, and indeed bit 10 is often correct for 100s of samples, I don't think it is SerDes singaling (timing or voltages). I'm using a test widget to pretend it is the synchronisation module, allowing the RAW FIFO to be setup and to grab some data. With test data this is perfect. The test widget starts RAW acquisition 800-ish clock cycles after the widget is enabled from the embedded software, and crucially grabs data into the FIFO just once, hence no problem with data writing when we are reading the FIFO etc.

Now, there are other things going on. For example the input sinusoid has an amplitude of 200mVac and frequency 200kHz with signal generator DC bias set to the same as the ADC's natural DC biasing point. However with no change in hardware, if I increase amplitude to 400mVac, I get almost a flat-line with +/-200 DN noise, no sinusoid!

The issue doesn't seem to go away if we reduce the ADC speed, however I suspect it is not LVDS SerDes signalling anyway, but somewhere in the 1:7:14 deserialisation process. I've re-coded the 7:17 user demultiplexers for more robust timing, demultiplexing in time rather than two 7bit registers with asynchronous multiplexers. I.e. increasing the demultiplexing action from tclk/2 for the case of asynchronous multiplexer CLBs to the full tclk for synchronous 7:14 demultiplexing.

Now, there are other issues, one being single sample glitches, the other being another periodic bit issue with the 14bit signing bit (MSB), hence the offsets that go from positive to negative etc. But I expect those are symptoms of the same  issue, i.e. fix the issue with bit 10 and bit 13 is also likely to be fixed.

Any pointers would be much appreciated. This is truly doing my head in...  :-//
I don't see how you sample the middle of the data eye. Bitslip alone only give you the word boundary.
In case you're not aware, there's a block called IDELAY that inserts delay into either the clock or data. And you have to scan it to get the whole eye. Then pick the middle value.


i love Melbourne
 

Offline NorthGuy

  • Super Contributor
  • ***
  • Posts: 3138
  • Country: ca
Re: SerDes headache on Xilinx FPGA
« Reply #6 on: June 22, 2017, 12:08:18 am »
To me, the picture looks like the 10-th bit (or whatever the bit is) isn't outputted at all - always zero.

Imagine, for example, the signal is 1023, then it needs to go to 1030, but the 10-th bit is always zero, so you get 1030 - 1024 = 6 instead. Then the curve continues at this new level until the time where the 10-th bit is supposed to be cleared, at which points the curbe jumps back to normal.
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: SerDes headache on Xilinx FPGA
« Reply #7 on: June 22, 2017, 12:35:58 am »
I was more thinking that the 10th bit was always the same as the 11th bit.....  e.g. 1024 to 2047 ends up as 0 to 1023, and -1024 to -2047 ends up as -1 to -1023.

This could be caused by the long run of 0s or 1s (e.g. 11110xx:xxxxxxx is reading as 11111xx:xxxxxxx and 00001xx:xxxxxxx is reading 00000xx:xxxxxxx due to the run of 1s or 0s causing it to miss-read).

It would be interesting to see if you have patterns like 0000000:1xxxxxx  where 0000001:xxxxxxx  actually fits the expected curve better.

It could also explain the odd glitches, were something like 111101x:xxxxxxx gets read as 111110x:xxxxxxx, causing spikes.



Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Online KE5FX

  • Super Contributor
  • ***
  • Posts: 1889
  • Country: us
    • KE5FX.COM
Re: SerDes headache on Xilinx FPGA
« Reply #8 on: June 22, 2017, 12:48:53 am »
What do your .ucf or .xdc timing constraints look like?
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: SerDes headache on Xilinx FPGA
« Reply #9 on: June 22, 2017, 12:49:49 am »
Oh, I don't know if it is applicable at these speeds or your setup, but at Gb/s speeds over longer cables, transmit preemphasis can solve problems like this.

https://www.maximintegrated.com/en/app-notes/index.mvp/id/5045
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline exmadscientist

  • Frequent Contributor
  • **
  • Posts: 342
  • Country: us
  • Technically A Professional
Re: SerDes headache on Xilinx FPGA
« Reply #10 on: June 22, 2017, 03:48:06 am »
can you setup the ADC to output a test pattern?
+1, this is EXACTLY what test patterns are for -- you will be amazed what they help you find!

Can you post the part numbers or datasheets for the things you're trying to interface? Knowing exactly what you are working with may reveal a few gotchas that aren't obvious to the untrained eye. Having spent far too much of my life working with Spartan-6 BITSLIPs (OK, it wasn't that long, but it was still too long!) I can say there's a ton of things that can go wrong. In particular your clocks have to be perfect. Even a single missing FIFO can cause huge troubles, and if the crossing clocks are only a few PPM apart, they can be rare troubles, which is always fun.

One thing that stands out is you say you're using "SerDes IP cores". Last time I did this I found it was less trouble to read UG381 and instantiate ISERDES2 blocks directly rather than try and figure out what the magic settings were to get the wizards to do what I wanted. If you haven't tried it this is something to consider. I might also be able to post known-working sample code for interfacing to an AD9637 (12-bit DDR LVDS) if that would be helpful to you.
 
The following users thanked this post: Someone

Offline AlexTopic starter

  • Regular Contributor
  • *
  • Posts: 175
  • Country: gb
Re: SerDes headache on Xilinx FPGA
« Reply #11 on: June 22, 2017, 08:52:03 pm »
Thank you all for your insightful responses. In summary, my immediate plan of attack is to setup an output pattern. Now for the other points,

First things first, I am using:
 - FPGA - Xilinx Spartan 6 XC6SLX45 in CSG324
 - ADC - Analog Devices AD9257 40MHz
 - PLL - Analog Devices AD9577 outputting 160MHz for ADCs and 140MHz for FPGA

I have attached:
- A diagram showing the SerDes clocking
- A diagram showing inside a single SerDes
- A picture of the 'User Demultiplexer' 7 to 14 bits. Let me know if this is not clear.

Any chance of seeing the raw data?
Sure. I have attached a .txt dump from a UART. This is imported and parsed by the two matlab files also attached marked 'MATLAB' and extension changed from .m to .txt. Note that:
- The 14 bit data is bit extended, in this case using the MSB
- The data is concatonated by action of the 16bit input 32bit output FIFO
- The data is interleaved in time into two FIFOs B0 and B1
- The data is output onto UART in hex, but is actually 14 bit 2's complement 

There also seems to be some sort of pattern of errors at a finer scale too.  maybe at the 256 or 128 counts.
You are right, so solving the issue with bit 10 might solve the other errors.

I might also be able to post known-working sample code for interfacing to an AD9637 (12-bit DDR LVDS) if that would be helpful to you.
Sure, if you are able. No harm done in scanning through it.

I found it was less trouble to instantiate ISERDES2 blocks directly rather than try and figure out what the magic settings were to get the wizards to do what I wanted. If you haven't tried it this is something to consider.
Yes the SerDes is manually instantiated. Granted though I cannot guarantee error-free.

What do your .ucf or .xdc timing constraints look like?
I can send the .ucf but it only really contains the pin and clocking resource placements, none of which can be changed as the PCB is custom and now fixed.
 

Online KE5FX

  • Super Contributor
  • ***
  • Posts: 1889
  • Country: us
    • KE5FX.COM
Re: SerDes headache on Xilinx FPGA
« Reply #12 on: June 22, 2017, 09:15:14 pm »
I can send the .ucf but it only really contains the pin and clocking resource placements, none of which can be changed as the PCB is custom and now fixed.

So there are no timing constraints at all...?
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26878
  • Country: nl
    • NCT Developments
Re: SerDes headache on Xilinx FPGA
« Reply #13 on: June 22, 2017, 09:21:16 pm »
No constraints is asking for trouble! For these kind of designs you need to setup constraints which specify the setup and hold timing (before & after) so the right delay can be set in the IOB. Every input and output needs a timing constraint (input to flipflops and flipflops to output)!
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: SerDes headache on Xilinx FPGA
« Reply #14 on: June 22, 2017, 09:55:20 pm »
No constraints is asking for trouble! For these kind of designs you need to setup constraints which specify the setup and hold timing (before & after) so the right delay can be set in the IOB. Every input and output needs a timing constraint (input to flipflops and flipflops to output)!

I see you use Altera :)

My suggestion would be to at least add a manually instantiated IDELAY2 primitive and allow the Microblaze to set the input delay and then experiment....

As long as your external clocks have a period set you should be fine - there is only one path from the pin, through the IDELAY2 to the SERDES block, and all the other clocks should be derived as needed by tools.

You could also just try playing with the phase settings on the DMC/PLL block to shift the sampling phase a little relative to the ADC clock.
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 
The following users thanked this post: Someone

Online BrianHG

  • Super Contributor
  • ***
  • Posts: 7725
  • Country: ca
Re: SerDes headache on Xilinx FPGA
« Reply #15 on: June 22, 2017, 10:29:31 pm »
No constraints is asking for trouble! For these kind of designs you need to setup constraints which specify the setup and hold timing (before & after) so the right delay can be set in the IOB. Every input and output needs a timing constraint (input to flipflops and flipflops to output)!

I see you use Altera :)

My suggestion would be to at least add a manually instantiated IDELAY2 primitive and allow the Microblaze to set the input delay and then experiment....

As long as your external clocks have a period set you should be fine - there is only one path from the pin, through the IDELAY2 to the SERDES block, and all the other clocks should be derived as needed by tools.

You could also just try playing with the phase settings on the DMC/PLL block to shift the sampling phase a little relative to the ADC clock.

Don't you need to do both?
Without specifying timing constraints, how will the chip fitter know if your source signal will fit in the timing window unless you just trust everything to sit on the clock edge perfectly?  (Though I did a pretty big non time constraint, other than FMAX, design once with Altera, it just happened to work great since all my IOs were forced to use dedicated clocked marcocells right at each IO pin before the data was fed to the rest of the FPGA.  Xilinx probably has a similar option to set for each IO pin.  The timing of each IO pin was right on the PLL with the tightest possible clocking.)
« Last Edit: June 22, 2017, 10:31:59 pm by BrianHG »
 

Online Someone

  • Super Contributor
  • ***
  • Posts: 4525
  • Country: au
    • send complaints here
Re: SerDes headache on Xilinx FPGA
« Reply #16 on: June 23, 2017, 01:04:22 am »
No constraints is asking for trouble! For these kind of designs you need to setup constraints which specify the setup and hold timing (before & after) so the right delay can be set in the IOB. Every input and output needs a timing constraint (input to flipflops and flipflops to output)!

I see you use Altera :)

My suggestion would be to at least add a manually instantiated IDELAY2 primitive and allow the Microblaze to set the input delay and then experiment....

As long as your external clocks have a period set you should be fine - there is only one path from the pin, through the IDELAY2 to the SERDES block, and all the other clocks should be derived as needed by tools.

You could also just try playing with the phase settings on the DMC/PLL block to shift the sampling phase a little relative to the ADC clock.

Don't you need to do both?
Without specifying timing constraints, how will the chip fitter know if your source signal will fit in the timing window unless you just trust everything to sit on the clock edge perfectly?  (Though I did a pretty big non time constraint, other than FMAX, design once with Altera, it just happened to work great since all my IOs were forced to use dedicated clocked marcocells right at each IO pin before the data was fed to the rest of the FPGA.  Xilinx probably has a similar option to set for each IO pin.  The timing of each IO pin was right on the PLL with the tightest possible clocking.)
With dynamic alignment of the sampling in a SERDES RX you don't need any constraint on the data timing only the clock rate. In a well designed high speed serial system (source synchronous or embedded clock) the RX handles all the complexity in realtime and aligns the data output transparently.
 

Online BrianHG

  • Super Contributor
  • ***
  • Posts: 7725
  • Country: ca
Re: SerDes headache on Xilinx FPGA
« Reply #17 on: June 23, 2017, 07:27:19 am »
No constraints is asking for trouble! For these kind of designs you need to setup constraints which specify the setup and hold timing (before & after) so the right delay can be set in the IOB. Every input and output needs a timing constraint (input to flipflops and flipflops to output)!

I see you use Altera :)

My suggestion would be to at least add a manually instantiated IDELAY2 primitive and allow the Microblaze to set the input delay and then experiment....

As long as your external clocks have a period set you should be fine - there is only one path from the pin, through the IDELAY2 to the SERDES block, and all the other clocks should be derived as needed by tools.

You could also just try playing with the phase settings on the DMC/PLL block to shift the sampling phase a little relative to the ADC clock.

Don't you need to do both?
Without specifying timing constraints, how will the chip fitter know if your source signal will fit in the timing window unless you just trust everything to sit on the clock edge perfectly?  (Though I did a pretty big non time constraint, other than FMAX, design once with Altera, it just happened to work great since all my IOs were forced to use dedicated clocked marcocells right at each IO pin before the data was fed to the rest of the FPGA.  Xilinx probably has a similar option to set for each IO pin.  The timing of each IO pin was right on the PLL with the tightest possible clocking.)
With dynamic alignment of the sampling in a SERDES RX you don't need any constraint on the data timing only the clock rate. In a well designed high speed serial system (source synchronous or embedded clock) the RX handles all the complexity in realtime and aligns the data output transparently.

OOOPS, I didn't realize the OP was using a SER RX with embedded clock.  In that case, it's up to the FPGA's capabilities for clock reconstruction and everything should be self correcting unless your source has noise, isn't the right voltage level, has a noisy clock reference, isn't terminated correctly or the FPGA PLL clock's analog supply isn't filtered properly, or, you signal traces aren't balanced properly.

In this case, we are no longer dealing with a SERDES problem in the FPGA firmware, but, a hardware problem on the PCB.
 

Online BrianHG

  • Super Contributor
  • ***
  • Posts: 7725
  • Country: ca
Re: SerDes headache on Xilinx FPGA
« Reply #18 on: June 23, 2017, 07:40:01 am »
Just out of curiosity, have you tried configuring these bits in the AD9257 to see what happens:

0x15 bits 0, 4&5 - Output drive strength and termination values for the LVDS port.
0x16 bits 0,1,2,3, Serial output phase adjust.

These controls might fix your bit problem without changing any of your FPGA code.  I know very well I had problems with Analog devices in the past with significantly larger more sophisticated IC with mega huge I2C control maps (like over 1000 controls with their high end HDMI interface ICs), but your issue may boil down to these few registers for such a simple chip.
« Last Edit: June 23, 2017, 07:57:12 am by BrianHG »
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: SerDes headache on Xilinx FPGA
« Reply #19 on: June 23, 2017, 07:44:21 am »
SERDES on Xilinx can not use an embedded clock - I assume that there is a second pair for the clock, or it is supplying the clock.

Only the Transciever blocks (GTP primatives) can do CDR, and IIRC they need much higher data rates (and also run length limited too, so the PLL can keep lock).
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Online BrianHG

  • Super Contributor
  • ***
  • Posts: 7725
  • Country: ca
Re: SerDes headache on Xilinx FPGA
« Reply #20 on: June 23, 2017, 07:47:32 am »
SERDES on Xilinx can not use an embedded clock - I assume that there is a second pair for the clock, or it is supplying the clock.

Only the Transciever blocks (GTP primatives) can do CDR, and IIRC they need much higher data rates (and also run length limited too, so the PLL can keep lock).

I know, 'Someone' didn't read that the OP was using an AD9257 and didn't check how the IC interfaced with the FPGA.
I still say that the configuration controls I mentioned 1 post earlier has a good chance of completely solving the OP's issue without changing his FPGA firmware.
 

Online BrianHG

  • Super Contributor
  • ***
  • Posts: 7725
  • Country: ca
Re: SerDes headache on Xilinx FPGA
« Reply #21 on: June 23, 2017, 08:02:38 am »
Out of curiosity, have you tried freeze-it on the FPGA or ADC after the 10 minutes of operation to see if it will quickly un-do the problem?
This wont tell you much more than you have a timing issue right on the edge.  Adjusting the ADC's drive strength and PLL phase may help this.

If you are using the DCO phase of 0011 (default), I would try adjusting the DCO phase to:
0010
0100

Just for clarification, are you clocking your SERDES from the CLK+/- input from the ACD or are you using the DCO+/- as your SERDES clock.  How do you capture the FCO+/-?
« Last Edit: June 23, 2017, 08:13:01 am by BrianHG »
 

Online Someone

  • Super Contributor
  • ***
  • Posts: 4525
  • Country: au
    • send complaints here
Re: SerDes headache on Xilinx FPGA
« Reply #22 on: June 23, 2017, 08:22:00 am »
SERDES on Xilinx can not use an embedded clock - I assume that there is a second pair for the clock, or it is supplying the clock.

Only the Transciever blocks (GTP primatives) can do CDR, and IIRC they need much higher data rates (and also run length limited too, so the PLL can keep lock).

I know, 'Someone' didn't read that the OP was using an AD9257 and didn't check how the IC interfaced with the FPGA.
I still say that the configuration controls I mentioned 1 post earlier has a good chance of completely solving the OP's issue without changing his FPGA firmware.
Read the original quote, I'll add some more emphasis.
No constraints is asking for trouble! For these kind of designs you need to setup constraints which specify the setup and hold timing (before & after) so the right delay can be set in the IOB. Every input and output needs a timing constraint (input to flipflops and flipflops to output)!

I see you use Altera :)

My suggestion would be to at least add a manually instantiated IDELAY2 primitive and allow the Microblaze to set the input delay and then experiment....

As long as your external clocks have a period set you should be fine - there is only one path from the pin, through the IDELAY2 to the SERDES block, and all the other clocks should be derived as needed by tools.

You could also just try playing with the phase settings on the DMC/PLL block to shift the sampling phase a little relative to the ADC clock.

Don't you need to do both?
Without specifying timing constraints, how will the chip fitter know if your source signal will fit in the timing window unless you just trust everything to sit on the clock edge perfectly?  (Though I did a pretty big non time constraint, other than FMAX, design once with Altera, it just happened to work great since all my IOs were forced to use dedicated clocked marcocells right at each IO pin before the data was fed to the rest of the FPGA.  Xilinx probably has a similar option to set for each IO pin.  The timing of each IO pin was right on the PLL with the tightest possible clocking.)
With dynamic alignment of the sampling in a SERDES RX you don't need any constraint on the data timing only the clock rate.





In a well designed high speed serial system (source synchronous or embedded clock) the RX handles all the complexity in realtime and aligns the data output transparently.
Dynamic alignment can be done on the IO SERDES (or just abstract this to any arbitrary high speed RX) and you don't specify the data relative to the clock since it could now be anywhere and still work.


.... other clause .....


The receiver design (wrapper, module, function, whatever your terminology) should deal with all of this internally no matter if its source synchronous or embedded clock.
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: SerDes headache on Xilinx FPGA
« Reply #23 on: June 23, 2017, 08:24:44 am »
Just out of curiosity, have you tried configuring these bits in the AD9257 to see what happens:

0x15 bits 0, 4&5 - Output drive strength and termination values for the LVDS port.
0x16 bits 0,1,2,3, Serial output phase adjust.

These controls might fix your bit problem without changing any of your FPGA code.  I know very well I had problems with Analog devices in the past with significantly larger more sophisticated IC with mega huge I2C control maps (like over 1000 controls with their high end HDMI interface ICs), but your issue may boil down to these few registers for such a simple chip.

^^^^^^ yes this! ^^^^^
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline AlexTopic starter

  • Regular Contributor
  • *
  • Posts: 175
  • Country: gb
Re: SerDes headache on Xilinx FPGA
« Reply #24 on: June 23, 2017, 05:42:23 pm »
So there are no timing constraints at all...?
Apologies, all timing constraints are in the .ucf file. I have attached it in .txt.

Out of curiosity, have you tried freeze-it on the FPGA or ADC after the 10 minutes of operation to see if it will quickly un-do the problem?
Just for clarification, are you clocking your SERDES from the CLK+/- input from the ACD or are you using the DCO+/- as your SERDES clock.  How do you capture the FCO+/-?
I have tried freezing the fpga back when the logic was considerably different, I will try it again.
SerDes is clocked using the DCO. The FCO is captured in the same way as the DCO. See attachments in post #12.

Just out of curiosity, have you tried configuring these bits in the AD9257 to see what happens:
0x15 bits 0, 4&5 - Output drive strength and termination values for the LVDS port.
0x16 bits 0,1,2,3, Serial output phase adjust.
No, another thing to try!

a hardware problem on the PCB.
Let's also have a look at the PCB and routing. There are two ADC, Left and Right., each feeding 8 channels to the FPGA for a total of 16 differential pairs. The PPL sits in the middle, providing two completely separate 160MHz clocks one for each ADC. The signals from the ADCs are Data 1-8, DCO and FCO. All LVDS. All signals from the Left ADC are routed to Bank 0. All signals from the Left ADC are routed to Bank 2.  So that is 10 LVDS pairs from each ADC, on different layers, as you can see in the PCB images attached.

The traces have been matched to within 2.5 mm worst case, which would be in the order of 20ps. The DDR clock is 580MHz so tclk = 1.72ns or 1720ps. 20ps as a proportion of 1720 is less than 1.2%. However, there are other features on these lines like vias and test pads. One via per trace was necessary due to the packages used, but the PCB design was constrained to 1 via per trace and all traces have one via. So the exact propagation difference is unknown, but it is controlled to some extent. In any case, this is a 16 channel DAQ and every channel will have a slightly different delay. From this perspective, any corrective measure (e.g. iDELAY) would have to be on a per-channel basis.

Now going inside the FPGA.
I don't see how you sample the middle of the data eye. Bitslip alone only give you the word boundary.
Not robustly enough. There is a fixed delay between data and sample clock using the ADC's timing specs *at the ADC's pads*.  ;D

There is also an issue with the 7:14 User Demux block I posted in post #12. I reuploaded the internal of the block here (the hand drawn image). This block is meant to split the input stream into a higher and lower nibbles. The clocking of the Post Mux has issues. Although the two DIV_CLK inputs are time constrained, the output of the toggle flip-flop is not. Overall this might cause a bit from one nibble in the serial stream to be muxed into the other. Even if this doesnt break the sampling timing, it certainly eats away from any headroom. So this needs to be fixed as a priority.


Pressing on...
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf