FPGA Clock (setup slack) issues.

[Joanna_H]

Hi all,

Further to the CPU I have been working on, I've moved from SRAM to SDRAM (cost being the controlling factor),  If I build the CPU on it's own, all is fine, it can run at a max of 50Mhz no isses.   If I build the SDRAM controller on it's own it can work at the required 133Mhz.

The primary incomming clock is 133Mhz and this is dropped down to 44.333333333Mhz using a PLL for the CPU.

The CPU is halted (internal clock cycles) if the SDRAM is either pending a read, pending a write or pending initialisation.



I've selected all the clocks as primary clocks and defined their appropriate frequencies.  Of course, the router makes it own choices based on what I suggest and comes up with:

The following 3 signals are selected as primary clocks :
    CP1/clk0 (driver: SLICE_749, clk load #: 344; quadrants: TL/TR/BL/BR)
    CP1/decodeClk (driver: CP1/SLICE_963, clk load #: 51; quadrants: TL/TR/BL/BR)
    clock_c (driver: clock, clk load #: 64; quadrants: TL/TR/BL/BR)

WARNING - The following 1 data signal has to use primary clock resource:

    cpuClock (driver: dc; quadrants: TL/TR/BL/BR)

The following 1 signal is selected as DCS clock :
    cpuClock (driver: dc, clk load #: 0; quadrants: TL/TR/BL/BR)

The following 3 signals are selected to use the secondary clock resources :
    CP1/flagsDirectRead_15__N_188 (driver: CP1/SLICE_158, clk load #: 0, sr load #: 27, ce load #: 0)
    CP1/ir_direct_out_0 (driver: CP1/SLICE_402, clk load #: 0, sr load #: 16, ce load #: 0)
    CP1/loadOperandB (driver: CP1/LCTRL/SLICE_415, clk load #: 0, sr load #: 0, ce load #: 24)

Signal reset_c is selected as Global Set/Reset.

I would have put cpuClock in the mainlist of clocks but it never appears as a valid clockNet to select for setup.

I'm using Lattice Diamond by the way.

My problem is that when routing it all out I find I have a -30us setup slack and I'm trying to figure out a way to resolve this.  Do any of you have any ideas or suggestions on how to resolve this issue??

Start NBR section for re-routing at 14:34:13 08/20/19
Level 4, iteration 1
0(0.00%) conflict; 0(0.00%) untouched conn; 29721127 (nbr) score;
Estimated worst slack/total negative slack<setup>: -22.216ns/-29721.128ns; real time: 1 mins 25 secs

Thanks for any help.

[SiliconWizard]

There may be some info missing, and I haven't thoroughly looked at what you gave here, but just an initial thought.

Your design doesn't look like having horribly long logic paths (but I haven't seen how your CPU is implemented), and with a negative slack of this order (-30µs?), this kinda smells of a clock domain crossing issue.

[Joanna_H]

There may be some info missing, and I haven't thoroughly looked at what you gave here, but just an initial thought.

Your design doesn't look like having horribly long logic paths (but I haven't seen how your CPU is implemented), and with a negative slack of this order (-30µs?), this kinda smells of a clock domain crossing issue.

FYI,

The CPU:



The SDRAM Controller:



Each will route on their own with their respective speeds fine, it's when I connect them together with the SDRAM Controller running at 133Mhz and the CPU running at 44.33333Mhz.

I'm wondering if it's somehow to do with the data/control line connections between them???   Is the CPU trying to respond at 133Mhz speeds on some of those bits??   Not sure how the router handles all this side of it.

[mfro]

...
Each will route on their own with their respective speeds fine, it's when I connect them together with the SDRAM Controller running at 133Mhz and the CPU running at 44.33333Mhz.

I'm wondering if it's somehow to do with the data/control line connections between them???   Is the CPU trying to respond at 133Mhz speeds on some of those bits??   Not sure how the router handles all this side of it.

How did you synchronize clock domain crossing between the two?

[Joanna_H]

...
Each will route on their own with their respective speeds fine, it's when I connect them together with the SDRAM Controller running at 133Mhz and the CPU running at 44.33333Mhz.

I'm wondering if it's somehow to do with the data/control line connections between them???   Is the CPU trying to respond at 133Mhz speeds on some of those bits??   Not sure how the router handles all this side of it.

How did you synchronize clock domain crossing between the two?

The incomming clock is 133, the cpu clock is 44.333333, the cpu clock is basically divide by 3 and uses a PLL , it is in phase with the incomming 133 clock.

The cpuClock can take a slow pulse 1hz to 10Mhz controll clock or the 44Mhz generated clock, controlled and sync'd using a DCS.

The cpu will only look at data lines on it's own clock, which is 1/3rd of the SDRAM clock max.

The internal cpu clocks will not toggle unless the SDRAM controller has told the CPU that is has completed the action it is performing, effectively the SDRAM controller can tell the CPU to halt as it's busy.  The cpu only samples the halt line on the pos/neg edge of the incomming raw cpu clock.

The cpuclock inside the CPU is divided into two, one inphase with the cpuClock and one 180deg out of phase.  One controls instruction decode and setup, the other controls instruction execution.

[mfro]

The incomming clock is 133, the cpu clock is 44.333333, the cpu clock is basically divide by 3 and uses a PLL , it is in phase with the incomming 133 clock.

That's good.

I guess the failing path is a CDC from the 133 MHz clock domain to the 44 MHz clock domain?

Then you need to make sure you transfer data on one of the clock edges of the fast clock where the slow clock has a chance to catch it.

You could either count clock cycles on the fast clock side and only transfer data on a clock edge where the slow clock can see it (but then you might run into hold time violations on the other side) or - if possible - make sure you keep the data stable for several clock cycles and define a multicycle timing exception.

As the negative slack is nearly a full cycle in the slow clock domain, it might also help to slightly offset the period by a few degrees

[asmi]

The incomming clock is 133, the cpu clock is 44.333333, the cpu clock is basically divide by 3 and uses a PLL , it is in phase with the incomming 133 clock.

The cpuClock can take a slow pulse 1hz to 10Mhz controll clock or the 44Mhz generated clock, controlled and sync'd using a DCS.

The cpu will only look at data lines on it's own clock, which is 1/3rd of the SDRAM clock max.

The internal cpu clocks will not toggle unless the SDRAM controller has told the CPU that is has completed the action it is performing, effectively the SDRAM controller can tell the CPU to halt as it's busy.  The cpu only samples the halt line on the pos/neg edge of the incomming raw cpu clock.

The cpuclock inside the CPU is divided into two, one inphase with the cpuClock and one 180deg out of phase.  One controls instruction decode and setup, the other controls instruction execution.

I'd suggest you to use FIFOs to cross clock domains - they are more reliable and they don't require any particular relationship between clocks. All SDRAM/DDRx controllers I've ever seen (and developed) use this approach - there is a command FIFO and two data FIFOs - one for read data, and another one for write data.

[asmi]

Your design doesn't look like having horribly long logic paths (but I haven't seen how your CPU is implemented), and with a negative slack of this order (-30µs?), this kinda smells of a clock domain crossing issue.

The only time I've ever seen that kind of slack was when I attempted to use HDL's builtin division operator, which Vivado happily turned into a crap ton of logic

[SiliconWizard]

I'd suggest you to use FIFOs to cross clock domains - they are more reliable and they don't require any particular relationship between clocks. All SDRAM/DDRx controllers I've ever seen (and developed) use this approach - there is a command FIFO and two data FIFOs - one for read data, and another one for write data.

Yes, I've never used Lattice SDRAM controllers though, so I don't know about them. But for Xilinx ones, that certainly is true. You can even have several ports, each with command, read and write FIFOs, and all can be accessed from different clock domains.

[SiliconWizard]

Just a note that is unrelated to your issue.

Unless you implement a cache system of some kind to improve things on average, if you use the SDRAM controller to directly write individual words at random addresses as you would with a SRAM, you will get nowhere near the max possible throughput of the SDRAM chip. At 133MHz, you will probably not even get close to the CPU clock (~44MHz).

[Joanna_H]

I think I'll just go back to my known working SRAM version for now.   These CDC issues are driving me insane.    

[SiliconWizard]

I've just taken a very quick look at Lattice's SDRAM controller IP, and it seems much simpler than what Xilinx offers. So you may indeed have to handle the CDC yourself. Not that it's very hard to do, but if you just start with Verilog (or any RTL for that matter), I can understand cumulating all that can be a bit intimidating. It will be interesting to learn how to use SDRAM in your project later on though. As I said, you will ideally have to also think about some kind of cache for RAM access. So, one thing at a time.

I don't know what size you're after, but if you're looking for fast SRAM chips that are still not too atrociously expensive, you could consider this: IS61WV102416DBLL-10TLI (recent product, 1Mx16, 10ns access time). Obviously still a lot more expensive than SDRAM, but it will be much easier to use and you'll get up to 200MByte/s easily.

[Joanna_H]

I was actually intendeding to use their 1Mx16 10n asycn chips, I liked how they worked in the datasheets, nice and simple.

[asmi]

I think I'll just go back to my known working SRAM version for now.   These CDC issues are driving me insane.    

You will have to learn how to deal with them sooner or later. Pretty much all external interfaces run on their own clocks, and use some sort of FIFO to get data into and out of FPGA. Personally as a rule I always create dedicated clocks for external interfaces so that I can adjust them (or main internal system clock) without affecting external chips (some of those require very specific frequency and will not work if you provide any other one).

Actually, designing your own SDRAM controller would be a good project for beginner in it's own right. Unlike some more modern memories like DDR3, SDRAM does not define minimum frequency so you can run it slower than rated speed if you want, you will just have to make sure timing for refresh command is met as otherwise you may lose memory contents - although some experiments show that in reality you can stretch this time VERY far and still have contents intact if you're not running the memory at some extreme conditions like in +70C ambient temp. If you've got logic analyzer, you can even observe signals to/from memory chip in real time if you run it slow enough - though you might want to use logic analyzer IP core for you if your FPGA vendor provides one.

[Joanna_H]

The worst thing about all this is..  1) I forgot to take a backup before I modded it..  2) I removed the SDRAM and everything stopped working..  3) I went mad trying to work out why the CPU was failing and tweaking things...   4)  I finally realised that I had an inout as an input only....  5) I've forgotten how my orignal system even worked....  lol

[asmi]

The worst thing about all this is..  1) I forgot to take a backup before I modded it..  2) I removed the SDRAM and everything stopped working..  3) I went mad trying to work out why the CPU was failing and tweaking things...   4)  I finally realised that I had an inout as an input only....  5) I've forgotten how my orignal system even worked....  lol

"There are two kinds of IT professionals - those who are making backups, and those who will be making them"

[SiliconWizard]

Damn.

Time to start using a version-control system. Now.

[Joanna_H]

The worst thing about all this is..  1) I forgot to take a backup before I modded it..  2) I removed the SDRAM and everything stopped working..  3) I went mad trying to work out why the CPU was failing and tweaking things...   4)  I finally realised that I had an inout as an input only....  5) I've forgotten how my orignal system even worked....  lol

"There are two kinds of IT professionals - those who are making backups, and those who will be making them"

30yrs of software dev, backup all the time...   2 weeks of playing with verilog... and ermm.. forgot... lol

[langwadt]

why not use a single clock and a clock enable for the slower parts?

[hamster_nz]

why not use a single clock and a clock enable for the slower parts?

Bacause the complexity of the logic in the slow domain has to be low enough to clock at the speed of the fast domain....

[langwadt]

why not use a single clock and a clock enable for the slower parts?

Bacause the complexity of the logic in the slow domain has to be low enough to clock at the speed of the fast domain....

what do you mean? using clock enable instead of dividing a clock is totally standard

[Someone]

why not use a single clock and a clock enable for the slower parts?

Bacause the complexity of the logic in the slow domain has to be low enough to clock at the speed of the fast domain....

what do you mean? using clock enable instead of dividing a clock is totally standard

Yes, but the multi cycle timing constraints to allow deep logic layers on the virtual "slow" clock is usually more complex and error prone than crossing domains.

[Joanna_H]

Well, finally got it all working again... LOL

Decided to focus on Microcode development for now and finish off the CPU, then come back to the memory issues.   Using a formal SRAM simulation model from IDT (10ns)

So, a quick test program:



And the simulation results:



And defining the microcode is nice and easy   Yeah, that PC to Internal address flag isn't needed on the step 3 of the load with index.  Just hadn't removed it when I took the screenshot...



Even the branch is easy to define.



And Step 0 is the same on all microcode, it's just the instruction fetch routine.



At least some progress was made

[SiliconWizard]

Nice, but are you using a VCS yet?

[Joanna_H]

Nice, but are you using a VCS yet?

Yeah, decided to just use Git as I had it installed.