DDR controller

[rubenvann]

I tried to be concise, but still this turned out to be quite a wall of text. The first paragraph can be skipped, it only provides information about my background .

I'm a computer engineer with a computer science background (basically, I don't know too much about electrical engineering: I'm trying to learn that on my own, but I'm struggling). Last year, I took a class where we had to work on the plasma softcore processor. This was my first encounter with FPGAs, and while I have to admit is was no 'love at first sight', I ended up buying a cheap FPGA (the mimas v2, which I can definitely recommend for people who want to experiment with FPGAs), downloaded Xilinx Webpack ISE, and started to do some really basic stuff. I first started messing around with the clock and VGA port, and I was able to generate a 640x480 VGA signal, and display a 128x128 picture using block RAM pretty easily. Ideally, I would want to have a fullscreen frame buffer so that I can output more interesting things (and in the end for higher resolutions too, but hey, baby steps!). Since there is not enough block RAM for a fullscreen buffer (and I would like memory with enough bandwidth to support at least one read/write port for a processor, and one read port to read the screen data), I started looking at the RAM module on the board (a Micron MT46H32M16LF or a Winbond W949D6CBHX6E, which seems to be a clone). I tried to use Xilinx MIG to generate an easier interface to access memory, but the interface that was generated had even more input signals than the RAM module. So, I downloaded the datasheets (Micron and Winbond), some examples (1, 2, 3, 4), and started learning about SDRAM and DDR controllers.

My goal is to make a minimal working DDR controller, (both to learn and for fun). After some trial (but mainly error*), I discovered this stackoverflow question that suggests you should use a functional model. I finally found one on the micron site (I use mobile_ddr.v, and manually edit it to include the right parameters, which involved mainly deleting ifdef-blocks, and ocasionally changing a parameter definition that deviates from the parameters in the datasheet), and now I'm at the point where I use that to design an LPDDR controller. It helped me to get the initialization right (at least, it doesn't throw errors so I assume it's correct). However, I still have some problems:
  1. The functional model spews out A LOT of setup/hold violation warnings
  2. Even when I don't use a preamble, and do not raise the data strobe after a first write command, I don't get any error or warning (while the datasheet suggests this should be done within 1.25 of a clock cycle ADDITIONAL INFO HERE).

I can think of two things that relate to the first problem. The first is that I use a normal simulation (not a post-map or a post-PAR one). Now, all signals change instantly, so of couse, hold time constraints are not satisfied. In Xilinx ISE, I have a process 'generate post-PAR simulation model', but I can' find an option to actually RUN a post-PAR simulation. This document from Xilinx suggests you need Modelsim for this (which seems weird to me: why would they ship their own simulator and then recommend people to use a product from another company? But I think their decision to not support windows 10 while Microsoft is basically forcing everybody to use it is a slap in the face of their users, and also hard to rationalize).

The second thing is that the LPDDR module needs a differential clock. A heavily simplified schematic view of the important signals:

(where the DDR controller changes the control signals on the rising edge of the clock)

I'm not sure if the buffer can lead to a problematic additional delay. If the controller changes its output signals faster than the buffer (I think this is referred to as a 'race condition'?) I think it is possible that the LPDDR module would miss commands. The controller logic should be complex enough to take longer than the buffer, but I would rather have a hard guarantee.

I'm really confused about the second problem (the functional model doesn't give a warning or error when I do not raise the data strobe after a write command). In the readme from the functional model, the only thing listed under 'limitations' is "". This piece of code looks like it tests for edges in dqs:

Code: [Select]

    // dqs edge checking
    always @ (posedge Sys_clk) begin
//        if (Write_pipeline[2] || Write_pipeline[1] || Data_in_enable) begin
        if (Write_pipeline[-1]) begin
            for (i=0; i<DQS_BITS; i=i+1) begin
                if (expect_neg_dqs[i]) begin
                    $display ("%m: At time %t ERROR: Negative DQS[%1d] transition required.", $realtime, i);
                end
                expect_neg_dqs[i] = 1'b1;
            end
        end else begin
           expect_neg_dqs = 0;
           expect_pos_dqs = 0;
        end
    end

    [[and here something similar for positive edges]]

So this both suggests that the functional model should test for edges in the data strobe signal. At the same time, there is no parameter for DQSS (defined as "Write command to 1st DQS latching
transition" in p51 of the winbond datasheet, and on p26 of the micron datasheet) or for the preamble.

[legacy]

the plasma softcore processor.

well, in my opinion, Plasma is not a good project, especially not well written
I don't like it very much 

[nctnico]

Most stuff from Opencores is poorly written. It can be useful but don't expect high clockspeeds.

Regarding a DDR controller: I designed one about a decade ago which was runtime configurable for different memory layouts because I used a PC memory module in that design. The easiest way is to use the DDR I/O flipflops in the FPGA IO fabric and keep the speed within sane limits (like a 100 to 150MHz DDR clock). There are examples which use FIFOs but the further you take an external signal into an FPGA the more difficult and critical the timing will get. The timing will vary over temperature and 3 gates in series will have a larger deviation than 1 gate. Also forget about using the DQ signals to clock the data back into the FPGA unless you can use local clock sources but you'll need to jump through many hoops. Still you'll need to get the timing constraints right for the design to work reliably.

[legacy]

may be the Papillo/Pro's repository is also good/acceptable?

[legacy]

sorry, my typo 

it's PapilioPro and this is the SDRAM controller that has been implement for this z80-super-SoC

[hamster_nz]

Hi - I authored that SDRAM controller linked to. The timing in it is a bit "guerrilla" - it works because of lots of underhanded tricks that are not self evident. It was a long while ago, and I might have forgotten a lot of the details when it comes to timing budgets for signals.

Alvaro Lopez has used this (and other resources) to write DDR his interface for his ZPU project - you can find them at

https://github.com/alvieboy/ZPUino-HDL/tree/master/zpu/hdl/zpuino/memory

I am sure Alvie will be happy to chat about them if you drop him an email too.

[nctnico]

What helps me to design complicated logic is to make a timing diagram for it. If you use software which can move rectangles it makes it easy to shuffle things around a bit. Either way a timing diagram makes it easy for me to see which needs to happen when and based on what condition.

[hamster_nz]

Awesome, thanks! Could you give me an example of such a trick?

- The time for the clock signal to leave the FPGA is 1.7 ns
- The time for the clock signal to get across the PCB to the SDRAM is about 0.2ns
- The SDRAM access time from clock edge is about 6ns
- The time for the data signal to get across the PCB to the FPGA is about 0.2ns
- The time for a signal to get back into the FPGA is about 1.42ns
- The setup time for the input flip flop is about 0.5ns.

That means that the data is back in the FPGA in about 10.02ns from the clock edge that triggers the SDRAM to present it - so at 100MHz the data is ready just in time for the next clock edge.

This is why why that controller only works between about 80MHz and 120MHz. The logic can work slower but you have to adjust how it captures the returning data to grab it  half a cycle earlier, then a full cycle earlier.

[sporadic]

What helps me to design complicated logic is to make a timing diagram for it. If you use software which can move rectangles it makes it easy to shuffle things around a bit. Either way a timing diagram makes it easy for me to see which needs to happen when and based on what condition.

I ran across WaveDrom the other day for this, pretty cool web based tool for timing diagrams.  Any other recommendations?

[nctnico]

I tried timing diagram tools but in the end Microsoft Visio does just fine.

[hamster_nz]

Awesome, thanks! Could you give me an example of such a trick?

- The time for the clock signal to leave the FPGA is 1.7 ns
- The time for the clock signal to get across the PCB to the SDRAM is about 0.2ns
- The SDRAM access time from clock edge is about 6ns
- The time for the data signal to get across the PCB to the FPGA is about 0.2ns
- The time for a signal to get back into the FPGA is about 1.42ns
- The setup time for the input flip flop is about 0.5ns.

That means that the data is back in the FPGA in about 10.02ns from the clock edge that triggers the SDRAM to present it - so at 100MHz the data is ready just in time for the next clock edge.

This is why why that controller only works between about 80MHz and 120MHz. The logic can work slower but you have to adjust how it captures the returning data to grab it  half a cycle earlier, then a full cycle earlier.

Interesting! I have very little experience with similar timing issues, so forgive me if I ask a stupid question, but how did you derive these timings? I would love to learn more about this.

Again, really appreciate the help

I got them from the FPGA's DC & switching datasheet (http://www.xilinx.com/support/documentation/data_sheets/ds162.pdf), the SDRAM's datasheet, and of course Google for the trace delay (about 6 inches per ns).

The most interesting thing is that the FPGA's tristate signal is relatively slow, and you can't turn around the data bus as quick as you might expect, this  requires an extra cycle here and there to allow enough time.

[nctnico]

For a Spartan 6 you will probably be better off by specifying the timing requirements in the UCF file using the OFFSET = IN ... constraints than trying to decipher the datasheet. There are configurable delays in the IO blocks which can be used to align the data with the clock properly.

[nctnico]

You'll need to constrain the delay from the OBUFDS to the outside clock. The outside signals will be synchronous with this clock + output delay. You can get this value from the datasheet or the timing report. This delay can be used to define a window (before / after) in which the incoming data is valid. Note that the internal DDR clock will need to be driven from a global clock net otherwise the timing will be way off. When using an internal PLL + bufg this will probably be the case but it is worth checking the RTL schematic to see if a BUFG is sourcing the clock signal.