BrianHG_DDR3_CONTROLLER open source DDR3 controller. NEW v1.50.

[BrianHG]

*****************************************************
*** NEW December 3, 2021.  BrianHG_DDR3_Controller V1.5 ***
*****************************************************
----------------------------------------------------------------
  A maddening over 11K lines of code.... 
----------------------------------------------------------------

A SystemVerilog DDR3 Memory Controller with up to 16 read/write ports with configurable width priority encoding.  Fully documented source code.

Simplified block diagram of BrianHG_DDR3_Controller V1.5:



(*** Just download the 'BrianHG_DDR3_README_V1.50.txt' and 'BrianHG_DDR3_README_V1.00.txt' for easy reading in a notepad. ***)

Note that the original v1.0 source files still exist, still function, and are backwards compatible, but understand that all the source files have been updated.  This includes the new .sdc files in the new demo projects.

In-depth instructions are located in the full v1.00 text release notes as well as all the parameters and ports are well documented within the source code and examples.


Written by Brian Guralnick.
For public use.
My GitHub repository release:
https://github.com/BrianHGinc/BrianHG-DDR3-Controller

--------------------------------------------------------------------------
BrianHG_DDR3_README_V1.00.txt Status/Revision Log, Instructions.
August 27, 2021.

Designed for Altera/Intel Quartus Cyclone V/10/MAX10 and others. (Unofficial Cyclone III & IV, may require overclocking.)
             Lattice ECP5/LFE5U series.  (Coming soon)
             Xilinx Artix 7 series.      (Coming soon)


*************************************************************
*** Release V1.00, August 27, 2021 **************************
*** Tested on Quartus Prime 20.1   **************************
*************************************************************

(Yes, 400MHz is official for -6 and 300MHz for -8 Cyclones/MAX10, -8 can be safely overclocked to 400MHz.)

Code: [Select]

Featured full Quartus Prime 20.1 projects:  (Except for 'BrianHG_DDR3_CIII_GFX_FMAX_Test_Q13.0sp1')
------------------------------------------
BrianHG_DDR3_DECA_GFX_DEMO                 400MHz, functional DDR3 System scrolling ellipse with optional RS232 debug port demo for Arrow DECA eval board.
BrianHG_DDR3_DECA_Show_1080p               400MHz, functional DDR3 System 1080p 32bit display with optional RS232 debug port demo for Arrow DECA eval board.
BrianHG_DDR3_DECA_RS232_DEBUG_TEST         400MHz, functional DDR3 System RS232 debug port demo for Arrow DECA eval board.
BrianHG_DDR3_DECA_only_PHY_SEQ             400MHz, functional DDR3 PHY Only controller with RS232 debug port demo for Arrow DECA eval board.
BrianHG_DDR3_CIV_GFX_FMAX_Test             400MHz, Hypothetical Cyclone IV DDR3 System scrolling ellipse build to verify FMAX.
BrianHG_DDR3_CIII_GFX_FMAX_Test_Q13.0sp1   400MHz, Hypothetical Cyclone III DDR3 System scrolling ellipse build to verify FMAX.  (Uses Quartus 13.0sp1)
BrianHG_DDR3_CV_GFX_FMAX_Fail              400MHz, Hypothetical Cyclone V-6 DDR3 System scrolling ellipse build to verify FMAX.  (FMAX FAILED)
BrianHG_DDR3_CV_GFX_FMAX_Test              300MHz, Hypothetical Cyclone V-6 DDR3 System scrolling ellipse build to verify FMAX.  (PASSED, but with features disabled)
BrianHG_DDR3_CV_PHY_ONLY_FMAX_Test         375MHz, Hypothetical Cyclone V-6 DDR3 PHY Only controller with RS232 debug port build to verify FMAX. (375MHz only, no multiport)
Source Folders:
---------------
BrianHG_DDR3                               Source code for BrianHG_DDR3 controller.
BrianHG_DDR3_GFX_source                    Source code for rendering random ellipses with a scrolling screen.

Have the SystemVerilog source and can be simulated in Modelsim outside Quartus.


Screenshots folder:
-------------------
LC-LUT_screenshots/                        Contains tables of the compiled LC&LUT usage for various clock frequency and feature builds.
FMAX_screenshots/                          Contains FMAX timing analyzer results screenshots of various FPGA builds.

Check here for compiled FMAX & LC/LUT usage stats:
https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/msg3649318/#msg3649318

[dmendesf]

Nice work. Is there a way to use it with VHDL under Quartus?

[BrianHG]

Though I am not familiar with VHDL, I do know that in either Verilog or VHDL, you can initiate either code type module.

Just google: How do I instantiate a SystemVerilog module inside a VHDL design?

There has got to be good examples.  You just need to find out how to pass 2 dimentional arrays if you will be using my multi-port module unless you make a simple smaller verilog module with only the ports and settings you want shrinking what you call in your VHDL code.

If crossing code is vendor specific, maybe the best place to ask would be on Intel's forum.
I know Intel has instructions on how to insert VHDL code/modules into verilog source code.

[ali_asadzadeh]

Thanks for sharing, Please make a reop on github too

[BrianHG]

Thanks for sharing, Please make a reop on github too

Coming in a few days once I fix the FMAX bottleneck.

[SiliconWizard]

Nice work. Is there a way to use it with VHDL under Quartus?

I don't use Quartus, but otherwise, the answer is usually, yes. Brian's controller is written in SystemVerilog, so first you need to check whether this is properly supported by the Quartus version you're using. If it's recent enough, I guess it is.

Mixing HDLs is usually no problem. You'll just need to define a component interface for the controller in VHDL following the SV interface.

[BrianHG]

SystemVerilog, so first you need to check whether this is properly supported by the Quartus version you're using. If it's recent enough, I guess it is.

My code should work all the way back to QuartusII V9.0 from 2005...
I did design it to run on Cyclone II & III which requires QII V13.x or earlier.

[dmendesf]

I just bought a Deca Max 10 (arrived today, my birthday... Perfect timing and plan to use this code with it, but with VHDL. I'll probably use the latest Quartus unless something requires an older version.

[SiliconWizard]

Question to Brian: you were talking about testing this controller on a Lattice ECP5 IIRC. Did you get to do this? If so, how did that turn out, and how many LUTs does it take?

[BrianHG]

Question to Brian: you were talking about testing this controller on a Lattice ECP5 IIRC. Did you get to do this? If so, how did that turn out, and how many LUTs does it take?

I just need to find a ECP5 board with at least 1 DDR3 ram chip.

The code was designed to be ported to old and new FPGAs alike, however, the capture read data method I used to be compatible across all basic FPGA limits the DDR3 controller to around 500MHz / 1gtps.  Higher than that and it would be recommended to change the read data sampling to using the DQS strobe input as a clock instead of as a latch-enable.

The DDR3 controller alone, 1 read, 1 write port, running a 16bit DDR3 512mb ram chip in Quartus uses:
3480 logic cells in the HDMI out ellipse demo.
512 LUT-Only LCs,
1806 Registers-Only LCs
1166 LUT/Registers LCs

The 3480 number may be inflated since it is connected to the Multiport module, and that one eats a crap-load of registers as it it has independent caches on each port and it's a huge cross-bar matrix.
In the Ellipse demo, it eats another ~3k logic cells.

[SiliconWizard]

Question to Brian: you were talking about testing this controller on a Lattice ECP5 IIRC. Did you get to do this? If so, how did that turn out, and how many LUTs does it take?

I just need to find a ECP5 board with at least 1 DDR3 ram chip.

Ah yes... I don't have one either. The one I have only has SDRAM. And right now, prices have inflated quite a bit, so ECP5 boards with DDR3 are not quite cheap...

The DDR3 controller alone, 1 read, 1 write port, running a 16bit DDR3 512mb ram chip in Quartus uses:
3480 logic cells in the HDMI out ellipse demo.
512 LUT-Only LCs,
1806 Registers-Only LCs
1166 LUT/Registers LCs

The 3480 number may be inflated since it is connected to the Multiport module, and that one eats a crap-load of registers as it it has independent caches on each port and it's a huge cross-bar matrix.
In the Ellipse demo, it eats another ~3k logic cells.

Ok, should give me a rough idea of what to expect. Doesn't look too bad.

[asmi]

Ah yes... I don't have one either. The one I have only has SDRAM. And right now, prices have inflated quite a bit, so ECP5 boards with DDR3 are not quite cheap...

Design you own? It won't be cheap either - at least initially - but it will surely be a lot of fun And if you team up with others, it will help to spread the NRE around, as well as speed up the process.

[BrianHG]

Ah yes... I don't have one either. The one I have only has SDRAM. And right now, prices have inflated quite a bit, so ECP5 boards with DDR3 are not quite cheap...

Design you own? It won't be cheap either - at least initially - but it will surely be a lot of fun And if you team up with others, it will help to spread the NRE around, as well as speed up the process.

I know how this may sound, but from my personal point of view, I would like to first get my controller working on Lattice, then worry about a custom PCB.  For me, having a PCB with a proven functional wired DDR3 setup allowing me to plug and play is a preferred first step.  With the DECA board, this was one thing I did not have to second guess any buggy behavior in my code right at the beginning just powering up the DDR3.  The initial failure of function was that I was using the DDR_IO primitive for Cyclone II/III/IV/V, not the newer primitive used by the MAX10.  Quartus did not complain.  It compiled and even simulated properly both at the logic level and gate level.  Yet, the DDR3 was doing nothing.  Using the Cyclone's DDR_IO primitive meant that nothing was outputting on the data lines in the real MAX10 FPGA, but, the input was still working.  This wasted over a week and if I encountered this problem on my home-made PCB, it might have taken me an extra month to figure out that the DDR_IO primitive which compiled and simulated fine was the culprit.

Lattice tools and FPGAs are new to me, so I do not know what would go wrong where.  An existing eval PCB is a preferred first step removing a piece in the debug equation.

[asmi]

You can always use a trial version of their DDR3 controller to verify the hardware. I would also use it to confirm that the pinout will work.

[SiliconWizard]

I tend to agree with BrianHG here. One thing to debug at a time...
Now I suppose if you're familiar with routing DDR3 stuff, going directly for a custom board should not be a problem. But I'm not. (Now I guess possibly the design could be shared, and someone else could do the routing...)

[asmi]

I tend to agree with BrianHG here. One thing to debug at a time...

Like I said, using a trial version of controller solves the hardware checkout problem. I use this method all the time, albeit with Xilinx devices.

Now I suppose if you're familiar with routing DDR3 stuff, going directly for a custom board should not be a problem. But I'm not. (Now I guess possibly the design could be shared, and someone else could do the routing...)

Or you can do the routing yourself and ask someone to check it once completed, and/or perhaps ask some questions in case DDR3 layout rules are not sufficiently clear. Otherwise it's going to be a classic chicken-and-egg problem when you won't attempt a DDR3 design because you have no experience, but you can't gain experience without actually doing it.

There is nothing particularly difficult about it, especially if you go for a relatively simple design - like a single DDR3 memory device, no ADDR/CTRL termination, and low'ish (as far as DDR3 standard goes) frequency. It's just a handful of rules you've got to follow, and that's pretty much it.

[dmendesf]

Asmi, I feel exactly that about using DDR 3. Do you have a list of layout rules and a description about how DDR3 ? I surely understand SRAM memories and more or less get dynamic memories, but I have no idea what other complications exists for synchronous memories.

[asmi]

Asmi, I feel exactly that about using DDR 3. Do you have a list of layout rules and a description about how DDR3 ? I surely understand SRAM memories and more or less get dynamic memories, but I have no idea what other complications exists for synchronous memories.

DDR3 memory interface consists of a bunch of traces, which are divided into several groups: a single group typically called "Address/control" (or "Command/address/control") which, as the name suggests, consists of address lines (A0-A15, depending on module's capacity), bank address lines (BA0-BA2), and command lines (CKE, RAS, CAS, WE, CS, ODT); and then one or more of "byte lanes" (also called "DQ group"), each one consisting of 8 DQ (data) lines, associated DQS and DQS# lines, and a data mask DM. Attached is a good image from iMX6 datasheet showing example rules for length matching in case of a single x16 DDR3 memory device.

0. Remember that the length matching is about signal length (sometimes also called electrical length) a.k.a. propagation delay, not necessarily physical length! This is important to keep in mind in case you route traces within a group on different layers - signals on outer layers travel faster than on internal ones. Also remember that a portion of via height that the signal is going along also needs to be included - it's typically called via z-length. Not all eCAD tools take this into account, so you have to be on a lookout for these things.
1. The clock line needs to be at least as long as address/control lines are. I typically match it as part of the group, but it can be a bit longer.
2. Traces within address/control lines need to be matched to ±10 ps.
3. All byte lanes has to be no longer than address/command traces.
4. All signals within a single byte lane need to be matched to ±10ps.
5. There is no requirement to match traces of different byte lanes.
6. Traces within differential pair (CK/CK#, DQSn/DQSn#) needs to be matched to ±2ps.

As far as impedance goes, depending on a frequency and a specific controller it can be 50 Ohm or as low as 40 Ohm (the latter is a Xilinx requirement for 7 series FPGAs for DDR3 frequencies above 666 MHz, 50 Ohm is good enough for speeds below that).

If you intend to use several memory devices in your interface, things get more complicated because with DDR3 there are two possible topologies for address/control lines - a balanced tree (like DDR2 and below), and a fly-by (new for DDR3). Technically all DDR3 controllers are supposed to support fly-by topology (which is easier to route), but in reality there are some which don't support it.

Finally, it might be required to implement a termination for address/control lines (DQ, DQS and DM lines don't need one because memory devices have dynamic on-die termination controlled by ODT input). Whether it's required or not can be determined by SI simulations, but typically you can get away without it for a single component which is close to the controller. For multi-chip interfaces you will more likely than not need to implement it.

[BrianHG]

FMAX bottleneck update in the DDR3_PHY_SEQUENCER:  Currently, I'm running individual serial shift timers which are tested before allowing a command request from the 'BrianHG_DDR3_CMD_SEQUENCER', which runs at half DDR_CK clock frequency, to go out.  There are 6 timers for each of the individual 6 main DDR3 commands which are selective reset to the new appropriate values when each new command is sent.  This design has allowed me to request any command at any time and the DDR3 would be controlled as quick as possible.  However, these timers have created a 'nexus' lump where an acknowledgement flag dependent on which command went out needs to get back to the BrianHG_DDR3_CMD_SEQUENCER's cmd out fifo as fast as possible which hits the FMAX as the compiler tries to work out the routing across 2 clock domains.

Currently, 1 test I am doing is to run the timers at half clock frequency on the same clock as the BrianHG_DDR3_CMD_SEQUENCER and pass a 'half-time' delay bit to the DDR3_CK clock command output stage.  This has erased the FMAX problem, and even allowed compiles where I get a legit 400MHz FMAX.  However, a good number of times, where we have delays which only require an ODD number of DDR3_CK clocks, say 5, have averaged up to 6.  This is not good as when using a 300MHz controller, every clock is precious.  I'm currently trying to debug and eliminate this 1 odd clock cycle penalty.  Once done, a beta version 0.95 will be uploaded where 350MHz should be easily achievable & 400MHz with compiler effort turned up to max and careful interfacing with my controller.

It is too bad the original code had this 1 problem as the solution was sweet without any coding hacks, though it was problematic to get Quartus to properly hit a >300MHz core with tc of 85 degrees.  Unless an idea spark comes in on how to solve the 'cmd_ack' bottleneck with the current design, I will once again have to muddy clean code for work-around solutions.

[BrianHG]

New release:
**********************************
Beta Release V0.95, August 4, 2021.
**********************************

Has now been uploaded at the top of this thread:
https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/
Changes are written there and the history is in the README file.
Please make sure you are downloading the _V0.95 as I kept the earlier revisions available for download as well.


Utilization report DDR3 controller inside the HDMI out ellipse demo:

Old V0.9.
3480  logic cells in the HDMI out ellipse demo,
512    LUT-Only LCs,
1806  Registers-Only LCs,
1166  LUT/Registers LCs.

New V0.95.
3100  logic cells in the HDMI out ellipse demo,
478    LUT-Only LCs,
1826  Registers-Only LCs,
796    LUT/Registers LCs.

New V0.95 True Stand-alone DDR3_PHY_SEQ.sv DDR3 controller with 128bit read and write port.
2082  logic cells,
515    LUT-Only LCs,
984    Registers-Only LCs,
583    LUT/Registers LCs.

[BrianHG]

A minor issue has bee found with 'DDR3_PHY_SEQ.sv' V0.95 when it swaps ram banks.  It just occasionally delays the next read or write command by 2 DDR_CK clocks.  This is minor as there is no data corruption and you may revert to the V0.90 if you truly need to.

I'm working on it now with a major improvement in FMAX and again a shrinkage of used logic cells.  It should be out in a day or 2, so, I wont release an intermediate patch.

[BrianHG]

Update:  Just cleaned up the main BrianHG_DDR3_PHY_SEQ.sv controller and sequencer.

All FMAX limitations and cross clock domain problems have been cleaned out.  Easily achieved proper 350MHz controller and 400MHz with only a few cross-clock domain signals not making the cut @ TC85 degrees by less the 0.070ns, yet it runs clean.  (The signals being some of the OEs for the write data, however, my codes is programmed to turn on the OE 1 clock cycle early, and turn it off 1 clock cycle late meaning an error here will not occur.  The core and all data + IO paths clear the timing analysis fine above 400MHz.)  The logic cell and LUT count has also shrunk.  The change in code has generated what appears as a minor occasional 1 DDR_CK clock delay occasionally when bursting after the first BL8, however, what is going on is if a read or write burst begins on an odd DDR_CK phase compared to it's half-clock interface clock, or the tRCD, tCAS happens to be an odd number, an alignment to the phase of the burst size of 8 may realign to the even matched phase after the first BL8 burst.  The old full speed V0.9 controller stacked a few additional commands in advance and would retain the 'ODD' alignments generating an unbroken consecutive burst making the appearance of a tighter ram controller by packing everything back-to-back.  After the release of version 1.0, I will see if there is a cheap method of packing the commands in this way once again without having to move the timers back to the full speed DDR_CK clock rate.

The one issue remaining is the multi-port handler 'BrianHG_DDR3_COMMANDER.sv'.  It's current FMAX has trouble passing 130MHz if it is configured with 2x128 bit read, 2x128bit write ports, all smart options enabled @ TC85 degrees.  I'm thinking of a way to get this one to compile robustly above 150MHz without any big compromise so you may at least use it with ram running at 300MHz and half rate.  Right now, it can work fine at 75MHz, or even at 100MHz with the ram at 400MHz.  (This means no timing violations at any temperature.  The 130MHz builds always work error free at 150MHz, but this is not what we want.)  If I cannot come up with a solution, I will leave it as it is and create a secondary FAST multi-port handler 'BrianHG_DDR3_COMMANDER_fast.sv' which will be a strict 2 read, 2 write port device targeting a 200MHz FMAX allowing half-rate support with 400MHz ram.  This multiport will be designed to be chain-able where you can use 3 of them to give you 4 read, 4 write ports, or 7 of them for 8 read, 8 write ports.

Note that the Altera's Cyclone & MAX FPGA fabrics are really slow and low power, my current code would probably be at least 50% faster on other vendor's FPGAs, or even Altera's Arria/Stratix FPGAs.  Achieving a full consistent 400MHz core builds without the paid version of Quartus for manually placing cells, even though that portion is truly only software serializers & IO port controller is still difficult as it still has to be controlled by a 200MHz section and bridge the 2 clock domains in both directions.

[Omega Glory]

Have you thought about hosting the code on GitHub? I bet a lot of people would find your work very helpful, and that would be an easy and free way to distribute it more widely than on this forum. In any case, amazing work!

[BrianHG]

Have you thought about hosting the code on GitHub? I bet a lot of people would find your work very helpful, and that would be an easy and free way to distribute it more widely than on this forum. In any case, amazing work!

We'll see in a few days after I release v1.00.
I never used GitHub before and would probably need to venture around first.

[BrianHG]

Get the new BrianHG_DDR3_Controller_V1.5 demo over here: https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/msg3785711/#msg3785711

          
  500MHz/1GTPS
          

Error free, well, on my DECA board anyways...
So much for Altera's software DDR3 300MHz limit.
Though, the reported FMAX at 0C reads only 461MHz.

If you download & program the attached ellipse-demo, scoping the SMD termination resistor tied to the DDR_CK line should show 500MHz.

Arrow DECA DEMO .sof programming file instructions:
(If the picture is still or scrolling noise, just flip 'Switch 0'.  You just powered up the demo in frozen picture mode and you are looking at the powered up random blank memory.)


Switch 0 = Enable/Disable drawing of ellipses.
Switch 1 = Enable/Disable screen scrolling.
Button 0 = Draw data from random noise generator.
Button 1 = Draw color image data from a binary counter.


Note: V3 just fixes a reset bug when using the RS232-Debugger.  Nothing else has changed.