BrianHG_DDR3_CONTROLLER open source DDR3 controller. NEW v1.60.

[BrianHG]

*****************************************************
*** NEW June 11, 2021.  BrianHG_DDR3_Controller V1.6 ***
*****************************************************
----------------------------------------------------------------
  A maddening over fnK lines of code.... 
----------------------------------------------------------------
My Github has the full latest v1.6 release package:
https://github.com/BrianHGinc/BrianHG-DDR3-Controller
https://github.com/BrianHGinc/BrianHG-DDR3-Controller/archive/refs/tags/v1.60.zip
-----------------------------------------------------
BrianHG_DDR3_Controller V1.6 Release, June 11, 2022.
Includes new BrianHG_GFX_VGA_Window_System.
-----------------------------------------------------
(Control ports are the same as v1.5)


Folder BrianHG_DDR3 now contains the new v1.6 controller.
Main source files:

Code: [Select]

- BrianHG_DDR3_v15_and_v16_Block_Diagram.png -> Illustration of module connections.

 - Includes these following sub-modules :
   - BrianHG_DDR3_CONTROLLER_v16_top.sv     -> v1.6 TOP entry to the complete project which wires the DDR3_COMMANDER_v16 to the DDR3_PHY_SEQ giving you access to all the read/write ports + access to the DDR3 IO pins.
   - BrianHG_DDR3_COMMANDER_v16.sv          -> v1.6 High FMAX speed multi-port read and write requests and cache, commands the BrianHG_DDR3_PHY_SEQ.sv sequencer.
   - BrianHG_DDR3_CMD_SEQUENCER_v16.sv      -> v1.6 Takes in the read and write requests, generates a stream of DDR3 commands to execute the read and writes.
   - BrianHG_DDR3_PHY_SEQ_v16.sv            -> v1.6 DDR3 PHY sequencer.          (If you want just a compact DDR3 controller, skip the DDR3_CONTROLLER_top & DDR3_COMMANDER and just use this module alone.)
   - BrianHG_DDR3_PLL.sv                    -> Generates the system clocks. (*** Currently Altera/Intel only ***)
   - BrianHG_DDR3_GEN_tCK.sv                -> Generates all the tCK count clock cycles for the DDR3_PHY_SEQ so that the DDR3 clock cycle requirements are met.
   - BrianHG_DDR3_FIFOs.sv                  -> Serial shifting logic FIFOs.

 - Includes the following test-benches :
   - BrianHG_DDR3_CONTROLLER_v16_top_tb.sv  -> Test the entire 'BrianHG_DDR3_CONTROLLER_v16_top.sv' system with Mircon's DDR3 Verilog model.
   - BrianHG_DDR3_COMMANDER_v16_tb.sv       -> Test just the commander_v16.  The 'DDR3_PHY_SEQ' is dummy simulated.  (*** This one will simulate on any vendor's ModelSim ***)
   - BrianHG_DDR3_CMD_SEQUENCER_v16_tb.sv   -> Test just the DDR3 command sequencer.                                 (*** This one will simulate on any vendor's ModelSim ***)
   - BrianHG_DDR3_PHY_SEQ_v16_tb.sv         -> Test just the DDR3 PHY sequencer with Mircon's DDR3 Verilog model providing logged DDR3 command results with any access violations listed.
   - BrianHG_DDR3_PLL_tb.sv                 -> Test just the PLL module.

 - IO port vendor specific modules :
   - BrianHG_DDR3_IO_PORT_ALTERA.sv         -> Physical DDR IO pin driver specifically for Altera/Intel Cyclone III/IV/V and MAX10.

 - Modelsim 'do' script files.
   - All setup_xxx.do files setup their associated Modelsim simulation.
   - All run_xxx.do   files quick re-compile and run their associated Modelsim simulation.


Folder 'BrianHG_DDR3_GFX_source_v16' contains my new BrianHG_GFX_VGA_Window_System multi-window system.
Main source files:

 - BrianHG_GFX_VGA_Window_System.pdf          -> Visual block diagram for the graphics system and layer-swapping illustration.
 - BrianHG_GFX_VGA_Window_System.txt          -> Full documentation for the VGA window system.

 - Includes these top hierarchy files:
   - BrianHG_GFX_VGA_Window_System.sv           -> Full window system where you drive the CMD_win_xxx controls via input ports.
   - BrianHG_GFX_VGA_Window_System_DDR3_REGS.sv -> Full window system where you drive the CMD_win_xxx controls via writing to DDR3 memory addresses through any multiport.

 - Modelsim 'do' script files.
   - All setup_xxx.do files setup their associated Modelsim simulation.
   - All run_xxx.do   files quick re-compile and run their associated Modelsim simulation.


New Arrow DECA board demo complete projects running the v1.6 BrianHG_DDR3_Controller conected to
the BrianHG_GFX_VGA_Window_System, all at 400MHz, all 100% timing requirements met.
Source folders:

 - BrianHG_DDR3_DECA_GFX_DEMO_v16_1_LAYER     -> Replaces the original ellipse demo, but now uses my new BrianHG_GFX_VGA_Window_System.
 - BrianHG_DDR3_DECA_GFX_DEMO_v16_2_LAYERS    -> Improved ellipse demo using 2 translucent windows scrolling at different speeds.
 - BrianHG_DDR3_DECA_GFX_HWREGS_v16_16_LAYERS -> Example 16 window layer system where writes to the DDR3 controls the window's regs.
 - BrianHG_DDR3_DECA_RS232_DEBUG_TEST_v16     -> Single port DDR3 controller example connected to my RS232 debugger.
 - BrianHG_DDR3_DECA_PHY_SEQ_only_v16         -> (No multiport controller.) Bare minimum DDR3 PHY_SEQ controller connected to my RS232 debugger.

Test hypothetical builds for Cyclone III,IV,V to see if we can meet FMAX.
 - BrianHG_DDR3_CIII_GFX_TEST_v16_1_LAYER_Q13.0sp1  -> Cyclone III example using Quartus 13.0 sp1.
 - BrianHG_DDR3_CIV_GFX_TEST_v16_1_LAYER            -> Cyclone IV example using Quartus 20.1.
 - BrianHG_DDR3_CV_GFX_TEST_v16_1_LAYER_350MHz      -> Cyclone V example running only at 350MHz using Quartus 20.1.


- Get new 2 window layer ellipse demo here:
https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/msg4230856/#msg4230856

- Get new VGA video system demo configured for up to 16 window layers driven by my RS232_Debugger here:
https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/msg4233016/#msg4233016


- User 'davemuscle' created an Avalon interface wrapper comparing my DDR3 PHY_SEQ_only free controller to Altera's expensive UniPHY IP, see here:
https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/msg4108963/#msg4108963
(Note that both controllers should have achieve double the bandwidth and my 'BrianHG_DDR3_DECA_GFX_DEMO_v16_2_LAYERS' demo already runs at double the efficiency.)

- User 'Nockieboy' has been working on integrating my controller with his Z80 8-bit GPU project, see his first multi-layer-window test here:
https://www.eevblog.com/forum/fpga/fpga-vga-controller-for-8-bit-computer/msg3980567/#msg3980567
and tile mode test:
https://www.eevblog.com/forum/fpga/fpga-vga-controller-for-8-bit-computer/msg4029019/#msg4029019



Check here for older compiled FMAX & LC/LUT usage stats:
https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/msg3649318/#msg3649318

[dmendesf]

Nice work. Is there a way to use it with VHDL under Quartus?

[BrianHG]

Though I am not familiar with VHDL, I do know that in either Verilog or VHDL, you can initiate either code type module.

Just google: How do I instantiate a SystemVerilog module inside a VHDL design?

There has got to be good examples.  You just need to find out how to pass 2 dimentional arrays if you will be using my multi-port module unless you make a simple smaller verilog module with only the ports and settings you want shrinking what you call in your VHDL code.

If crossing code is vendor specific, maybe the best place to ask would be on Intel's forum.
I know Intel has instructions on how to insert VHDL code/modules into verilog source code.

[ali_asadzadeh]

Thanks for sharing, Please make a reop on github too

[BrianHG]

Thanks for sharing, Please make a reop on github too

Coming in a few days once I fix the FMAX bottleneck.

[SiliconWizard]

Nice work. Is there a way to use it with VHDL under Quartus?

I don't use Quartus, but otherwise, the answer is usually, yes. Brian's controller is written in SystemVerilog, so first you need to check whether this is properly supported by the Quartus version you're using. If it's recent enough, I guess it is.

Mixing HDLs is usually no problem. You'll just need to define a component interface for the controller in VHDL following the SV interface.

[BrianHG]

SystemVerilog, so first you need to check whether this is properly supported by the Quartus version you're using. If it's recent enough, I guess it is.

My code should work all the way back to QuartusII V9.0 from 2005...
I did design it to run on Cyclone II & III which requires QII V13.x or earlier.

[dmendesf]

I just bought a Deca Max 10 (arrived today, my birthday... Perfect timing and plan to use this code with it, but with VHDL. I'll probably use the latest Quartus unless something requires an older version.

[SiliconWizard]

Question to Brian: you were talking about testing this controller on a Lattice ECP5 IIRC. Did you get to do this? If so, how did that turn out, and how many LUTs does it take?

[BrianHG]

Question to Brian: you were talking about testing this controller on a Lattice ECP5 IIRC. Did you get to do this? If so, how did that turn out, and how many LUTs does it take?

I just need to find a ECP5 board with at least 1 DDR3 ram chip.

The code was designed to be ported to old and new FPGAs alike, however, the capture read data method I used to be compatible across all basic FPGA limits the DDR3 controller to around 500MHz / 1gtps.  Higher than that and it would be recommended to change the read data sampling to using the DQS strobe input as a clock instead of as a latch-enable.

The DDR3 controller alone, 1 read, 1 write port, running a 16bit DDR3 512mb ram chip in Quartus uses:
3480 logic cells in the HDMI out ellipse demo.
512 LUT-Only LCs,
1806 Registers-Only LCs
1166 LUT/Registers LCs

The 3480 number may be inflated since it is connected to the Multiport module, and that one eats a crap-load of registers as it it has independent caches on each port and it's a huge cross-bar matrix.
In the Ellipse demo, it eats another ~3k logic cells.

[SiliconWizard]

Question to Brian: you were talking about testing this controller on a Lattice ECP5 IIRC. Did you get to do this? If so, how did that turn out, and how many LUTs does it take?

I just need to find a ECP5 board with at least 1 DDR3 ram chip.

Ah yes... I don't have one either. The one I have only has SDRAM. And right now, prices have inflated quite a bit, so ECP5 boards with DDR3 are not quite cheap...

The DDR3 controller alone, 1 read, 1 write port, running a 16bit DDR3 512mb ram chip in Quartus uses:
3480 logic cells in the HDMI out ellipse demo.
512 LUT-Only LCs,
1806 Registers-Only LCs
1166 LUT/Registers LCs

The 3480 number may be inflated since it is connected to the Multiport module, and that one eats a crap-load of registers as it it has independent caches on each port and it's a huge cross-bar matrix.
In the Ellipse demo, it eats another ~3k logic cells.

Ok, should give me a rough idea of what to expect. Doesn't look too bad.

[asmi]

Ah yes... I don't have one either. The one I have only has SDRAM. And right now, prices have inflated quite a bit, so ECP5 boards with DDR3 are not quite cheap...

Design you own? It won't be cheap either - at least initially - but it will surely be a lot of fun And if you team up with others, it will help to spread the NRE around, as well as speed up the process.

[BrianHG]

Ah yes... I don't have one either. The one I have only has SDRAM. And right now, prices have inflated quite a bit, so ECP5 boards with DDR3 are not quite cheap...

Design you own? It won't be cheap either - at least initially - but it will surely be a lot of fun And if you team up with others, it will help to spread the NRE around, as well as speed up the process.

I know how this may sound, but from my personal point of view, I would like to first get my controller working on Lattice, then worry about a custom PCB.  For me, having a PCB with a proven functional wired DDR3 setup allowing me to plug and play is a preferred first step.  With the DECA board, this was one thing I did not have to second guess any buggy behavior in my code right at the beginning just powering up the DDR3.  The initial failure of function was that I was using the DDR_IO primitive for Cyclone II/III/IV/V, not the newer primitive used by the MAX10.  Quartus did not complain.  It compiled and even simulated properly both at the logic level and gate level.  Yet, the DDR3 was doing nothing.  Using the Cyclone's DDR_IO primitive meant that nothing was outputting on the data lines in the real MAX10 FPGA, but, the input was still working.  This wasted over a week and if I encountered this problem on my home-made PCB, it might have taken me an extra month to figure out that the DDR_IO primitive which compiled and simulated fine was the culprit.

Lattice tools and FPGAs are new to me, so I do not know what would go wrong where.  An existing eval PCB is a preferred first step removing a piece in the debug equation.

[asmi]

You can always use a trial version of their DDR3 controller to verify the hardware. I would also use it to confirm that the pinout will work.

[SiliconWizard]

I tend to agree with BrianHG here. One thing to debug at a time...
Now I suppose if you're familiar with routing DDR3 stuff, going directly for a custom board should not be a problem. But I'm not. (Now I guess possibly the design could be shared, and someone else could do the routing...)

[asmi]

I tend to agree with BrianHG here. One thing to debug at a time...

Like I said, using a trial version of controller solves the hardware checkout problem. I use this method all the time, albeit with Xilinx devices.

Now I suppose if you're familiar with routing DDR3 stuff, going directly for a custom board should not be a problem. But I'm not. (Now I guess possibly the design could be shared, and someone else could do the routing...)

Or you can do the routing yourself and ask someone to check it once completed, and/or perhaps ask some questions in case DDR3 layout rules are not sufficiently clear. Otherwise it's going to be a classic chicken-and-egg problem when you won't attempt a DDR3 design because you have no experience, but you can't gain experience without actually doing it.

There is nothing particularly difficult about it, especially if you go for a relatively simple design - like a single DDR3 memory device, no ADDR/CTRL termination, and low'ish (as far as DDR3 standard goes) frequency. It's just a handful of rules you've got to follow, and that's pretty much it.

[dmendesf]

Asmi, I feel exactly that about using DDR 3. Do you have a list of layout rules and a description about how DDR3 ? I surely understand SRAM memories and more or less get dynamic memories, but I have no idea what other complications exists for synchronous memories.

[asmi]

Asmi, I feel exactly that about using DDR 3. Do you have a list of layout rules and a description about how DDR3 ? I surely understand SRAM memories and more or less get dynamic memories, but I have no idea what other complications exists for synchronous memories.

DDR3 memory interface consists of a bunch of traces, which are divided into several groups: a single group typically called "Address/control" (or "Command/address/control") which, as the name suggests, consists of address lines (A0-A15, depending on module's capacity), bank address lines (BA0-BA2), and command lines (CKE, RAS, CAS, WE, CS, ODT); and then one or more of "byte lanes" (also called "DQ group"), each one consisting of 8 DQ (data) lines, associated DQS and DQS# lines, and a data mask DM. Attached is a good image from iMX6 datasheet showing example rules for length matching in case of a single x16 DDR3 memory device.

0. Remember that the length matching is about signal length (sometimes also called electrical length) a.k.a. propagation delay, not necessarily physical length! This is important to keep in mind in case you route traces within a group on different layers - signals on outer layers travel faster than on internal ones. Also remember that a portion of via height that the signal is going along also needs to be included - it's typically called via z-length. Not all eCAD tools take this into account, so you have to be on a lookout for these things.
1. The clock line needs to be at least as long as address/control lines are. I typically match it as part of the group, but it can be a bit longer.
2. Traces within address/control lines need to be matched to ±10 ps.
3. All byte lanes has to be no longer than address/command traces.
4. All signals within a single byte lane need to be matched to ±10ps.
5. There is no requirement to match traces of different byte lanes.
6. Traces within differential pair (CK/CK#, DQSn/DQSn#) needs to be matched to ±2ps.

As far as impedance goes, depending on a frequency and a specific controller it can be 50 Ohm or as low as 40 Ohm (the latter is a Xilinx requirement for 7 series FPGAs for DDR3 frequencies above 666 MHz, 50 Ohm is good enough for speeds below that).

If you intend to use several memory devices in your interface, things get more complicated because with DDR3 there are two possible topologies for address/control lines - a balanced tree (like DDR2 and below), and a fly-by (new for DDR3). Technically all DDR3 controllers are supposed to support fly-by topology (which is easier to route), but in reality there are some which don't support it.

Finally, it might be required to implement a termination for address/control lines (DQ, DQS and DM lines don't need one because memory devices have dynamic on-die termination controlled by ODT input). Whether it's required or not can be determined by SI simulations, but typically you can get away without it for a single component which is close to the controller. For multi-chip interfaces you will more likely than not need to implement it.

[BrianHG]

FMAX bottleneck update in the DDR3_PHY_SEQUENCER:  Currently, I'm running individual serial shift timers which are tested before allowing a command request from the 'BrianHG_DDR3_CMD_SEQUENCER', which runs at half DDR_CK clock frequency, to go out.  There are 6 timers for each of the individual 6 main DDR3 commands which are selective reset to the new appropriate values when each new command is sent.  This design has allowed me to request any command at any time and the DDR3 would be controlled as quick as possible.  However, these timers have created a 'nexus' lump where an acknowledgement flag dependent on which command went out needs to get back to the BrianHG_DDR3_CMD_SEQUENCER's cmd out fifo as fast as possible which hits the FMAX as the compiler tries to work out the routing across 2 clock domains.

Currently, 1 test I am doing is to run the timers at half clock frequency on the same clock as the BrianHG_DDR3_CMD_SEQUENCER and pass a 'half-time' delay bit to the DDR3_CK clock command output stage.  This has erased the FMAX problem, and even allowed compiles where I get a legit 400MHz FMAX.  However, a good number of times, where we have delays which only require an ODD number of DDR3_CK clocks, say 5, have averaged up to 6.  This is not good as when using a 300MHz controller, every clock is precious.  I'm currently trying to debug and eliminate this 1 odd clock cycle penalty.  Once done, a beta version 0.95 will be uploaded where 350MHz should be easily achievable & 400MHz with compiler effort turned up to max and careful interfacing with my controller.

It is too bad the original code had this 1 problem as the solution was sweet without any coding hacks, though it was problematic to get Quartus to properly hit a >300MHz core with tc of 85 degrees.  Unless an idea spark comes in on how to solve the 'cmd_ack' bottleneck with the current design, I will once again have to muddy clean code for work-around solutions.

[BrianHG]

New release:
**********************************
Beta Release V0.95, August 4, 2021.
**********************************

Has now been uploaded at the top of this thread:
https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/
Changes are written there and the history is in the README file.
Please make sure you are downloading the _V0.95 as I kept the earlier revisions available for download as well.


Utilization report DDR3 controller inside the HDMI out ellipse demo:

Old V0.9.
3480  logic cells in the HDMI out ellipse demo,
512    LUT-Only LCs,
1806  Registers-Only LCs,
1166  LUT/Registers LCs.

New V0.95.
3100  logic cells in the HDMI out ellipse demo,
478    LUT-Only LCs,
1826  Registers-Only LCs,
796    LUT/Registers LCs.

New V0.95 True Stand-alone DDR3_PHY_SEQ.sv DDR3 controller with 128bit read and write port.
2082  logic cells,
515    LUT-Only LCs,
984    Registers-Only LCs,
583    LUT/Registers LCs.

[BrianHG]

A minor issue has bee found with 'DDR3_PHY_SEQ.sv' V0.95 when it swaps ram banks.  It just occasionally delays the next read or write command by 2 DDR_CK clocks.  This is minor as there is no data corruption and you may revert to the V0.90 if you truly need to.

I'm working on it now with a major improvement in FMAX and again a shrinkage of used logic cells.  It should be out in a day or 2, so, I wont release an intermediate patch.

[BrianHG]

Update:  Just cleaned up the main BrianHG_DDR3_PHY_SEQ.sv controller and sequencer.

All FMAX limitations and cross clock domain problems have been cleaned out.  Easily achieved proper 350MHz controller and 400MHz with only a few cross-clock domain signals not making the cut @ TC85 degrees by less the 0.070ns, yet it runs clean.  (The signals being some of the OEs for the write data, however, my codes is programmed to turn on the OE 1 clock cycle early, and turn it off 1 clock cycle late meaning an error here will not occur.  The core and all data + IO paths clear the timing analysis fine above 400MHz.)  The logic cell and LUT count has also shrunk.  The change in code has generated what appears as a minor occasional 1 DDR_CK clock delay occasionally when bursting after the first BL8, however, what is going on is if a read or write burst begins on an odd DDR_CK phase compared to it's half-clock interface clock, or the tRCD, tCAS happens to be an odd number, an alignment to the phase of the burst size of 8 may realign to the even matched phase after the first BL8 burst.  The old full speed V0.9 controller stacked a few additional commands in advance and would retain the 'ODD' alignments generating an unbroken consecutive burst making the appearance of a tighter ram controller by packing everything back-to-back.  After the release of version 1.0, I will see if there is a cheap method of packing the commands in this way once again without having to move the timers back to the full speed DDR_CK clock rate.

The one issue remaining is the multi-port handler 'BrianHG_DDR3_COMMANDER.sv'.  It's current FMAX has trouble passing 130MHz if it is configured with 2x128 bit read, 2x128bit write ports, all smart options enabled @ TC85 degrees.  I'm thinking of a way to get this one to compile robustly above 150MHz without any big compromise so you may at least use it with ram running at 300MHz and half rate.  Right now, it can work fine at 75MHz, or even at 100MHz with the ram at 400MHz.  (This means no timing violations at any temperature.  The 130MHz builds always work error free at 150MHz, but this is not what we want.)  If I cannot come up with a solution, I will leave it as it is and create a secondary FAST multi-port handler 'BrianHG_DDR3_COMMANDER_fast.sv' which will be a strict 2 read, 2 write port device targeting a 200MHz FMAX allowing half-rate support with 400MHz ram.  This multiport will be designed to be chain-able where you can use 3 of them to give you 4 read, 4 write ports, or 7 of them for 8 read, 8 write ports.

Note that the Altera's Cyclone & MAX FPGA fabrics are really slow and low power, my current code would probably be at least 50% faster on other vendor's FPGAs, or even Altera's Arria/Stratix FPGAs.  Achieving a full consistent 400MHz core builds without the paid version of Quartus for manually placing cells, even though that portion is truly only software serializers & IO port controller is still difficult as it still has to be controlled by a 200MHz section and bridge the 2 clock domains in both directions.

[Omega Glory]

Have you thought about hosting the code on GitHub? I bet a lot of people would find your work very helpful, and that would be an easy and free way to distribute it more widely than on this forum. In any case, amazing work!

[BrianHG]

Have you thought about hosting the code on GitHub? I bet a lot of people would find your work very helpful, and that would be an easy and free way to distribute it more widely than on this forum. In any case, amazing work!

We'll see in a few days after I release v1.00.
I never used GitHub before and would probably need to venture around first.

[BrianHG]

Get the new BrianHG_DDR3_Controller_V1.5 demo over here: https://www.eevblog.com/forum/fpga/brianhg_ddr3_controller-open-source-ddr3-controller/msg3785711/#msg3785711

          
  500MHz/1GTPS
          

Error free, well, on my DECA board anyways...
So much for Altera's software DDR3 300MHz limit.
Though, the reported FMAX at 0C reads only 461MHz.

If you download & program the attached ellipse-demo, scoping the SMD termination resistor tied to the DDR_CK line should show 500MHz.

Arrow DECA DEMO .sof programming file instructions:
(If the picture is still or scrolling noise, just flip 'Switch 0'.  You just powered up the demo in frozen picture mode and you are looking at the powered up random blank memory.)


Switch 0 = Enable/Disable drawing of ellipses.
Switch 1 = Enable/Disable screen scrolling.
Button 0 = Draw data from random noise generator.
Button 1 = Draw color image data from a binary counter.


Note: V3 just fixes a reset bug when using the RS232-Debugger.  Nothing else has changed.