DDR3 Refresh and Micron DDR3 model

[ddr_controller]

I am trying to implement a DDR3 controller for Arty A7 and have found many useful threads in this forum, however, one question I have not been able to solve is refresh timings.

I initialized the DDR and performed a Write and close the row without errors, then I ran the simulation for at least 10ms but the model does not show any error regarding not refreshing during this time. From the docs. I read that on average refresh commands should be sent every 7.8us (page 121).

Do I have to activate the error message with some sort of #define or it just doesn't check for refresh violations?

[BrianHG]

That's the maximum average.  This is the slowest if you wish to evenly space your refreshes.
You only need to refresh the entire chip every 64ms if the chip is set to maximum 85deg C.
Check out the max ^tRFC in the datasheet.  It is actually 70.2ms.

Run your sim for 70.2ms without any refresh commands, and it should spit out an error.

Remember, it is possible to wait 60ms without any refreshing and then burst refresh commands back-back to do the entire chip in one shot without any error.

See Micron 4gb DDR3 data sheet, page 86, table 58, 'Refresh Timing'.
Download this data sheet: Micron DDR3 datasheet
Careful, running the DDR3 at 105 deg C shortens this 64ms time to 16ms meaning you also need to shrink the average refresh period if you run the DDR3 in this mode.

(Doesn't Xilinx have a free DDR3 ram controller?)

[ddr_controller]

Shouldn't it be 70.2us? In table 58 t_RFC MAX = 70,200ns

[BrianHG]

Read the next page, and I did say the max was 70.2 at the top of my post...
I shrunk the time down to 60ms to give you time to do all 8192 back to back refresh commands before the 64ms time runs out.
Actually, with the larger DDR3 chips having a slower refresh, cut my 60ms down to <50ms.

[ddr_controller]

I think I replied before your edit 

About Xilinx free controller: I am trying to learn how a DRAM controller works so I want to build one...

Anyway, I think I just don't understand when and how refreshed are done. I got lost in the documentation, can you point me to the key pages on the docs. where I can find a complete description of how to refresh?

[BrianHG]

If you are learning about writing a DDR3 controller, just read the entire document.
You will find all your answers there as I have.
My DDR3 controller: https://github.com/BrianHGinc/BrianHG-DDR3-Controller
You do not necessarily need to take it as far as I did.

To refresh the entire chip, it must be done 8192 times within 64ms as stated on page 87:

^tREFI @tc<85deg =  7.8us (64ms/8192)

In other words, 8192 refresh commands X 7.8us = 63.9ms.

All that is important is that every 1 of the 8192 refresh commands must repeat within 64ms.  You are permitted to stuff all those 8192 refresh commands all the way at the end of the 64ms so long as the first of the 8192 happens still less than 64ms since that last time that one of 8192 was called and there are 8192 which need to be called since 8192 rows need to be refreshed.

IE: It takes 8192 refresh commands to refresh the entire DDR3 ram chip.

There is a complete section on refresh in the data sheet.

[ddr_controller]

On the note of simulation. I have run 80ms after precharge and no refresh error has popped. Is this the expected behaviour of the Micron ddr3.v?

[BrianHG]

Try opening a closed bank & row and try a read BL8.
Sorry, I do not have time to see what happens with my controller's refresh forced off.

[ddr_controller]

No worries, you are being very helpful! Thanks a lot!

You said that there is an entire section talking about refresh but I only found page 123 and it doesn't give me many clues...

I know that there are multiple ways of refreshing, RAS# based, CAS# (CBR)... But I can't find the description of how each of them works. In my mind, I just imagine that the REFRESH command just refreshes all the banks and that if you want to target specific lines you just open and close that line, but then you have to keep track of the time to know the next refresh.

[asmi]

I know that there are multiple ways of refreshing, RAS# based, CAS# (CBR)... But I can't find the description of how each of them works. In my mind, I just imagine that the REFRESH command just refreshes all the banks and that if you want to target specific lines you just open and close that line, but then you have to keep track of the time to know the next refresh.

No, that's not how it works. What exactly gets refreshed is none of your controller's business, all it needs to do is issue a refresh command every 7.8 us on average, addressing is generated internally, in other words, memory device tracks what is to be refreshed by itself. Here is the relevant section from the datasheet:

The REFRESH command is used during normal operation of the DRAM and is analogous to CAS#-before-RAS# (CBR) refresh or auto refresh. This command is nonpersistent, so it must be issued each time a refresh is required. The addressing is generated by the internal refresh controller. This makes the address bits a “Don’t Care” during a REFRESH command. The DRAM requires REFRESH cycles at an average interval of 7.8μs (maximum when TC ≤ 85°C or 3.9μs maximum when TC ≤ 95°C). The REFRESH period begins when the REFRESH command is registered and ends tRFC (MIN) later.

To allow for improved efficiency in scheduling and switching between tasks, some flexibility in the absolute refresh interval is provided. A maximum of eight REFRESH commands can be posted to any given DRAM, meaning that the maximum absolute interval between any REFRESH command and the next REFRESH command is nine times the maximum average interval refresh rate. Self refresh may be entered with up to eight REFRESH commands being posted. After exiting self refresh (when entered with posted REFRESH commands), additional posting of REFRESH commands is allowed to the extent that the maximum number of cumulative posted REFRESH commands (both preand post-self refresh) does not exceed eight REFRESH commands.

At any given time, a maximum of 16 REFRESH commands can be issued within 2 x tREFI.

[BrianHG]

I know that there are multiple ways of refreshing, RAS# based, CAS# (CBR)... But I can't find the description of how each of them works. In my mind, I just imagine that the REFRESH command just refreshes all the banks and that if you want to target specific lines you just open and close that line, but then you have to keep track of the time to know the next refresh.

The refresh command just refreshes 1 row group (only a few rows can be refreshed at once) in all the banks simultaneously out of 8192 row groups which each bank has.
Internally in the DDR3, every time you send a refresh that row counter increases by 1.

See block diagram figure 5 on page 16 of the Micron data sheet.  You will see the block 'refresh counter' which has 13 bits, hence 8192 possible refresh address positions.  (Note that the other 2 block diagrams has the refresh counter at 16 bits.  I'm not sure what is going on here.  The slower refresh time for the larger ram chips may just be internally incrementing this counter during the longer refresh period to make up for more rows.)

[trossin]

DRAMs do more than refresh in response to a refresh command so they don’t like you to save up more than 8  or 16 ( I can’t remember) before busting them.  There is this ugly thing called row hammer that vendors fixed up by doing some magic during refresh. They also do some internal calibration magic.

Unless you are super worried about latency, there is not much reason to delay refresh. Just stop what you are doing, close your open pages and issue the refresh.  I’m not a fan of open page mode for applications that don’t have much locality as there are a crap load of rows per bank. It takes longer to close the page and open a new one which sucks for throughput.

DDR5 gave us per bank set(same bank of all groups) refresh which is a fun way to cut down on latency as you are not blocking traffic to the entire device.  It complicates your blocking logic though.

Don’t forget to be a nice person and do ZQCal 8 times a second.