OpenHBMC. Open-source AXI4-based HyperBus memory controller

[VGN]

Hello everyone!

I'm developing a high performance AXI4-based HyperBus memory controller for Xilinx 7-series FPGAs.
This memory controller is a small part of my another project: https://www.eevblog.com/forum/thermal-imaging/openirv-isc0901b0-(autoliv-nv3-flir-e4568)-based-opensource-thermal-camera/
Anyway, I thought that, probably, it is worth to publish it as a separate project.

This IP-core is packed for Vivado 2020.1 for easy block design integration, though you can use raw HDL files.

This is a first release. I haven't tested it well, but could successfully pass continues memory test at ~162MHz, i.e. 325MB/s on a custom devboad with Spartan-7 and a single HyperRAM. I'm going to beat the level of 200MHz (400MB/s), as soon as I get a board with HyperRAM capable 200MHz, as right now, the memory I have is limited by 166MHz. Soon I'm going to test long burst transfers on hardware, a DMA is needed for this, as Microblaze cannot initate long burst transfers.

Resource utilization: 565 LUT, 678 FF, 1.5 BRAM (1.5 x RAMB36E1 = 3 x RAMB18E1)

Feel free to ask question, criticize the design, report bugs and donate if you like this IP-core)

Link to repo: https://github.com/OVGN/OpenHBMC

[KE5FX]

Good stuff.    I found it surprisingly hard to get my own HyperRAM state machine working.  The data sheets are pretty awful and the interface itself is more complicated than it has any right to be.  (What's up with all the fixed-versus-variable latency cruft, for instance... why can't I just poll RWDS to find out when the device is ready to read or write...?)

[asmi]

Good stuff.    I found it surprisingly hard to get my own HyperRAM state machine working.  The data sheets are pretty awful and the interface itself is more complicated than it has any right to be.  (What's up with all the fixed-versus-variable latency cruft, for instance... why can't I just poll RWDS to find out when the device is ready to read or write...?)

I found it very easy and documentation very good and helpful. Had no problems implementing interface. As for latency, by default the memory starts up in fixed double-latency mode, so you don't have to worry about it at all. This is also the only possible mode for dual-die 128M chips. The option to turn on variable latency is there in case you want to trade lower latency for increased controller complexity.
Fixed latency is also great when you want to scale your memory interface horizontally by using multiple chips in parallel lockstep, so effectively getting a multiple of bandwidth (2x for using 2 chips, 4x for 4, etc). This is necessary for example when you want to use these chips as a frame buffer - 720p@60Hz requires 1280*720*4*60~211 MB/s of bandwidth, and you will need at least double of that (so that you can read and write at the same time). In my design I used a pair of chips in parallel for 720p, for 1080p@60Hz (~475 MB/s) you will want 3 or 4 chips.
That said, I gotta say I prefer using LPDDR1 modules for smaller designs when using MIG is not an option due to its' rather high resource requirements. These chips are available in variety of capacities from 128Mbit to 2Gbit with x16 or x32 data bus, can go up to 200 MHz DDR at CL3, and it's protocol is quite simple to implement (though not as simple as hyperbus).

[VGN]

Good stuff.    I found it surprisingly hard to get my own HyperRAM state machine working.  The data sheets are pretty awful and the interface itself is more complicated than it has any right to be.  (What's up with all the fixed-versus-variable latency cruft, for instance... why can't I just poll RWDS to find out when the device is ready to read or write...?)

Thanks! Yes, the interface is a bit complicated, but not as much for me. Ha-ha, polling the RWDS to find out when the device is ready is the next generation DRAM, we should wait for this another 10 years))

Had no problems implementing interface.

It is very interested how did you solve the problem of data transfer from RWDS clock domain to internal FPGA clock?

[asmi]

It is very interested how did you solve the problem of data transfer from RWDS clock domain to internal FPGA clock?

I used ISERDES in memory mode and fed rwds clock to CLK/CBLB pins of SERDES, while feeding free-running clocks from MCMM to OCLK/OCLKB/CLKDIV pins as described in UG471. I also considered using IN_FIFO to help close timing (it can act as 1:2 deserializer as well as regular FIFO, which is how it's used for DDR2/3 controllers), but for -2 device and 166 MHz this proved unnecessary. With this arrangement (no IN_FIFO) I was able to close timing at 200 MHz DDR for LPDDR1 x16 controller for -2 device by carefully choosing which IO pins to use for dq and dqs signals, but I suspect that for -1 device those FIFOs might be necessary. So I think it shouldn't be very hard to get new 200 MHz HB2 chips to work. Also note that my design used 2 64Mbit chips in parallel, so I had 2 parallel HB buses and so 2 separate dqs clocks.
I love how you went straight to the crux of the problem, it shows that you invested quite a bit of time getting this to work! Commendable

[KE5FX]

Ha-ha, polling the RWDS to find out when the device is ready is the next generation DRAM, we should wait for this another 10 years

My impression is that they could have done just that if they hadn't been forced to overload RWDS as a byte mask for writes.  In the simplest case, where I write a single 16-bit word, I ended up with this goofy logic:

   

Code: [Select]

RAM_M_WR:
            begin
               case (RAM_edge_cnt)
                  0:  RAMDQ_OUT_reg <= RAM_CA_words[0];   // Write addr [47:40] prior to first rising edge
                  1:  RAMDQ_OUT_reg <= RAM_CA_words[1];   // 5 ns after first rising edge (and 5 ns prior to first falling edge): write addr[39:32]
                  2:  RAMDQ_OUT_reg <= RAM_CA_words[2];   // 5 ns after first falling edge: write addr[31:24]
                  3:  RAMDQ_OUT_reg <= RAM_CA_words[3];   // 5 ns after second rising edge: write addr[23:16]
                  4:  RAMDQ_OUT_reg <= RAM_CA_words[4];   // 6-clock tACC initial latency period began 5 ns ago at edge 4, after row address is known   
                  5:  RAMDQ_OUT_reg <= RAM_CA_words[5];   // This period is 6 cycles long, ending at edge 16 when the fixed additional latency period begins
                  27: RAMRWDS_Z_reg <= 1'b0;              // Write-enable RWDS to unmask byte writes (RWDS_OUT_reg is 0)
                  28: RAMDQ_OUT_reg <= BYTE_1;            // End of additional 6-cycle latency period that began at edge 16: start writing 16-bit word
                  29:
                      begin
                        RAMDQ_OUT_reg  <= BYTE_0;         // Write rest of 16-bit word
                        RAM_state      <= RAM_END_CMD;
                      end
               endcase
            end   
What would have made more sense:

Code: [Select]

         
RAM_M_WR:
            begin
               case (RAM_edge_cnt)
                  0:  RAMDQ_OUT_reg <= RAM_CA_words[0];   // Write addr [47:40] prior to first rising edge
                  1:  RAMDQ_OUT_reg <= RAM_CA_words[1];   // 5 ns after first rising edge (and 5 ns prior to first falling edge): write addr[39:32]
                  2:  RAMDQ_OUT_reg <= RAM_CA_words[2];   // 5 ns after first falling edge: write addr[31:24]
                  3:  RAMDQ_OUT_reg <= RAM_CA_words[3];   // 5 ns after second rising edge: write addr[23:16]
                  4:  RAMDQ_OUT_reg <= RAM_CA_words[4];   
                  5:  RAMDQ_OUT_reg <= RAM_CA_words[5];
                  6:  begin
                           RAMDQ_OUT_reg <= BYTE_1;       // Put first byte on data bus
                           RAM_state <= RAM_WRITE_B0;     // Wait until RWDS indicates that it was accepted
                      end
               endcase
            end

RAM_WRITE_B0:
            begin
               if (RWDS) begin
                  RAMDQ_OUT_reg <= BYTE_0;                // First byte accepted; now write second byte
                  RAM_state     <= RAM_END_CMD;           // (RWDS was asserted for only one cycle, no need to spin on it)
               end     
            end
I have to believe that masked byte writes are an obscure corner case with this type of RAM, which is targeted primarily at framebuffers and FIFOs and the like.  I certainly don't need them myself.  They should've been handled with an entirely separate mode, instead of balkanizing all memory operations into fixed and variable latency versions. 

The fact that the whole DCARS option was necessary is an indication that the bus interface design should have been thought through a little further.

[asmi]

Soon I'm going to test long burst transfers on hardware, a DMA is needed for this, as Microblaze cannot initate long burst transfers.

Forgot to mention - if you turn on data and instruction caches in Microblaze, it will issue burst transactions as it will try to read/commit entire cache line in a single operation. This obviously won't create a sustained load on memory (unless you write the code such that it will force constant cache evictions), but it will allow you to verify in hardware that bursts are working as they should. You can also use AXI Traffic Generator IP to generate any traffic on AXI bus you desire, and it also has presets to mimic some typical use cases like video streaming, PCIE, Ethernet traffic, etc.

[VGN]

I have to believe that masked byte writes are an obscure corner case with this type of RAM, which is targeted primarily at framebuffers and FIFOs and the like.  I certainly don't need them myself.  They should've been handled with an entirely separate mode, instead of balkanizing all memory operations into fixed and variable latency versions.

To be honest, I don't see any problem with variable latency. Yes, it is weird a bit, but pretty easy to handle.

The fact that the whole DCARS option was necessary is an indication that the bus interface design should have been thought through a little further.

But DCARS is not necessary and not mandatory. More over, I doubt that you can purchase parts that support DCARS mode.

[VGN]

Forgot to mention - if you turn on data and instruction caches in Microblaze, it will issue burst transactions as it will try to read/commit entire cache line in a single operation. This obviously won't create a sustained load on memory (unless you write the code such that it will force constant cache evictions), but it will allow you to verify in hardware that bursts are working as they should. You can also use AXI Traffic Generator IP to generate any traffic on AXI bus you desire, and it also has presets to mimic some typical use cases like video streaming, PCIE, Ethernet traffic, etc.

Thanks, vivado provides some AXI Verification IP (VIP), I'm going to try it. BTW, initiating single transfers 8/16/32-bit manually over microblaze is easy, but is there any way to initiate wrapped cache transfer manually?

I used ISERDES in memory mode and fed rwds clock to CLK/CBLB pins of SERDES, while feeding free-running clocks from MCMM to OCLK/OCLKB/CLKDIV pins as described in UG471.

You didn't mention the IDELAY. Am I right, that you also had used IDELAY to delay the RWDS strobe before feeding it to CLK/CBLB pins of SERDES?
You have quite interesting implementation, though I don't understand how does it work stable, as your free-running clocks are still not synchronouse to the rwds domain. I don't understand how data is synchronized when going through the CLK->OCLK path inside the ISERDES in MEMORY_MODE...

[asmi]

Thanks, vivado provides some AXI Verification IP (VIP), I'm going to try it.

I use it a lot, and it's great for simulating all kinds of AXI transactions!

BTW, initiating single transfers 8/16/32-bit manually over microblaze is easy, but is there any way to initiate wrapped cache transfer manually?

I honestly don't remember off top of my head. Try skimming through the user guide for MB - I remember it is very detailed so I'm sure it covers everything.

You didn't mention the IDELAY. Am I right, that you also had used IDELAY to delay the RWDS strobe before feeding it to CLK/CBLB pins of SERDES?

You may or may not need them depending on what your pinout looks like and which clock buffers you use. I don't have that design handy so I can't look it up, but I remember that clock buffers have rather large insertion delay, such that you might need to add IDELAY to data lines to compensate for that. In my 200 MHz LPDDR1 design I have IDELAY blocks on both clock and data lines. BTW - in case you need more resolution for delay, there is undocumented IDELAYE2_FINEDELAY component, which can provide finer delays (though that is only really useful at much higher speeds, because documented version has a resolution of 78 ps at 200 MHz).

You have quite interesting implementation, though I don't understand how does it work stable, as your free-running clocks are still not synchronouse to the rwds domain. I don't understand how data is synchronized when going through the CLK->OCLK path inside the ISERDES in MEMORY_MODE...

This mode has been specifically designed to support strobe-based memory interfaces (which is what DDRx interfaces are, and so is HB1/2 as far as reads are concerned). They provide functional schematics of how it works in UG471 (it's basically a version of a classic double-flop synchronization CLK->OCLK->CLKDIV, latter two are required to be phase-aligned), but you can just assume that it does, since this is a hard silicon block, provided that you use it exactly how they are suggesting. Carefully read entire section of that user guide which talks about ISERDESE2, you might need to read it several times to "get it" - as was in my case - because it's not exactly the cleanest explanation, and the component itself is rather complex and covers many different use cases.
As for stability - this is what IO timing constraints are for. If you constrain your design properly, timing analysis can guarantee proper functioning of interface in all conditions as Vivado will adjust placement and routing to help satisfy them (basically it can add quite a bit of delay by choosing a longer connection path inside the fabric's interconnect backbone, or move components around to ensure both setup and hold constraints are satisfied). As you increase your interface speed, at some point timing margins will become so small that you won't be able to perform a static capture using constraints, and in this case you will need to implement/perform some sort of calibration to ensure that you sample data right in the middle of the data eye, but in my experience with right pinout and some effort on your part you can successfully achieve static capture at 200 MHz DDR and below.
Finally - I'm sure you know this, but someone reading this might not - Cypress provides simulation models for these chips, so you can run full functional simulations to make sure it works like it should in all cases and scenarios.

[AcM]

Hi all,

the hyperram IP is great but seems to be a little sensible to the project, sometime seems that the data read is "delayed" by a clock period.
Found after some read errors, maybe it needs a little calibration procedure at the beginning?
Anyway great job VGN 

[VGN]

First, asmi, thank you very much for such a detailed post!   And sorry for such a delayed answer from me!

[VGN]

Hi, AcM! Yes, this IP core is far from perfect, I'm still working on it.

In previous version v1.0 there was an AXI4 protocol handshake bug! I have fixed it. Now continues bursts from DMA along with simultaneous single transactions from CPU to the memory looks to be working properly.

the hyperram IP is great but seems to be a little sensible to the project, sometime seems that the data read is "delayed" by a clock period.
Found after some read errors, maybe it needs a little calibration procedure at the beginning?

Which core version are you using? I have updated it to v1.1 (rev.40) several days before. I have added an idelay static configuration options. Probably you don't meet timings constraints. Checkout "...\constrs\OpenHBMC.xdc" and make sure, that the clock period value of RWDS is correct, it should be equal to the memory clock period (clk_hbmc_0 or clk_hbmc_270). I know that proper value should be automatically calculated over block design input parameters, but Xilinx doesn't provide enough documentation about IP core design  . I found out that a special .ttcl file should be added, but didn't try it yet.

In fact the core constraints file is not finished, I added only input delay constraints. Also there is a negative slack at some asynchronous chains, that should be ignored. Hope to fix this soon.

Generally, try for now to reduce the clock frequency to 125MHz and apply a proper clock period value of 8.000ns to "OpenHBMC.xdc".

[ali_asadzadeh]

Thanks for sharing

[kmike]

Thanks!

Great work!

Ramtest usin Microblaze passed on a Trenz 0725 Board (Artix7 + 64MBit HyperRam) @ 100MHz! Update: Test also passed at 166MHz

Now I will test it using video DMA 

br,
mike

[VGN]

Thanks!

Great work!

Ramtest usin Microblaze passed on a Trenz 0725 Board (Artix7 + 64MBit HyperRam) @ 100MHz! Update: Test also passed at 166MHz

Hi, kmike , thanks for feedback, glad to hear that. BTW, I'm not completely sure, but rwds signal position is critical. As rwds is used as a clock signal, it should be connected to clock capable pin of the bank, i.e. SRCC or MRCC. I have two boards with Spartan-7 XC7S100 and XC7S50. On the first board with XC7S100 the rwds is connected to the SRCC and I also could pass memtest at 162MHz. But at the second board with XC7S50 the rwds is not at clock capable pin and I can't meet timings at frequencies >125MHz.

Nevertheless, I would like to run this memory controller at maximum speed that memory part supports (for my W956D8MBYA5I the limit is 200MHz) with slowest 7-series FPGA, i.e. Spartan-7 family with -1 speedgrade. I have an idea of completely another way of incomming data capturing over ISERDESE2 in OVERSAMPLE mode.
Yes, this is a bit more complex, but for my opinion this solution is going to be much faster, more reliable and probably there will be no need to make calibrations or even use idelay primitives. Details further...

[kmike]

Hi VGN,

on the Trenz board the RWDS signal ist not connected to a CC pin, but still happily passes ramtest @ 166MHz. I can not increase the frequency, because the RAM chip (S27KS0641) on this board is not capable of 200MHz. The FPGA is an Artix-7 35t -2 speed grade.

I will use the memory at 100MHz for this project (camera to TFT adapter), because my pixel clock is only 5MHz 

br,
mike

[VGN]

Hello everyone!

Some updates. New HyperRAM controller version. Achieved 200MHz clock speed on Spartan-7 (-1 speed grade) with W956D8MBYA5I part.
Repo: https://github.com/OVGN/OpenHBMC

[gkite]

Hello, can your hardware board be sold? I want to buy a set of boards。

[glenenglish]

Well done. compatibility can be an issue , not all HyperRAMs are created equal.