Mixing DDR3 data lines between bytes

[Boscoe]

Hi all, I just realised (mid through my PCBs being fabricated) that I mixed the data lines on my DDR3 memory and Zynq between the byte lanes. I understand that data lines can be mixed to some extent to optimise PCB layout but will this cause me problems when accessing single bytes from memory? Attached is a picture of the schematic.

If this is a problem are there anyways to mitigate it?

Thanks,
George

[Berni]

Yep this will likely cause problems, you can only safely swap within the same byte. Mostly because timings are also based in groups of 8.

[asmi]

If this is a problem

Yes it is. And the biggest problem is figuring out how such glaring error managed to pass unnoticed through schematic checkups/reviews/etc.

are there anyways to mitigate it?

If the length of bytes lanes is sufficiently similar, you might be able to get away with it, possibly along with lowering the memory frequency. Otherwise you will have to do a respin.

[Boscoe]

Thank you for the reply. Is the timing the only concern here? THe the byte masking not be an issue as well? If there's a single byte access?

[asmi]

Thank you for the reply. Is the timing the only concern here? THe the byte masking not be an issue as well? If there's a single byte access?

I don't think Zynq does per-byte accesses because it's got cache on chip, and cache controller typically reads/writes entire cache lines at a time. Not sure how it would behave if your have a PL peripheral which would request a narrow transfer directly to/from DDR controller though. But sinse that's probably within your control, you can either design those peripherals to ensure that doesn't happen, or use Cache IP to eliminate such accesses (this can lead to cache coherence problems, so make sure you know what you are doing and have a plan on how to deal with these issues before they manifest themselves, otherwise you will be in a world of pain).

[BrianHG]

Yes, every 8bits of DQ and DM and DQS all need to be matched.
So, you can swap the data bits within an 8 bit group, or swap an entire 8 bit bank with DQS and DM in parallel.
You cannot cross the data lines from 1 8bit bank to another.

[BrianHG]

If your controller wires the UDM and LDM pins, Data Mask, then somewhere within, it probably can write individual bytes.  Just wire it correctly.

Also, the UDQS and LDQS also need to be in order.

If you have an 8bit soft core CPU and want to quickly write bytes without having to do a read modify write step, then yes once again, just wire the 2 8 bit bytes with their properly associated DM and DQS pins correctly.

[Boscoe]

Again thank you for all your replies.

I’m now at the point of deciding whether it’s worth populating this board with about €250 of components. The boards are complex and have a lot more circuitry than just the Zynq so I could get some value by testing the rest at the same time. The bonus could be that the DDR3 does work despite the mixed data lines, the down side is I’ll likely lose €250 of components.

What are you opinions on that?

[asmi]

Again thank you for all your replies.

I’m now at the point of deciding whether it’s worth populating this board with about €250 of components. The boards are complex and have a lot more circuitry than just the Zynq so I could get some value by testing the rest at the same time. The bonus could be that the DDR3 does work despite the mixed data lines, the down side is I’ll likely lose €250 of components.

What are you opinions on that?

It depends on what were your target lengths for length matching to begin with, and if you are into BGA desoldering/reballing or not. If so, you can populate the minimum set of parts required to get Zynq up and running and see if it works, and if it doesn't work - you desolder, reball and re-use them later. Or you can do a PCB respin, and pray that there are no further mistakes in the schematics and/or PCB layout. Oh, and hang one of those boards on the wall next to your work place to remind you the value of a proper schematic checkup/review for your future projects.

In your shoes the first order of business would be a complete schematics audit - perferably by someone who did not participate in the design, because not smelling your own sh*t is a real thing that actually happens in the real life, and so a fresh set of eyes would go a long way. I'm telling this to you as someone who regularly suffers from "the curse of a first revision", and is a serial offender of making stupid mistakes and not seeing them before submitting an order to manufacturing.

[asmi]

If your controller wires the UDM and LDM pins, Data Mask, then somewhere within, it probably can write individual bytes.  Just wire it correctly.

Also, the UDQS and LDQS also need to be in order.

All of that is kind of obvious, this is not the subject of this topic. The question is can this PCB be salvaged, or it isn't worth even trying.

If you have an 8bit soft core CPU and want to quickly write bytes without having to do a read modify write step, then yes once again, just wire the 2 8 bit bytes with their properly associated DM and DQS pins correctly.

Context is important, he is talking about a Zynq SoC, which is an ARM Cortex A9 CPU, and from the part of schematics it can be seen that he screwed up wiring to the hard memory controller, so no 8 bit cores.

[Boscoe]

Yes, I totally agree with this, however this is an ambitious personal project done humbly. Due to some of the costs involved I was thinking of paying someone to review however finding someone with the right knowledge that's affordable is not easy. I have been through a lot of rounds of review myself but as you say you become blind to your own work - this mistake was due to a whole in my knowledge.

I'm also using Eagle which doesn't have sophisticated error checking for this kind of thing.

[Boscoe]

Again thank you for all your replies.

I’m now at the point of deciding whether it’s worth populating this board with about €250 of components. The boards are complex and have a lot more circuitry than just the Zynq so I could get some value by testing the rest at the same time. The bonus could be that the DDR3 does work despite the mixed data lines, the down side is I’ll likely lose €250 of components.

What are you opinions on that?

It depends on what were your target lengths for length matching to begin with, and if you are into BGA desoldering/reballing or not. If so, you can populate the minimum set of parts required to get Zynq up and running and see if it works, and if it doesn't work - you desolder, reball and re-use them later. Or you can do a PCB respin, and pray that there are no further mistakes in the schematics and/or PCB layout. Oh, and hang one of those boards on the wall next to your work place to remind you the value of a proper schematic checkup/review for your future projects.

In your shoes the first order of business would be a complete schematics audit - perferably by someone who did not participate in the design, because not smelling your own sh*t is a real thing that actually happens in the real life, and so a fresh set of eyes would go a long way. I'm telling this to you as someone who regularly suffers from "the curse of a first revision", and is a serial offender of making stupid mistakes and not seeing them before submitting an order to manufacturing.

I think I will populate the whole PCB. There power sequencing for the FPGA and rest of the system requires another MCU and some other bits. At this point I might as well do it all. The whole 16 bit data bus is length matched as well as the relevant lines such as DQSx etc.

[asmi]

Yes, I totally agree with this, however this is an ambitious personal project done humbly. Due to some of the costs involved I was thinking of paying someone to review however finding someone with the right knowledge that's affordable is not easy. I have been through a lot of rounds of review myself but as you say you become blind to your own work - this mistake was due to a whole in my knowledge.

I'm also using Eagle which doesn't have sophisticated error checking for this kind of thing.

If it's a personal project, you can just post schematics here and I'm sure you will get some feedback.

[Berni]

Yeah if you never write individual bytes and the whole 16bit bus is length matched together, then it should still work (Tho not optimal since these DQS groups are also internally matched)

You can still try running the DDR bus slower if there are issues (and often they don't run super fast on FPGAs anyway)

[Boscoe]

Hi all,

I just wanted to give an update. I assembled the HW a few days ago and been slowly verifying all the elements of the electronics. This evening was the Zynq memory tests. I really didn't have a lot of hope because I also didn't know the delay timings of the PCB, I used common values however I will revise these at a later date.

I ran the full 256MB Xilinx memory test with zero errors 

I also ran the Xilinx read and write eye test / optimisation, I got a result between 65% and 75% which is not amazing but also maybe to do with my generic timing numbers.

I'm operating the RAM at the full 525MHz.

I'm very happy and also very surprised. This will still be fixed in the next revision of this PCB.

I also ran a test program that wrote to the DDR3 through heap allocation. I allocated 1MiB of memory and assigned each byte an increasing value from 0 to 255. I then read that memory with the same algorithm to verify the memory had the correct values - it did!

[asmi]

The best test of memory interface is a successful boot up of Lunux. Build a quick petalinux image and see if it reliably boots. Also remember that things tend to change with temperature,  so make sure to run a long term test.

[Berni]

Nice, congrats.

And yeah i also find that booting linux is a very good test due to how varied the access is, so it also exposes issues with address lines, power delivery, crosstalk..etc

Indeed also try doing tests at different temperatures, i seen RAM work fine, but stop working when things get hot.

[TomS_]

I find something that helps with reviewing my own schematics is to step away for a couple of days, maybe work on another project for a bit. Basically you just need to forget about it for a little bit and then come back to review it again with a fresher perspective.

Yes it slows things down when you really just want to hit the order button and get it moving, but you also have an increased chance of finding errors.

A couple of months ago I built my most complex board yet, it had 6 layers and a bunch of parts I'd never worked with before. After reviewing on and off for a couple of weeks I found a couple of errors that would have absolutely ended up requiring a respin to fix.

And after all that I made a last minute addition which ended up shorting all of the internal power planes together due to a KiCad quirk. I was able to drill out the offending vias and rescue the board which worked perfectly otherwise.

I guess long story short, it pays to slow things down a bit. If you're working intensely on something you'll fatigue yourself, and that will lead to mistakes.

[Boscoe]

I find something that helps with reviewing my own schematics is to step away for a couple of days, maybe work on another project for a bit. Basically you just need to forget about it for a little bit and then come back to review it again with a fresher perspective.

Yes it slows things down when you really just want to hit the order button and get it moving, but you also have an increased chance of finding errors.

A couple of months ago I built my most complex board yet, it had 6 layers and a bunch of parts I'd never worked with before. After reviewing on and off for a couple of weeks I found a couple of errors that would have absolutely ended up requiring a respin to fix.

And after all that I made a last minute addition which ended up shorting all of the internal power planes together due to a KiCad quirk. I was able to drill out the offending vias and rescue the board which worked perfectly otherwise.

I guess long story short, it pays to slow things down a bit. If you're working intensely on something you'll fatigue yourself, and that will lead to mistakes.

Hi Tom, thank you for sharing this great piece of wisdom! This is something I already practice and did with this design finding many mistakes before commuting - long are the days I order from pure excitement! Although that is difficult not to do. I made this mistake having reviewed it multiple times because it was simply a gap in my knowledge. It’s completely obvious to me now but I didn’t think this was a problem before I coincidently read about it while my PCBs were getting fabbed.