Custom Spartan-7 FPGA board for beginners

[asmi]

Hi,

I've recently designed a simple board with Spartan-7 FPGA. The board was specifically designed to be very simple so that it could be used by others as a starting point for their own design, and also to prove that you can actually design a decent board using free tools. It was designed from the ground up in KiCAD 5, and full sources are publicly available in my Github repository. The board was designed for 4-layer JLCPCB.com process as the cheapest 4-layer boards with controlled impedance I was able to find, specifically for the JLC2313 spackup. The board is 72x89 mm in size, and 10 board is only $29 (+delivery), though I'd recommend to select ENIG finish which would raise the price to $43.5.
Initially I was going to make just a bare minimum board without any peripherals, but soon realized that it won't be very useful. So I've added 256 Mbytes of DDR2 memory (128Mx16). The reason I've chosen DDR2 over DDR3 is that it allowed me to save one DC-DC converter because DDR2 uses the same 1.8 V rail that is used by the FPGA itself, while DDR3 uses 1.5 or 1.35 V. There is also a coupe of user buttons (aside from CFG_RESET used to reset FPGA configuration, and active-low SYS_RESET used for system reset without reconfiguration), and a pair of RGB LEDs.
In order to make sure I'm providing a sources for the board that actually works, I've assembled the board and fully tested it. I've attached photos of that board. I've also uploaded pre-made Gerber packages ready to be uploaded to the JLCPCB.com (current revision is D), as well as schematics PDF, and assembly document which I used for manual assembly. I've installed XC7S50-2FTGB196C FPGA with MT47H128M16RT-25E:C DDR2 memory chip, which is pretty much the maximum configuration. I confirmed that the board is able to output 1080p@60Hz via HDMI, but the FPGA heated up enough that I need to place a small heatsink over it. I also confirmed that DDR2 memory works at 333 MHz without any bit errors, which indicates that my manual length matching in KiCAD was good enough.

Like I said, the idea for this board to be as simple as possible yet be useful, as well as being the basis for other designs, so I didn't pack the board as tight as possible to make manual assembly easier. The real challenge for assembly is 18 ultra-small 0201 0.47 uF capacitors. I tried to use 0402 size instead, but they don't fit between vias under FPGA without violating manufacturing tolerances.
It is possible to customize the hardware on board - you can install smaller capacity DDR2 chip as long as the data bus width is still 16 bit (make sure it's supported by MIG!) - or don't install any at all, or any Spartan-7 chip in 1mm pitch FTGB196 package (I wouldn't recommend using devices smaller than S25 if you keep DD2 memory as memory controller will consume some resources), you can opt to not fit any HDMI-related parts if you don't want. As a matter of fact, if you are not confident about BGA soldering, you can assemble the first board with the smallest S6 device to see just how easy it is to work with BGA chips.

KiCAD source for the board can be found here: https://github.com/asmi84/kicad-projects
It uses a library which is available here: https://github.com/asmi84/kicad-library

If you have any questions or suggestions, feel free to post them here.

[SiliconWizard]

Nice work.

The layout job for the FPGA looks like it must have been no picnic.

[asmi]

Nice work.
The layout job for the FPGA looks like it must have been no picnic.

Thanks! Actually it wasn't all that hard, I think it took me like a couple of evenings to get the DDR2 layout down, once that was completed everything else was quite simple as this package was designed for 4 layer boards, plus very generous 3.5 mil (~0.09 mm) tolerances provided by JLCPCB process made escapes (which often are a pain for BGAs) pretty much a non-issue.
As for DDR2 layout, the biggest hurdle was differential traces (CK, DQS) as built-in trace matching tools just don't work quite right for diff pairs. I did length matching manually, only used length-matching tools to see what is current trace length so that I will know which way I needed to adjust it. Once you get a hand on it, it's actually not nearly as bad as I was afraid it will be when I was just coming into it. One thing I did not do though is account for via length when matching traces laid on a top layer with ones on the bottom layer, but for 333 (or even 400) MHz this is not a problem, which was confirmed by running a memory test for about an hour without encountering any bit errors.

[mark03]

Thanks for this.  I plan to design my own Spartan-7 board soon with the same FTGB196 package, so it's nice to have an example.

For the PCB houses I've used (OSHPark and DirtyPCBs), the limiting factor for BGAs is usually the overall via diameter (drill plus annular ring).  Sometimes this bumps up against the limit and folks "cheat" a bit to make it work.  What size did you use for the dogbone vias?

Do you print paste on the BGA pads or flux only?  If paste, leaded or lead free?  I heard cautions about mixing leaded paste and lead-free balls; not sure if there's anything to that.

[asmi]

Thanks for this.  I plan to design my own Spartan-7 board soon with the same FTGB196 package, so it's nice to have an example.

That's great! Let me know if you have any questions or need any help - I'd be more than happy to help out. Or if you use KiCAD yourself, you can just take my design, get rid of parts you don't need and add your own components. The primary reason I've published the full sources is that I would like to dispel the myth that FPGAs are some kind of black voodoo magic, and demonstrate that it's not very hard to get basic board designs up and running without too much effort and money. Now - don't get me wrong, some FPGA designs are insanely complex, but that doesn't mean all of them are.

For the PCB houses I've used (OSHPark and DirtyPCBs), the limiting factor for BGAs is usually the overall via diameter (drill plus annular ring).  Sometimes this bumps up against the limit and folks "cheat" a bit to make it work.  What size did you use for the dogbone vias?

This board was designed around JLCPCB's 4 layer process, so I used 0.2/0.45 mm vias (which is the minimum they can produce). The advantage of this fab is that it is the only one in it's price class that offers impedance control at no additional charge, which is important for high-speed interfaces like DDR2/3 or HDMI. This via size makes for an easy escape of 0.8 mm BGA (which is what DDR2 is), not to mention larger 1 mm pitch of the FPGA, with no cheating required. On top of that, they also allow traces down to 3.5 mil (~0.09 mm), which allows to fit two traces between 1 mm pitch vias and so you will need less layers to escape. Even though on this board I used 0.1 mm minimum traces, which was enough for this design. JLC2313 stackup they are offering means that 50 Ohm impedance traces are only 5.78 mils wide (I used 0.15 mm wide traces), which is great for "dense" buses like DDRX as there are a lot of traces.

Do you print paste on the BGA pads or flux only?  If paste, leaded or lead free?  I heard cautions about mixing leaded paste and lead-free balls; not sure if there's anything to that.

This board was assembled using room temperature stable lead free paste (this one), and I used stencils for both sides (since the board is quite small, I could fit stencils for both sides on a single sheet of metal) - I ordered it also at jlcpcb together with boards. That is just because I have the equipment (manual stencil printer, and reflow oven), but I tried to design the board such that it could be assembled with just a hot air gun. You will need some kind of magnification as 0201 caps are just too small (0.6x0.3 mm IIRC). I'm using a stereo microscope from AmScope (SE410), and it is just great because it has a large working distance so that you can comfortably use soldering iron under it, while still being quite affordable (I think I got it for around $200 for a full set of eyepieces - x5/x10/x15/x20).

Though in the past I used leaded solder paste with lead-free BGAs and they turned out just fine too. One of the boards i used it on is now about 3 years old, and it still works OK today. So if you prefer using leaded paste, it should be fine as well.

[Dmeads]

Seriously cool! Thanks for putting this together!

Was power to the FPGA a hassel? Were the DC-DC converters switching?

Overall really amazing design.

[SiliconWizard]

Nice work.
The layout job for the FPGA looks like it must have been no picnic.

Thanks! Actually it wasn't all that hard, I think it took me like a couple of evenings to get the DDR2 layout down, once that was completed everything else was quite simple as this package was designed for 4 layer boards,

Yeah, not that it's very hard, but all this BGA fan-out is very tedious and I just hate doing that (usually have it done by someone else/subcontract layouts like this )

Some EDA packages have some kind of automatic BGA fan-out, I don't know whether KiCad does yet?

[langwadt]

Nice work.
The layout job for the FPGA looks like it must have been no picnic.

Thanks! Actually it wasn't all that hard, I think it took me like a couple of evenings to get the DDR2 layout down, once that was completed everything else was quite simple as this package was designed for 4 layer boards,

Yeah, not that it's very hard, but all this BGA fan-out is very tedious and I just hate doing that (usually have it done by someone else/subcontract layouts like this )

Some EDA packages have some kind of automatic BGA fan-out, I don't know whether KiCad does yet?

I believe there are some python scripts around to do automatic fanout of BGAs

[SiliconWizard]

I believe there are some python scripts around to do automatic fanout of BGAs

I'll have a look!

[asmi]

Yeah, not that it's very hard, but all this BGA fan-out is very tedious and I just hate doing that (usually have it done by someone else/subcontract layouts like this )

Actually I like that part the most - it's like playing chess when you've got to think and plan ahead. Especially in cases where you know you can't do a full breakout, when you need to be very strategic.

Some EDA packages have some kind of automatic BGA fan-out, I don't know whether KiCad does yet?

They all suck, as in most cases you need not just "a fanout", but a very specific one because things like DDRX or LVDS require certain way of doing it. On this board I didn't pay too much attention because I know it will run at only 333 MHz (as PHY in that package is only rated at that speed), and I know from my past experience with DDR3 memory that at these low frequencies I can get away with this. But I also know that's not how one would go about fanout when he would be laying out DDR3-1866 working at 933 MHz (for example for Kintex-7 board), as small things start to matter at that frequency. That is the hard part.
If you don't have any hi-speed or interface requirements (which I kinda doubt), than doing just a simple fanout is so fast in just about any EDA I worked in (and in the past year alone I worked in KiCAD, Orcad Pro and Altium), that any specific tool is just not needed.

[SiliconWizard]

Yeah, not that it's very hard, but all this BGA fan-out is very tedious and I just hate doing that (usually have it done by someone else/subcontract layouts like this )

Actually I like that part the most - it's like playing chess when you've got to think and plan ahead. Especially in cases where you know you can't do a full breakout, when you need to be very strategic.

Ok, to each their own!

Do you take contract work?

[asmi]

Do you take contract work?

Yes I do, when I'm not busy with different contract

[OwO]

I've done DDR3-1066 on 4 layers too and got it working reliably. The tricks are:

• All DDR traces must form a microstrip or stripline against the closest ground plane (usually 0.1mm away) which must be solid and have no breaks under the traces
• All ground and power pins on both the SoC and DDR chip should be accounted for; either they need to go to a plane, or have a decoupling cap right under the balls that goes to a plane. Every ground ball should have its own via, but a cluster of e.g. 4 ground balls are allowed to be connected together and grounded using 3 vias spread apart.
• If using a power plane, it must be stitched to the ground plane using a capacitor every 1-2cm, with higher density near signal layer transitions. For example, if a few DDR signals go from top to inner layer, you need a stitching capacitor or two around the via transition.

The stackup at the DDR area was: top (DDR traces); in1 (ground plane); in2 (DDR traces); bottom (VCCDDR plane). The rest of the board has ground planes on both in1 and bottom, with in1 being solid and bottom being broken by components. Therefore high speed signals are either routed on the top layer, or in2 while the bottom ground plane is cleared of components where striplines are routed. It goes without saying that stitching is critical.

All 3 boards tested so far passes memtests running under Linux for a few days. Btw the memtest that comes with Xilinx embedded SDK example projects is not as demanding as just booting Linux. I've had one inadequate BGA reflow that still passes the Xilinx memtest but Linux will fail to boot every other time and running any user application typically locks it up. The issue was fixed by adding more flux and redoing the reflow.

The design is on GitHub under gabriel-tenma-white/sdr5.
In the future I may design a Kintex-7 board with faster DDR3.

[Mechatrommer]

If you have any questions or suggestions, feel free to post them here.

for beginner its not easy to assemble that BGA package FPGA and RAM. it will need advanced tools such as stencil or rework station. so trying to send 4 layer PCB to fab house (more expensive for small qty) is another matter, assembling them is another hard matter. for someone who only just learning to play/program with FPGA, i will be interested at reasonably priced ready made dev board + step by step on how to program it and what toolchains etc... do you think about selling it?

[asmi]

for beginner its not easy to assemble that BGA package FPGA and RAM. it will need advanced tools such as stencil or rework station. so trying to send 4 layer PCB to fab house (more expensive for small qty) is another matter, assembling them is another hard matter.

That is the exact myth I'm trying to dispel with this project. No, you don't need any advanced or expensive tools, the only tool you need is a hot air gun, which can be had for about $100 on Aliexpress, ebay or Amazon. That, and some tacky flux + some solder for the rest of components.
Seriously. The cheapest XC7S6-1FTGB196C FPGA is only $14.7, get it and buy the rest of component on a board except memory, and give it a go. You will be pleasantly surprised how easy BGAs are to work with, compared with other packages (there is a TSSOP package with exposed pad on this board which is IMHO harder to solder as it sits on a ground plane for thermal reasons).

do you think about selling it?

I will need to get a reasonable volume (at least one tray, which is 90 chips IIRC) before I can approach Xilinx sales rep and get a good price. As it is, the BOM for a single board - as I assembled it - is around $80 (including PCB, but excluding passives as I have literally reels of them), and so the retail board price is going to be $100+ once I factor in all costs. And for that price, this is a very specialized board with a rather limited functionality (which, of course, is by design as I wanted to keep it simple), basically, you can connect a bunch of sensors and/or perhaps some kind of wireless transceiver, and display information gathered to a some kind of HDMI display or TV. Or implement some kind of display adapter in FPGA and connect it to a some external controller board.

That said, since I have few boards laying around (they are of REV B, which have some issues, but nothing that can't be fixed by a sharp knife, as I haven't ordered REV C yet due to this whole virus thing going on), if someone is interested in getting a board (assembled or not), leave a post here, and we can work something out. I can either send a bare PCB, or assemble it with whatever parts you want if you cover the cost of parts and postage. FPGAs are export-controlled, so unless someone tells me how to deal with these issues, I can only send assembled boards to Canada and perhaps US (as I imagine there shouldn't be too much trouble for re-exporting FPGAs back in there).

[SiliconWizard]

for beginner its not easy to assemble that BGA package FPGA and RAM. it will need advanced tools such as stencil or rework station. so trying to send 4 layer PCB to fab house (more expensive for small qty) is another matter, assembling them is another hard matter.

That is the exact myth I'm trying to dispel with this project. No, you don't need any advanced or expensive tools, the only tool you need is a hot air gun, which can be had for about $100 on Aliexpress, ebay or Amazon. That, and some tacky flux + some solder for the rest of components.
(...)
You will be pleasantly surprised how easy BGAs are to work with, compared with other packages (there is a TSSOP package with exposed pad on this board which is IMHO harder to solder as it sits on a ground plane for thermal reasons).

Agreed. And you don't need stencils to solder BGAs if you're going to do it with a hot air gun, just flux?

Yes packages with thermal pads are much more annoying to hand-solder. TSSOPs are not the worst IMO. The absolute worst are the QFNs with thermal pads, as it's very easy to have bad connections to some outter pads due to the package not laying quite flat, and it's much harder to inspect (and fix). 

[OwO]

BGAs are easy to *solder*, you are right about that one. However they are difficult to troubleshoot, impossible to probe, and on any project with lots of BGA components I have to budget around twice total development time and cost.

First iteration of sdr5 had an issue with a missing signal, and as I can not physically bodge a connection I can not verify that this was indeed the issue. That means extra board spins which may or may not fix the problem. I also can not verify with good confidence whether the BGA soldering was fault free, so I have to build multiple copies of every prototype iteration to rule out soldering issues.

Lead free balls makes the whole process far more painful because there is a narrow window of temperature between unreliable reflow and potential component damage. Sometimes I will specifically buy reballed chips because they usually come with leaded balls and are easier to deal with.

Given this I would absolutely recommend to cut BGA use to a minimum and only when it's justified (some parts are only available in BGA, size constraints, etc).

[asmi]

Agreed. And you don't need stencils to solder BGAs if you're going to do it with a hot air gun, just flux?

Yep. I only apply solder paste if I do a full board reflow.

Yes packages with thermal pads are much more annoying to hand-solder. TSSOPs are not the worst IMO. The absolute worst are the QFNs with thermal pads, as it's very easy to have bad connections to some outter pads due to the package not laying quite flat, and it's much harder to inspect (and fix). 

On this board I use TPS65581 triple DC-DC buck converter IC, which is TSSOP-20EP with exposed thermal pad, which sits on a solid chunk of ground plane with a bunch of thermal vias to ensure good thermal conductivity, so soldering it using lead-free solder would be somewhat tricky with hot air gun, but since these are prototypes, you can use leaded solder and that will make things much easier. These ground planes are obviously not an issue for an oven since entire board is heated up to soldering temperature. This chip is great for many reasons - very easy to use due to no need for external compensation, reasonably good efficiency (it's mostly 70+% for main 1.0 V rail and 80+% for other rails), but my experience with it shows it can heat up pretty severely under full load on all rails - hence the need for robust thermal path.

[asmi]

First iteration of sdr5 had an issue with a missing signal, and as I can not physically bodge a connection I can not verify that this was indeed the issue. That means extra board spins which may or may not fix the problem. I also can not verify with good confidence whether the BGA soldering was fault free, so I have to build multiple copies of every prototype iteration to rule out soldering issues.

It's easy - stop buying garbage chips that came from god knows where. Only buy from the vendor or reputable resellers. I never had any soldering failures with BGA that were sourced from reputable places, but few parts which I bought from aliexpress et al. all had to be baked before soldering as I had zero confidence in their solder balls, and was half-expecting them to not work at all - but I got lucky so far with no failures after 2 hrs baking at 125°C just before reflow.

Given this I would absolutely recommend to cut BGA use to a minimum and only when it's justified (some parts are only available in BGA, size constraints, etc).

I do the exact opposite - for my designs I use BGAs whenever I can as their advantages far outweigh their limitations. I will take BGA over QFN any day, for example.

[OwO]

No, all the soldering "failures" I've had were because of too low temperature/inadequate reflow and were fixed by a second reflow. This has nothing to do with the chips. You just can NOT know if your BGA soldering was actually sufficient and when you do encounter bugs you have to do many copies to rule out soldering issues. This also doesn't address the lack of debugging access to pins, which makes it impossible to test fixes and requires a board respin for every change. It's impractical to have test points on every possible connection, which wouldn't be necessary on a QFP because you can attach bodge wires to the pins.

[asmi]

No, all the soldering "failures" I've had were because of too low temperature/inadequate reflow and were fixed by a second reflow. This has nothing to do with the chips.

Of course it does. Garbage chips tend to come with garbage solder balls, so you either bake them, then cross your fingers and hope for the best, or you reballs them to known-good balls.

You just can NOT know if your BGA soldering was actually sufficient and when you do encounter bugs you have to do many copies to rule out soldering issues.

If course you can. You just need to be a tad smarter about this. Like test the board before soldering, including measuring trace impedance if that is going to be a problem (like for 10G links). If you know the board is good, and so are chips (because you bought them from reputable source, and not from some shady guy who will be nowhere to be found a day after the sale) and logic (say, you use MIG which pretty much guarantees the logic is correct), so by exclusion you can determine that the reason for the bug is indeed soldering. In a lot of cases there are easier ways to rule out soldering issues like using JTAG boundary scan.

This also doesn't address the lack of debugging access to pins, which makes it impossible to test fixes and requires a board respin for every change. It's impractical to have test points on every possible connection, which wouldn't be necessary on a QFP because you can attach bodge wires to the pins.

This is a moot point as I don't have equipment to "debug" anything in high-speed interfaces anyways, and for issues which are slow enough that equipment I have is sufficient, I can always scrape off a solder mark from via or track and get access that way.
But if I would have the right tools, there are ways to have test points on high-speed traces and not degrade SI too much.
Oh, and last point - care to show me QFP chip with DDR3 interface? Once you NOT find one, you might want to read up exactly why, and see if that has anything to do with BGAs.

[OwO]

for beginner its not easy to assemble that BGA package FPGA and RAM. it will need advanced tools such as stencil or rework station. so trying to send 4 layer PCB to fab house (more expensive for small qty) is another matter, assembling them is another hard matter. for someone who only just learning to play/program with FPGA, i will be interested at reasonably priced ready made dev board + step by step on how to program it and what toolchains etc... do you think about selling it?

There is a XC7A100T dev board with DDR3 from QMtech for about $80:
https://www.aliexpress.com/item/4000170003461.html

It's very hard to beat that if you buy parts at list price. There are also Zynq boards for even cheaper. There is a lot of documentation available if you look it up.

[OwO]

Of course it does. Garbage chips tend to come with garbage solder balls, so you either bake them, then cross your fingers and hope for the best, or you reballs them to known-good balls.

What does baking have to do with solder balls? The chips I got had date codes only a few months prior, and this is verifiable with device DNA over JTAG. They are not reballed chips, and actually I've had better luck with reballed chips because they have leaded balls. I can show you ball uniformity by pressing to a piece of glass with a bit of liquid but you still won't be convinced.

The signal that I needed bodged wasn't a high speed signal. I didn't say use a QFP FPGA. I said use BGA only when justified, such as the FPGA and DDR3 chips. However I frown upon gratuitous use of BGA such as power management ICs etc which isn't aimed at your design but rather in general. I still assert my points about BGA being much more difficult to troubleshoot so I think it's reserved for the more experienced designer. When troubleshooting I'd rather rule out the soldering first, rather than having to rule out everything else (which is usually much harder to do).

[asmi]

What does baking have to do with solder balls? The chips I got had date codes only a few months prior, and this is verifiable with device DNA over JTAG. They are not reballed chips, and actually I've had better luck with reballed chips because they have leaded balls.

What does crappy solder balls have to do with soldering defects? Is that a serious question?
Moisture and solder balls oxidation are two sure ways to get solder failures with BGAs. Baking alleviates the first one, but does nothing for the second one. Good flux (either stand-alone, or as part of solder paste) will help with removing oxidation, but only to the point.
Read up on MSL to get more information on this if you're interested. And if you continue using crap chips from shady sources, at least bake chips before reflow to reduce your chances of failures - this gets worse as ball size decreases, so you can get away with a lot on 1 mm pitch chips, but as you go down in pitch, your risk of getting a failure increases.

I can show you ball uniformity by pressing to a piece of glass with a bit of liquid but you still won't be convinced.

You don't need to show anything to me - as I said above, I've got some BGA chips off aliexpress so I know what I'm talking about. Just look at those balls under x20 microscope, and compare them to brand new chip that just came off a sealed package, and you will clearly see the difference.

The signal that I needed bodged wasn't a high speed signal. I didn't say use a QFP FPGA. I said use BGA only when justified, such as the FPGA and DDR3 chips. However I frown upon gratuitous use of BGA such as power management ICs etc which isn't aimed at your design but rather in general. I still assert my points about BGA being much more difficult to troubleshoot so I think it's reserved for the more experienced designer. When troubleshooting I'd rather rule out the soldering first, rather than having to rule out everything else (which is usually much harder to do).

Well using BGAs for PMIC is a stupid idea in my opinion, or in any package without thermal pad for that matter, unless it's a really low power one when thermals are not a factor. But for example for for QSPI flash chips I prefer 1 mm pitch BGA-24 package, not the least because there is a greater range of pin-compatible capacities available. I can't use 0.5 mm and below pitch packages because PCB tech required is way out of my price range, but 0.65 mm and up is a fair game.

[Mechatrommer]

and so the retail board price is going to be $100+ once I factor in all costs.
..if someone is interested in getting a board (assembled or not), leave a post here, and we can work something out. I can either send a bare PCB, or assemble it with whatever parts you want if you cover the cost of parts and postage.

FPGAs are export-controlled, so unless someone tells me how to deal with these issues, I can only send assembled boards to Canada and perhaps US (as I imagine there shouldn't be too much trouble for re-exporting FPGAs back in there).

bummer. this line destroyed what i'm about to reply on the above quote... long story short... for $100+ with step by step programming guide, that should be reasonable. i have $5 china altera fpga breakout laying for years and altera blaster bought separately, manually downloaded the fpga SW in many GB few years later (i dont even remember the name ) but dont have time to figure out how to start (namely the SW/VHDL/Verilog part), maybe due to lack of need, the toolchain maybe obsolete or invalidated by now. so they are unsolved puzzles. i need a puzzle that shows me how to solve it in quick time, like... this is the HW, this is the SW/IDE (download from <here>), you'll need this programmer, and this is the sample program, you compile it its surefire to do led blink. that kind of thing. i dont need another unsolved puzzle. good luck to you anyway. cheers.

[asmi]

bummer. this line destroyed what i'm about to reply on the above quote... long story short... for $100+ with step by step programming guide, that should be reasonable. i have $5 china altera fpga breakout laying for years and altera blaster bought separately, manually downloaded the fpga SW in many GB few years later (i dont even remember the name ) but dont have time to figure out how to start (namely the SW/VHDL/Verilog part), maybe due to lack of need, the toolchain maybe obsolete or invalidated by now. so they are unsolved puzzles. i need a puzzle that shows me how to solve it in quick time, like... this is the HW, this is the SW/IDE (download from <here>), you'll need this programmer, and this is the sample program, you compile it its surefire to do led blink. that kind of thing. i dont need another unsolved puzzle. good luck to you anyway. cheers.

Well that's the thing. Since I don't make any profit off this board, it makes no sense for me to even get involved into all these export shenanigans as that will not yield me anything except a lot of headaches. But then again - you will need to pass this "export check" in order to even download Vivado/Vitis IDE, and without it the board would be absolutely useless anyway.

[iMo]

There is a XC7A100T dev board with DDR3 from QMtech for about $80:
https://www.aliexpress.com/item/4000170003461.html
It's very hard to beat that if you buy parts at list price. There are also Zynq boards for even cheaper.

That is the key message any Maker outside China should remember.

[asmi]

I bit of a joke to lighten the mood:
Apparently US bureaucrats think that terrorists can't solder BGAs So if you can - you're not a terrorist     

[asmi]

There is a XC7A100T dev board with DDR3 from QMtech for about $80:
https://www.aliexpress.com/item/4000170003461.html
It's very hard to beat that if you buy parts at list price. There are also Zynq boards for even cheaper.

That is the key message any Maker outside China should remember.

The key message here is to do your research before you buy any board! It sucks in so many ways, but I'll just mention a couple - 1) it's main rail (Vccint) power supply can only provide up to 3 Amps of current, while the device can require up to double that current, 2) it's got NO peripherals at all.

[asmi]

Hmmm, I looked at my packing list, and it seems that the FPGA I used is not export-controlled (see screenshot)... Not sure what to make of it.

[Mechatrommer]

for beginner its not easy to assemble that BGA package FPGA and RAM. it will need advanced tools such as stencil or rework station. so trying to send 4 layer PCB to fab house (more expensive for small qty) is another matter, assembling them is another hard matter. for someone who only just learning to play/program with FPGA, i will be interested at reasonably priced ready made dev board + step by step on how to program it and what toolchains etc... do you think about selling it?

There is a XC7A100T dev board with DDR3 from QMtech for about $80:
https://www.aliexpress.com/item/4000170003461.html

It's very hard to beat that if you buy parts at list price. There are also Zynq boards for even cheaper. There is a lot of documentation available if you look it up.

oh yeah that thing actually exist. exactly what i was describing... thanks.

[c64]

Initially I was going to make just a bare minimum board without any peripherals, but soon realized that it won't be very useful. So I've added 256 Mbytes of DDR2 memory (128Mx16). The reason I've chosen DDR2 over DDR3 is that it allowed me to save one DC-DC converter because DDR2 uses the same 1.8 V rail that is used by the FPGA itself, while DDR3 uses 1.5 or 1.35 V.

Have you considered to just solder SO-DIMM connector, so if you need memory - you can just plug a DDR2 stick from laptop? One less BGA to solder

[asmi]

Have you considered to just solder SO-DIMM connector, so if you need memory - you can just plug a DDR2 stick from laptop? One less BGA to solder

SODIMMs have 64 bit data bus, while this board only uses (and really only has pins for) 16 bit, not to mention SODIMM connector is rather large so it would require larger PCB (==more expensive). To implement full 64 bit memory interface, I would need to use at least FGGA484 package, and these chips are quite a bit more expensive, not to mention it will require to design at least 6 layer board, also SODIMM modules consume quite a bit more power than a single chip, so I will need to beef up the DC-DC converter for the memory rail, which again flies in the face of my goal of avoiding dedicated power rail for the memory. So it comes full circle - I specifically wanted this board to be cheap in both PCB manufacturing and BOM cost so that more beginners would want to give it a try. You can use S25 device instead of S50 (the former is half the price of the latter), and use smallest 64 MByte DDR2 chip (which is about a quarter price of 256 MByte chip I used), so the whole thing will be about twice as cheap to assemble as mine was (I wanted to confirm max configuration works as this will guarantee that smaller ones will have no problems).
I'm thinking about designing a more complex board with something like DDR3 SODIMM for those who want a bit more ooumph for their designs, but it will be significantly more expensive to manufacture and assemble.

[c64]

Can you use only 16 bits and leave the rest unused? Not sure about power consumption though, probably it's not possible to disable refresh on unused chips.

SODIMM connector is less than 100mm long so it should fit onto the 100*100 board (which is the max size to get a good price from LCSC).

[Gerhard_dk4xp]

No, all the soldering "failures" I've had were because of too low temperature/inadequate reflow and were fixed by a second reflow. This has nothing to do with the chips. You just can NOT know if your BGA soldering was actually sufficient and when you do encounter bugs you have to do many copies to rule out soldering issues. This also doesn't address the lack of debugging access to pins, which makes it impossible to test fixes and requires a board respin for every change. It's impractical to have test points on every possible connection, which wouldn't be necessary on a QFP because you can attach bodge wires to the pins.

A customer of mine uses the curve tracer to check the functionality of the pin protection diodes.
That makes definitely sure that the soldering for the pin was a success.
OK, someone will now state the case of multiple parallel pins.

I'm playing with the idea of marrying a FPGA with GTX transceivers to a Beaglebone Black or AI.
That would separate the potential Linux problems/support and DDRn from my soldering art and
would let me concentrate on these fast JESD-204B ADCs, DACs and SDR.

I'm still stuck at the point wether I can get peripherals in the FPGA memory mapped
to a window in the ARM address space.
There must be a 16 bit multiplexed bus possible at least, but I have never seen it used.
The FPGA configuration could be bit banged by the BBB directly from a .bit file, no need for a dongle.

Cheers, Gerhard
 

[asmi]

Can you use only 16 bits and leave the rest unused? Not sure about power consumption though, probably it's not possible to disable refresh on unused chips.

SODIMM connector is less than 100mm long so it should fit onto the 100*100 board (which is the max size to get a good price from LCSC).

I don't quite see what advantages would SODIMM have over current solution - even if all technical issues were somehow resolved. I seriously doubt that SODIMM connector + module combination would cost less than just a single chip, and it would certainly be more cumbersome to use. What all these troubles are for?

[Bassman59]

No, all the soldering "failures" I've had were because of too low temperature/inadequate reflow and were fixed by a second reflow. This has nothing to do with the chips. You just can NOT know if your BGA soldering was actually sufficient and when you do encounter bugs you have to do many copies to rule out soldering issues. This also doesn't address the lack of debugging access to pins, which makes it impossible to test fixes and requires a board respin for every change. It's impractical to have test points on every possible connection, which wouldn't be necessary on a QFP because you can attach bodge wires to the pins.

Ya know, it's funny, I work at a small company ( < 50 people) and we solder BGAs to boards every day. Our most common sizes are the 256- and 484-ball jobs. Since we've been doing this for, oh, 20 years now, we have it ironed out and BGA attachment is not a problem.

Yes, that long experience helps. But if you simply say that it'll never work and you don't try, your prophecy fulfills itself.

[NorthGuy]

I don't quite see what advantages would SODIMM have over current solution - even if all technical issues were somehow resolved. I seriously doubt that SODIMM connector + module combination would cost less than just a single chip, and it would certainly be more cumbersome to use. What all these troubles are for?

SODIMM DDR3 is dirt cheap - compare prices to what you pay for ICs. The connector is probably $3-5 which will give you up to 16 G. The 16G module is under $100 now.

[asmi]

SODIMM DDR3 is dirt cheap - compare prices to what you pay for ICs. The connector is probably $3-5 which will give you up to 16 G. The 16G module is under $100 now.

No it won't. Take a look at schematics, and you will see why.

[dmendesf]

You stole my joke!   

I bit of a joke to lighten the mood:
Apparently US bureaucrats think that terrorists can't solder BGAs So if you can - you're not a terrorist     

[c64]

I don't quite see what advantages would SODIMM have over current solution - even if all technical issues were somehow resolved. I seriously doubt that SODIMM connector + module combination would cost less than just a single chip, and it would certainly be more cumbersome to use. What all these troubles are for?

Connector can be found for < $1
OK maybe no advantages for you. But I see few advantages for others:
- one less BGA to solder
- easy to upgrade/downgrade
- most people probably already have some spare memory modules

[asmi]

Connector can be found for < $1

Where?

OK maybe no advantages for you. But I see few advantages for others:
- one less BGA to solder

I don't see it as advantage, especially since one of the goals of this board was to practice BGA soldering.

- easy to upgrade/downgrade

What's the reason for that?

- most people probably already have some spare memory modules

I don't have any

But you're missing the bigger picture - for SODIMM you will need to add a dedicated DC-DC converter (existing one is only rated for 1.5 A, and Vccaux can consume up to 0.6 A of that depending on your design), and once you do that you might as well go for DDR3 instead (the sole reason I went for DDR2 was to avoid adding another DC-DC rail for memory, all else being equal, DDR3 wins every time - less power, faster speed, more capacity, cheaper), you will also need to add a termination controller (which is not required for bare chips). This will also expose another problem, which I alluded to in my previous post - insufficient amount of ADDR/CTRL pins in FPGA bank to route all SODIMM address lines - normally you'd need 3 memory byte lanes for that, but since I only have a single bank, I can only dedicate 2 memory lanes for ADDR/CTRL. And I still don't know how to "terminate" unused data lines - couldn't find any information on that in memory's datasheet.

That said, since design is an open source design, you are more than welcome to take it, remove the memory chip (and any other things you don't need) and add your memory connector to see if you can get it all to work. I don't have high hopes for that because of everything I said above, but I'd love to be proven wrong.

[c64]

Forgive me if I say something stupid, I have no experience with SDRAM whatsoever. But SODIMM has multiple chips and if you only use one of them, all the others will not consume any power. Yes it will be a waste of chips, but if module is free - no problem

Regarding insufficient amount of lines in FPGA - do you need to use all the lines from the module? And if the module contains same chip as you currently use, will it have same number of address lines?

Regarding termination controller - i don't know what it is but sounds scary. 

Connector can be found for < $1

Where?

In China 

[kmike]

Dear asmi,

first of all thank you for your great work!

Is there a constraint file available? I dont know wich IO standard to use for the DDR RAM pins

thanks!

br,
mike

[asmi]

Dear asmi,

first of all thank you for your great work!

Is there a constraint file available? I dont know wich IO standard to use for the DDR RAM pins

thanks!

br,
mike

When you create a memory controller using MIG, it will assign proper constraints automatically. Just make sure you get pinout right (exactly like on schematics). If my memory serves me (no pun intended), MIG uses SSTL18_II standard for single-ended signals (with 50 Ohm ODT for bidirectional signals), and DIFF_SSTL18_II for differential ones.
I've attached a xdc file (rename the file and remove  the ".txt" extension because forum doesn't allow xdc files) which you can load into MIG to save you time for manual entry of all signals, but I'm not 100% sure that is accurate, so please verify that pin assignments are correct. If you install smaller capacity memory chip than what I did, drop the highest address line(s) as necessary.
As for the rest of it, please note that user buttons are connected to the same IO bank as memory, so you will need to use LVCMOS18 standard. HDMI data/clock pins use TMDS_33 standard, everything else is LVCMOS33 (unless you customize the board and/or connect something to IO bank 14 which uses a different standard).

[kmike]

Thank you!

do you clock the DDR chip with 333MHz?

br,
mike

[asmi]

Thank you!

do you clock the DDR chip with 333MHz?

br,
mike

Yes, that's the maximum clock rate for this package. I feed 166.667 MHz and 200 MHz clocks into MIG, which are generated by the "Clocking" IP from the system clock of 100 MHz.

[kmike]

Thank you!

do you clock the DDR chip with 333MHz?

br,
mike

Yes, that's the maximum clock rate for this package. I feed 166.667 MHz and 200 MHz clocks into MIG, which are generated by the "Clocking" IP from the system clock of 100 MHz.

Just to make sure, that this horse is dead, 166.667 MHz goes to the system clock of the MIG and 200 MHz is the reference clock?
I need an additional 10MHz clock in my design, should I generate it with the clocking IP or the MIG? Vivado keeps telling me that my internal reset is not in sync with the clk_ui of the MIG...

[asmi]

Just to make sure, that this horse is dead, 166.667 MHz goes to the system clock of the MIG and 200 MHz is the reference clock?
I need an additional 10MHz clock in my design, should I generate it with the clocking IP or the MIG? Vivado keeps telling me that my internal reset is not in sync with the clk_ui of the MIG...

I've attached the diagram which shows the correct clocking. Note that reset polarity for clocking IP needs to be changed to active-low as that's how the sysreset signal is wired on a board. It also shows 10 MHz output I've added for you.
For reset, ui_clk_sync_rst is a reset signal (sysreset) synchronized to ui_clk. But if you need a reset for your 10 MHz blocks, you will need to synchronize sysreset to it. You can use Processor System Reset IP for that, or do you manually in HDL.

[kmike]

Thank you!

Got it finally sorted out. Vivado automatically puts an AXI clock converter into the AXI interconnect (if I have different clocks in the design), but always(!) assigns 100MHz to to master interface, even if I use 83MHz CPU clock...
Solution is to put a clock converter manually at each AXI slave, that is running on a different clock than the CPU. Looks like automation cannot solve everything 

Br,
mike

[asmi]

Just for the heck of it, I decided to try loading Linux on this board. It turned out to be harder than I expected, as u-boot's compilation fails when the system is configured without network support (as the board doesn't contain any network hardware). The workaround is to force network support on in u-boot even if it's disabled in a kernel.
Anyways, I got Linux to boot successfully. I've connected external UART-USB module as this way I can use terminal like putty, as opposed to jtagterminal function of xsdb as the latter is more limited. The full bootlog is here for those curious:

Code: [Select]

U-Boot 2020.01 (Dec 05 2020 — 14:15:39 +0000)

Model: Xilinx MicroBlaze
DRAM: 256 MiB
WDT: Not found!
Loading Environment from SPI Flash... Invalid bus 0 (err=-19)
*** Warning — spi_flash_probe_bus_cs() failed, using default environment

In: serial
Out: serial
Err: serial
Model: Xilinx MicroBlaze
Net: Net Initialization Skipped
No ethernet found.
U-BOOT for sp50

Hit any key to stop autoboot: 0
U-Boot>?
? — alias for 'help'
askenv — get environment variables from stdin
base — print or set address offset
bdinfo — print Board Info structure
boot — boot default, i.e., run 'bootcmd'
bootd — boot default, i.e., run 'bootcmd'
bootelf — Boot from an ELF image in memory
bootm — boot application image from memory
bootp — boot image via network using BOOTP/TFTP protocol
bootvx — Boot vxWorks from an ELF image
cmp — memory compare
coninfo — print console devices and information
cp — memory copy
crc32 — checksum calculation
dhcp — boot image via network using DHCP/TFTP protocol
dm — Driver model low level access
echo — echo args to console
editenv — edit environment variable
env — environment handling commands
exit — exit script
ext2load — load binary file from a Ext2 filesystem
ext2ls — list files in a directory (default /)
ext4load — load binary file from a Ext4 filesystem
ext4ls — list files in a directory (default /)
ext4size — determine a file's size
false — do nothing, unsuccessfully
fatinfo — print information about filesystem
fatload — load binary file from a dos filesystem
fatls — list files in a directory (default /)
fatsize — determine a file's size
fdt — flattened device tree utility commands
fsinfo — print information about filesystems
fsload — load binary file from a filesystem image
fsls — list files in a directory (default /)
fstype — Look up a filesystem type
go — start application at address 'addr'
gpio — query and control gpio pins
help — print command description/usage
iminfo — print header information for application image
imxtract — extract a part of a multi-image
interrupts- enable or disable interrupts
irqinfo — print information about IRQs
itest — return true/false on integer compare
led — manage LEDs
ln — Create a symbolic link
load — load binary file from a filesystem
loadb — load binary file over serial line (kermit mode)
loads — load S-Record file over serial line
loadx — load binary file over serial line (xmodem mode)
loady — load binary file over serial line (ymodem mode)
loop — infinite loop on address range
ls — list files in a directory (default /)
md — memory display
mii — MII utility commands
mm — memory modify (auto-incrementing address)
mw — memory write (fill)
nfs — boot image via network using NFS protocol
nm — memory modify (constant address)
part — disk partition related commands
ping — send ICMP ECHO_REQUEST to network host
printenv — print environment variables
pxe — commands to get and boot from pxe files
reset — Perform RESET of the CPU
run — run commands in an environment variable
save — save file to a filesystem
saveenv — save environment variables to persistent storage
saves — save S-Record file over serial line
setenv — set environment variables
sf — SPI flash sub-system
showvar — print local hushshell variables
size — determine a file's size
sleep — delay execution for some time
source — run script from memory
sspi — SPI utility command
sysboot — command to get and boot from syslinux files
test — minimal test like /bin/sh
tftpboot — boot image via network using TFTP protocol
true — do nothing, successfully
version — print monitor, compiler and linker version
U-Boot>version
U-Boot 2020.01 (Dec 05 2020 — 14:15:39 +0000)

microblazeel-xilinx-linux-gcc (GCC) 9.2.0
GNU ld (GNU Binutils) 2.32.0.20190204
U-Boot>boot
## Booting kernel from Legacy Image at 80000000 ...
Image Name: Linux-5.4.0-xilinx-v2020.2
Image Type: MicroBlaze Linux Kernel Image (uncompressed)
Data Size: 9085444 Bytes = 8.7 MiB
Load Address: 80000000
Entry Point: 80000000
Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 82e00000 ...
Image Name: petalinux-image-minimal-microbla
Image Type: MicroBlaze Linux RAMDisk Image (uncompressed)
Data Size: 6930085 Bytes = 6.6 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 81e00000
Booting using the fdt blob at 0x81e00000
Loading Kernel Image
Loading Ramdisk to 8e964000, end 8efffea5 ... OK
Loading Device Tree to 8e95d000, end 8e963193 ... OK

Starting kernel ...

Ramdisk addr 0x8e964000,
FDT at 0x8e95d000
earlycon: uartlite_a0 at MMIO 0x40600000 (options '115200n8')
printk: bootconsole [uartlite_a0] enabled
cma: Reserved 16 MiB at 0x8d800000
Linux version 5.4.0-xilinx-v2020.2 (oe-user@oe-host) (gcc version 9.2.0 (GCC)) #1 PREEMPT Fri Dec 4 04:46:15 UTC 2020
setup_memory: max_mapnr: 0x10000
setup_memory: min_low_pfn: 0x80000
setup_memory: max_low_pfn: 0x90000
setup_memory: max_pfn: 0x90000
Zone ranges:
DMA [mem 0x0000000080000000-0x000000008fffffff]
Normal empty
HighMem empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000080000000-0x000000008fffffff]
Initmem setup node 0 [mem 0x0000000080000000-0x000000008fffffff]
setup_cpuinfo: initialising cpu 0
setup_cpuinfo: Using full CPU PVR support
wt_msr_noirq
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists, mobility grouping on. Total pages: 64960
Kernel command line: console=ttyUL0,115200 earlycon root=/dev/ram0 rw
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 228324K/262144K available (5400K kernel code, 177K rwdata, 1456K rodata,
167K init, 564K bss, 17436K reserved, 16384K cma-reserved, 0K highmem)
Kernel virtual memory layout:
* 0xfffea000..0xfffff000 : fixmap
* 0xff800000..0xffc00000 : highmem PTEs
* 0xff7ff000..0xff800000 : early ioremap
* 0xf0000000..0xff7ff000 : vmalloc & ioremap
rcu: Preemptible hierarchical RCU implementation.
Tasks RCU enabled.
rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
irq-xilinx: /amba_pl/interrupt-controller@41200000: num_irq=4, sw_irq=0, edge=0x9
xilinx_timer_init: Timer base: 0xf0020000, Clocksource base: 0xf0020010
clocksource: xilinx_clocksource: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
sched_clock: 32 bits at 100MHz, resolution 10ns, wraps every 21474836475ns
/amba_pl/timer@41c00000: irq=1, cpu_id 0
xilinx_timer_shutdown
xilinx_timer_set_periodic
Calibrating delay loop... 48.74 BogoMIPS (lpj=243712)
pid_max: default: 4096 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
rcu: Hierarchical SRCU implementation.
devtmpfs: initialized
random: get_random_u32 called from bucket_table_alloc.isra.0+0x70/0x218 with crng_init=0
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 16 (order: -4, 448 bytes, linear)
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic allocations
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(0.350:1): state=initialized audit_enabled=0 res=1
clocksource: Switched to clocksource xilinx_clocksource
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 6144 bytes, linear)
TCP established hash table entries: 2048 (order: 1, 8192 bytes, linear)
TCP bind hash table entries: 2048 (order: 3, 40960 bytes, linear)
TCP: Hash tables configured (established 2048 bind 2048)
UDP hash table entries: 128 (order: 0, 6144 bytes, linear)
UDP-Lite hash table entries: 128 (order: 0, 6144 bytes, linear)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Trying to unpack rootfs image as initramfs...
random: fast init done
Freeing initrd memory: 6764K
workingset: timestamp_bits=14 max_order=16 bucket_order=2
Key type cifs.idmap registered
romfs: ROMFS MTD (C) 2007 Red Hat, Inc.
io scheduler mq-deadline registered
io scheduler kyber registered
XGpio: gpio@40000000: registered, base is 510
XGpio: gpio@40000000: dual channel registered, base is 504
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
40600000.serial: ttyUL0 at MMIO 0x40600000 (irq = 4, base_baud = 0) is a uartlite
printk: console [ttyUL0] enabled
printk: console [ttyUL0] enabled
printk: bootconsole [uartlite_a0] disabled
printk: bootconsole [uartlite_a0] disabled
brd: module loaded
loop: module loaded
spi-nor spi0.0: s25fl128l (16384 Kbytes)
4 fixed-partitions partitions found on MTD device spi0.0
Creating 4 MTD partitions on "spi0.0":
0x000000000000-0x000000220000 : "fpga"
0x000000220000-0x0000002a0000 : "boot"
0x0000002a0000-0x0000003a0000 : "bootenv"
0x0000003a0000-0x0000009a0000 : "kernel"
libphy: Fixed MDIO Bus: probed
NET: Registered protocol family 17
Key type encrypted registered
Freeing unused kernel memory: 164K
This architecture does not have kernel memory protection.
Run /init as init process
INIT: version 2.88 booting
mount: mounting proc on /proc failed: Device or resource busy
mount: mounting sysfs on /sys failed: Device or resource busy
/etc/rcS.d/S04mdev: line 10: /proc/sys/kernel/hotplug: No such file or directory
Sat Dec 5 14:17:57 UTC 2020
random: dd: uninitialized urandom read (512 bytes read)
Configuring packages on first boot....
(This may take several minutes. Please do not power off the machine.)
Running postinst /etc/rpm-postinsts/100-sysvinit-inittab...
update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)
Removing any system startup links for run-postinsts ...
/etc/rcS.d/S99run-postinsts
INIT: Entering runlevel: 5
Configuring network interfaces... Cannot find device "eth0"
Starting Dropbear SSH server: random: dropbearkey: uninitialized urandom read (32 bytes read)
Generating 2048 bit rsa key, this may take a while...
random: dropbearkey: uninitialized urandom read (32 bytes read)
random: crng init done
Public key portion is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Ixdq8q05F/6974vKDs6onvP/35jPb0e3LdkOUDFp
Q64CGJVgww1sJj9MacLYlRNIqmt5dDtoHE5MHjQTJG4yyaIXO072xVGiUz9bPJKwXxy/loVMYDLeJbGH
7cLW85DQh9isR4CDgnRJzYRV0he6FMTKmLRcb+7FLhHOQ8LCDAZqf1St114LXqK54Qzbpy+7k+tTLqK8
QUSXys5ad52WzzYhRArUYO/78NLPXSYsjciULBGnmaqnC2Yfp2ls3M+KiAYvVuzgSr1B6iIgPdIoY0iH
ulkn8qS/Vz2JWFYKZs5CYa35X39ZukreVGc3mjpmzyi6h9Ws6PaXdb2btMG7 root@sp50
Fingerprint: sha1!! fe:ab:fb:e7:42:2d:13:d8:ae:8d:cf:9e:d0:0f:b8:ab:e7:1b:e4:1c
dropbear.
Starting internet superserver: inetd.
Starting syslogd/klogd: done
Starting tcf-agent: OK

PetaLinux 2020.2 sp50 /dev/ttyUL0

sp50 login: root
Password:
root@sp50:~# uname -a
Linux sp50 5.4.0-xilinx-v2020.2 #1 PREEMPT Fri Dec 4 04:46:15 UTC 2020 microblaze GNU/Linux
root@sp50:~# uname -a
Linux sp50 5.4.0-xilinx-v2020.2 #1 PREEMPT Fri Dec 4 04:46:15 UTC 2020 microblaze GNU/Linux
root@sp50:~#

Now I'm thinking it would be cool to design a small addon board with 100 Mbit ethernet PHY and a UART-USB bridge - there should be just enough GPIOs to do that, and 100 Mbit PHY uses MII bus running at 25 MHz, so lack of length-matching should not be a big deal.
Right now I'm booting everything through JTAG as the system as-built does not fit into 128 Mbit QSPI flash device I have installed on a board. I think it's possible to cut away enough fat to make it fit, or alternatively it's possible to replace the flash with 256 Mbit device in the same SOIC-8 package.
Also Xilinx recently added experimental support of SMP into petalinux, would be interesting to see if I can get it to work. I've created a dual-core design, and it fits into S50 device I have on a board along with 128K of L2 cache (implemented using System Cache IP).

[SiliconWizard]

I didn't know MicroBlaze was supported by Linux (but I've never used it, so it's just that I didn't know!)

[asmi]

I didn't know MicroBlaze was supported by Linux (but I've never used it, so it's just that I didn't know!)

Linux support was initially implemented by the company PetaLogix (hence the name - "PetaLinux"), but it was acquired by Xilinx back in 2012.
I think Antel's NIOS is also supported by Linux, though as far as I remember only for paid versions as free version lacks MMU.

[DiTBho]

What do you need to implement in order to support SMP or AMP?
Is there an IP, or do you have to manually implement code?

[asmi]

What do you need to implement in order to support SMP or AMP?
Is there an IP, or do you have to manually implement code?

Since this support is experimental, there is next to zero official (or unofficial) documentation, and the code will not even compile as-is if you turn on SMP, so you will need to make some code changes to make the build work.

As for system design, from what I've been able to gather, it requires identical Linux-grade cores with ACE for their I/D buses, all connected to a System Cache IP instance which works as a unified L2 cache for all cores, you also need a dedicated interrupt controller and dedicated timer for each core, and also a single "global" timer for entire system, and all of that is on top of regular requirements for some kind of non-volatile memory (QSPI being the typical example), a minimum of 32 MB of RAM, and some kind of UART or network interface, so you can actually interact with the system. All peripherals needs to be connected to a single peripheral bus accessible to all CPU cores. Also as interrupt assignment to individual cores is not implemented yet, all PICs need have all interrupts wired the same way.

Here is the link to relevant commit to official petalinux github repository: https://github.com/Xilinx/linux-xlnx/commit/17cbc3a2e0bebb8c0fe7d5ae060e3412fd1a0be6 As it's all is very experimental, be prepared to dive in head first into the source code and debug things.

[asmi]

Just in case anyone is interested - I've designed a little addon board with 100 Mbps Ethernet PHY and USB-UART bridge (because using mdm's uart submodule is not very convenient with Linux shell). Like the main board, the source files are in my github rep, and the license is WTFPL , so feel free to do what you want with it. And as is typical of me, I've screwed up the first revision by mixing up USB D+ and D- lines , but the second revision works like charm, and it works in petalinux (tested in version 2020.2) as well as in lwip stack for baremetal applications (tested using lwip-echo sample in Vitis 2020.2 using the same bitstream that I used for petalinux build) because I use the same PHY chip as Arty board (I've done this exactly to minimize a chance for some kind of incompatibility).

[rstofer]

for someone who only just learning to play/program with FPGA, i will be interested at reasonably priced ready made dev board + step by step on how to program it and what toolchains etc... do you think about selling it?

There are literally hundreds of boards, some with better reference material than others.
I picked a board designed for university classes even though I know there are cheaper boards because of the Reference Material.

https://reference.digilentinc.com/reference/programmable-logic/basys-3/start?redirect=1

See in the list where it talks about "Getting Started With Vivado"?  This is a very friendly introduction for newcomers.  There is a link to installing Vivado as a separate page.

Digilent has terrific documentation but their boards are expensive.  You could start with the Lattice IceStick and follow the instructions at VHDLwhiz.com.  It is definitely inexpensive:

https://www.mouser.com/new/lattice-semiconductor/lattice-icestick-kit/

NandLand.com has an inexpensive board but I haven't tried it:

https://www.nandland.com/goboard/index.html

There seems to be plenty of documentation.

Here's an interesting company with a number of boards.  I have a couple of the 2.01 boards and the Debug adapter board.

https://www.ztex.de/usb-fpga-2/

The OP's board is another possibility if he gets to production quantities.  It's nice to have a built-in forum at eevBlog

[NorthGuy]

Just in case anyone is interested - I've designed a little addon board with 100 Mbps Ethernet PHY and USB-UART bridge

No pictures?

[asmi]

No pictures?

Here you go It's not very interesting - just two chips (PHY and USB-UART) and some support components.
FPGA die temperature hovers around 60°C when Linux is running, so heatsink is definitely required to prevent it from overheating.

[asmi]

Oh - one more thing. Since I got some blank PCBs for both the main board and that addon, if someone wants to make one for himself, pls let me know and I can ship a board for free (you cover shipping). PCBs are from JLCPCB, so the soldermask is not very durable, but you shouldn't have too much trouble assembling it using hot air gun and soldering iron. You will need a microscope though, as 0201 are too damn small to even see with naked eyes, even if you have a very good vision.
I can probably ship an assembled one too, but only in Canada as I don't want to deal with export BS at this point as QSPI flash device I use is export-controlled (yea, I was surprised to find this out - FPGA itself is EAR99, but the freaking flash chip is not ). I still don't have PnP machine, so don't expect fast shipping as I will be assembling to order - well unless someone orders like few 100's of them so I will have to see into some mass-production options

[PointyOintment]

I might be interested if not for the fact that I already have a Xilinx FPGA board (Papilio Duo) sitting around unused. That's probably the case for a lot of people here.

How is the heatsink attached? It looks like a clip that mechanically attaches to the board, but I don't see any holes in the board in the pictures in the first post.

[asmi]

How is the heatsink attached? It looks like a clip that mechanically attaches to the board, but I don't see any holes in the board in the pictures in the first post.

It's attached to the BGA package itself. Here is the mounting mechanism: https://www.qats.com/Heat-Sink/superGRIP  The specific heatsink I use is this one: ATS-X50150G-C1-R0 It's not cheap, but it ships with all mounting hardware, requires minimal PCB space (~5mm around BGA package), no holes, only takes a few seconds to mount, and the mounting is reversible, meaning you can remove the heatsink without destroying anything.

[gnuarm]

Oh - one more thing. Since I got some blank PCBs for both the main board and that addon, if someone wants to make one for himself, pls let me know and I can ship a board for free (you cover shipping). PCBs are from JLCPCB, so the soldermask is not very durable, but you shouldn't have too much trouble assembling it using hot air gun and soldering iron. You will need a microscope though, as 0201 are too damn small to even see with naked eyes, even if you have a very good vision.
I can probably ship an assembled one too, but only in Canada as I don't want to deal with export BS at this point as QSPI flash device I use is export-controlled (yea, I was surprised to find this out - FPGA itself is EAR99, but the freaking flash chip is not ). I still don't have PnP machine, so don't expect fast shipping as I will be assembling to order - well unless someone orders like few 100's of them so I will have to see into some mass-production options

The Avnet web page says the part is ECCN: 3A991.D.  I've always had trouble figuring this stuff.  The government docs are inscrutable.  The only reliable source of info is the vendors that I can see.

The odd thing is the requirement in that section is "Field programmable logic devices having a maximum number of single-ended digital input/outputs between 200 and 700".  So a 196 pin package would not apply.  Maybe they count the I/Os on the die as it is easy to repackage a die i expect.  But when I look up 3A991.d it refers to column AT1 in the CCL Country Chart which is entirely empty!  Why have a classification that has no restrictions?  Are there other restrictions? 

The other category this might fit is 3A001.a.7 which is for devices with more than 700 I/Os, so again not a match and again a reference to column AT1, the null set. 

So far I've been safe with my products because I supply to a networking company who doesn't want to go to jail when they ship their products with my board inside.  Actually, their electronics is much more advanced, but apparently covered by the same rules. 

[asmi]

The Avnet web page says the part is ECCN: 3A991.D.  I've always had trouble figuring this stuff.  The government docs are inscrutable.  The only reliable source of info is the vendors that I can see.

On DK website, where I actually bought the device I'm using, it's listed as EAR99: https://www.digikey.ca/en/products/detail/xilinx-inc/XC7S50-2FTGB196C/8040781
But in general - this is exactly why I don't want to deal with export controls. I though about doing a Kickstarter for a few FPGA projects, but when I realized the amount of bureaucracy I will have to go through to ship internationally, I decided that it was more than I was prepared to handle 

[gnuarm]

At one time I had to ship my product to Mexico where my customer assembles their systems.  What a huge PITA!  I tried filing the paper work myself to save the $10 FedEx wanted, but only once!  Everything about using the government web site is a big hassle.  Just generating a password was ridiculous.  They had some seven or eight rules and I followed them all.  It said I didn't do it right, I had to include special characters.  Try again, no, not THAT special character.  Try again, no, not THAT special character either, here's the list of what you can use.  Try again, no, you aren't doing it right!!!  No explanation.  Turns out their list of special characters was not correct.  Then you had to change it every 30 days which means I had to write it down, which is a no-no for security actually.  Then once I got locked out of the account because I didn't change it in time I had to speak to someone to get back in and that took two freaking days to get someone on the phone!!!   

I gladly paid FedEx their $10 from then on. 

My customer ended up with a different assembler who had me ship to Laredo, TX and they trucked it across the border.  I gave them the whatever it's called number for the product once and they never bothered me about it again.  Ahhhh!   

[gnuarm]

Do you have a list of the design rules you used on this board?  I want to consider using the 196 pin package and want to get some pricing from my assembly house.  I'm not sure what solder mask rules to use among others.

[BrianHG]

No pictures?

Here you go It's not very interesting - just two chips (PHY and USB-UART) and some support components.
FPGA die temperature hovers around 60°C when Linux is running, so heatsink is definitely required to prevent it from overheating.

Funny, you seem to have compensated for the length of trace between the IC U2 and your connector, but it doesn't seem to take into account the odd length of the traces on the Sparta board to it's connector.

[asmi]

Funny, you seem to have compensated for the length of trace between the IC U2 and your connector, but it doesn't seem to take into account the odd length of the traces on the Sparta board to it's connector.

I did length matched them on a main board too (see rev D) and even manufactured these boards, but it turned out that was unnecessary for MII interface. Now I'm thinking maybe I should attempt 1G RGMII with rev D...

[asmi]

Do you have a list of the design rules you used on this board?  I want to consider using the 196 pin package and want to get some pricing from my assembly house.  I'm not sure what solder mask rules to use among others.

This board was designed around JLCPCB's 4 layer process (as the absolute cheapest way to get 4 layer boards with controlled impedance) and their JLC2313 stackup as more suitable for high-speed designs (thinner prepreg allows for thinner traces for the same impedance), but I didn't go all the way down to 0.09 mm traces, rather the smallest traces/spacings on the board are 0.1 mm. I use 0.2 mm hole/0.45 mm pad vias. As for soldermask, I have set 0.05 mm clearance and 0.1 mm min width, and that seemed to work OK.

[gnuarm]

Do you have a list of the design rules you used on this board?  I want to consider using the 196 pin package and want to get some pricing from my assembly house.  I'm not sure what solder mask rules to use among others.

This board was designed around JLCPCB's 4 layer process (as the absolute cheapest way to get 4 layer boards with controlled impedance) and their JLC2313 stackup as more suitable for high-speed designs (thinner prepreg allows for thinner traces for the same impedance), but I didn't go all the way down to 0.09 mm traces, rather the smallest traces/spacings on the board are 0.1 mm. I use 0.2 mm hole/0.45 mm pad vias. As for soldermask, I have set 0.05 mm clearance and 0.1 mm min width, and that seemed to work OK.

I find JLC's web page on capabilities to be unclear.  That's why I asked.  When I've tried asking the current assembly house about this they always say they will build whatever I give them.  lol   They don't seem to get into the specs department.   I'll try asking them what they can build cheaply and see if their pwb house can come up with specs.

[asmi]

I find JLC's web page on capabilities to be unclear.  That's why I asked.  When I've tried asking the current assembly house about this they always say they will build whatever I give them.  lol   They don't seem to get into the specs department.   I'll try asking them what they can build cheaply and see if their pwb house can come up with specs.

If your PCB fab can do 0.1/0.1 mm traces/spacings, 0.2/0.45mm vias, controlled impedance and the right stackup, you will be good to go. I recommend you to not dwell too much on a soldermask, as it's not super-critical here, and in my experience fabs which can match or exceed the geometry requirements we're talking about here, will have good enough soldermask process to successfully assemble boards. Stackup is super-important to get correct impedances, if you will have to use a different one, you will need to adjust all CI traces' width (which are DDR2 and HDMI on this board) to match whatever width your stackup requires to achieve 50 Ohm single-ended/100 Ohm differential impedance.
And this is why nowadays designing a "fab-agnostic" board is almost impossible for anything high-speed. Thankfully, quite a bit of fabs are willing to manufacture boards to your stackup specs as long as you didn't spec materials which either don't exist (for example speccing out a prepreg sheets of thickness which doesn't actually exist), or hard to source. Some won't even charge extra for custom stackup - like WellPCB, which is my fab of choice for production boards, as well as prototypes for 6 layers and above.

[gnuarm]

It's easy to say "don't worry about it", but if I am asking for a quote I need to give them a spec.  It's not like I have a board laid out they can measure from.  That's ok.  I have the Xilinx numbers and have derived values from that.  I was just looking for some confirmation.

[asmi]

It's easy to say "don't worry about it", but if I am asking for a quote I need to give them a spec.  It's not like I have a board laid out they can measure from.  That's ok.  I have the Xilinx numbers and have derived values from that.  I was just looking for some confirmation.

Well if cheap Chinese fabs nowadays have no trouble manufacturing 0.05 mm solder mask expansion and 0.1 mm wide slivers, whatever fab you use should be able to do that too. Or just order them at JLCPCB and call it a day. I avoid using them for production runs (like I said, I prefer WellPCB, they are a bit more expensive, but offer significantly higher quality), but for 4 layer prototypes they are great.

[gnuarm]

It's easy to say "don't worry about it", but if I am asking for a quote I need to give them a spec.  It's not like I have a board laid out they can measure from.  That's ok.  I have the Xilinx numbers and have derived values from that.  I was just looking for some confirmation.

Well if cheap Chinese fabs nowadays have no trouble manufacturing 0.05 mm solder mask expansion and 0.1 mm wide slivers, whatever fab you use should be able to do that too. Or just order them at JLCPCB and call it a day. I avoid using them for production runs (like I said, I prefer WellPCB, they are a bit more expensive, but offer significantly higher quality), but for 4 layer prototypes they are great.

I don't buy bare PWB.  I buy assembled boards.  That's why I was looking for specs to give to them rather than asking what they can do.  They can do whatever I ask them to do, they just use an appropriate PWB fabricator.  If I want a quote, I give them my specs.

They are presently building my board at a very good price.  I don't want to screw that up by asking them to build a board they will have to use a different service with. 

Also, I don't know what JLC can or can't do.  Their capabilities page is complex and ill defined with many errors in the illustrations.  Not what I think of as a bench mark of any kind.

[nockieboy]

@asmi - just wondering, did you place the DDR2 further away from the FPGA in the PCB for a reason?  Looks like it could have been placed a lot closer?

[asmi]

@asmi - just wondering, did you place the DDR2 further away from the FPGA in the PCB for a reason?  Looks like it could have been placed a lot closer?

There are few reasons:
1. This board was designed for hand-assembly, I wanted to leave more space between BGA devices for easier hot air gun soldering.
2. I only had 2 signal layers to work with (and even those are not fully accessible due to part placement), length matching requires quite a bit of space, so I wanted to have lots of it.
3. Making compact board was not a design goal (as it in most cases goes against point 1), anything at 10x10 cm or below was good enough. I could've made it more compact, but in that case it would be next to impossible to assemble it without a reflow oven.

Also, they are not that far away from each other - only about 12.5 mm apart. The scale on photos and in eCAD can be very deceptive if you don't have some reference to see the actual scale. All DDR2 traces are only about 36 mm long, which is far enough from critical length (interface is running at 333 MHz due to package limitations) to be reasonably sure it will work with no termination for address/control lines without doing actual board-level simulations (which I can can't do in KiCAD, I would have to import the PCB into Orcad Pro somehow).

[moreasm]

Hi asmi,

I did a DDR2 simulation (400 MT/s) with HyperLynx and I'd say the result is excellent.

Attached you will find the report.

Enjoy the reading.

moreasm

[asmi]

Hi asmi,

I did a DDR2 simulation (400 MT/s) with HyperLynx and I'd say the result is excellent.

Attached you will find the report.

Enjoy the reading.

moreasm

Thanks! I wish I would have access to tools like this... But they are waaaay outside of my budget.
It looks like under/overshoot peaks are a bit out of hand, which is kind of expected given the lack of termination, but the over/undershoot areas still within the limits.

[moreasm]

No problem.

I enjoyed taking the test.

The hardest part was successfully exporting the "HyperLynx" format from KiCad. HyperLyinx generated an import error due to the absence of the stackup in the exported file, once added manually (with a little effort) the import was successful.

Then I assigned the IBIS models and the DDRx batch tool did the rest (once configured).

Tomorrow I will make another report with all the images of the signals not just the ones that don't go through.

I will also see to do the "Power Integrity" test.

Congratulations on your work ... 

moreasm

[moreasm]

Hi,

this is a full DDR2 simulation (400 MT/s) with HyperLynx.

The PDF is large, use this link for download: HyperLynx DDR2 Spartan-7 Custom Board Full Report

moreasm

[koba30]

On this board I use TPS65581 triple DC-DC buck converter IC, which is TSSOP-20EP with exposed thermal pad, which sits on a solid chunk of ground plane with a bunch of thermal vias to ensure good thermal conductivity, so soldering it using lead-free solder would be somewhat tricky with hot air gun, but since these are prototypes, you can use leaded solder and that will make things much easier. These ground planes are obviously not an issue for an oven since entire board is heated up to soldering temperature. This chip is great for many reasons - very easy to use due to no need for external compensation, reasonably good efficiency (it's mostly 70+% for main 1.0 V rail and 80+% for other rails), but my experience with it shows it can heat up pretty severely under full load on all rails - hence the need for robust thermal path.

Hi, is this TPS65581 recommended to power the board with a 5v power supply? Or should it be replaced with another converter with a lower input voltage range?

[asmi]

Hi, is this TPS65581 recommended to power the board with a 5v power supply? Or should it be replaced with another converter with a lower input voltage range?

The datasheet says it should be OK, however there is an LDO which generates 5 V from 12 V for HDMI out and a voltage level converter. If you plan to power the board from 5 V, you can remove that LDO as you can provide 5 V to HDMI out and a voltage level converter directly from your input.

[koba30]

The datasheet says it should be OK, however there is an LDO which generates 5 V from 12 V for HDMI out and a voltage level converter. If you plan to power the board from 5 V, you can remove that LDO as you can provide 5 V to HDMI out and a voltage level converter directly from your input.

That’s great, thanks for your advice, I do not plan to use the HDMI so there is not required the 5v power supply.