DDR3 using cheap PCB Fabs?

[PhillyGreg]

Hi Everyone,

I'm working on planning out a board for a camera project I've been working on for awhile now.
Eventually will be targeting a Kintex device. But for a first go at something like this I'm going to go with an Artix in a 484 pin BGA.
Figuring I'll probably screw something up the first time around.

I've done some higher speed designs in the past. Mostly image sensors and DSP + ADC, 200MHZ range.
But this is the first time I'm working with DDR3, and the first time I'm looking at using BGAs that are $100+ a pop.
(Though the 15T is $40 and pin compatible, so might be a cheap way to validate the DDR3)


The current plan is an Artix 7, DDR3 SODIMM, the MGT's will go to cable drivers for 3G SDI, and 32 LVDS pairs for the image sensor.

What I've been running into is all of the inexpensive fabs(<$500USD for a couple boards) all use rather thick prepreg compared to any of the tech notes I've been reading.
Originally was looking at a pair of x16 chips for the RAM, but given the 12 -15 mil traces needed to get to 50 ohm traces...not going to work well there.
It seems somewhat possible to get 4 byte lanes, length matched to 5mil with 12mil traces routed to a SODIMM on a 4 layer board. I'm not opposed to a 6 layer, but it seems the inner layer spacing isn't really any better.

I was curious if anyone here has produced a working FPGA + DDR3 board with any of the low cost, thick stackups?
Any advise, recommendations? Or am I just being super paranoid on this given the Artix maxes out at 1066MT?

Thanks
Greg

[asmi]

I've recently found a fab which quoted me $237 for 10 6-layer boards 10x10 cm with 0.1/0.1 mm traces, 0.2 mm drills, and impedance control on all signal layers. I've chosen the stackup myself (using prepregs and cores that are commonly used in the industry), and they were able to make it almost exactly what I asked for - they even ran field solver to verify that traces I calculated do indeed have correct width for the kind of stackup that is to be used). BTW that board is for Artix-7/DDR3 as well

The boards are being manufactured right now, I'm supposed to receive them some time next week (placed an order last Friday, with lead time of 7 days). I intend to write up a review of the boards once I receive them and have some time to inspect them under my microscope, as there seem to be great interest in finding inexpensive manufacturer who have the process that is good enough for 0.8 mm BGA (and up). They also offered 4-layer boards with the same as above process for only $142 - and their quotes include DHL shipping! I consider it to be a bargain as just about everywhere else I was getting into $200+ territory as soon as I mention things like "0.1 mm traces" or "controlled impedance".

I would be more than glad to recommend it to you (and everyone), but I take my integrity seriously and never recommend products or services that I didn't use personally, so I'd wait until I see the results of that order to see if it's any good. Their sales rep didn't seem to understand half of questions I've asked him, but their supporting engineers were very good and thorough.

Also - if you want to achieve 1066 MT/s with Artix, you will need to use their fastest (-3) speed grade, which is expensive. They support 667 MT/s with -1 grade (as well as -1LE), 800 MT/s with -2, and 1066 with -3.

[ali_asadzadeh]

There is a 4 layer design based on spartan 6 and DDR3 on opencores.org, it has PCIe and sata too, the guy used a clever way of routing to achieve low cost, so it's doable

[asmi]

There is a 4 layer design based on spartan 6 and DDR3 on opencores.org, it has PCIe and sata too, the guy used a clever way of routing to achieve low cost, so it's doable

Can you please provide the link? I'd love to see how he has done it. The problem with DDR is not routing per se - it's the breakout of DDR3 chip which is 0.8 mm pitch BGA.

[PhillyGreg]

I've recently found a fab which quoted me $237 for 10 6-layer boards 10x10 cm with 0.1/0.1 mm traces, 0.2 mm drills, and impedance control on all signal layers. I've chosen the stackup myself (using prepregs and cores that are commonly used in the industry), and they were able to make it almost exactly what I asked for - they even ran field solver to verify that traces I calculated do indeed have correct width for the kind of stackup that is to be used). BTW that board is for Artix-7/DDR3 as well

Also - if you want to achive 1066 MT/s with Artix, you will need to use their fastest (-3) speed grade, which is expensive. They support 667 MT/s with -1 grade (as well as -1LE), 800 MT/s with -2, and 1066 with -3.

That sounds like a great deal. I was seeing prices around that for 6 layer with much larger traces and vias.
Would be very interested to hear how your test turns out. I'll keep an eye out.

I'm taking my time planning this one so I'm not expecting to try and fab anything till well after Christmas time.

Thanks for the reminder there, I knew there was a reason 1066MT sounded fast. (I've been reading way too many data sheets lately.) 800MT is perfectly fine for me, I need the -2 to meet timing anyway.

Glad to know someone else is working with similar parts, everyone I know thinks I'm nuts trying to design this stuff without a large prototype budget and HyperLynx.

There is a 4 layer design based on spartan 6 and DDR3 on opencores.org, it has PCIe and sata too, the guy used a clever way of routing to achieve low cost, so it's doable

This is one I've looked at, seems like it matches up.
https://opencores.org/project,spartan6_pcie

Source files are Altium.

I've looked at it a bit. Other than length matching it seems to ignore most of the "rules" of DDR3 layout.
Looks like .3mm drils and .1mm annular ring (.5mm total diameter), .15mm traces.
He doesn't mention stackup, on the page. (In Altium, it shows .320mm cores and prepreg)

Other point I noted was the layer arrangement.

Top layer is a ground flood, L2 ground plane with a cut for the DDR3 data and some address, L2 ground plane cut for DQS and Clock, Address/CTL, Bottom DDR3 1.5V plane

I don't really have the experience to judge if this is a good practice or not.
According to the page it works, but generally violates most of the "rules" I read, so without having good SI tools at hand I was a little hesitant to try something along these lines.

[asmi]

That sounds like a great deal. I was seeing prices around that for 6 layer with much larger traces and vias.
Would be very interested to hear how your test turns out. I'll keep an eye out.

Unfortunately I don't have any instrumentation to verify if the fab got CI traces right (and I'm not willing to sell my car for one of those), so I plan to verify it in old-school way - assemble it and see if it works. There is also USB3.0 diff traces on board (which are 5Gbps each) so if those will work, I will assume that the fab has done great job. I plan to use XC7A35T-2FTG256I and x16 DDR3L chip for the first board to minimise damage in case I screwed something up - this is the first revision, and I don't have a very good history of getting my boards to work on a first try

Glad to know someone else is working with similar parts, everyone I know thinks I'm nuts trying to design this stuff without a large prototype budget and HyperLynx.

I wanted to do something akin to what I'm doing right now for quite a while, but I didn't fancy spending $400+ on a 6-board, so I've invested quote a lot of time into looking for affordable fab that would be able to make boards to the specs I wanted. Now spending $240 doesn't sound great either, but at least it's something I can actually afford to spend every so often - especially since BOM cost for just a single board isn't that far off the price of PCBs. And since I design boards myself, I can't possibly design more than one 6-layer board per month anyway - so this pricing is more-or-less OK with me.
The thing is - if this fab will hold true to it's promise, it will open a lot of possibilities - like making boards for Zynq, just about any MPU in existence (they all are in 0.8 mm pitch BGA or even finer), and a whole lot of other 0.8 mm ICs. This is why I decided it's worth to take a risk and placed an order even though some initial exchanges left me a bit uneasy.
BTW - this is the stackup I ended up using:

As you can see it's pretty good for HS traces. All coppers are 35 um (1 Oz).
And just for reference - this is the stackup I've requested:

It's almost spot-on (to ~0.01mm tolerance). I used information from here to design stackup such that it could be manufactured.

I've looked at it a bit. Other than length matching it seems to ignore most of the "rules" of DDR3 layout.
Looks like .3mm drils and .1mm annular ring (.5mm total diameter), .15mm traces.
He doesn't mention stackup, on the page. (In Altium, it shows .320mm cores and prepreg)

Other point I noted was the layer arrangement.

Top layer is a ground flood, L2 ground plane with a cut for the DDR3 data and some address, L2 ground plane cut for DQS and Clock, Address/CTL, Bottom DDR3 1.5V plane

I don't really have the experience to judge if this is a good practice or not.
According to the page it works, but generally violates most of the "rules" I read, so without having good SI tools at hand I was a little hesitant to try something along these lines.

Yea it looks a bit fishy. I've recently purchased Orcad PCB Editor Pro (they were kind enough to offer me a massive discount since I use it for hobby projects only), which does include SI tools like IBIS/HSPICE sim, and thanks to them I feel a little better about that first board I'm waiting for now.

[darrell]

I've recently found a fab which quoted me $237 for 10 6-layer boards 10x10 cm with 0.1/0.1 mm traces, 0.2 mm drills, and impedance control on all signal layers. I've chosen the stackup myself (using prepregs and cores that are commonly used in the industry), and they were able to make it almost exactly what I asked for - they even ran field solver to verify that traces I calculated do indeed have correct width for the kind of stackup that is to be used). BTW that board is for Artix-7/DDR3 as well

I'd be curious to know the name of the manufacturer if it works out. I've used Eurocircuits (about $400 for 8L 0./1mm track/space) in the past for a Zynq DDR3 design which was successful. Their stack wasn't ideal, so I couldn't hit the desired impedance. It worked and had good signal integrity since the traces were short. The local fab charges about $1600 for a minimum order of 8 layer with a custom stack which I've done a time or two.

[PhillyGreg]

The thing is - if this fab will hold true to it's promise, it will open a lot of possibilities - like making boards for Zynq, just about any MPU in existence (they all are in 0.8 mm pitch BGA or even finer), and a whole lot of other 0.8 mm ICs.

I had a bit of tunnel vision reading the specs you posted, and completely missed the .1mm trace spec.
The Zynq devices that one of the forum members has had for sale, have been mighty tempting.

Also makes for really nice routing on the 1mm devices.
Looks like three rows out on the first layer. Depending on the drill to copper, possibly on the inner layers.
Suddenly I'm not wasting half the pins on the 484 pin devices.

The first stackup looks great, I'd be more than happy there. (Length matching 12 - 13mil traces is not fun)
The second even better. I've had a couple $1000 quotes that didn't come up with anything that nice.

On another note, how are the length matching tools in Orcad?
Altium won't use delays for the bond lengths, I've ended up using a spreadsheet to figure out target trace lengths.
(I was surprised by the variation within the designated lane groups on the Artix. Given how picky Xilinx is I figured they would be grouped a little better)

I'd be curious to know the name of the manufacturer if it works out. I've used Eurocircuits (about $400 for 8L 0./1mm track/space) in the past for a Zynq DDR3 design which was successful. Their stack wasn't ideal, so I couldn't hit the desired impedance. It worked and had good signal integrity since the traces were short. The local fab charges about $1600 for a minimum order of 8 layer with a custom stack which I've done a time or two.

Hmm, interesting.
Your local fabs pricing is better than some of the quotes I've gotten. Just well out of the budget for a personal hobby project. Unfortunately

I beginning to get the impression that the DDR3 interface is pretty robust these days.
And with short traces and good routing, that getting reasonably close to 50 ohms is good enough for a prototype/dev system.

[asmi]

I had a bit of tunnel vision reading the specs you posted, and completely missed the .1mm trace spec.
The Zynq devices that one of the forum members has had for sale, have been mighty tempting.

Yea, I'm also really tempted to buy few of those.

Also makes for really nice routing on the 1mm devices.
Looks like three rows out on the first layer. Depending on the drill to copper, possibly on the inner layers.
Suddenly I'm not wasting half the pins on the 484 pin devices.

I used 0.2/0.45 mm vias, and consequently was able to route 2 traces between vias. This made full breakout of BGA256 a breeze with no difficulties whatsoever - unlike my previous 4-layer design, which I had to almost completely re-lay several times because I routed myself into the corner.

The first stackup looks great, I'd be more than happy there. (Length matching 12 - 13mil traces is not fun)
The second even better. I've had a couple $1000 quotes that didn't come up with anything that nice.

My boards are already on their way! Hoping to get them tomorrow or a day after that. As soon as I have a chance to inspect them, I will post an update. Probably won't get around to assembling them for another week or two though as I will have other things to do during weekends.

On another note, how are the length matching tools in Orcad?
Altium won't use delays for the bond lengths, I've ended up using a spreadsheet to figure out target trace lengths.
(I was surprised by the variation within the designated lane groups on the Artix. Given how picky Xilinx is I figured they would be grouped a little better)

In Orcad/Allegro it really depends on the license level. I have Professional level license, and it doesn't include z-height if vias into length, but constraint manager allows to specify additional offsets to tolerance on a per-trace level within match groups, so I just manually set those limits according to which layers traces go to. On higher-level licenses (Allegro baselinse, High-speed option) this is accounted for automatically.
I also absolutely love their diff traces handling, where they not only match total trace length, but also can check to ensure that signals arrive to vias in phase - essentially it brakes down the entire traces into sub-traces based on pins and vias, and enforces correct static phase at each endpoint.
I actually kind of like Cadence's approach to setting license limits in such a way that most of higher-level features can be "emulated" on lower-level licenses, even if this will be somewhat labor-intensive - but at least it's possible at all.
I've purchased Pro level license for the sole purpose of trace length matching (initially I was looking at their standard level license, but it wasn't of much use to me without match groups). It is VERY good and can be as precise as you want it to be. They also support tree-style routing via user-defined route schedules and T-points where trace branches off. No Excel is needed. Watch some YT videos of that in action (can't like those as I'm at work now and YT is banned here) - it's very impressive!

Hmm, interesting.
Your local fabs pricing is better than some of the quotes I've gotten. Just well out of the budget for a personal hobby project. Unfortunately

I beginning to get the impression that the DDR3 interface is pretty robust these days.
And with short traces and good routing, that getting reasonably close to 50 ohms is good enough for a prototype/dev system.

From what I read up on the subject of DDR3, exact impedance match is not as big of a deal as people seem to think - infact the best impedance for it is 40 Ohm, but such traces are unpractical in many stackups and at speeds of 1066 MT and slower 50 Ohm is perfectly OK. If you read Xilinx recommendations for DDR3 routing, you will see that they require 40 Ohm traces for 1333 MT and higher speeds (this is obviously for their Kintex/Virtex devices).
Here is the screenshot straight from MIG user guide (UG586):

[asmi]

Also on a subject of DDR3 layout. After reading up on a subject, I've got impression that DQ/DQS lines are the most important to get right, as they can switch up to twice per clock cycle.
However after running IBIS sims even with grossly out-of-spec traces, I realised that ODT will take care even of most extreme cases for writes, and similarly FPGA side's IN_TERM is pretty good at ensuring good SI for reads.
What you really need to watch out for is address/control signals groups, as they don't have ODT, so if you opt to not implement any sort of external termination (this approach is common for FPGA designs with just a single memory IC that is physically close to FPGA), you really need to run IBIS sim for these traces to make sure overshoot is in check. So help you with that, you may need to use _R "reduced strength" drivers on FPGA side. Also for the clock you've got to make sure the eye is open enough. I botched up that part, and only realised the problem after I sent the board for manufacturing, so I will have to solder 0402 termination resistor directly to vias of DDR3 IC (IBIS sim shows the best results when it's just past receiver, so bottom side of the board straight under DDR3 chip seems like the best place, and because those vias happened to be diagonal, I've got 1.3 mm of distance between centers, so 0402 part can easily fit in there).

[asmi]

Small update - DHL managed to lose my package somewhere between their depot and dropoff location , looks like I'm not getting it until at least early next week

[asmi]

Another update - DHL managed to find my shipment and delivered it today. I've posted my impressions here: https://www.eevblog.com/forum/manufacture/ourpcb-affordable-multi-layer-board-manufacturer-my-experience/

[diyaudio]

There is a 4 layer design based on spartan 6 and DDR3 on opencores.org, it has PCIe and sata too, the guy used a clever way of routing to achieve low cost, so it's doable

Can you please provide the link? I'd love to see how he has done it. The problem with DDR is not routing per se - it's the breakout of DDR3 chip which is 0.8 mm pitch BGA.

This is very true, I watched a presentation by ti proving this can be done with a 4 layer board.
see this link.

[krho]

Moronic is that they have a patent on that, but the only thing they did is took out some balls so there is a place for vias and more traces.

[PhillyGreg]

Thanks for all of the input everyone.
Got sidetracked working on the power supply board that will go with this one. Wanted to get it done before the free shipping deal was over.

I was over in another thread talking about bringing up a Zynq board with DDR3x16 - as of today I have what seems to be a working design for 4-layer boards.

Thats awesome to know, I've been reading a lot of 8 layer recommendations, and haven't had the time to really look at the pinout yet.

However after running IBIS sims even with grossly out-of-spec traces, I realised that ODT will take care even of most extreme cases for writes, and similarly FPGA side's IN_TERM is pretty good at ensuring good SI for reads.

Good to know on the clock, worth adding a couple spots for resistors to be safe.
I've read so many datasheets and tech notes, I'd forgotten the 50Ohm was for the low speeds. Thanks for the reminder.

Sounds like ORCAD is a pretty sweet package, I've been eying the deal that I've seen advertised...but not the best time of year to be buying more software.

I did spend some time with Altium this weekend, I know everyone pretty universally feels that it's SI tools are useless.
I agree UI wise they absolutely suck. And it can't use ebd models for the SODIMM.
I figure if they have it in the software, it can't be absolutely wrong. Not as refined, accurate, or misleading I can believe.

(Now if it can't properly model a square wave, a transmission line, and a termination...I'm very concerned about the rest of the program as a whole.)

I threw together a quick schematic with 4 x8 DDR3 chips with both spec and out of spec traces and what I saw matches your description.
Out of spec traces do have an effect, and at over 1066 it is rather dramatic on the Addr/Control lines, but not nearly as bad as I was expecting to see.
The reduced strength is a great tip, that may do the trick.

Given this, I may just finish off the SODIMM layout I've mostly finished. ALLPCB's calculator is showing like $60 and free shipping.
Since I've never done an Artix layout, I'm thinking keep it super cheap with the smallest Artix.
Spend the holidays thrashing it and get it right the first go with ourPCB.

[asmi]

Good to know on the clock, worth adding a couple spots for resistors to be safe.

Be very careful with that as high-speed traces really don't like having stubs. If you will end up having such spots take great care to ensure stubs are as short as humanly possible. Otherwise they might screw you over and make matter worse just by themselves.

Sounds like ORCAD is a pretty sweet package, I've been eying the deal that I've seen advertised...but not the best time of year to be buying more software.

I did spend some time with Altium this weekend, I know everyone pretty universally feels that it's SI tools are useless.
I agree UI wise they absolutely suck. And it can't use ebd models for the SODIMM.
I figure if they have it in the software, it can't be absolutely wrong. Not as refined, accurate, or misleading I can believe.

At this point I can be sure that at least Orcad's field solver is very good. As OurPCB did ran some more advanced (and expensive I'm sure) field solver to figure out my trace's impedance during pre-production prep and it's results were pretty much spot-on with Orcad's, so they didn't have to adjust anything and just released files to production.

Since I've never done an Artix layout, I'm thinking keep it super cheap with the smallest Artix.

If it's going to be your first Artix design, download XMP277 document from Xilinx website. It is pure GOLD, it will ask you for some basic application parameters, and will generate a schematic checklist to ensure you got configuration right! I was able to get my FPGA to be recognized by the Vivado IDE (through Digilent's cable), boot it up, burn the SPI flash and boot from it, as well as working in-circuit debug on a first try! Just follow it to the letter, and mark off items as you go through verification. It will walk you though pin connections, getting right signals to right pins to get config right, etc. Very useful tool!

[PhillyGreg]

If it's going to be your first Artix design, download XMP277 document from Xilinx website. It is pure GOLD

That's an awesome find. Covers most of the things I was tearing my hair out going through manuals to make sure I covered everything.

The first Spartan 6 board I did was a mess, it was my first go at an FPGA and I think everything I could do wrong I did.
One part I miss with QFP parts, easy to cut traces and wire to pins forgotten

Also realized I nearly forgot the XADC for the DDR3 temp sensor.

[asmi]

Also realized I nearly forgot the XADC for the DDR3 temp sensor.

When I was designing my first Artix board, in addition to the checklist, I used schematics of Digilent's Arty board as a reference and also to get an idea of what kind of components to use (for example idea to use REF3012 as XADC's voltage reference came straight from Arty's schematics). Same can be said of using two beads to "clean up" +1V8 power rail so that it can be used to power XADC. There is no need to reinvent the wheel.

[asmi]

In case anyone's looking for just plain prototype quantities of smallish four-layer boards, OSH Park is unbeatable in the States. Just over $10 per square inch for three copies, and the 4layer process is 5/5mil with 0.25mm drill, purple soldermask with ENIG by default. Even the crappiest China producer is no less than 3x that for the minimum quantity.

I think it was already discussed recently. There are three issues with Oshpark - 1) unless you break their DRC and specify slightly smaller annular ring than they require, you can't breakout 0.8 mm BGAs (though somebody did this with 17 mils instead of required 18 mils and got away with it, but YMMV), 2) on typical FPGA board just PMIC takes up at least few square inches, as result, the board tends to be rather large and as such expensive, and finally 3) they don't offer controlled impedance option (due to shared panel I imagine). Also for some reason delivery to Canada via USPS takes forever - at least it took almost full month for my order to arrive. And by the time you order 9 of them (in most Chinese shops ordering anything less than 10 doesn't make sense as it costs same money) they don't appear as cheap anymore. To give you some context, when I asked OurPCB about 4 layer boards with 0.1/0.1 mm (3.9 mil) traces and 0.2 mm drills with controlled impedance and ENIG finish, they quoted me something like $170 + some change with DHL delivery included for 10 boards 10x10 cm - and you get to choose whatever stackup you want! No other PCB fab I know of can get even close to those specs and price.

[texaspyro]

OSHPARK is only price competitive for very small boards.  For 4x6" boards you get 3 four layer boards for $240 in 2-3 weeks.  I do use them quite a bit for smaller boards, but it does not take much before the cost gets out of control.

From gojgo.com in China you get 50 boards for $220 (or 5 boards for $95).  I get boards from them in 8 days with DHL shipping.  There are less expensive places in China than gojgo, but they do really good work and their contact guy is very fluent in English.  Plus, you can send them Eagle/Kicad files and let them muck about making and naming Gerbers.  I have not checked what their minimum design rules are...  it's very rare that I go below 10/10 mil.

If the design fits in a 50x50 or 100x100 mm board, they are $40/$60 for 5, shipped.

One disadvantage of using Chinese board houses for prototypes... you should see my pile 'o "coasters" when the design did not work out and I ordered a 50 board prototype run.

[asmi]

Shrinking the solder mask helps a bit as well. I think it's probably reasonable to get away with a slightly thinner annular ring, and for a prototype I can't really say that I'm hugely worried about it. It seems quite wasteful to have ten boards built when you may find the design is faulty after the first prototype.

Actually I'm now exactly in this position where I got some 6-layer boards which are non-functional due to mistakes in layout. Of 10 boards that I've got, I've trashed one for testing really old LF solder paste that I happen to have to see if it's still usable (the answer is - it's not, got to throw it away and buy a fresh one). Another board was partially assembled with power supply-related parts only (this part of the board is not affected by layout mistakes) to confirm that all calculations I made for voltage- and current-setting resistors are correct and will not burn down expensive parts when I will fully assemble the board. So these boards were definitely not wasted, as I extracted useful information out of them. I will likely use a couple more boards to tune reflow profile to new LF paste, as well as I plan to practice placing tiny 0201 parts as I never actually used such small parts before (I'm fully comfortable with 0402 and bigger - placed 100's of them with no problems). So I wouldn't call them a complete waste even if the use will obviously be not what it was intended.
Now, the notion of "prototype" is very theoretical when it comes to high-speed multilayer boards because getting design to actually work when boards are made by specific manufacturer is very important part of "prototyping". Each manufacturer tends to use different materials/stackups, and these things have huge effect on the way you design the board. So switching manufacturer mid-project might very well be akin to near-complete redesign, as all high-speed traces will need to have different widths/spacings to accommodate new materials and stackup.

Point 2 - hmm. The PMIC area on the PYNQ Z1 I have takes up just about a square inch. If you abandon DDR and run everything except the core at 3.3V, you can get away with quite a bit less space.

I suspect PYNQ board has quite a bit more layers than 4. Have you actually tried to design PMIC design that would fit on 1 sq inch? And no, you can't "run everything except the core at 3.3V". At the very least you will need another DC-DC converter for 1.8 V rail to power things like PLL, which actually are quite power-hungry and can easily consume half amp of power or even more each. For example, right now I'm looking at quite simple design consisting of MB soft-core with DDR3 controller, and power estimation tool tells me that AUX rail will consume almost 0.3 A of power. You can perhaps get away with linear regulator in marginal cases (if you don't care about power efficiency), but most real-world designs I've seen use switchers for that rail.
Not to mention that foregoing external memory also severely limits usability of FPGAs. There is a reason that almost any board out there has at least some kind of external memory.

Essentially, if I have only $50 to spend on prototype PCBs, OSHPark is the only company that will sell me any 4 layer ENIG boards at all.

PCBWay will make 10 10x10 cm 4 layer boards with ENIG for $75 + shipping (granted it will be only 0.15/0.15/0.3 process). I actually have working Artix 4 layer board with ENIG that was manufactured by them (and assembled by me). It has two 64M HyperRAM memory chips as it's impossible to place DDR3 with such process.

And there was #3, which is also quite important If you don't order this option and instead rely on manufacturer to always manufacture same stackup, with higher speed you run a risk of your design exhibiting intermittent failures if it was marginal to begin with. You might get away with it for relatively robust buses like DDR2/DDR3, but things like LVDS, and especially 1Gbps+ lines (for example USB3 tx/rx lines, SATA, PCIE, etc.) are almost guaranteed to screw you over if you won't get your trace impedance/diff impedance right.

[asmi]

The Pynq is a six-layer board - the PMIC is a TPS65400, a very nice chip with two 4A outputs and two 2A outputs. The 1.8V rail is required, as you point out, but that's essentially all. It's not a terrible hard chip to lay out if you assume 5V 2A input.

This is the chip I use as main PMIC as well. I did my best trying to lay it out as compact as possible, but still it took 26x33 mm, or approximate 1.3 sq inch. If you add an area that is taken by PMIC reset button as well as power jack - it's already over 2 sq inch. If you want to power the board from wall wart, you will probably need another DC-DC converter to provide +5V rail so that you can power the board either from USB or from dedicated power supply. On my current board I also needed a boost converter to power LCD backlight. In total everything power-related occupies an area of about 3 sq inch. As you can see it's fairly tightly packed, and most of passive components are as small as they could possibly be.

[asmi]

Check it out, this guy put a 0.8mm 217-pin BGA and some DDR2 on a 4-layer OSHPark board: http://hforsten.com/making-embedded-linux-computer.html

Yes I saw that. But that is beside the point as he ultimately just got lucky. He built a very marginal design with no length matching of DDR2 traces, out-of-spec traces, no termination resistors. This is not a way to design a robust circuit with good margins. And he didn't need most of IO pins broken out - which is what ultimately made it possible. No good for most FPGAs designs.

Basically until I see a devboard with Artix, DDR3-800 x16 and some assorted peripherals made by OSHPark and actually working, I'm very skeptical about all that. Designs with FPGAs are complicated enough as they are, no need to further complicate them by limiting yourself only to what OSHPark can do. It might be an interesting challenge to someone, but not my forte for sure. I like to build robust, reliable designs that just work without any unnecessary drama.

[PhillyGreg]

Check it out, this guy put a 0.8mm 217-pin BGA and some DDR2 on a 4-layer OSHPark board: http://hforsten.com/making-embedded-linux-computer.html

For that size board OSHPark can be a good deal.

I've seen the blog on pushing the design rules, I've done it with them a couple times on small 2 layer boards.
But when it gets to be $150 for three, I prefer to stick to the rules.

Yes I saw that. But that is beside the point as he ultimately just got lucky.

Given the note at the bottom about the random bit error...maybe not so lucky.

I like to build robust, reliable designs that just work without any unnecessary drama.

Exactly, It's no fun debugging designs with hardware that works most of the time.

Point 2 - hmm. The PMIC area on the PYNQ Z1 I have takes up just about a square inch.

Made me giggle a little. Here the discussion is how small the PS can be.

I decided to break the power supply on my design to a second board.
One so I can reuse the expensive pieces when my first couple boards don't work perfectly. (I don't trust trying to setup 6 rails on my bench supplies and not fry something once)
And two then I can set it up for current measurements, and I2C monitoring etc.

In the end it fills up a 10cm x 10cm card rather well...

On the plus side, I've got a couple days off this week so I'm hoping to finally get version one of the FPGA board sent off.
I've managed to convince myself that I stand a pretty good chance of having my DDR3 layout actually work.

The power supply boards should be here this week.
(Still have no clue how they make money with the free shipping, but figured I might as well take them up on it)

[asmi]

Given the note at the bottom about the random bit error...maybe not so lucky.

To be fair to him, the specific MCU he has chosen to use does not support ODT for DDR2, and you already know how much of a difference does it make - so I'm surprised it worked at all. This is why I think he got lucky. Even though we was running memory only at 133 MHz.

I decided to break the power supply on my design to a second board.

If you do that, make sure you've got enough capacitance near power rails that power high-speed interfaces to hold them steady until power supply has a chance to react to abrupt change of current drawn. I can't find the source now, but I remember reading that it's strongly recommended to have onboard supplies for high-speed interfaces to improve transient response. Also if you use termination to Vddr3 / 2 - make sure Vtt tracking supply is on the board as it need to track Vddr3 rail in case in dips or bounces to make sure the eye is as open as it can be (SSTL signals are references to midrail, so if rail jumps - so does the mid-rail, and Vref should follow it).
This is especially important if you want to use memory in DDR3L mode as there is less of a margin - Vref ± 0.15 V as opposed to Vref ± 0.175 V for regular DDR3.

One so I can reuse the expensive pieces when my first couple boards don't work perfectly. (I don't trust trying to setup 6 rails on my bench supplies and not fry something once)

The major problem with using lab supplies is getting power sequencing right. FPGA may fail to initialize if it's grossly wrong. I had this happened (with different FPGA though).

(Still have no clue how they make money with the free shipping, but figured I might as well take them up on it)

They don't. My understanding is that it's time-limited promotion.