Layout considerations for wired ETH

[Sudo_apt-get_install_yum]

Hello everyone.

I'm working on a project with wired ETH, its my first time working with it so I've done a lot of reading but have a few questions that I think more experience designers might know the answer to!

The design is split over two PCBs due to physical space constrains. The boards will be galvanically isolated from each other and connected through a <10 cm FFC.
I've done a mockup drawing, left board is the processor board with MAC an the right board is the I/O-board.

The issue is that I would prefer if all connectors would be on the same PCB (I/O-board) for easier access and maintainability but I'm not sure of running ETH through a semi long FFC.
The processor contains the MAC and an external PHY-chip is connected to the processor through RMII, I might use MII but would prefer RMII due to fewer pins.
My understanding is that the trace length between the magnetics and connector should be as short as possible (< 25 mm)  and not deviate to much from 100Ω while the length between the PHY and magnetics should be longer than 25 mm.
The issue is that I don't want to run both isolated signals and non-isolated signals in the same FFC especially when one is "high speed" ETH (10 Mbit/s).
And running high speed PHY through a FFC cant be to good either? The specifications & design guidelines for RMII doesn't seem to crazy. But it might just be my lack of knowledge with ETH.

Is this a dead end and am I forced to place the ETH connector on the processors PCB or is there a good way to route PHY or twisted-pair semi far through a FFC while preferably maintaining the same signal reference in the FFC?
Is it possible to route the PHY through its own FCC or is it to unreliable?
MII should be better in such a case compared to RMII but I've never done this in practice so its hard to be certain.
Is there any other "slimmed" internal cabling one could use to connect the ethernet from the processor board to a connector on the I/O-board?

The layout isn't done, I've only placed everything on the board to get a rough ide of component placement.

What are your guys thoughts on the matter?
Thanks for any help!

[T3sl4co1l]

My understanding is that the trace length between the magnetics and connector should be as short as possible (< 25 mm)  and not deviate to much from 100Ω while the length between the PHY and magnetics should be longer than 25 mm.

This is a myth and, I think we went through it semi-recently but I don't recall the thread.  Or maybe it was on Stack...

Perhaps someone else can find the thread.

PHY to magnetics only matters if there's troublesome noise coming from the magnetics, and that somehow manages to couple into the PHY.  Toroid cores are used so the external magnetic field is minimal; this would only be due to common mode, electric field from the media side itself.  So it seems a stretch that it would be a problem, even in the worst case (very sensitive PHY, noisy wire right over top it).

And this is irrelevant for integrated magnetics (the connector is shielded), for example.  I suppose you'll be using discrete magnetics with the non-standard connector, so that doesn't apply at least.  But it still seems unlikely there would be a problem.

Conversely, PHY to magnetics distance is constrained by stub length, particularly for traditional 10/100 wiring (CT to AVCC, either side to termination resistors to PHY).  It might be strictly better for this distance to be short... but the length scale that actually matters is still quite large (i.e. a fractional bit time, or say a ns or so -- 10s of cm).

I forget if there were other opinions about this, regarding the purported 25mm limit.

To be clear: in general, my knee-jerk reaction is to seriously doubt or disregard anything that 1. isn't justified by realistic and quantitative (even if rather loose) measures or explanation, and 2. that no such explanation is obviously given by the quantity in question.  Given, of course, that I have adequate knowledge about possible explanations to meet #2 -- this accounts for the Dunning-Kruger gap, at least past the initial peak.

So, if something says "keep 25mm" but doesn't explain why, I would just as well ignore it.  Why, does it have cooties?  Is it radioactive?  Does it have radioactive cooties?  Where does such an arbitrary number come from?  Why is it so suspiciously close to a round, hand-wavey number (25.4mm == 1in)?  How long has this number been repeated, can earlier references be found?  Maybe one of them had an explanation.  Maybe the original case is completely inapplicable and no one remembers that.

Basic critical thinking stuff.  But, it helps also that I'm generally experienced enough to have a feel for what signals, attenuations, distances, etc. are needed in a given application, or at least in the applications I've worked on specifically; that's experience that's unfortunately not so easy to confer by text alone.  So, you can only take this so seriously, I'm afraid.

The issue is that I don't want to run both isolated signals and non-isolated signals in the same FFC especially when one is "high speed" ETH (10 Mbit/s).
And running high speed PHY through a FFC cant be to good either? The specifications & design guidelines for RMII doesn't seem to crazy. But it might just be my lack of knowledge with ETH.

Definitely don't put RMII on it, that's a huge EMI/RFI mess already.  Those signals are simply LVCMOS logic level, so, 3.3V 20mA ~1ns rise sorts of things.  Even with a ground plane, putting it over a flex cable is dubious.  Plain unshielded, suicide.  (You would at least interleave ground between every signal, as for any other ribbon cable; perhaps it would pass tests alright for say a modest length (up to 10"?) and with a few ferrite beads clipped on, and preferably with the boards sharing an RF ground some other way (Y caps and standoffs?) to reduce the common-mode burden on the cable.)

And I would avoid PHY for the above reason (stub length).

Media however, I mean, it's just extending cable, who freaking cares.  That's the other thing about the "advice", distance from magnetics to connector is irrelevant, it's all the same wiring.  Keep the differential impedance around 100 ohms and nothing's the wiser.  Well, keep CM impedance high, don't run over circuit GND plane (can make local plane referenced to centertap / Bob Smith terminators though), simple enough.  Maybe getting Z_DM low enough is hard for planar (diff pair no GND plane), in which case length should be kept short, that's fine.



Tim

[Sudo_apt-get_install_yum]

Thanks for the lengthy reply!

My understanding is that the trace length between the magnetics and connector should be as short as possible (< 25 mm)  and not deviate to much from 100Ω while the length between the PHY and magnetics should be longer than 25 mm.

This is a myth and, I think we went through it semi-recently but I don't recall the thread.  Or maybe it was on Stack...

That might be the case but the reason for making this post is mostly due to what I perceive as contradicting and scattered information on the web.
My intuition is that it would be better to run longer traces from the magnetics since it would be an "extension" of the external cable, just internal and without the twisted pair and shielding.
But reading forum posts and application notes suggests the opposite. The RMII specifications make it seem like it is a very robust and "loose" interface, maybe a bit less than regular MII while specifications and internet posts make it seem like "twisted pair" is incredibly sensitive.

I'm not too worried regarding the EMI aspect since the box is made out of cast aluminum and grounded to earth. Assuming your talking about radiated emissions through long internal cables?
I'm reusing a lot from my other designs that have passed EMC tests but ETH is something I've never touched.

So, if something says "keep 25mm" but doesn't explain why, I would just as well ignore it.  Why, does it have cooties?  Is it radioactive?  Does it have radioactive cooties?  Where does such an arbitrary number come from?  Why is it so suspiciously close to a round, hand-wavey number (25.4mm == 1in)?  How long has this number been repeated, can earlier references be found?  Maybe one of them had an explanation.  Maybe the original case is completely inapplicable and no one remembers that.

I get where you're coming from and I mostly agree. Its a lot of different reputable companies recommending and stand by the "< 25 mm" rule and I would assume that they would have done their homework and are more knowledgeable than I am in this subject.
To just ignore something like that takes either a lot of hubris or deep knowledge and experience in the subject.

The below image was the original idea. Keeping the ETH-PHY and magnetics on the processor PCB and sending the twisted-pair down through the FFC to the I/O-board. It would save a lot of conductors in the FFC and would be galvanically isolated from the processor PCB.
FFC signals would be the just the four for the twisted pair with some surrounding ground while the RMII would be 9 signals + 10 ground signals interleaved between the signals.
Avoiding stubs and crossing ground/power planes is just standard layout practice, especially when running different references/galvanically isolated?
I planned on placing ferrites close to the output connector just in case, id keep/remove them deepening on test results.

[T3sl4co1l]

That might be the case but the reason for making this post is mostly due to what I perceive as contradicting and scattered information on the web.
My intuition is that it would be better to run longer traces from the magnetics since it would be an "extension" of the external cable, just internal and without the twisted pair and shielding.
But reading forum posts and application notes suggests the opposite. The RMII specifications make it seem like it is a very robust and "loose" interface, maybe a bit less than regular MII while specifications and internet posts make it seem like "twisted pair" is incredibly sensitive.

AFAIK, MII and RMII are identical in signaling, i.e. basic garden variety LVCMOS, treat it as you would any other GPIO or SPI or whatever signal you would do on-board.  Use source termination on long traces, keep stub length and length matching within reason (pretty lax as the clock rate isn't terrifically high, double check the setup/hold times to be sure; stub lengths, compare with rise/fall time if available), keep normal or common mode noise away so as not to violate noise margins, etc.

RMII is basically double the clock rate for half the signals, so of course is a bit more strict.  I don't see how it would be any more flexible/tolerant, unless it really is something different and special.  Admittedly I haven't looked into it in depth, but as I recall it didn't look like anything different from this basic description, so I have no reason to suspect it's more than this?

I'm not too worried regarding the EMI aspect since the box is made out of cast aluminum and grounded to earth. Assuming your talking about radiated emissions through long internal cables?
I'm reusing a lot from my other designs that have passed EMC tests but ETH is something I've never touched.

Oh, OK.  Do you have additional filtering at the input then -- what's the input anyway, just power?  Any other signals?  Just Ethernet?  So, a glorified router (presumably with some additional processing function which motivates this build)?  Maybe an LCD or pushbuttons or something not shown (but, beware that such openings can emit RFI too)?

I get where you're coming from and I mostly agree. Its a lot of different reputable companies recommending and stand by the "< 25 mm" rule and I would assume that they would have done their homework and are more knowledgeable than I am in this subject.
To just ignore something like that takes either a lot of hubris or deep knowledge and experience in the subject.

Yep, exactly.

Let me see here...

Here's one that disregards it, https://e2e.ti.com/support/interface-group/interface/f/interface-forum/949043/dp83826i-differential-pair-trace-length-from-phy-to-magnetics-of-25mm-is-ok but you might take TI's first-level service engineers with a grain of salt as well, I've seen spotty results from them.  Again, no explanation given.

Oh yeah, it was Stack, I had commented on this one: https://electronics.stackexchange.com/questions/635694/how-to-design-pcb-layout-for-ethernet-to-spi-phy-correctly
Which links to: https://electronics.stackexchange.com/questions/256333/ethernet-distance-from-phy-to-magnetics
which in turn links to Micrel who give no explanation.

Here's a slightly older reference, that uses similar numbers but as maximums instead: https://hub.digi.com/dp/path=/support/asset/technical-note-tn266---pcb-layout-for-the-ethernet-phy-interface/
Still no explanation or reference.

Likely you have to go back very far, back around when 100BASE-T was introduced in the first place.  Maybe IEEE 802.3 itself has clues, I don't know.  This is unfortunately something the internet is very, very poor at researching, so I won't try too hard here.

The below image was the original idea. Keeping the ETH-PHY and magnetics on the processor PCB and sending the twisted-pair down through the FFC to the I/O-board. It would save a lot of conductors in the FFC and would be galvanically isolated from the processor PCB.

Yes, that looks fine.  Hm, there should be another box for PHY in there, I guess it's implied as part of Magnetics.

Tim

[Sudo_apt-get_install_yum]

Thanks Tim.

RMII is basically double the clock rate for half the signals, so of course is a bit more strict.  I don't see how it would be any more flexible/tolerant, unless it really is something different and special.  Admittedly I haven't looked into it in depth, but as I recall it didn't look like anything different from this basic description, so I have no reason to suspect it's more than this?

This was sort of what i was getting at. RMII runs at twice the clock rate compared to MII and half the "buss width" 2-bits vs 4-bits so id assume that it is less immune than MII when running it in 10 Mbit mode. But on the other hand, MII uses two separate clock lines for RX and TX which might be worse than the single/shared clock line that RMII uses and using twice the data lines makes it easier to get timing mismatches. Its obviously more complex than this but this is just my general thoughts regarding this.
They are electrically the same though but conceptually different.

Oh, OK.  Do you have additional filtering at the input then -- what's the input anyway, just power?  Any other signals?  Just Ethernet?  So, a glorified router (presumably with some additional processing function which motivates this build)?  Maybe an LCD or pushbuttons or something not shown (but, beware that such openings can emit RFI too)?

It’s far from a glorified router, but depends on perspective...
The ETH interface is just one of the communication interfaces used. I’m using two RS485 busses for ModBus RTU and would like to incorporate ModBus TCP, the main difference being the PHY, RS485 vs ETH.
I have loads of filtering and protection on all inputs. I use "digital I/O" and some "analog I/O" that support >30 M cabling so lightning protection and heavy filtration is a must due to EMC requirement, lightning, immunity through cables etc.
Luckily, I’m reusing these parts from my other designs that have already passed these test so it shouldn’t be too much of a headache. At least I hope so

Yes, that looks fine.  Hm, there should be another box for PHY in there, I guess it's implied as part of Magnetics.

Exactly, should have just added it for sake of clarity.
Thanks for the links, got a bit of reading to do

[T3sl4co1l]

Ah; gateway, fine, close enough. 

Still very available though -- is the lightning protection and filtering not feasible as an add-on or external thing then?

I guess that means it's also an endpoint, at least in part, so that's probably fine.  Well, I don't know offhand how many precooked analog/digital IO boxes there are out there, but maybe all together it's still worth it.

Tim

[Sudo_apt-get_install_yum]

I’ve looked at PLCs and other types of the shelf solutions. I haven’t been able to find anything that that comes close to filling all the criteria. Il be making several dozen of these a year and talked to several large PLC manufacturers, none are willing to guarantee availability for the next two years. Id not be comfortable limiting myself in that way, luckily I haven’t found an of the shelf solution that would work anyways.

The lightning protection and filtering are a must due to EMC regulations. Any product with a specified cable length longer than 30m is required to pass lightning strike tests. Formfactor is very important so external protection, power adapters, filters and so on are a no go. It also has to pass a bunch of other tests like IP-67, chemical resistance (airborne & water soluble) and vibration tests so relying on external protection and the types of materials the manufacturer chooses to use is not optimal. A lot of work for a single person, but I like it 
There are of course a bunch of other details but that’s a different story : )

This is a version two of a previous thing I designed. Its conceptually identical but with more updated electronics and a bit more feature!

[ajb]

PHY to magnetics only matters if there's troublesome noise coming from the magnetics, and that somehow manages to couple into the PHY.

I have a vague recollection of this being the specific issue cited for the >25mm rule in at least one appnote somewhere at some point.  My suspicion is that this advice dates back to older PHYs and older, possibly non-toroidal transformer styles, where it may have been a somewhat more valid concern.  I could see hanging on to that advice as a manufacturer of eth magnetics wanting to reduce support burden when their products are used with arbitrary PHYs of unknown resiliency to external magnetic fields. 

This was sort of what i was getting at. RMII runs at twice the clock rate compared to MII and half the "buss width" 2-bits vs 4-bits so id assume that it is less immune than MII when running it in 10 Mbit mode. But on the other hand, MII uses two separate clock lines for RX and TX which might be worse than the single/shared clock line that RMII uses and using twice the data lines makes it easier to get timing mismatches. Its obviously more complex than this but this is just my general thoughts regarding this.
They are electrically the same though but conceptually different.

There's no particular reason that two separate clock lines should be better or worse than one, and a careful layout should be sufficient to prevent timing issues regardless of the number of data lines.  If you're truly only running in 10Mbit mode then the clock continues to run at the same frequency but the data lines are changed only every 10th clock.  The RMII spec (which turns out to be just on the internet here) requires that the data be valid for every single one of those ten clock cycles though, so fundamentally the electrical requirements are the same in all the ways that a PCB designer cares about between 10Mb and 100Mb modes, because the data still has to be right by the first clock edge for each of those bits. 

But in any case, running the MDI over FFC is going to be a simpler solution than running MII, so that seems like the way to go.  Standard 0.5mm polyester FFC jumpers are pretty close to 100R differential, so they should actually work pretty well.