Author Topic: Gigabit Ethernet Differential Pair Routing  (Read 12547 times)

0 Members and 1 Guest are viewing this topic.

Offline cheeseburgerTopic starter

  • Contributor
  • Posts: 30
  • Country: de
Gigabit Ethernet Differential Pair Routing
« on: August 24, 2022, 01:00:00 pm »
Hello

I'm currently trying to get a design with Gigabit Ethernet to work properly. Gigabit Ethernet requires 100 Ω differential routing. To achieve this, I used the built-in Calculator in KiCad.

1573138-0

In general, it works and can establish a Link with 1 Gig, but when exchanging multiple Gigabyte of Data it starts to cut out and collect RX Errors. Sometimes it also needs multiple seconds and attempts to even establish a Link.

Attached is a screenshot of the relevant Schematic and PCB sections. The PCB is 4 Layers, with the GND Layer under the Ethernet Signals. The Ethernet pairs are length matched. Track width of the Ethernet pairs is 0.24 mm. PCB Thickness is 1.6 mm.

The PCB is the JLC7628 Stackup: https://cart.jlcpcb.com/impedance

The PHY/Device itself is a Raspberry Pi CM4.

Any ideas what I could try next to improve this?

1573144-1

1573150-2
 

Offline MarginallyStable

  • Regular Contributor
  • *
  • Posts: 92
  • Country: us
Re: Gigabit Ethernet Differential Pair Routing
« Reply #1 on: August 24, 2022, 03:32:45 pm »
I'm not an expert, but does the ground pour on the same layer make this more of a waveguide structure? Might affect the impedance calculations.
« Last Edit: August 24, 2022, 03:35:21 pm by MarginallyStable »
 
The following users thanked this post: karpouzi9, cheeseburger

Online AndyC_772

  • Super Contributor
  • ***
  • Posts: 4559
  • Country: gb
  • Professional design engineer
    • Cawte Engineering | Reliable Electronics
Re: Gigabit Ethernet Differential Pair Routing
« Reply #2 on: August 24, 2022, 04:25:21 pm »
What are the length and type of cable you're using? What's it connected to at the other end?

Is the PCB actually fabricated on a controlled impedance process, and can you get test coupons that demonstrate that you have indeed got the 100R you're aiming for?

If you don't need the full 100m cable length, Ethernet is pretty forgiving of impedance mismatches in short traces at the ends. If you do, then the whole thing becomes a lot more critical. I wonder whether you've got a cabling or ground noise problem, rather than there being anything wrong with your SI on the Ethernet pairs.
 
The following users thanked this post: cheeseburger

Offline jc101

  • Frequent Contributor
  • **
  • Posts: 871
  • Country: gb
Re: Gigabit Ethernet Differential Pair Routing
« Reply #3 on: August 24, 2022, 05:01:07 pm »
Grab the KiCad files from Raspberry Pi for their compute module IO board and see how they implemented it.

https://www.raspberrypi.com/products/compute-module-4-io-board/
 

Offline wraper

  • Supporter
  • ****
  • Posts: 19475
  • Country: lv
Re: Gigabit Ethernet Differential Pair Routing
« Reply #4 on: August 24, 2022, 05:30:25 pm »
Even if there is impedance/length mismatch, I doubt it would be enough to cause noticeable problems. Especially if you don't try to run it with a cable of maximum length.
 

Offline nvmR

  • Regular Contributor
  • *
  • Posts: 92
  • Country: il
Re: Gigabit Ethernet Differential Pair Routing
« Reply #5 on: August 24, 2022, 06:40:29 pm »
Hey OP,

I would look at two points:
1) Biasing the ethernet port correctly - I did not find a datasheet for the phy, or a reference design in a quick look, but did find an old Pi schematic - check it out:
https://www.raspberrypi.org/app/uploads/2012/04/Raspberry-Pi-Schematics-R1.0.pdf
page 3 is the phy (albeit an old 10/100 one)
and also this:
http://wiki.amperka.ru/_media/products:waveshare-raspberry-pi-ethernet-usb-hub-hat:waveshare-raspberry-pi-ethernet-usb-hub-hat-schematic.pdf
and this (which shows a simple connection actually)
https://datasheets.raspberrypi.com/rpi4/raspberry-pi-4-reduced-schematics.pdf

2) Check if your pi isn't overheating, and thus dropping.
« Last Edit: August 24, 2022, 06:44:45 pm by nvmR »
 

Offline cheeseburgerTopic starter

  • Contributor
  • Posts: 30
  • Country: de
Re: Gigabit Ethernet Differential Pair Routing
« Reply #6 on: August 25, 2022, 07:40:15 pm »
What are the length and type of cable you're using? What's it connected to at the other end?

The cable is a shielded 2M CAT6 Cable going to a Cisco Switch. Other devices on the same Switch and also with the same Cable never had any issues.

Is the PCB actually fabricated on a controlled impedance process, and can you get test coupons that demonstrate that you have indeed got the 100R you're aiming for?

The PCB is manufactured with the mentioned Stackup which I also used for my calculations. Does controlled impedance in the ordering process change something else than the Stackup?

If you don't need the full 100m cable length, Ethernet is pretty forgiving of impedance mismatches in short traces at the ends. If you do, then the whole thing becomes a lot more critical. I wonder whether you've got a cabling or ground noise problem, rather than there being anything wrong with your SI on the Ethernet pairs.

It often doesn't even establish the Link with a 2M cable. So there is definitely something going horribly wrong.

I will try to power the Raspberry Pi with a different Power Supply and disconnect all connected peripheral to rule out a Ground issue.
« Last Edit: August 25, 2022, 07:42:28 pm by cheeseburger »
 

Online AndyC_772

  • Super Contributor
  • ***
  • Posts: 4559
  • Country: gb
  • Professional design engineer
    • Cawte Engineering | Reliable Electronics
Re: Gigabit Ethernet Differential Pair Routing
« Reply #7 on: August 25, 2022, 08:03:35 pm »
If you order a board with controlled impedance, the manufacturer should control the tolerances (especially layer thicknesses) to a much higher degree. They'll also fabricate test coupons on the edges of the panel, which can be tested to confirm that the stack-up of the panel really does result in the calculated impedance (+/- some reasonable tolerance).

For this reason, a controlled impedance board is usually quite a bit more expensive than an uncontrolled one.

I don't think this is your problem, though. A poor impedance match might break the link with a 100m cable, where you're operating at the limits of the system's noise margin and ability to echo cancel. At 2m there's plenty of margin, and I'd be very surprised indeed if the PCB alone were the problem.

Can you share a schematic, including how the RJ45 and magnetics are terminated?
 
The following users thanked this post: cheeseburger

Offline cheeseburgerTopic starter

  • Contributor
  • Posts: 30
  • Country: de
Re: Gigabit Ethernet Differential Pair Routing
« Reply #8 on: August 25, 2022, 08:14:48 pm »
According to the Support of the PCB manufacturer (JLCPCB) they just default to the Stackup which I've used for calculation when not choosing controlled impedance. Not sure if this is really true and if they do any test coupons at all.

Sure, part of the schematic is already in the first post.

I'm using a Raspberry Pi CM4, which already contains the Gigabit PHY. The RJ45 Jack is a Würth Elektronik 74991104402. The Magnetics are already integrated into the RJ45 Jack. Datasheet to the RJ45 Jack https://www.we-online.com/catalog/datasheet/74991104402.pdf

Maybe the Jack itself is the Issue? I tried with a Würth Elektronik 74991104402 and a Würth Elektronik 7499110123 (similar design) but both had similar issues with not working properly with even just 2M cable.
« Last Edit: August 25, 2022, 08:28:51 pm by cheeseburger »
 

Offline twospoons

  • Frequent Contributor
  • **
  • Posts: 323
  • Country: nz
Re: Gigabit Ethernet Differential Pair Routing
« Reply #9 on: August 25, 2022, 08:48:34 pm »
You definitely need to pull that top copper pour away from the diff pairs. As has been mentioned,  you need roughly 5x the distance to the ground plane underneath. For the stackup you have used that means 1mm gap from ground pour to diff pair.   I've seen a similar issue with USB, where pulling back the top layer ground pour was the difference between passing or not passing a compliance check.

Its also imperative to tell the PCB manufacturer if you have controlled impedance traces - otherwise the process control will be a lot looser.
« Last Edit: August 25, 2022, 08:53:55 pm by twospoons »
 
The following users thanked this post: cheeseburger

Offline cheeseburgerTopic starter

  • Contributor
  • Posts: 30
  • Country: de
Re: Gigabit Ethernet Differential Pair Routing
« Reply #10 on: August 25, 2022, 08:53:27 pm »
You definitely need to pull that top copper pour away from the diff pairs. As has been mentioned,  you need roughly 5x the distance to the ground plane underneath.

Ok, thanks. What would you recommend between the differential pairs? Also 5-10x trace width spacing without copper pour or do I need a copper pour between the differential pairs?
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17761
  • Country: fr
Re: Gigabit Ethernet Differential Pair Routing
« Reply #11 on: August 25, 2022, 08:55:05 pm »
Note that a number of baseboards for the CM4 add ESD protection to the Ethernet pairs. I think that's a good idea. (Not that it would change anything for your problem here, unless it came from an ESD damage to the CM4.)
 
The following users thanked this post: cheeseburger

Offline twospoons

  • Frequent Contributor
  • **
  • Posts: 323
  • Country: nz
Re: Gigabit Ethernet Differential Pair Routing
« Reply #12 on: August 25, 2022, 09:07:03 pm »
You don't need copper between the diff pair, but you do need to keep them separated to avoid crosstalk.

There are a few design guides about, such as https://www.ti.com/lit/an/snla387/snla387.pdf?ts=1661430097636&ref_url=https%253A%252F%252Fwww.google.com%252F and https://www.ti.com/lit/an/spraar7i/spraar7i.pdf?ts=1661406367506&ref_url=https%253A%252F%252Fwww.google.com%252F

One of the interesting points is to check the location of you length matching wiggles. See the linked docs.

You haven't posted an image of your ground plane, but I assume it is unbroken under the diff pairs?
 
The following users thanked this post: cheeseburger

Offline Gribo

  • Frequent Contributor
  • **
  • Posts: 678
  • Country: ca
Re: Gigabit Ethernet Differential Pair Routing
« Reply #13 on: August 26, 2022, 12:14:04 pm »
Is the RJ45 connector a vertical one? you might want to remove the pour from below the connector.
I am available for freelance work.
 
The following users thanked this post: cheeseburger

Offline PeteH

  • Supporter
  • ****
  • Posts: 121
  • Country: ca
Re: Gigabit Ethernet Differential Pair Routing
« Reply #14 on: August 26, 2022, 12:52:12 pm »
I've seen issues sometimes when swapping media side vs PHY side common mode chokes, however odd that sounds.

If you can, see if one is available that swaps the common mode chokes to the PHY side in the jack. (I.e. like the magjack on the CM4 I/O board reference design)
 

Online AndyC_772

  • Super Contributor
  • ***
  • Posts: 4559
  • Country: gb
  • Professional design engineer
    • Cawte Engineering | Reliable Electronics
Re: Gigabit Ethernet Differential Pair Routing
« Reply #15 on: August 26, 2022, 01:58:14 pm »
Sure, part of the schematic is already in the first post.

OK, but that just shows that two connectors are joined together, not which active components actually connect to each signal, sadly. It's one of the downsides of using a pre-designed module. Can you trace the signals as far as the PHY and any termination components? Is it possible that, say, both your connector and the module both include termination components, whereas only one of them actually should? Or perhaps neither does, and they're required separately?

Do you know for certain that the problem is anything to do with your board? Can you plug the module into any other PCB that provides a physical network connection, and show it running error free? Do you know for certain that it's not, say, a timing issue on the GMII / RGMII side (assuming that's what the PHY uses)?

I can't help but feel that, if the link won't stay up reliably even with a 2m cable, we're not looking for a layout or SI problem here at all. It may well be the case that adjusting the stack-up or the topology of the ground plane will help a little, but I can't see such small issues breaking the port to the extent that it won't even come up with a really short cable.

 
The following users thanked this post: cheeseburger

Offline cheeseburgerTopic starter

  • Contributor
  • Posts: 30
  • Country: de
Re: Gigabit Ethernet Differential Pair Routing
« Reply #16 on: August 26, 2022, 10:00:33 pm »
Do you know for certain that the problem is anything to do with your board? Can you plug the module into any other PCB that provides a physical network connection, and show it running error free? Do you know for certain that it's not, say, a timing issue on the GMII / RGMII side (assuming that's what the PHY uses)?

I can't help but feel that, if the link won't stay up reliably even with a 2m cable, we're not looking for a layout or SI problem here at all. It may well be the case that adjusting the stack-up or the topology of the ground plane will help a little, but I can't see such small issues breaking the port to the extent that it won't even come up with a really short cable.

Thanks for again suggesting that the issue might be somewhere completely else than where I'm currently looking at.

Yes, I have the official Carrier Board from Raspberry Pi and I don't recall any Ethernet issues with the same modules I'm now using on my custom Board. Even thought, it might be a good idea to retest and verify if there are really no RX issues with the official Carrier Board.

The Schematics for the official Carrier Board are open source. One very interesting point is that they mention a MagJack in the Schematic, but have a Trxcom Jack in the PCB and also populated on the Carrier Board (https://datasheets.raspberrypi.com/cm4io/cm4io-datasheet.pdf Page 11 - Trxcom TRJG0926HENL).

I will now ask Raspberry Pi if they can provide me the full datasheet for the Jack they've used, so I can check if it contains some internal termination or other differences.
 

Offline dmendesf

  • Frequent Contributor
  • **
  • Posts: 399
  • Country: br
Re: Gigabit Ethernet Differential Pair Routing
« Reply #17 on: August 27, 2022, 12:36:35 am »
Just remove the CM module from the carrier board and use a multimeter to figure out what's going on in the RJ45 connector.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 30091
  • Country: nl
    • NCT Developments
Re: Gigabit Ethernet Differential Pair Routing
« Reply #18 on: August 27, 2022, 01:38:40 am »
Hello

I'm currently trying to get a design with Gigabit Ethernet to work properly. Gigabit Ethernet requires 100 Ω differential routing. To achieve this, I used the built-in Calculator in KiCad.

(Attachment Link)

In general, it works and can establish a Link with 1 Gig, but when exchanging multiple Gigabyte of Data it starts to cut out and collect RX Errors. Sometimes it also needs multiple seconds and attempts to even establish a Link.

Attached is a screenshot of the relevant Schematic and PCB sections. The PCB is 4 Layers, with the GND Layer under the Ethernet Signals. The Ethernet pairs are length matched. Track width of the Ethernet pairs is 0.24 mm. PCB Thickness is 1.6 mm.

The PCB is the JLC7628 Stackup: https://cart.jlcpcb.com/impedance

The PHY/Device itself is a Raspberry Pi CM4.

Any ideas what I could try next to improve this?
The ground pour is too close to the traces. I'd say have a minimum clearance of 4x the width of the traces or 4x the distance between the pairs (whatever is greater).

Does the Rpi CM module have ESD protection and EMI filtering? If not, then I suggest to add TVS diodes. Having 10 Ohm @100 MHz 0402 ferrite beads in series with the ethernet signals also helps to avoid EMC problems caused by cheap & crappy ethernet phys.
« Last Edit: August 27, 2022, 01:41:56 am by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline bson

  • Supporter
  • ****
  • Posts: 2755
  • Country: us
Re: Gigabit Ethernet Differential Pair Routing
« Reply #19 on: August 28, 2022, 07:33:19 pm »
Two opposing currents result in a mutual impedance; they obstruct one another.  The PCB calculator takes this into account when calculating the trace geometries, although you specify the impedance and it indirectly solves for the geometry.  But the opposing currents is the source of this impedance.

When a signal current passes a 0A ground plane it looks like an opposing current.  The higher the signal level, the more impedance it sees.  Hence, you get an attenuation of the signal, or a loss, or dissipation - and this dissipation goes somewhere: onto the ground plane.  Here it forms on opposing current for other differential pairs, and in effect it's a crosstalk antenna.

A ground pour is fine as it keeps the board from warping due to the large copper-free areas otherwise.  But I'd remove it altogether from the differential pairs, and space those out a little more for good measure since you have plenty of board area to work with.  They can get in close around the magjack pins, that's fine.

Do you have a scope?  If so scope out the supply ripple at the magjack.  You may find decoupling it slightly will reduce this.  I don't think it's a huge deal, but it's so easy to check and add a few caps as needed when you're going to spin another board rev anyway.

But fix the ground pour and maybe tweak the spacing first.  There's little reason to worry about other problems until the obvious ones are fixed.

Ideally, you'd use a high-speed differential probe to look at the pairs, trigger on one of the (R)MII error signals, and confirm crosstalk.  Then you can confirm it's fixed in your rev 2 boards.

Just my $.02...
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf