Any limit to number of ethernet switches in series ?

[langwadt]

how about something nuts, a leaky coax and wifi?

[nctnico]

I'm looking at  new project which will have a large number (100-200) of nodes ( probably based on RasPi compute module) that need to be linked with ethernet.
For various reasons, it's not practical to have each one going to a switch, we ideally want them in a number of series chains.

I'm looking at each node having a switch chip integrated, e.g.
https://www.lcsc.com/product-detail/span-style-background-color-ff0-Ethernet-span-Switches_IC-Plus-IP175G_C80220.html
(as it's the cheapest documented switch chip I could immediately find)

This onboard switch would have 3 ports - one to the node, and an in & out connecting to adjacent nodes. Possible more to allow branching.

As others mentioned: a standard (store and forward) ethernet switch doesn't have a real limit beyond the MAC learning table.

About the IP175G: this chip exists in many incarnations where each new version seems to be a die-shrink and has some new features. I have used the C version a long time ago and looked into using F/L versions but didn't go through with it due to chip shortage. The only IP175 I managed to find where desoldered ones. I strongly recommend contacting an IC-plus distributor for additional information. There are some interesting non-public appnotes available (including EMC compliance). I also recall the power consumption is rather high so a heatsink might be needed.  One of the interesting features is that the MAC on this chip works down to DC so you can bit-bang the MAC interface from a microcontroller. Or use an FPGA without being picky about the frequency used for the MAC interface. The IP175 takes care of all the clock domain crossing & buffering issues.

I ended up using the LAN9303 from Microchip. This chip has 2 integrated phys and 1 MAC so one of the ports will need to interface with a MAC and the MAC interface on the LAN9303 needs to see the right frequency otherwise it operates in a different mode.

[linux-works]

[old guy mode=enabled]

I worked at DEC in the mid 80's up to the early 90's.  networking was my thing.  10base5 was a thing and 'thickwire' with vampire taps was a thing

anyway, iirc, there was a limit to the number of repeaters and then bridges.  there were no 'switches' back then; it was called a bridge and you'd be lucky if it came close to wire-speed (even at 10mb, I dont think the dec gear could run at 100% link speed).

there were 3 repeaters in series and then no more.  unless the next was a bridge and then that 'reset' the repeater count.  then there was the question about how many bridges.  I think the number at the time was 7.  and you could have some repeaters between those bridges (max of 3, again).

for a long time, these were the layer1 and layer2 rules.  had to do with round trip time (ttl) and end to end delays.  the 'jam signal' had to be seen by all and that was really only for true broadcast domains (layer1 repeated hubs).  today you cant even find a 'hub' if you try; everything, even $5 multiport boxes are switches.  (I used to like having real hubs as you could do port sniffing easier; now we have port-mirroring but its still NOT the same thing).

for today, I do think the limits are nowhere the same but there are still limits.  things will break if you slow extend latency and transit delay.  they'll work until they dont.  (who's law is that?)

time for the old man to nap.  later...

[linux-works]

switches are not always store-forward

there's also 'cut-thru' and you may not be able to know which is which.

smaller consumer boxes are often bridges, so that means s-f.

but high density data center switches can also be cut-thru.

these days, there might even be strange hybrids of the two.

networking companies have run out of ideas (about 30 years ago) so they all try to break^Hchange things in special ways.

but 'switch' can mean many things.

[brucehoult]

[old guy mode=enabled]

I worked at DEC in the mid 80's up to the early 90's.  networking was my thing.  10base5 was a thing and 'thickwire' with vampire taps was a thing

Old ... pffft.

DEC was my dream when I was at university, using the 11/34, 11/70, 11/780.

But then I got a job and they had a DG Eclipse MV/10000 and I used those for a decade, ending on MV/2000. Actually, the job started as a summer holiday job. The first task was to decide what compiler my employer should buy -- and what database -- as they'd never had an in-house programmer. I spent the first two weeks (in December 1984) in the local Data General office playing with the COBOL, FORTRAN, PL/I compilers and DG/SQL and some non-relational database. I advised my summer employer to get the PL/I and the SQL -- which was already quite advanced, with precompiled queries, proper use of PL/I variables in the compiled query (not just textual interpolation like so many things do today), and also proper referential integrity.

Data General actually offered me a job at the end of that two weeks, but I was already committed to the financial company I was evaluating the stuff for.

I'd previously spent the summer two years earlier (82/3) doing COBOL on a Pr1me as the only in-house programmer in a small city council ... somewhat under the wing of programmers at the bureau that owned the Pr1me. That was at the end of my 2nd year at university. I did 9-5 programming at the city council and then worked around 6 pm to midnight picking up hay bales (by hand, on to a truck) and carting them to the barn. Oh, the endless energy at 20 years old!

So, yeah ... you can't possibly be old, because I'm not.

[ejeffrey]

switches are not always store-forward

there's also 'cut-thru' and you may not be able to know which is which.

Several people have mentioned "store-and-forward" but it's not relevant in this question at all.

Store and forward means that the switch waits for the entire packet before it beings transmitting.  Cut-through switches sometimes (but not always) begin transmitting before the packet is completely received, shortly after the headers are received.  Other than shaving a couple microseconds of latency, it doesn't really affect how it works at a network level.

[Someone]

Is there anything intrinsic to ethernet preventing a series string of, say, 15 switches working reliably?

No problem at all in theory. Just the loss of reliability from added failure points.

I've seen various device to device networks with realtime audio or video (so dedicated bandwidth, and not over shared paths) happily work with sustained high bandwidth and low latency demands in 6-8 deep layers of switches. From what you've described it should be no problem going for dozens of hops.

Also recall you can get a handle on this from internet traceroute. I'm already 6 hops to leave the physical site, and ending at a total of 15-20 hops to some website endpoints. Loading a website across that still works fine.

[madires]

Also recall you can get a handle on this from internet traceroute. I'm already 6 hops to leave the physical site, and ending at a total of 15-20 hops to some website endpoints. Loading a website across that still works fine.

That's IP, not Ethernet. But the links between the routers are often ethernet.

[madires]

Store and forward means that the switch waits for the entire packet before it beings transmitting.  Cut-through switches sometimes (but not always) begin transmitting before the packet is completely received, shortly after the headers are received.

They will fall back to store-and-forward if, for example, the link speed of the output port is lower then the link speed of the input port.

Other than shaving a couple microseconds of latency, it doesn't really affect how it works at a network level.

One drawback is the forwarding of bad frames. Only in store-and-forward mode the switch can check the FCS at the end of the frame and discard the frame if it's bad. In most cases it doesn't matter much.

[magic]

They will fall back to store-and-forward if, for example, the link speed of the output port is lower then the link speed of the input port.

With lower you could start transmitting right away and buffer the excess in a FIFO.
Higher is the tricky one

In reality, I'm not familiar with such gear, but I wouldn't be surprised if they don't bother handling either.

[paulca]

I also don't think there is a hard limit, but YMMV eventually.

That said, when you look at other case studies where that kind of scale is reached readily, such as office buildings they often start out as a random collection of desktop/consumer grade switches across the floors and only gets to pro-level switching in the building core.

The company I started my career with fell into this state.  They had 3 large floors, each floor did have a Cisco switch in the "core cabinet" where the vertical cabling went.  However, the "glands" in the floor where cables passed down to power extensions and network switches typically had little "Net Gear" style switches.

It turned out when they finally got the worst case switching loop that created a broadcast storm disabling half the network, there were something like 60 switches in the building.  With the majority of them being under the control of the dev teams and office workers.  All it had taken was someone to connect a switch in a loop and without any fancy spanning tree and anti-looping it caused a cascade failure.

The solution was to rewire the floors so that ALL ethernets terminate back in the locked cabinet into 48 port cisco switches.

[mikeselectricstuff]

OK so looks like the answer is basically no.
The architecture has changed a bit now some more thought has been applied.
Now looking likely to be a few strings of maybe six to ten 8-port switches, each feeding 6 nodes. As we have control of the switch design there is a scope for tweaking any of its parameters that might be useful to optimise things - still need to investigate this.
There will likely be a low-speed side-channel, probably RS485, linking our switch boards for various reasons, so we can remotely play with switch configs easily.

Thanks for the input everyone.

[paulca]

For a bit of random.   In one cluster I worked on we used optical splitters and duplicated MAC addresses to distribute data to nodes.  Layer 1 baby.

You might be surprised to find that 16/32/64 optical splitters are readily available.  Mostly passive too.

[madires]

I also don't think there is a hard limit, but YMMV eventually.

AFAIK, only the classic STP has a hard limit. Modern versions like MSTP don't, but it depends on the specific implementation. There are also proprietary modifications (-> vendor lock-in). An important point for larger networks (applies also to ethernet with any sort of STP) is convergence time. It will increase with complexity, depth, and number of network elements. Exceeding some network size the whole network can become unstable, i.e. a small change can cause havoc. The typical solution is to partition a large network into smaller ones.

It turned out when they finally got the worst case switching loop that created a broadcast storm disabling half the network, there were something like 60 switches in the building.  With the majority of them being under the control of the dev teams and office workers.  All it had taken was someone to connect a switch in a loop and without any fancy spanning tree and anti-looping it caused a cascade failure.

A classic!

[linux-works]

it does.  cut thru can simply drop frames.  see, layer 2 is not guaranteed delivery no matter what anyone tells you.  there used to be llc2 and llc3 but again that's ancient.  like x.25 ancient

everything 'south' of the tcp layer can drop frames.  hell, even udp can drop frames.  its parent (whatever sits on udp at the time) has to do timeout and retries.  if you sit on top of tcp, tcp does all that.  but tcp does a LOT of work since it can assume layer2 and ip (3) can both drop pdu's and be allowed to, if it comes down to it.

if you can avoid dropping at lower layers, the upper layers have a 'smoother time'.

so yes, it kind of does matter if you store/forward or drop and let 'higher ups' resend.

[linux-works]

also, some newcomers (like avc bridging, aka, 'time sensitive networking or TSN) will have to deal with special hardware to get the guaranteed delivery *timing* (not just getting every frame there but getting them 'on schedule') will make things even worse or harder for network engineers.

in car networking, TSN is getting to be more and more of a thing.  with the hope of having level3-5 'self drive' in the future, you need the next level of redundant and time-sensitive networking.  in those cases, you dont just go adding more network nodes or switch/routers.

https://en.wikipedia.org/wiki/Time-Sensitive_Networking

for home networking, which is probably what the orig question was about, I still would not cross more than, say, 3 bridges or switches unless you really have to.  and 3 layers should be more tree hier than anyone at home would need (3 layers of roll-up in the tree hier).   at some point, you break things into subnets and route them, and btw, 'routers reset the count' (lol).

[linux-works]

lol - a still unopened bag of 50ohm coax 'thinwire' (10base2) terminators.

(OT: it really *was* the coolest place to work.  you put in a purchase req for anything in the DECdirect catalog and it just got expensed from one cost center to another.  DEC was its own world, we even had our own telephone numbers called DTN (dec telephone network).  I dont think there is anything like that, company wise, left anymore.  we had our own set of satellites linking US, europe and what we called GIA for 'general international area')