General > General Technical Chat
Ethernet as System Comm Link
fourfathom:
I sometimes reminisce about past design decisions, and one that still bugs me, 25 years later, is an inter-board communications link in a telecom product (the Cerent 454, a SONET optical/electrical network switch).
I was director of systems engineering, and essentially the product architect, and we were figuring out how to have the individual interface and switching boards and common control boards communicate. I had proposed using point-to-point 100BASE-T ethernet (backplane connected) with Ethernet switches on the redundant common control cards. However, my proposal was shot down by our software team, with them trotting out the "Ethernet is an unreliable protocol" argument. I tried to explain how the underlying physical layer was just as reliable as any other physical interface and protocol -- this wasn't thinwire Ethernet with vampire taps or Tees where collisions were possible, and the switch ICs had more than adequate buffering. If they wanted more reliability then they were free to add an upper-layer protocol on top of the physical layer I was providing.
But they were adamant. "Ethernet Is Unreliable", and with me being a hardware engineer and them supposedly being the networking experts, I finally had to give in. Instead, I gave them a custom design, built into the ASICS we were designing -- in my opinion no more reliable than the Ethernet-switch physical layer I had proposed, but apparently more than adequate for our purposes.
But it still bugs me now. I believe that the software team had absorbed the "UNRELIABLE" concept during their education, and hadn't really thought about what that meant. I was already fighting a lot of battles at that stage, but I wish I had stuck to my guns on that one. It would have been a cleaner solution.
So am I wrong? If so, why? I would truly like to learn.
madires:
No, your idea of using Ethernet was perfectly reasonable. Around the same time Juniper did excatly that. They used Ethernet for connecting the routing engine modules (basically industrial PCs).
Benta:
I think you're being a bit unfair here.
Back then, several network technologies were in play, especially in Telecoms.
Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.
Especially in Telecoms, where correct data package time delivery is essential, I understand their concern.
Other technologies in the running back then were Token-Ring (deterministic) and FDDI (deterministic), a few others, plus any amount of proprietary formats.
fourfathom:
--- Quote from: Benta on March 05, 2022, 09:13:01 pm ---I think you're being a bit unfair here.
Back then, several network technologies were in play, especially in Telecoms.
Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.
Especially in Telecoms, where correct data package time delivery is essential, I understand their concern.
Other technologies in the running back then were Token-Ring (deterministic) and FDDI (deterministic), a few others, plus any amount of proprietary formats.
--- End quote ---
But this wasn't network payload data, just internal configuration and housekeeping data. They did indeed say "unreliable".
The product used traditional STS-1 and VT-1.5 circuit-switching internally (we designed our own STS-1-granularity 60 GBit/s throughput switch ASICS, and high port-count VT-1.5 switch ASICS, as well as customer-port interface ASICs). One of our claims to fame was being able to map GigE into SONET, with multi-port switched ethernet on our cards. Of course we also had OC-3 to OC-192, as well as 12-channel DS3 and DS1 cards). Any card in any slot (but some slots were more equal than others). I did some ASIC design, particularly a high-density DS3 chip, using digital clock synthesis and PLL techniques, but mostly I was responsible for system-level design and board-level advise.
You're right, this was all happening at the time when the traditional networks used guaranteed circuit-switched timeslotted channels, and ATM / cells were to be the hoped-for modernization. TCP/IP was non-deterministic and still being studied for use in traditionally-deterministic applications. We designed and built an ATM card, but it got dropped before anyone actually deployed it (and good riddance!)
We got acquired by Cisco, and I spent many meetings having to explain circuit-switched networks and various deterministic redundant network failover topologies to the router guys. Our product actually used those new-fangled network discovery methods to figure out how to best configure itself in a semi-random network topology (it wasn't always BLSR or similar, sometimes it was more of a mesh).
Fun times! (I told you I have been reminiscing!)
Cerebus:
--- Quote from: fourfathom on March 05, 2022, 08:06:38 pm ---..., and we were figuring out how to have the individual interface and switching boards and common control boards communicate.
--- End quote ---
You're talking about moving control plane traffic around, right? One presumes that getting the SONET data plane traffic around was the whole point of the switching cards.
--- Quote ---But they were adamant. "Ethernet Is Unreliable", and with me being a hardware engineer and them supposedly being the networking experts, I finally had to give in. Instead, I gave them a custom design, built into the ASICS we were designing -- in my opinion no more reliable than the Ethernet-switch physical layer I had proposed, but apparently more than adequate for our purposes.
But it still bugs me now. I believe that the software team had absorbed the "UNRELIABLE" concept during their education, and hadn't really thought about what that meant. I was already fighting a lot of battles at that stage, but I wish I had stuck to my guns on that one. It would have been a cleaner solution.
--- End quote ---
I'm suspecting that at some point one of more of them had heard "Ethernet is not a reliable medium" and failed to realise that means "does not have guaranteed delivery" not has "has high failure rates" or "has high bit error rates". In a chassis environment, where you're in control of everything there is no reason for ethernet to be less reliable [in the common sense] than any other layer 2 protocol/medium. Indeed, the likelihood of systematic failures in a home spun ASIC is likely to be higher than in commodity ethernet chips that have been tested in 1000s of applications rather than just one.
--- Quote from: Benta on March 05, 2022, 09:13:01 pm ---Back then, several network technologies were in play, especially in Telecoms.
--- End quote ---
You're saying that to a man talking about designing a SONET switch is a bit like saying "Now, Grandma, you hold the egg like this and suck here".
--- Quote ---Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.
--- End quote ---
Firstly I suspect that he knows what his colleagues said and putting words into their mouths that suit your argument ought to be a bit infra dig. Secondly this is a custom build of a switched ethernet network inside a product, it isn't a general network. It can be made as deterministic as one wishes, if deterministic is even a necessary property for this particular network application.
Navigation
[0] Message Index
[#] Next page
Go to full version