10GBASE-T - Too much speed for home use?

[ve7xen]

It can suck, but it doesn't have to. Like with everything, if you invest a bit of money in proper gear (not those stupid consumer "mesh" products), you can actually get pretty amazing performance, even in RF noisy environments with a bit of tweaking. You'll notice most consumer devices, they like to default to channels 36 to 48 and sometimes around channel 155 down the other end of the 5 GHz ISM band. They often avoid the DFS channels altogether, even if they are better.

There's a reason I qualified it as 'relative to wired Ethernet'. It can be fine, if the equipment is okay and the setup is okay, but it's still a shared medium, with a lower maximum throughput, shorter range, and is generally much less stable even when working well. Wired 'just works'.

[james_s]

Sorry for being a bit pedantic, but switching and routing are two different concepts in the networking world. Switching happens at OSI layer 2, while routing happens at OSI layer 3. A classic ethernet switch (or bridge) switches ethernet frames between segments at the data-link layer. An IP router routes IP packets between networks at the network layer. The IP packet is stacked on top of the ethernet frame. Or in other words, the IP packet is the payload of the ethernet frame. A so called L3 switch (L3 for OSI layer 3) can do both.

Yes I know this and yes I think it's very pedantic. I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird. It's grammatically correct to say that a switch (or router or anything else that has some kind of control over where data goes) "routes" traffic. Traffic signals, signs, traffic officers and other methods "route traffic" on roads.

[metebalci]

Sorry for being a bit pedantic, but switching and routing are two different concepts in the networking world. Switching happens at OSI layer 2, while routing happens at OSI layer 3. A classic ethernet switch (or bridge) switches ethernet frames between segments at the data-link layer. An IP router routes IP packets between networks at the network layer. The IP packet is stacked on top of the ethernet frame. Or in other words, the IP packet is the payload of the ethernet frame. A so called L3 switch (L3 for OSI layer 3) can do both.

Yes I know this and yes I think it's very pedantic. I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird. It's grammatically correct to say that a switch (or router or anything else that has some kind of control over where data goes) "routes" traffic. Traffic signals, signs, traffic officers and other methods "route traffic" on roads.

I dont think it is very pedantic. As this is a technical forum, when switch and route is used in the same sentence in a post about networking, I think it should be clarified.

Also I dont think traffic signals and officers are a good first analogy, a switch (or router) does not block or interrupt a traffic (leaving extra features like ACLs aside).

[2N3055]

Sorry for being a bit pedantic, but switching and routing are two different concepts in the networking world. Switching happens at OSI layer 2, while routing happens at OSI layer 3. A classic ethernet switch (or bridge) switches ethernet frames between segments at the data-link layer. An IP router routes IP packets between networks at the network layer. The IP packet is stacked on top of the ethernet frame. Or in other words, the IP packet is the payload of the ethernet frame. A so called L3 switch (L3 for OSI layer 3) can do both.

Yes I know this and yes I think it's very pedantic. I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird. It's grammatically correct to say that a switch (or router or anything else that has some kind of control over where data goes) "routes" traffic. Traffic signals, signs, traffic officers and other methods "route traffic" on roads.

I am with you here.. I understood perfectly what you meant and that kind of terminology is used like that in high level LAN talk all the time.
Not perfect and technically not exactly right terminology, but I understood what you meant. In that context "routes data" meant "moves data" not "it performs TCP/IP routing on it"
Context matter a lot. That is why English is so confusing sometimes...

Make it even more confusing by introducing Layer 3 switching.. What do you call data transfers that happen there.....

If we want to be pedantic, then we should use "data forwarding" and "data redirecting" or something of that sort and use layer it happens on (2 or 3).....
Which can be left to highly technical discussions. 

[madires]

I am with you here.. I understood perfectly what you meant and that kind of terminology is used like that in high level LAN talk all the time.
Not perfect and technically not exactly right terminology, but I understood what you meant. In that context "routes data" meant "moves data" not "it performs TCP/IP routing on it"
Context matter a lot. That is why English is so confusing sometimes...

I only explained very basic networking terminology. And in this context routing is a function at OSI layer 3. Also, it doesn't imply TCP/IP, it can be any other L3 network protocol too. If you're dealing with switches and routers, and talk about routing data, which of both device types do you think is meant or involved?

[madires]

I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird.

A good alternative would be 'forwarding' as 2N3055 pointed out. Forwarding applies to layer 2 and 3.

[2N3055]

I am with you here.. I understood perfectly what you meant and that kind of terminology is used like that in high level LAN talk all the time.
Not perfect and technically not exactly right terminology, but I understood what you meant. In that context "routes data" meant "moves data" not "it performs TCP/IP routing on it"
Context matter a lot. That is why English is so confusing sometimes...

I only explained very basic networking terminology. And in this context routing is a function at OSI layer 3. Also, it doesn't imply TCP/IP, it can be any other L3 network protocol too. If you're dealing with switches and routers, and talk about routing data, which of both device types do you think is meant or involved?

I didn't say you are wrong.
"routing" and "switching" are words being commonly used to signify a work done by routers and switches. But in English language "routing messages" is what mail server does... And it is not network device... Word routing made sense in that sentence despite being "nontechnical". That is my point.
Another point is that when discussing something there is no need to always go into details. Quite the contrary, it is an art of saying only necessary to make it quick and efficient.
Otherwise we might start always discussing that there are hubs, bridging, routing, store&forward switching, cut through switching, asymmetric switching, spanning tree protocol etc etc...

Important part is that usually when using switches, switch exchanges data based on MAC of network interface on devices connected, while router makes distinction between subnets and decides on packet transfer decisions based on IP address of the host. Once packet arrives at destination subnet (by whatever means) then final L3 to L2 resolution is done (via ARP) and packet reaches it's destination. Also distinction is that switch has switching matrix that allows paralel (isolated) connections between hosts that serves to increase performance.
L3 switches are intelligent enough to keep both IP address and MAC address tables directly so they can do switching by shortcutting routing. Also no difference logically, just much faster because it uses parallel switching matrix instead of serialized traffic through router interfaces....

As for other L3 protocols, it's been decades since I worked on anything else than TCP/IP so I simply ignore them in general conversation. You are, again, correct, but it doesn't matter. For all the practical purposes if someone asks you about networking today, it is going to be TCP/IP..

Best,

[sokoloff]

It’ll be an IP network, but with a mix of TCP and other traffic (mostly UDP).

[tszaboo]

A few things:
There are internet providers here that offer more than 1 Gbps speeds.
AC Wifi is faster than gigabit, so if you want a wired extender than having more than a gigabit makes sense.
On the other hand I don't think you need more than gigabit in 2023. The worst offenders are game updates which are dozens of gigabytes, and that are handled in mere minutes. I think right now it makes sense to get cables that would handle 10 gig networking, and only switch to it when it's more standard. It makes more economic sense. Especially in walls, get something that is future proof, CAT 7 or similar, because replacing it is more pain.

[metebalci]

A few things:
There are internet providers here that offer more than 1 Gbps speeds.
AC Wifi is faster than gigabit, so if you want a wired extender than having more than a gigabit makes sense.
On the other hand I don't think you need more than gigabit in 2023. The worst offenders are game updates which are dozens of gigabytes, and that are handled in mere minutes. I think right now it makes sense to get cables that would handle 10 gig networking, and only switch to it when it's more standard. It makes more economic sense. Especially in walls, get something that is future proof, CAT 7 or similar, because replacing it is more pain.

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

[madires]

There are internet providers here that offer more than 1 Gbps speeds.

Yep, for example 2.5Gbps GPON.

Especially in walls, get something that is future proof, CAT 7 or similar, because replacing it is more pain.

IIRC, the current maximum for TP is 25GBASE-T (Cat8 and a maximum segment length of 30m). Installing Cat7 and Cat8 cables is quite effortful (if done to specs). So it could make more sense to go for glass fiber anyway when rewiring your house, in case you plan to go beyond 10Gpbs.

[madires]

"routing" and "switching" are words being commonly used to signify a work done by routers and switches. But in English language "routing messages" is what mail server does... And it is not network device... Word routing made sense in that sentence despite being "nontechnical". That is my point.

We shouldn't forget the scope of 'routing messages'. When I write a letter to a friend in another town the post office will route it to my friend's town. But when I write a letter to a neighbor a few houses down the street I don't post it at the local post office. Instead I'll simply throw the letter into the neigbor's mailbox, i.e. no routing at all.

Important part is that usually when using switches, switch exchanges data based on MAC of network interface on devices connected,

Endpoints (or servers) exchange data with other endpoints (or servers). Switches exchange data when negotiating links and other network related things, like STP.

L3 switches are intelligent enough to keep both IP address and MAC address tables directly so they can do switching by shortcutting routing. Also no difference logically, just much faster because it uses parallel switching matrix instead of serialized traffic through router interfaces....

For routing vendors implement a routing table which contains all the routing information (RIB) and a forwarding table (FIB) which is a list of destinations and their corresponding egress interfaces. The RIB can contain inactive routes (known, but not used because there's a better route). The active routes are put into the FIB. Since routing is a lot of effort vendors came up with ideas to simplify the routing process by making it more like switching, e.g. by creating hashes for flows and using a special table which is faster/shorter than the standard FIB. Cisco's CEF is such an example. These techniques are also implemented in ASICs. But you can't switch IP packets as simply as ethernet frames. Maybe the router has to fragment IPv4 packets or has to deal with IPv6 option headers. This leads us to MPLS which tags IP packets (also other protocols) with switching labels to allow intermediate routers to switch traffic instead of costly routing.

To give you an idea about the scale of ethernet switching and IP routing: A standard inexpensive SOHO switch chipset usually has an 8k MAC address table. A full IP routing table has currently 950k IPv4 routes and about 150k IPv6 routes.

[james_s]

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

Mine doesn't. It's fast for wireless, but especially when there are multiple devices the gigabit ethernet is significantly faster.

[james_s]

We shouldn't forget the scope of 'routing messages'. When I write a letter to a friend in another town the post office will route it to my friend's town. But when I write a letter to a neighbor a few houses down the street I don't post it at the local post office. Instead I'll simply throw the letter into the neigbor's mailbox, i.e. no routing at all.

You're routing it yourself and not relying on the post office, it's still being routed. You're not blasting out a massive quantity of letters into all of the mailboxes or just throwing it into the wind and hoping it finds its destination.

[ejeffrey]

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

I don't think they can in single client operation, but my understanding is that it is technically possible in ideal benchmarks with multiple clients and an access point with enough MIMO channels.
 
Wifi 6E access points, if they have separate radios for 5 and 6 GHz could probably exceed 1 Gb/s more easily, but still not for a single client.

[madires]

We shouldn't forget the scope of 'routing messages'. When I write a letter to a friend in another town the post office will route it to my friend's town. But when I write a letter to a neighbor a few houses down the street I don't post it at the local post office. Instead I'll simply throw the letter into the neigbor's mailbox, i.e. no routing at all.

You're routing it yourself and not relying on the post office, it's still being routed. You're not blasting out a massive quantity of letters into all of the mailboxes or just throwing it into the wind and hoping it finds its destination.

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

[tszaboo]

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

I don't think they can in single client operation, but my understanding is that it is technically possible in ideal benchmarks with multiple clients and an access point with enough MIMO channels.
 
Wifi 6E access points, if they have separate radios for 5 and 6 GHz could probably exceed 1 Gb/s more easily, but still not for a single client.

There is a wide variety of speeds under the term AC
https://en.wikipedia.org/wiki/IEEE_802.11ac-2013#Advertised_speeds

At least it works in most cases and not confusing like USB 3.1 (version 2) 1x1 PD- 67W.

[JohanH]

PON and GPON are standards made by ITU, the telecom industry. It's all made for feeding commercials and dumb TV content to as many users as possible, disregarding their ability to send data back. So the connection is shared among users, often passively and in most cases asymmetric. This saves some fibers, but the equipment is proprietary, expensive and there is a high threshold to upgrade the equipment. Later GPON standards are better, with better upstream speed, but if you have a small village on PON (typically restricted to 100/30 Mbit per user + bandwidth is shared!), you are lucky if they upgrade it in 20 years.

I work from home and would hate having a restricted upstream. Luckily we have an ISP that uses standard IEEE ethernet switches and none of the ITU crap. So I have the option for 100/100 Mbit or 1000/1000 Mbit over fiber (and options in between). Higher speeds than 1 Gbit aren't on the table today, but if you pay... they only have to change two tranceivers and your home fiber terminal. Might be a matter of time.

IEEE standard equipment is the only true networking and Internet equipment. PON and GPON is just disguised TV on life support and a surrogate for true end-to-end Internet. Engineers stick to IEEE standards.

[ve7xen]

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

If you don't know which house she lives in, are you making copies for all your neighbours and hoping that everyone discards letters not addressed to them? 

This saves some fibers, but the equipment is proprietary, expensive and there is a high threshold to upgrade the equipment. Later GPON standards are better, with better upstream speed, but if you have a small village on PON (typically restricted to 100/30 Mbit per user + bandwidth is shared!), you are lucky if they upgrade it in 20 years.

It's not so much about saving fibre, though that is part of it because otherwise you will lay a lot of fibre that never gets used, or if you want to avoid that, you will have to do relatively a lot of splicing work, but this is relatively cheap. It's more about saving the number of ports and PHYs you need to service the customers. Every classical Ethernet port requires its own switch port and own PHY, and this requires more physical space, more optics, more power, and more capacity in the switching backplane, which all costs fairly substantially at scale since we're talking about like 16:1 or 32:1 here.

XGS-PON (10G/2.5G) is widely deployed here and the ILEC is selling 1000/1000 service, and I have not seen any significant reports of congestion, and based on my experience in consumer ISPs, I find the risk of it unlikely. They are running 16:1, so it isn't really much of an oversubscription ratio. Upstream of the PON the oversubscription ratio is quickly going to increase.

Reality is that actual traffic patterns of residential users are still highly asymmetric preferring the downstream direction, so that is what ISPs are optimizing their networks for. It has nothing to do with TV, especially since the majority of ISPs deploying PON come from the telecom side, not the cable TV side, so while they all have TV offerings these days, it's not their heritage nor that significant a part of their business.

[JohanH]

The ITU/IEEE rivalry might be a bit exaggerated. Mostly tongue in cheek nowadays. But there are some background.

Conditions may vary and it's good that newer standard PON is deployed. I have seen bad networks around by the big ISPs.

There are lots of smaller fiber cooperatives around here that were built 15 years ago when the big operators removed their copper and offered only 3G/4G instead. All of these countryside and small town networks were built with network switches, AFAIK. I've some insight about ten such networks built around here in different towns. We tried to start our own fiber network in my village also; we were verbally threatened by the then local old telecom ISP; ultimately we didn't get enough interested users and didn't start digging. Fiber has become so cheap, so you don't save anything with PON, especially in rural, more widespread networks. Big ISPs might have easier deployment with PON in big cities with lots of departments.

[ve7xen]

The ITU/IEEE rivalry might be a bit exaggerated. Mostly tongue in cheek nowadays. But there are some background.

Conditions may vary and it's good that newer standard PON is deployed. I have seen bad networks around by the big ISPs.

There are lots of smaller fiber cooperatives around here that were built 15 years ago when the big operators removed their copper and offered only 3G/4G instead. All of these countryside and small town networks were built with network switches, AFAIK. I've some insight about ten such networks built around here in different towns. We tried to start our own fiber network in my village also; we were verbally threatened by the then local old telecom ISP; ultimately we didn't get enough interested users and didn't start digging. Fiber has become so cheap, so you don't save anything with PON, especially in rural, more widespread networks. Big ISPs might have easier deployment with PON in big cities with lots of departments.

Yeah, this is definitely true. There hasn't been a lot of development in small-scale PON (though there has been some), so it doesn't really start to make sense until a certain inflection point is reached that I'd guess is around 1000 homes passed. Many of these community ISPs are grassroots efforts, and often start at very small scale. It definitely doesn't make sense to run a PON shelf that can service 1000s of customers and costs 10s of thousands when you're only passing 100 homes and could do the job with a cheap 48-port switch. It's also much easier technically to get going using classical Ethernet, and more people understand that stuff, know how to source the equipment (good luck getting a vendor to return your calls about buying a single PON shelf, never mind the contract for management software etc. etc.), understand the automation options available and so on.

Both ways are doable, and can be done profitably, but PON is definitely more efficient at the scale of a major ISP.

Whether it offers a good experience is more down to competition and business decisions than the technology, and I guarantee that if your experience sucks on PON it's going to suck on any last mile technology.

Big cities and apartments are actually the one place where classical ethernet can compete with PON because you can home equipment at the building itself and cabling is relatively trivial.

[madires]

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

If you don't know which house she lives in, are you making copies for all your neighbours and hoping that everyone discards letters not addressed to them? 

I'm too lazy to make all the copies to flood my neighborhood. Instead I would ask a neighbor I know. BTW, flooding (BUM traffic) can be a real problem for larger LANs.

[Halcyon]

It can suck, but it doesn't have to. Like with everything, if you invest a bit of money in proper gear (not those stupid consumer "mesh" products), you can actually get pretty amazing performance, even in RF noisy environments with a bit of tweaking. You'll notice most consumer devices, they like to default to channels 36 to 48 and sometimes around channel 155 down the other end of the 5 GHz ISM band. They often avoid the DFS channels altogether, even if they are better.

There's a reason I qualified it as 'relative to wired Ethernet'. It can be fine, if the equipment is okay and the setup is okay, but it's still a shared medium, with a lower maximum throughput, shorter range, and is generally much less stable even when working well. Wired 'just works'.

I would generally agree with that statement. Wired Ethernet is certainly more "plug and play" than Wi-Fi.

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

If you don't know which house she lives in, are you making copies for all your neighbours and hoping that everyone discards letters not addressed to them? 

For those playing along at home, this is essentially what a hub is (as opposed to a switch). A hub is essentially just a dumb device that repeats all packets across all ports, which is why collision domains can be an important consideration in network design that incorporates hubs. I do not miss those days.

[madires]

An ethernet switch does it too when it doesn't know yet how to reach the destination MAC. This is the 'U' (unknown unicast) in BUM traffic. It can also be leveraged for attacks, e.g. MAC flooding.

WiFi is unreliable by default because is uses ISM bands. At any time a neighbor can plug in a new gadget which occupies an ISM band, forcing you to change channels or the band. The bands can become so crowded that you barely get any usable throughput. And you can't do anything about this as long as the devices involved adhere to local radio regulations.

[David Hess]

IMHO it's only worth if you have some shared storage which needs fast access from multiple places.

In the past I had 10/100 Ethernet for my internet connected network, and gigabit Ethernet for local connections to my NAS.  Now I have gigabit Ethernet for my internet connected network, and 2 x 1 gigabit networks in parallel for my local connection to my NAS for a total of 3 gigabits for file transfers when using SMB 3.

I considered 10 gigabit Ethernet, however it would have cost roughly $130 for a switch, $65 for each SFP+ network interface card, and maybe $30 per cable.

Depends what you're doing in your home, but 10G is slower than the read/write speed of a mediocre SSD these days, so if you're using one PC as a file server for another one (or have a NAS) you could saturate that easily enough.

With consumer hardware there are usually other performance limitations.  10G USB connected SATA3 SSDs seem to top out at about 250 MB/s.  Direct connected SATA3 SSDs are somewhat faster, but my 4 and 8 drive hard disk RAID arrays are just as fast.  If you are using Microsoft Storage Spaces then forget performance.

If you have only short runs of ethernet cable <= 30m good old Cat5e usually works also fine with 10GBASE-T. Alternatively there's 2.5GBASE-T and 5GBASE-T in case you need just a small boost. Beyond that I'd go for glass fiber anyway because of less trouble (high speed ethernet can be quite finicky about TP) and the upgrade path.

2.5GBASE-T and 5GBASE-T might be acceptable in some environments, like Windows only, however driver support is bad and even Intel's 2.5GBASE-T solution has problems.