10GBASE-T - Too much speed for home use?

[AndyBeez]

10GBASE-T Ethernet ports are finding their way onto domestic computers. For example, '10G' is a plus $100 order option at the Apple Store. One question though; is anyone fully leveraging the stupid-fast speed of 802.3an/10GBASE-T in a [non-enterprise] household environment? And if so, what are you connecting this too? Did it cost $s to upgrade your network devices and cables? 

https://en.wikipedia.org/wiki/10_Gigabit_Ethernet

[kripton2035]

I don't think this has any use for home network. my gigabit network from year 2000 is still plenty enough.
may be if you have 10 kids watching netflix in 4k simultaneously ?

[wraper]

IMHO it's only worth if you have some shared storage which needs fast access from multiple places.

[brucehoult]

Depends what you're doing in your home, but 10G is slower than the read/write speed of a mediocre SSD these days, so if you're using one PC as a file server for another one (or have a NAS) you could saturate that easily enough.

The fastest home internet you can get -- at least in my nearest town -- is 1 gig, with actual speed in the mid 900s. So if it's just for sharing internet then 10 G is not needed at present.

[brucehoult]

I don't think this has any use for home network. my gigabit network from year 2000 is still plenty enough.
may be if you have 10 kids watching netflix in 4k simultaneously ?

Netflix says their 4K streams are 15 Mbps, so you'd need well over 50 kids to trouble 1 gig ethernet -- and you only need that for the upstream from their router to the modem (if one device isn't doing both)

[AndyBeez]

Simply, a 10G network requires a 10G infrastructure. Server's need to be 100G monsters just to keep up with the packets.

I don't think this has any use for home network. my gigabit network from year 2000 is still plenty enough.
may be if you have 10 kids watching netflix in 4k simultaneously ?

Netflix says their 4K streams are 15 Mbps, so you'd need well over 50 kids to trouble 1 gig ethernet -- and you only need that for the upstream from their router to the modem (if one device isn't doing both)

A motivation for me to pose this question is a networking speed freak who believes 10G is NOT fast enough for his 4K TV on Netflix! His TV is a 58 inch juggernaut and thus, it needs more speed because it is a bigger screen. Guy's like him are a gift to the marketing suits in marketing suites.

The only 'domesticated' use case for such bandwidth is a $10,000 MacPro video studio, with 8K video stored on a SAN/Raid array loaded with M2 EVO ssd modules. But outside of the Hollywood hills, there is little justification for this level of investment. Maybe in the near-instantaneous future world of the Metaverse 10G will be 'bufferingly' slow, but for now, even inside the enterprise, it's a too much speed for desktop users.

btw My fastest outbound-upstream link is a mere 100Mbps on a 330Mbps FTTP link. When I started networking, 1Mbps over coax was regarded a top enterprise level speed - and was priced accordingly.

[wraper]

His TV is a 58 inch juggernaut and thus, it needs more speed because it is a bigger screen. Guy's like him are a gift to the marketing suits in marketing suites.

It's not even that large. I'd say barely above mainstream 55" size. I have 65" OLED which I don't find that large and my friend has 77" OLED which is something closer to "juggernaut" but you can get 85" LCD monster for like 1000 quid.

[sokoloff]

I have two 10G networks in my house, both for storage. One is for storage replication and VM migration traffic and only covers 3 nodes in a single homelab rack. The other connects the storage arrays to a 10G switch with just 1 10Gbps client and a 2.5Gbps switch for our family desktops to connect to the storage.

None of that relates to internet, which I have only 300 Mbps down and 20 up.

2.5 and 10 Gbps equipment is fairly cheap now. It lets me use a single highly redundant and backed up storage array on all our PCs without it feeling super-slow and just have a smallish NVMe local volume on each PC.

[dobsonr741]

10G makes sense for normal home networks too, for NAS and faster backups. Not just for home labs anymore.

I do Time Machine backups to TrueNAS, and just wish my gear would be 10G instead of 1G.

[james_s]

10G is faster than most people need, but there is no such thing as too much speed. My home network is gigabit and it is adequate for my needs but I do saturate the connection to individual machines regularly while pushing large files around.

[wraper]

The question is how it's supposed to help said TV when it has a 1 Gbit wired interface at best (most likely 100 mbit) and not the fastest WIFI?

[sokoloff]

The exact same way that cryogenically stress-relieved^* HDMI cables will deliver a crisper image quality to that same guy's TV, obviously.

^* - or whatever other things he's been hoodwinked by

[py-bb]

I wouldn't bother with 10G stuff unless you need more than 110mib/sec (like 114mb/sec) sustained speeds.

You can however have one link 10G, for example I got a stack of SolarFlare cards cheap, but 10G switches are really expensive.

10G base T also has higher latency (both in real terms and expressed in terms of packets) due to error correction methods, but we're talking microseconds so this can probably be ignored.

[SiliconWizard]

If you copy/access large files over the LAN, it's definitely worth it.
Otherwise, not so much.
But for $100, I would definitely consider buying the option. Even if you equip your lan for 10G only later on.

[nightfire]

On workstations a 10 GBit/s port is beneficial when you have to push lots of data over the network to a NAS or similar storage unit. This can happen for backup purposes (and not to overesxceed the backup window) or if you are working with photo editing or video cut, where most of the data reside on a NAS or similar server, or second workstation.
But: To do this, you need stable switches- the ordinary 500$ halfmanaged switch can also go into hiccup due to small buffers.

[Halcyon]

I use 10 Gig to all my servers and the trunk from router to switch. The gear is actually quite affordable. I deliberately wired my house with Cat 6 cable knowing that I would be upgrading to 10 Gig at some point. I ended up doing that a few years ago.

I don't use it to its maximum advantage but it's good knowing I have the headroom should I need it. I was finding I was maxing out 1 Gigabit quite often, especially when dealing with large files on my NAS.

[py-bb]

I use 10 Gig to all my servers and the trunk from router to switch. The gear is actually quite affordable. I deliberately wired my house with Cat 6 cable knowing that I would be upgrading to 10 Gig at some point. I ended up doing that a few years ago.

I don't use it to its maximum advantage but it's good knowing I have the headroom should I need it. I was finding I was maxing out 1 Gigabit quite often, especially when dealing with large files on my NAS.

Link/model of switch? (Affordable 10G gear that isn't a lucky auction on ebay - link me!)

[Halcyon]

I'm using the Ubiquiti USW-Pro-48 with a Netgate 7100 router which I realise isn't a cheap solution for home users, but there are cheaper switches out there which will give you a 10 Gig "core" to your network for example the Mikrotik CSS610-8G-2S+IN

The 10 Gig Ubiquiti DAC cables are dirt cheap.

[sokoloff]

DAC cables are one of those things that seem like they should be expensive but are remarkably cheap, so I use them for my short backup network.

Fs.com is a good vendor to know about. Fast and fair pricing (maybe not the lowest on everything, but time has value as well).

[metebalci]

I have 10G LAN and WAN at home. Other than special cases, for home use, it only matters if you have a reasonably fast NAS and/or 10G+ WAN (Internet) connection.

At the moment, I have one all 10G, one hybrid 1/10G managed QNAP switches, two 1G (HP and Cisco) managed switches and a Cisco AP. Although QNAP would not be my first choice normally, it is the most quiet 10G switch I tried so I keep it. I have CAT 6 cabling inside walls, and it works fine. DAC is an easy solution for short distance and SFP+ switches are cheaper and more quiet but ethernet is still more flexible. Having a switch with at least a few dual personality ports is nice to have.

[mariush]

Link/model of switch? (Affordable 10G gear that isn't a lucky auction on ebay - link me!)

You say you're in Afghanistan ... so kinda tough for people to ship you switches there.

If you're in US or some bigger country, I'd suggest emailing/contacting unixsurplus.com to give you a list of switches with 10gbs sfp+ ports they have available and more affordable.

They don't list everything they have on their website. Last year around September I asked what they could do in $300 and they had 2 units of Juniper EX4200 EX4200 (24 x 10g SFP+ and 4x1g SFP with two power supplies) for $275  .. specs here : https://www.juniper.net/content/dam/www/assets/datasheets/us/en/switches/ex4200-ethernet-switch-with-vitual-chassis-technology.pdf

Something interesting they have is a Quanta T5032 with 32 QSFP+ 40g ports for $419 : https://unixsurplus.com/quanta-t5032-ly6-switch/ 
Description says it uses Atom but it's PPC : https://www.qct.io/product/index/Switch/Ethernet-Switch/T5000-Series/QuantaMesh-T5032-LY6#specifications

40g to 4 x10g SFP+ DAC cables start from around $40 for the 1 meter versions, so around $10 per 10g port.

40g doesn't seem that popular anymore, as people moved to QSFP28, with 10/25/50/100 gbps ... you can get qsfp+ cards for $50 ex. https://unixsurplus.com/mellanox-cx353a--adapter/
qsfp28 can be bought at around $110 ... example 109$ for low profile 100g card : https://unixsurplus.com/Mellanox-MCX455A-Single-Port-100GB-QSFP18-Low-Profile/


Servethehome forum - the Great Deals section is also worth checking out : https://forums.servethehome.com/index.php?forums/great-deals.8/
from time to time very good deals (which go away fast) are posted there.

Look also at previous posts for switches and bookmark those sellers or add those switch models you like to your ebay searches / bookmarks etc

[madires]

If you have only short runs of ethernet cable <= 30m good old Cat5e usually works also fine with 10GBASE-T. Alternatively there's 2.5GBASE-T and 5GBASE-T in case you need just a small boost. Beyond that I'd go for glass fiber anyway because of less trouble (high speed ethernet can be quite finicky about TP) and the upgrade path.

[wraper]

If you have only short runs of ethernet cable <= 30m good old Cat5e usually works also fine with 10GBASE-T. Alternatively there's 2.5GBASE-T and 5GBASE-T in case you need just a small boost. Beyond that I'd go for glass fiber anyway because of less trouble (high speed ethernet can be quite finicky about TP) and the upgrade path.

From what I've read it works fine up to about 45m.

[DavidAlfa]

I don't think this has any use for home network. my gigabit network from year 2000 is still plenty enough.

Gbit in the year 2000? By then we were still going from 10M to 100M ethernet, no way!

[madires]

From what I've read it works fine up to about 45m.

Yep, Cat5e and Cat6 up to 50m under good conditions. Cat6a or better up to 100m.

[madires]

Gbit in the year 2000? By then we were still going from 10M to 100M ethernet, no way!

IIRC, we got the first GigE ports (via GBICs) in 1998/99, and 1000Base-T a bit later.

[jc101]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

[Monkeh]

Gbit in the year 2000? By then we were still going from 10M to 100M ethernet, no way!

IIRC, we got the first GigE ports (via GBICs) in 1998/99, and 1000Base-T a bit later.

802.3z (1000BASE-CX, SX, LX, all fibre) in '98, 802.3ab with 1000BASE-T in '99 - wouldn't see much heavy deployment for a year or two after, but Apple had gigabit NICs in their G4 machines in 2000. I wasn't exactly paying attention at the time, but I imagine vendors hedged their bets and had hardware on the market, not necessarily on the shelves at your local PCWorld, up to a year or two prior to standardisation.

As is usual with networking standards, it takes a while to filter to consumer use - we've had 10GBASE-T since 2006 and the fibre standards since 2002, but almost nobody uses it yet because they don't need it. Back in 2000, most people had precious little need for anything faster than their 56k modems. Meanwhile, in the backhaul and server world, 40Gb got old a few years back.

[metebalci]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

How do you saturate 10G with only 6 HDDs ? Is it a single raid-0/stripe ? It doesnt seem to be possible in any other combination.

[sokoloff]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

[jc101]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

How do you saturate 10G with only 6 HDDs ? Is it a single raid-0/stripe ? It doesnt seem to be possible in any other combination.

It's a RAID5 Stripe, but it also has an SSD cache too.  Plenty of RAM also helps.  My typical file sizes are 1Tb to 10's Tb, seems to chew through them without a hitch.

[py-bb]

Link/model of switch? (Affordable 10G gear that isn't a lucky auction on ebay - link me!)

You say you're in Afghanistan ... so kinda tough for people to ship you switches there.

If you're in US or some bigger country, I'd suggest emailing/contacting unixsurplus.com to give you a list of switches with 10gbs sfp+ ports they have available and more affordable.

They don't list everything they have on their website. Last year around September I asked what they could do in $300 and they had 2 units of Juniper EX4200 EX4200 (24 x 10g SFP+ and 4x1g SFP with two power supplies) for $275  .. specs here : https://www.juniper.net/content/dam/www/assets/datasheets/us/en/switches/ex4200-ethernet-switch-with-vitual-chassis-technology.pdf

Something interesting they have is a Quanta T5032 with 32 QSFP+ 40g ports for $419 : https://unixsurplus.com/quanta-t5032-ly6-switch/ 
Description says it uses Atom but it's PPC : https://www.qct.io/product/index/Switch/Ethernet-Switch/T5000-Series/QuantaMesh-T5032-LY6#specifications

40g to 4 x10g SFP+ DAC cables start from around $40 for the 1 meter versions, so around $10 per 10g port.

40g doesn't seem that popular anymore, as people moved to QSFP28, with 10/25/50/100 gbps ... you can get qsfp+ cards for $50 ex. https://unixsurplus.com/mellanox-cx353a--adapter/
qsfp28 can be bought at around $110 ... example 109$ for low profile 100g card : https://unixsurplus.com/Mellanox-MCX455A-Single-Port-100GB-QSFP18-Low-Profile/


Servethehome forum - the Great Deals section is also worth checking out : https://forums.servethehome.com/index.php?forums/great-deals.8/
from time to time very good deals (which go away fast) are posted there.

Look also at previous posts for switches and bookmark those sellers or add those switch models you like to your ebay searches / bookmarks etc

Europe. Lol where'd you get Afghanistan from?

[metebalci]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

I dont know 15K drives but enterprise SATA is around 250MB/s max at sequential read. So 6 disks combined would be 1.5GB/s at a theoretical maximum. I also have a 8 HDD striped mirrors/raid10 and still not easy to saturate also probably because of other factors.

[Monkeh]

Europe. Lol where'd you get Afghanistan from?

Well you're flying their flag.

[metebalci]

It's a RAID5 Stripe, but it also has an SSD cache too.  Plenty of RAM also helps.  My typical file sizes are 1Tb to 10's Tb, seems to chew through them without a hitch.

So it is 5 HDDs omitting the raid5 overhead should be just around 10G / 1.25GB/s. I also have a 8 HDD striped mirros/raid10 and not easy to saturate 10G still.

[sokoloff]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

I dont know 15K drives but enterprise SATA is around 250MB/s max at sequential read. So 6 disks combined would be 1.5GB/s at a theoretical maximum. I also have a 8 HDD striped mirrors/raid10 and still not easy to saturate also probably because of other factors.

1.5 GB/s of data transfer from the array is a bit over 12 Gbps on the wire. (1 Big B = 8 little b)

[metebalci]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

I dont know 15K drives but enterprise SATA is around 250MB/s max at sequential read. So 6 disks combined would be 1.5GB/s at a theoretical maximum. I also have a 8 HDD striped mirrors/raid10 and still not easy to saturate also probably because of other factors.

1.5 GB/s of data transfer from the array is a bit over 12 Gbps on the wire. (1 Big B = 8 little b)

I am aware but it is theoretical, on the paper.

[jc101]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

[metebalci]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

I think 900 MB/s is a bit pessimistic for 10G. I have tested an NVMe pool before and 1.1GB/s was possible (from windows, smb copy) without jumbo frames.

[sokoloff]

The 4-drive RAID 1+0 array on my Proxmox server benches at 411-425MB/s sequential read on the box and 388-406MB/s from the neighboring 10Gbps DAC-connected backup server, obviously well above what 1 or 2.5Gbps ethernet could handle.

That's with just 4 ancient 15K 2.5" SAS drives (model EH0300JDYTH).

[jc101]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

I think 900 MB/s is a bit pessimistic for 10G. I have tested an NVMe pool before and 1.1GB/s was possible (from windows, smb copy) without jumbo frames.

No doubt it is pessimistic, it's supposed to be.  When designing LAN and IT systems, it means I have built-in headroom.  If things are always on the local LAN, no problem, when you start to traverse many switches and routers latency creeps in to slow things down.

[metebalci]

The 4-drive RAID 1+0 array on my Proxmox server benches at 411-425MB/s sequential read on the box and 388-406MB/s from the neighboring 10Gbps DAC-connected backup server, obviously well above what 1 or 2.5Gbps ethernet could handle.

That's with just 4 ancient 15K 2.5" SAS drives (model EH0300JDYTH).

Yes I think this is the most important (and for many I guess the only) benefit of moving to 10G at home. If one has such a NAS (even 4 disks like you showed), 10G is a great upgrade.

[james_s]

Europe. Lol where'd you get Afghanistan from?

The flag under your name indicating your location says Afghanistan. If that is incorrect you should edit your profile to fix it, otherwise people are going to assume you're in Afghanistan when answering your questions.

[metebalci]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

I think 900 MB/s is a bit pessimistic for 10G. I have tested an NVMe pool before and 1.1GB/s was possible (from windows, smb copy) without jumbo frames.

No doubt it is pessimistic, it's supposed to be.  When designing LAN and IT systems, it means I have built-in headroom.  If things are always on the local LAN, no problem, when you start to traverse many switches and routers latency creeps in to slow things down.

OK, I agree. I have a home environment in my mind, maybe 2 switches at most.

[metebalci]

Europe. Lol where'd you get Afghanistan from?

The flag under your name indicating your location says Afghanistan. If that is incorrect you should edit your profile to fix it, otherwise people are going to assume you're in Afghanistan when answering your questions.

You might have found a bug in the forum software. He wrote something for location not a country, and I think the first country in country code list is Afghanistan, hence the flag I guess.

[ve7xen]

I think 2.5/5G makes more sense in the home setting. No fancy cabling requirements, cheaper PHYs, lower power, and still multiples faster. 2.5G is starting to be widespread in home networking equipment. If you're going to buy 10G ports in your home gear, I'd make sure they also support mGig (it is a much newer standard, despite being slower, so 10G doesn't imply 2.5G or 5G). 10GBASE-T kind of sucks as a standard.

Either way, storage is really the only typical use case, and even there, most home users aren't moving large files around on the regular.

[Monkeh]

Europe. Lol where'd you get Afghanistan from?

The flag under your name indicating your location says Afghanistan. If that is incorrect you should edit your profile to fix it, otherwise people are going to assume you're in Afghanistan when answering your questions.

You might have found a bug in the forum software. He wrote something for location not a country, and I think the first country in country code list is Afghanistan, hence the flag I guess.

Location and country are separate fields. The bug is in the person filling them.

[SiliconWizard]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

How do you saturate 10G with only 6 HDDs ? Is it a single raid-0/stripe ? It doesnt seem to be possible in any other combination.

Recent HDDs with decent performance and several TB of capacity can routinely transfer over 200 MBytes/s. I don't know how he has configured his setup either, but obviously 6 times that would exceed the throughput of a 10Gbps link.

[nightfire]

At work, I have a new NAS with 8 Enterprise HDDs  as Raid6, and this easily can saturate the 10 Gbit/s link in linear operation. No problem here, depends a bit on the bang the CPU delivers for checksum calculation.

[Halcyon]

I don't think this has any use for home network. my gigabit network from year 2000 is still plenty enough.

Gbit in the year 2000? By then we were still going from 10M to 100M ethernet, no way!

Yep, that's about right. You started to see Gigabit NICs in consumer computers from about 2000 onwards. It was probably a few more years before they started to be adopted widely in the home user segment as the cost of Gigabit switches came down. From memory I ditched 10 Megabit ethernet sometime around 1996/1997 when we bought our first switch to replace the old ethernet hub at home.

[SiliconWizard]

At home I switched to a 1Gb LAN around 2006 IIRC. Yeah.
Still not on 10Gb. But I plan on redoing my LAN with fiber, 10Gb or over. Still has a cost.

[ejeffrey]

The only 'domesticated' use case for such bandwidth is a $10,000 MacPro video studio, with 8K video stored on a SAN/Raid array loaded with M2 EVO ssd modules.

That's pretty much as big of an exaggeration as the person who needs > 1 Gb networking for their 4k tv.

A commodity file server can easily exceed 1 Gb even with ordinary mechanical hard drives much less SSDs.  If you have a local file server and  either multiple wired clients or a single client with a 10 G nic you can easily can easily use that speed.  And multi gigabit internet options are... Not super common but not exactly rare either. 

Do you "need" that?  Depends, but probably not.  But it's definitely possible to use that bandwidth with relatively common hardware.

But outside of the Hollywood hills, there is little justification for this level of investment.

What level of investment?  10GBase-T NICs are under $100 and switches < $50 / port.  It's not as cheap as 1 Gb networking equipment but it's not a huge premium to build a 10 G network.  I think that if you are building a network that will have a home file server you should strongly consider 10 G. 

btw My fastest outbound-upstream link is a mere 100Mbps on a 330Mbps FTTP link. When I started networking, 1Mbps over coax was regarded a top enterprise level speed - and was priced accordingly.

Good for you.

[Halcyon]

As much as I bag out the National Broadband Network (NBN) in Australia, largely based on previous years of butt-hurt, I'm fortunate enough to have fibre to the premises now.

It's not uncommon to see Gigabit (or higher) WAN links to homes these days depending on which part of the world you live in. Commencing this year, NBN are already planning to expand residential fibre services beyond the 1 Gigabit limit.

Consumer routers such as those made by Asus and Netgear are already offering 10 Gigabit ports and 802.11ax Wi-Fi is already capable of saturating a Gigabit link.

[py-bb]

Europe. Lol where'd you get Afghanistan from?

Well you're flying their flag.

sorry what?

[py-bb]

A modern 15krm SAS drive will give you 370mib/sec (320mib/sec on track) speeds give or take. So you'll see an average of 320mib/sec but it's really 370mib/sec with pauses for the track changes.

You can saturate network connections much more easily with SSDs though, SAS comes in a few speeds and is usually dual connector - so SAS 12gbps (bit) (gen III - we're on 24 which is really 22.5 in main deployment - but most drive stuff is still 12gbps max really) will give you 24gbps per SSD (both connectors) - now you can saturate stuff.

And the big proper enterprise SSDs are your best chance at them sustaining that performance too.

[NiHaoMike]

One use case I can think of would be wireless VR, in combination with 802.11ay or whatever short range, high bandwidth Wifi standard becomes popular by then. In fact, 10G might not be enough for high resolutions and 25G or 40G might be needed to make it work.

[mariush]

Europe. Lol where'd you get Afghanistan from?

Well you're flying their flag.

sorry what?

Change the country / flag from your profile ... it shows up on the left side

[metebalci]

A modern 15krm SAS drive will give you 370mib/sec (320mib/sec on track) speeds give or take. So you'll see an average of 320mib/sec but it's really 370mib/sec with pauses for the track changes.

You can saturate network connections much more easily with SSDs though, SAS comes in a few speeds and is usually dual connector - so SAS 12gbps (bit) (gen III - we're on 24 which is really 22.5 in main deployment - but most drive stuff is still 12gbps max really) will give you 24gbps per SSD (both connectors) - now you can saturate stuff.

And the big proper enterprise SSDs are your best chance at them sustaining that performance too.

SATA SSD yes more easy to saturate 10G and PCIe SSD can saturate even 40G easily, even the not performance oriented PM9A3 I am using has >6GB/s sequential read spec. I was talking about HDDs.

[metebalci]

At work, I have a new NAS with 8 Enterprise HDDs  as Raid6, and this easily can saturate the 10 Gbit/s link in linear operation. No problem here, depends a bit on the bang the CPU delivers for checksum calculation.

Just wondered if this is hardware raid, Linux mdraid or zfs ?

[py-bb]

A modern 15krm SAS drive will give you 370mib/sec (320mib/sec on track) speeds give or take. So you'll see an average of 320mib/sec but it's really 370mib/sec with pauses for the track changes.

You can saturate network connections much more easily with SSDs though, SAS comes in a few speeds and is usually dual connector - so SAS 12gbps (bit) (gen III - we're on 24 which is really 22.5 in main deployment - but most drive stuff is still 12gbps max really) will give you 24gbps per SSD (both connectors) - now you can saturate stuff.

And the big proper enterprise SSDs are your best chance at them sustaining that performance too.

SATA SSD yes more easy to saturate 10G and PCIe SSD can saturate even 40G easily, even the not performance oriented PM9A3 I am using has >6GB/s sequential read spec. I was talking about HDDs.

SATA SSD wont, they're limited to 6gbps which is ~500mib/sec give or take (you'll see 480mib/sec at least in practice) - nowhere near the 1.10gib/sec of 10G

[madires]

When you don't need 10Gbps, but a bit more than 1Gbps would be nice, then link aggregation could be an alternative. There are a lot of inexpensive semi-managable SOHO switches supporting LACP/LAGs. NASes with two ethernet ports usually support it too, and all modern OSes anyway.

[bitwelder]

One more caution for 10 Gbps copper cables: if the line is terminated at some SFP+ transceivers, do check that the transceivers you use are rated for the length of cable you plan to use.
We happen to have a 10Gbps copper line which as measured by the switch should be around 40m, and we used transceivers rated for 30m.
The 10 Gbps link works, but the switch detects some transmission error.

[sokoloff]

When you don't need 10Gbps, but a bit more than 1Gbps would be nice, then link aggregation could be an alternative. There are a lot of inexpensive semi-managable SOHO switches supporting LACP/LAGs. NASes with two ethernet ports usually support it too, and all modern OSes anyway.

Link aggregation increases aggregate throughtput for multiple clients, but many configurations do not increase speed for single client/single connection. (That depends on the hashing algorithm chosen; if the connection used is hashed by MAC address pairing, a single client-server pair will only use one of the links.)

[Halcyon]

When you don't need 10Gbps, but a bit more than 1Gbps would be nice, then link aggregation could be an alternative. There are a lot of inexpensive semi-managable SOHO switches supporting LACP/LAGs. NASes with two ethernet ports usually support it too, and all modern OSes anyway.

Link aggregation increases aggregate throughtput for multiple clients, but many configurations do not increase speed for single client/single connection. (That depends on the hashing algorithm chosen; if the connection used is hashed by MAC address pairing, a single client-server pair will only use one of the links.)

Correct and a common mistake people make. Almost always it's MAC or IP based, which means even if you have a LAG set up on a client PC, the maximum throughput you'll get is that of one single NIC. Where it really becomes useful is where you have multiple clients accessing a single resource (or you want redundancy in your links).

[metebalci]

A modern 15krm SAS drive will give you 370mib/sec (320mib/sec on track) speeds give or take. So you'll see an average of 320mib/sec but it's really 370mib/sec with pauses for the track changes.

You can saturate network connections much more easily with SSDs though, SAS comes in a few speeds and is usually dual connector - so SAS 12gbps (bit) (gen III - we're on 24 which is really 22.5 in main deployment - but most drive stuff is still 12gbps max really) will give you 24gbps per SSD (both connectors) - now you can saturate stuff.

And the big proper enterprise SSDs are your best chance at them sustaining that performance too.

SATA SSD yes more easy to saturate 10G and PCIe SSD can saturate even 40G easily, even the not performance oriented PM9A3 I am using has >6GB/s sequential read spec. I was talking about HDDs.

SATA SSD wont, they're limited to 6gbps which is ~500mib/sec give or take (you'll see 480mib/sec at least in practice) - nowhere near the 1.10gib/sec of 10G

Naturally I meant multiple SATA SSDs. I just meant it is easier to saturate 10G comparing to HDDs, which has almost the half-speed.

[metebalci]

When you don't need 10Gbps, but a bit more than 1Gbps would be nice, then link aggregation could be an alternative. There are a lot of inexpensive semi-managable SOHO switches supporting LACP/LAGs. NASes with two ethernet ports usually support it too, and all modern OSes anyway.

Link aggregation increases aggregate throughtput for multiple clients, but many configurations do not increase speed for single client/single connection. (That depends on the hashing algorithm chosen; if the connection used is hashed by MAC address pairing, a single client-server pair will only use one of the links.)

Correct and a common mistake people make. Almost always it's MAC or IP based, which means even if you have a LAG set up on a client PC, the maximum throughput you'll get is that of one single NIC. Where it really becomes useful is where you have multiple clients accessing a single resource (or you want redundancy in your links).

This opens up another topic. A 10G NIC probably have to use multiple queues/RSS to be able to reach this speed. Call this queues, flows, channels, threads, all are more or less about the same topic. If you use SMB for example, it is normally single threaded, hence it is hard to reach 10G depending on the NIC using SMB. SMB v3 brings multichannel support but it sometimes does not run properly out of the box. All of these high speed network tech is mainly for concurrent access, so increasing the capacity of a single pipeline (from a single NIC of PC to another single NIC on a NAS) becomes more and more difficult after a point.

[SiliconWizard]

When you don't need 10Gbps, but a bit more than 1Gbps would be nice, then link aggregation could be an alternative. There are a lot of inexpensive semi-managable SOHO switches supporting LACP/LAGs. NASes with two ethernet ports usually support it too, and all modern OSes anyway.

Note that 2.5Gb is becoming relatively common on new gear and less expensive than 10Gb - this may be a good alternative, 2.5x the throughput.

[AndyBeez]

Thinking about this, the ethernet port on domestic hardware may go extinct - Just like Firewire did. Remember how blisteringly fast those were back in 2005? Soon everything non-wireless will connect via USB-C. Or at least, the adapters will. I intended wiring my solid walled house with Cat6 cable, drill holes, run conduit and fit mounting boxes. Just like a real network guy! Instead, I purchased a couple of former enterprise 5Ghz repeaters and I have speed everywhere. And no relationships were harmed in the installation process either. 

[madires]

Link aggregation increases aggregate throughtput for multiple clients, but many configurations do not increase speed for single client/single connection. (That depends on the hashing algorithm chosen; if the connection used is hashed by MAC address pairing, a single client-server pair will only use one of the links.)

Correct and a common mistake people make. Almost always it's MAC or IP based, which means even if you have a LAG set up on a client PC, the maximum throughput you'll get is that of one single NIC.

... in the worst case. It also depends on the specific network traffic patterns and protocols used. So it can be anything between 1Gbps and the total throughput of the LAG.

[ve7xen]

Thinking about this, the ethernet port on domestic hardware may go extinct - Just like Firewire did. Remember how blisteringly fast those were back in 2005? Soon everything non-wireless will connect via USB-C. Or at least, the adapters will. I intended wiring my solid walled house with Cat6 cable, drill holes, run conduit and fit mounting boxes. Just like a real network guy! Instead, I purchased a couple of former enterprise 5Ghz repeaters and I have speed everywhere. And no relationships were harmed in the installation process either.

This has happened already, more or less. Most people don't have any wired Ethernet devices other than their modem/router itself these days, and if you want such a port on most modern laptops, the only option is a dongle.

But wireless still sucks relative to wired, and USB-C isn't a suitable replacement or competitor for wired Ethernet, so it will live on for a long time yet, just become less and less relevant in the home setting.

[james_s]

I still use wired gigabit for everything I can, wireless is reserved for portable devices. I can easily saturate the whole wireless network by copying large files from my laptop. The wired network has switches that route the traffic so transfering between two machines doesn't bog down the rest of the network. For the average consumer it's probably not noticeable but if you're copying around tens of gigabytes of video files or whatever it becomes very obvious.

[dobsonr741]

If you have thunderbolt ext storage and want to share it to another Mac certainly would need 10G, as Thunderbolt is already at 40gig.

[Halcyon]

This has happened already, more or less. Most people don't have any wired Ethernet devices other than their modem/router itself these days, and if you want such a port on most modern laptops, the only option is a dongle.

But wireless still sucks relative to wired, and USB-C isn't a suitable replacement or competitor for wired Ethernet, so it will live on for a long time yet, just become less and less relevant in the home setting.

It can suck, but it doesn't have to. Like with everything, if you invest a bit of money in proper gear (not those stupid consumer "mesh" products), you can actually get pretty amazing performance, even in RF noisy environments with a bit of tweaking. You'll notice most consumer devices, they like to default to channels 36 to 48 and sometimes around channel 155 down the other end of the 5 GHz ISM band. They often avoid the DFS channels altogether, even if they are better.

For my set up at home, I have two internal access points (Ubiquiti U6-LR) and one outside access point to provide 5 GHz coverage throughout my backyard. I use channels 106, 138 and 155 respectively, as there are no other networks on those channels. To avoid weather radar forcing your access points to jump to a non-DFS channel, jump on your local communications authority and search their database on any transmitters within 100 kilometers on the same channel(s) you plan to use. In Australia it's ACMA and you can do a frequency range search online: https://web.acma.gov.au/rrl/assignment_range.search

I actually switched to Wi-Fi on my main TV as the 100 Mbps NIC was getting overwhelmed with some of the 4K content coming off my network (which averaged at 110 Mbps). (Why Sony decided to only include a 10/100 NIC in one of their expensive TVs defies all logic).

[madires]

The wired network has switches that route the traffic so transfering between two machines doesn't bog down the rest of the network.

Sorry for being a bit pedantic, but switching and routing are two different concepts in the networking world. Switching happens at OSI layer 2, while routing happens at OSI layer 3. A classic ethernet switch (or bridge) switches ethernet frames between segments at the data-link layer. An IP router routes IP packets between networks at the network layer. The IP packet is stacked on top of the ethernet frame. Or in other words, the IP packet is the payload of the ethernet frame. A so called L3 switch (L3 for OSI layer 3) can do both.

[M0HZH]

At the office we're running ~100 clients with Hyper-V, SAN, CRM server and a bunch of other stuff, entirely on 1Gb.

I doubt you would really need 10Gb at home for anything other than video editing on a NAS or something like that. Even 2.5Gb is hard to justify (although it's relatively cheap now).

[ve7xen]

It can suck, but it doesn't have to. Like with everything, if you invest a bit of money in proper gear (not those stupid consumer "mesh" products), you can actually get pretty amazing performance, even in RF noisy environments with a bit of tweaking. You'll notice most consumer devices, they like to default to channels 36 to 48 and sometimes around channel 155 down the other end of the 5 GHz ISM band. They often avoid the DFS channels altogether, even if they are better.

There's a reason I qualified it as 'relative to wired Ethernet'. It can be fine, if the equipment is okay and the setup is okay, but it's still a shared medium, with a lower maximum throughput, shorter range, and is generally much less stable even when working well. Wired 'just works'.

[james_s]

Sorry for being a bit pedantic, but switching and routing are two different concepts in the networking world. Switching happens at OSI layer 2, while routing happens at OSI layer 3. A classic ethernet switch (or bridge) switches ethernet frames between segments at the data-link layer. An IP router routes IP packets between networks at the network layer. The IP packet is stacked on top of the ethernet frame. Or in other words, the IP packet is the payload of the ethernet frame. A so called L3 switch (L3 for OSI layer 3) can do both.

Yes I know this and yes I think it's very pedantic. I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird. It's grammatically correct to say that a switch (or router or anything else that has some kind of control over where data goes) "routes" traffic. Traffic signals, signs, traffic officers and other methods "route traffic" on roads.

[metebalci]

Sorry for being a bit pedantic, but switching and routing are two different concepts in the networking world. Switching happens at OSI layer 2, while routing happens at OSI layer 3. A classic ethernet switch (or bridge) switches ethernet frames between segments at the data-link layer. An IP router routes IP packets between networks at the network layer. The IP packet is stacked on top of the ethernet frame. Or in other words, the IP packet is the payload of the ethernet frame. A so called L3 switch (L3 for OSI layer 3) can do both.

Yes I know this and yes I think it's very pedantic. I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird. It's grammatically correct to say that a switch (or router or anything else that has some kind of control over where data goes) "routes" traffic. Traffic signals, signs, traffic officers and other methods "route traffic" on roads.

I dont think it is very pedantic. As this is a technical forum, when switch and route is used in the same sentence in a post about networking, I think it should be clarified.

Also I dont think traffic signals and officers are a good first analogy, a switch (or router) does not block or interrupt a traffic (leaving extra features like ACLs aside).

[2N3055]

Sorry for being a bit pedantic, but switching and routing are two different concepts in the networking world. Switching happens at OSI layer 2, while routing happens at OSI layer 3. A classic ethernet switch (or bridge) switches ethernet frames between segments at the data-link layer. An IP router routes IP packets between networks at the network layer. The IP packet is stacked on top of the ethernet frame. Or in other words, the IP packet is the payload of the ethernet frame. A so called L3 switch (L3 for OSI layer 3) can do both.

Yes I know this and yes I think it's very pedantic. I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird. It's grammatically correct to say that a switch (or router or anything else that has some kind of control over where data goes) "routes" traffic. Traffic signals, signs, traffic officers and other methods "route traffic" on roads.

I am with you here.. I understood perfectly what you meant and that kind of terminology is used like that in high level LAN talk all the time.
Not perfect and technically not exactly right terminology, but I understood what you meant. In that context "routes data" meant "moves data" not "it performs TCP/IP routing on it"
Context matter a lot. That is why English is so confusing sometimes...

Make it even more confusing by introducing Layer 3 switching.. What do you call data transfers that happen there.....

If we want to be pedantic, then we should use "data forwarding" and "data redirecting" or something of that sort and use layer it happens on (2 or 3).....
Which can be left to highly technical discussions. 

[madires]

I am with you here.. I understood perfectly what you meant and that kind of terminology is used like that in high level LAN talk all the time.
Not perfect and technically not exactly right terminology, but I understood what you meant. In that context "routes data" meant "moves data" not "it performs TCP/IP routing on it"
Context matter a lot. That is why English is so confusing sometimes...

I only explained very basic networking terminology. And in this context routing is a function at OSI layer 3. Also, it doesn't imply TCP/IP, it can be any other L3 network protocol too. If you're dealing with switches and routers, and talk about routing data, which of both device types do you think is meant or involved?

[madires]

I was using the word "route" in a different context, it doesn't make sense in the English language to say that the switch "switches traffic" that just sounds weird.

A good alternative would be 'forwarding' as 2N3055 pointed out. Forwarding applies to layer 2 and 3.

[2N3055]

I am with you here.. I understood perfectly what you meant and that kind of terminology is used like that in high level LAN talk all the time.
Not perfect and technically not exactly right terminology, but I understood what you meant. In that context "routes data" meant "moves data" not "it performs TCP/IP routing on it"
Context matter a lot. That is why English is so confusing sometimes...

I only explained very basic networking terminology. And in this context routing is a function at OSI layer 3. Also, it doesn't imply TCP/IP, it can be any other L3 network protocol too. If you're dealing with switches and routers, and talk about routing data, which of both device types do you think is meant or involved?

I didn't say you are wrong.
"routing" and "switching" are words being commonly used to signify a work done by routers and switches. But in English language "routing messages" is what mail server does... And it is not network device... Word routing made sense in that sentence despite being "nontechnical". That is my point.
Another point is that when discussing something there is no need to always go into details. Quite the contrary, it is an art of saying only necessary to make it quick and efficient.
Otherwise we might start always discussing that there are hubs, bridging, routing, store&forward switching, cut through switching, asymmetric switching, spanning tree protocol etc etc...

Important part is that usually when using switches, switch exchanges data based on MAC of network interface on devices connected, while router makes distinction between subnets and decides on packet transfer decisions based on IP address of the host. Once packet arrives at destination subnet (by whatever means) then final L3 to L2 resolution is done (via ARP) and packet reaches it's destination. Also distinction is that switch has switching matrix that allows paralel (isolated) connections between hosts that serves to increase performance.
L3 switches are intelligent enough to keep both IP address and MAC address tables directly so they can do switching by shortcutting routing. Also no difference logically, just much faster because it uses parallel switching matrix instead of serialized traffic through router interfaces....

As for other L3 protocols, it's been decades since I worked on anything else than TCP/IP so I simply ignore them in general conversation. You are, again, correct, but it doesn't matter. For all the practical purposes if someone asks you about networking today, it is going to be TCP/IP..

Best,

[sokoloff]

It’ll be an IP network, but with a mix of TCP and other traffic (mostly UDP).

[tszaboo]

A few things:
There are internet providers here that offer more than 1 Gbps speeds.
AC Wifi is faster than gigabit, so if you want a wired extender than having more than a gigabit makes sense.
On the other hand I don't think you need more than gigabit in 2023. The worst offenders are game updates which are dozens of gigabytes, and that are handled in mere minutes. I think right now it makes sense to get cables that would handle 10 gig networking, and only switch to it when it's more standard. It makes more economic sense. Especially in walls, get something that is future proof, CAT 7 or similar, because replacing it is more pain.

[metebalci]

A few things:
There are internet providers here that offer more than 1 Gbps speeds.
AC Wifi is faster than gigabit, so if you want a wired extender than having more than a gigabit makes sense.
On the other hand I don't think you need more than gigabit in 2023. The worst offenders are game updates which are dozens of gigabytes, and that are handled in mere minutes. I think right now it makes sense to get cables that would handle 10 gig networking, and only switch to it when it's more standard. It makes more economic sense. Especially in walls, get something that is future proof, CAT 7 or similar, because replacing it is more pain.

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

[madires]

There are internet providers here that offer more than 1 Gbps speeds.

Yep, for example 2.5Gbps GPON.

Especially in walls, get something that is future proof, CAT 7 or similar, because replacing it is more pain.

IIRC, the current maximum for TP is 25GBASE-T (Cat8 and a maximum segment length of 30m). Installing Cat7 and Cat8 cables is quite effortful (if done to specs). So it could make more sense to go for glass fiber anyway when rewiring your house, in case you plan to go beyond 10Gpbs.

[madires]

"routing" and "switching" are words being commonly used to signify a work done by routers and switches. But in English language "routing messages" is what mail server does... And it is not network device... Word routing made sense in that sentence despite being "nontechnical". That is my point.

We shouldn't forget the scope of 'routing messages'. When I write a letter to a friend in another town the post office will route it to my friend's town. But when I write a letter to a neighbor a few houses down the street I don't post it at the local post office. Instead I'll simply throw the letter into the neigbor's mailbox, i.e. no routing at all.

Important part is that usually when using switches, switch exchanges data based on MAC of network interface on devices connected,

Endpoints (or servers) exchange data with other endpoints (or servers). Switches exchange data when negotiating links and other network related things, like STP.

L3 switches are intelligent enough to keep both IP address and MAC address tables directly so they can do switching by shortcutting routing. Also no difference logically, just much faster because it uses parallel switching matrix instead of serialized traffic through router interfaces....

For routing vendors implement a routing table which contains all the routing information (RIB) and a forwarding table (FIB) which is a list of destinations and their corresponding egress interfaces. The RIB can contain inactive routes (known, but not used because there's a better route). The active routes are put into the FIB. Since routing is a lot of effort vendors came up with ideas to simplify the routing process by making it more like switching, e.g. by creating hashes for flows and using a special table which is faster/shorter than the standard FIB. Cisco's CEF is such an example. These techniques are also implemented in ASICs. But you can't switch IP packets as simply as ethernet frames. Maybe the router has to fragment IPv4 packets or has to deal with IPv6 option headers. This leads us to MPLS which tags IP packets (also other protocols) with switching labels to allow intermediate routers to switch traffic instead of costly routing.

To give you an idea about the scale of ethernet switching and IP routing: A standard inexpensive SOHO switch chipset usually has an 8k MAC address table. A full IP routing table has currently 950k IPv4 routes and about 150k IPv6 routes.

[james_s]

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

Mine doesn't. It's fast for wireless, but especially when there are multiple devices the gigabit ethernet is significantly faster.

[james_s]

We shouldn't forget the scope of 'routing messages'. When I write a letter to a friend in another town the post office will route it to my friend's town. But when I write a letter to a neighbor a few houses down the street I don't post it at the local post office. Instead I'll simply throw the letter into the neigbor's mailbox, i.e. no routing at all.

You're routing it yourself and not relying on the post office, it's still being routed. You're not blasting out a massive quantity of letters into all of the mailboxes or just throwing it into the wind and hoping it finds its destination.

[ejeffrey]

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

I don't think they can in single client operation, but my understanding is that it is technically possible in ideal benchmarks with multiple clients and an access point with enough MIMO channels.
 
Wifi 6E access points, if they have separate radios for 5 and 6 GHz could probably exceed 1 Gb/s more easily, but still not for a single client.

[madires]

We shouldn't forget the scope of 'routing messages'. When I write a letter to a friend in another town the post office will route it to my friend's town. But when I write a letter to a neighbor a few houses down the street I don't post it at the local post office. Instead I'll simply throw the letter into the neigbor's mailbox, i.e. no routing at all.

You're routing it yourself and not relying on the post office, it's still being routed. You're not blasting out a massive quantity of letters into all of the mailboxes or just throwing it into the wind and hoping it finds its destination.

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

[tszaboo]

Does 802.11ac really give >1Gbps anywhere in a normal size home ? I am using 802.11ac with relatively new mobile devices and I havent seen anything above 500Mbps and that is also only when very close and direct line of sight of the AP (no mesh, single AP) ?

I don't think they can in single client operation, but my understanding is that it is technically possible in ideal benchmarks with multiple clients and an access point with enough MIMO channels.
 
Wifi 6E access points, if they have separate radios for 5 and 6 GHz could probably exceed 1 Gb/s more easily, but still not for a single client.

There is a wide variety of speeds under the term AC
https://en.wikipedia.org/wiki/IEEE_802.11ac-2013#Advertised_speeds

At least it works in most cases and not confusing like USB 3.1 (version 2) 1x1 PD- 67W.

[JohanH]

PON and GPON are standards made by ITU, the telecom industry. It's all made for feeding commercials and dumb TV content to as many users as possible, disregarding their ability to send data back. So the connection is shared among users, often passively and in most cases asymmetric. This saves some fibers, but the equipment is proprietary, expensive and there is a high threshold to upgrade the equipment. Later GPON standards are better, with better upstream speed, but if you have a small village on PON (typically restricted to 100/30 Mbit per user + bandwidth is shared!), you are lucky if they upgrade it in 20 years.

I work from home and would hate having a restricted upstream. Luckily we have an ISP that uses standard IEEE ethernet switches and none of the ITU crap. So I have the option for 100/100 Mbit or 1000/1000 Mbit over fiber (and options in between). Higher speeds than 1 Gbit aren't on the table today, but if you pay... they only have to change two tranceivers and your home fiber terminal. Might be a matter of time.

IEEE standard equipment is the only true networking and Internet equipment. PON and GPON is just disguised TV on life support and a surrogate for true end-to-end Internet. Engineers stick to IEEE standards.

[ve7xen]

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

If you don't know which house she lives in, are you making copies for all your neighbours and hoping that everyone discards letters not addressed to them? 

This saves some fibers, but the equipment is proprietary, expensive and there is a high threshold to upgrade the equipment. Later GPON standards are better, with better upstream speed, but if you have a small village on PON (typically restricted to 100/30 Mbit per user + bandwidth is shared!), you are lucky if they upgrade it in 20 years.

It's not so much about saving fibre, though that is part of it because otherwise you will lay a lot of fibre that never gets used, or if you want to avoid that, you will have to do relatively a lot of splicing work, but this is relatively cheap. It's more about saving the number of ports and PHYs you need to service the customers. Every classical Ethernet port requires its own switch port and own PHY, and this requires more physical space, more optics, more power, and more capacity in the switching backplane, which all costs fairly substantially at scale since we're talking about like 16:1 or 32:1 here.

XGS-PON (10G/2.5G) is widely deployed here and the ILEC is selling 1000/1000 service, and I have not seen any significant reports of congestion, and based on my experience in consumer ISPs, I find the risk of it unlikely. They are running 16:1, so it isn't really much of an oversubscription ratio. Upstream of the PON the oversubscription ratio is quickly going to increase.

Reality is that actual traffic patterns of residential users are still highly asymmetric preferring the downstream direction, so that is what ISPs are optimizing their networks for. It has nothing to do with TV, especially since the majority of ISPs deploying PON come from the telecom side, not the cable TV side, so while they all have TV offerings these days, it's not their heritage nor that significant a part of their business.

[JohanH]

The ITU/IEEE rivalry might be a bit exaggerated. Mostly tongue in cheek nowadays. But there are some background.

Conditions may vary and it's good that newer standard PON is deployed. I have seen bad networks around by the big ISPs.

There are lots of smaller fiber cooperatives around here that were built 15 years ago when the big operators removed their copper and offered only 3G/4G instead. All of these countryside and small town networks were built with network switches, AFAIK. I've some insight about ten such networks built around here in different towns. We tried to start our own fiber network in my village also; we were verbally threatened by the then local old telecom ISP; ultimately we didn't get enough interested users and didn't start digging. Fiber has become so cheap, so you don't save anything with PON, especially in rural, more widespread networks. Big ISPs might have easier deployment with PON in big cities with lots of departments.

[ve7xen]

The ITU/IEEE rivalry might be a bit exaggerated. Mostly tongue in cheek nowadays. But there are some background.

Conditions may vary and it's good that newer standard PON is deployed. I have seen bad networks around by the big ISPs.

There are lots of smaller fiber cooperatives around here that were built 15 years ago when the big operators removed their copper and offered only 3G/4G instead. All of these countryside and small town networks were built with network switches, AFAIK. I've some insight about ten such networks built around here in different towns. We tried to start our own fiber network in my village also; we were verbally threatened by the then local old telecom ISP; ultimately we didn't get enough interested users and didn't start digging. Fiber has become so cheap, so you don't save anything with PON, especially in rural, more widespread networks. Big ISPs might have easier deployment with PON in big cities with lots of departments.

Yeah, this is definitely true. There hasn't been a lot of development in small-scale PON (though there has been some), so it doesn't really start to make sense until a certain inflection point is reached that I'd guess is around 1000 homes passed. Many of these community ISPs are grassroots efforts, and often start at very small scale. It definitely doesn't make sense to run a PON shelf that can service 1000s of customers and costs 10s of thousands when you're only passing 100 homes and could do the job with a cheap 48-port switch. It's also much easier technically to get going using classical Ethernet, and more people understand that stuff, know how to source the equipment (good luck getting a vendor to return your calls about buying a single PON shelf, never mind the contract for management software etc. etc.), understand the automation options available and so on.

Both ways are doable, and can be done profitably, but PON is definitely more efficient at the scale of a major ISP.

Whether it offers a good experience is more down to competition and business decisions than the technology, and I guarantee that if your experience sucks on PON it's going to suck on any last mile technology.

Big cities and apartments are actually the one place where classical ethernet can compete with PON because you can home equipment at the building itself and cabling is relatively trivial.

[madires]

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

If you don't know which house she lives in, are you making copies for all your neighbours and hoping that everyone discards letters not addressed to them? 

I'm too lazy to make all the copies to flood my neighborhood. Instead I would ask a neighbor I know. BTW, flooding (BUM traffic) can be a real problem for larger LANs.

[Halcyon]

It can suck, but it doesn't have to. Like with everything, if you invest a bit of money in proper gear (not those stupid consumer "mesh" products), you can actually get pretty amazing performance, even in RF noisy environments with a bit of tweaking. You'll notice most consumer devices, they like to default to channels 36 to 48 and sometimes around channel 155 down the other end of the 5 GHz ISM band. They often avoid the DFS channels altogether, even if they are better.

There's a reason I qualified it as 'relative to wired Ethernet'. It can be fine, if the equipment is okay and the setup is okay, but it's still a shared medium, with a lower maximum throughput, shorter range, and is generally much less stable even when working well. Wired 'just works'.

I would generally agree with that statement. Wired Ethernet is certainly more "plug and play" than Wi-Fi.

When I deliver my letter to my neighbor's mailbox it's like a LAN with an ethernet switch. I don't have to know anything about routing, because that neighbor lives in the third house to the right, across the street (my MAC address table). I don't write my neighbor's full address on the envelope, just her name.

If you don't know which house she lives in, are you making copies for all your neighbours and hoping that everyone discards letters not addressed to them? 

For those playing along at home, this is essentially what a hub is (as opposed to a switch). A hub is essentially just a dumb device that repeats all packets across all ports, which is why collision domains can be an important consideration in network design that incorporates hubs. I do not miss those days.

[madires]

An ethernet switch does it too when it doesn't know yet how to reach the destination MAC. This is the 'U' (unknown unicast) in BUM traffic. It can also be leveraged for attacks, e.g. MAC flooding.

WiFi is unreliable by default because is uses ISM bands. At any time a neighbor can plug in a new gadget which occupies an ISM band, forcing you to change channels or the band. The bands can become so crowded that you barely get any usable throughput. And you can't do anything about this as long as the devices involved adhere to local radio regulations.

[David Hess]

IMHO it's only worth if you have some shared storage which needs fast access from multiple places.

In the past I had 10/100 Ethernet for my internet connected network, and gigabit Ethernet for local connections to my NAS.  Now I have gigabit Ethernet for my internet connected network, and 2 x 1 gigabit networks in parallel for my local connection to my NAS for a total of 3 gigabits for file transfers when using SMB 3.

I considered 10 gigabit Ethernet, however it would have cost roughly $130 for a switch, $65 for each SFP+ network interface card, and maybe $30 per cable.

Depends what you're doing in your home, but 10G is slower than the read/write speed of a mediocre SSD these days, so if you're using one PC as a file server for another one (or have a NAS) you could saturate that easily enough.

With consumer hardware there are usually other performance limitations.  10G USB connected SATA3 SSDs seem to top out at about 250 MB/s.  Direct connected SATA3 SSDs are somewhat faster, but my 4 and 8 drive hard disk RAID arrays are just as fast.  If you are using Microsoft Storage Spaces then forget performance.

If you have only short runs of ethernet cable <= 30m good old Cat5e usually works also fine with 10GBASE-T. Alternatively there's 2.5GBASE-T and 5GBASE-T in case you need just a small boost. Beyond that I'd go for glass fiber anyway because of less trouble (high speed ethernet can be quite finicky about TP) and the upgrade path.

2.5GBASE-T and 5GBASE-T might be acceptable in some environments, like Windows only, however driver support is bad and even Intel's 2.5GBASE-T solution has problems.

[tkamiya]

I've done this with two low end servers with 10GB SFP+ on both ends.  OS was Ubuntu Linux.
 Connection was with optical fiber.  Created RAM disk on both sides and measured the transfer speed.  The best I've seen was 3GB and only briefly.  Average was about 2GB. 

Whether if jumping onto 10GB or not depends on rest of the network.  If you are using good servers, perhaps.  But in my case, it wasn't worth it. 

[vad]

I've recently rebuilt my homelab. It is now a Proxmox cluster with Ceph distributed storage, all running on 3 Intel NUCs. I wish NUCs had SFP+ because the Ceph cluster network would definitely benefit from 10 Gbps. Instead, each NUC has three inexpensive USB3 2.5GbE adapters that work fine.

OP, try to avoid 10GBASE-T if possible. SFP+ might be more cost-effective on the used market, and it's also more energy-efficient.

[Veteran68]

At home I run 10GBASE-T between my main desktop, main rackmount server which (among other things) is my home media server, and my QNAP NAS appliance (~58TB storage). So no, definitely not "too much" speed for home use. Is there even such a thing?

I've done this with two low end servers with 10GB SFP+ on both ends.  OS was Ubuntu Linux.
 Connection was with optical fiber.  Created RAM disk on both sides and measured the transfer speed.  The best I've seen was 3GB and only briefly.  Average was about 2GB. 

Whether if jumping onto 10GB or not depends on rest of the network.  If you are using good servers, perhaps.  But in my case, it wasn't worth it. 

Then there's definitely something wrong with your setup. I've measured transfer between my desktop and NAS at ~9Gbps raw throughput, and that's strictly over twisted pair CAT6A. No fiber involved. Now not every application can max that bandwidth so YMMV, but for raw data transfer there should be no reason why you can't do much better than 3Gbps unless there's some crappy cable over extremely long and noisy distances.